betaFeedback

WorksMathematics

Measuring Coherence in Predictive Models

Abstract

Many sequential decision-making procedures, such as active learning, implicitly assume that a model's current predictions agree with those it expects to make after updating on new data, a property we call coherence. Most models in use do not update by exact conditioning and are not guaranteed to be coherent, which can lead to worse decisions. Yet there is currently no general way to measure incoherence or to tell when it matters. We develop tools to measure incoherence and characterise when it affects decisions. We introduce the incoherence ratio, an MMD-based measure of how far any updating procedure departs from coherence, computed only from the procedure's own predictions and updates, and accompany it with an anytime-valid test of coherence. In Bayesian experimental design, acquisition objectives that are equivalent for coherent models choose different data under every approximate model we study, and the incoherence ratio tracks how far incoherence distorts them. We also introduce action coherence, which captures only the incoherence that changes a decision. What a resulting disagreement costs, however, is determined primarily by predictive accuracy. Sequential decisions depend on how a model updates as well as on what it predicts, so coherence, like accuracy, should be checked rather than assumed.

Introduction

In sequential decision-making problems, actions are often chosen using a model that is updated as new data arrive. Active learning [1, 2], for example, selects the data whose acquisition will most improve the model’s predictions after updating. Making this choice requires the model to anticipate how its own predictions will change, which relies on its current predictions agreeing with those it expects to make after updating. Following Fong et al. [ [3], we call such a model coherent. Many methods for sequential decision-making are derived assuming exact Bayesian conditioning [4, 5], which guarantees coherence by construction. The models they are applied to, however, rarely update by exact conditioning, and so need not be coherent. Rather than assuming coherence, we measure how severely it is violated, and identify when and how incoherence affects decisions.

Bayesian experimental design shows how incoherence can reach a decision. The expected predictive information gain (EPIG) [6] selects the design whose outcome most reduces uncertainty in the model’s predictions at a target of interest. It can be written in several standard forms: a reduction in predictive entropy, a mutual information, or an expected divergence from the current predictive, and these are used interchangeably because they agree under coherence. For an incoherent model, however, they become different acquisition rules that can rank candidate designs in opposite orders (§ 3). Empirically, the entropy form acquires worse data on average than the mutual-information form for every approximate model we study (§ 6).

Coherence has a long history in both philosophy [7, 8] and statistics [9], where it corresponds to the martingale property of predictive distributions [3] and underpins the predictive approach to Bayesian inference [de Finetti, 1937]. Most models central to machine learning do not update by explicit conditioning, and a growing body of work asks whether their updates nonetheless behave like a Bayesian update. Prediction rules have been measured against exchangeability [11], in-context learning in large language models and tabular foundation models has been shown to depart from the martingale property [12, 13], and approximate Bayesian neural networks have been found to update incoherently [14]. These works establish that coherence can fail in important models, using diagnostics specific to each setting, most of which need data from the task or a reference Bayesian model. Separately, work on sequential decisions has shown that decision quality depends on a model’s joint predictive distribution, rather than its marginal predictions [15, 16]. A model’s update rule implicitly defines such a joint, and coherence is the requirement that it agree with the model’s current predictions. Because coherence concerns how predictions update rather than how accurate they are, a model can predict accurately at every input and still be incoherent. What is still missing is a general way to measure the incoherence of an arbitrary updating procedure using only the procedure itself, and an account of when incoherence matters for the decisions these models are used to make.

We address both. We first set out what coherence means for any procedure that predicts and updates, without reference to a prior, likelihood or parameter posterior (§ 2). The requirement is local to the history, target and update relevant to the problem, and, in keeping with a predictivist view, concerns only predictions of observable quantities. To measure incoherence, we introduce the incoherence ratio Ψ\Psi, a scale-invariant measure of how far a procedure’s updates depart from conditioning, estimated from its own predictions and updates alone (§ 4). We also give an anytime-valid sequential test of coherence (§ 5). To determine when incoherence matters, we introduce action coherence, a relaxation that asks only whether incoherence changes the action a procedure selects, and so isolates exactly what reaches a decision (§ 3).

Our experiments show what incoherence reveals about models and when it reaches decisions (§ 6). Our measure of incoherence falls as an approximation is refined and ranks models consistently across datasets. For every approximate model we study, objectives that agree under coherence select different data on a substantial share of decisions, so incoherence reaches the action. The measure tracks how susceptible a model’s decisions are to this disagreement, while accuracy determines what it costs. Incoherence is therefore a warning rather than a verdict. Sequential decisions rely on how a model updates, not only on what it predicts, and coherence is the property that captures this.

Predictive coherence

A predictive procedure comprises a probabilistic model and its prescribed update rule. At a realised history hh, the procedure returns predictive laws for the variables required by the prediction and update problem. We write YY for the prediction target and ZZ for the future observations that will be incorporated before the procedure is queried for YY again. We call ZZ the update block, which may contain one or more observations. An ordered update design d=(d1,…,dK)d=(d_1,\ldots,d_K) specifies which observations comprise the block and their order. The update is written U(h,z)U(h,z) and maps a history and an observed update block to an updated history. In supervised learning, for example, the initial history may be the observed dataset h^=(x1:n,y1:n)\hat{h}=(x_{1:n},y_{1:n}), and the target YY the response at a fixed test input x∗x_*. An update design d=xn+1:n+Kd=x_{n+1:n+K} specifies a collection of additional inputs, with corresponding update block Z=Yn+1:n+KZ=Y_{n+1:n+K}. Here the components of ZZ are responses of the same kind as YY. More generally, however, the construction assumes neither that the observations are exchangeable nor that the components of ZZ take values in the same space as YY.

We write PY(⋅;h)P_Y(\cdot;h) and PZ(⋅;h)P_Z(\cdot;h) for the procedure’s current predictive laws for the target and the complete update block, respectively. When the procedure does not supply a joint law for Z=(Z1,…,ZK)Z=(Z_1,\ldots,Z_K), we construct one sequentially [17] in the order specified by the design dd, as in recursive resampling [3]: PZ(dz;h)=∏k=1KPZk(dzk;hk−1)P_Z(dz;h)=\prod_{k=1}^{K}P_{Z_k}(dz_k;h_{k-1}), where h0≡hh_0\equiv h and hk:=U(hk−1,zk)h_k:=U(h_{k-1},z_k) is the updated history after observing z≤kz_{\leq k}. We demonstrate throughout that updating on the whole block agrees with updating component-wise along dd, so that hK=U(h,z)h_K=U(h,z). Throughout, a semicolon indicates that a predictive law is indexed by a history, and we reserve the vertical bar for probabilistic conditioning within a specified joint distribution. Subscripts name the variable whose law is described and superscripts carry the indices on which the law depends, namely the update design and its realisation. We suppress any index that is held fixed in the surrounding discussion, and reinstate it where it varies. In this section the design is fixed and suppressed.

Update-induced joint. The model’s predictive law for the update block and its post-update target predictions together define the update-induced joint,

Q(dy,dz;h):=PY(dy;U(h,z)) PZ(dz;h).(1)Q(dy,dz;h) := P_Y(dy;U(h,z))\,P_Z(dz;h). \tag*{(1)}

By construction, QZ(⋅;h)=PZ(⋅;h)Q_Z(\cdot;h)=P_Z(\cdot;h) and QY∣Z=z(⋅;h)=PY(⋅;U(h,z))Q_{Y\mid Z=z}(\cdot;h)=P_Y(\cdot;U(h,z)) for PZ(⋅;h)P_Z(\cdot;h)-almost every zz. Thus QQ is the unique joint distribution whose ZZ-marginal represents the model’s current beliefs about the update block and whose conditional distributions represent the predictions that the procedure will actually return after updating. The target marginal of this joint is

QY(dy;h)=∫PY(dy;U(h,z)) PZ(dz;h).(2)Q_Y(dy;h)=\int P_Y(dy;U(h,z))\,P_Z(dz;h). \tag*{(2)}

This is the model’s post-update predictive averaged under its current beliefs about the update block, which we call the expected future predictive. The update-induced joint is the operationally relevant distribution for a procedure which first observes ZZ and then YY. When the history hh is fixed, we suppress it, writing PYP_Y and PZP_Z for the current predictives, PYz:=PY(⋅;U(h,z))P_Y^z := P_Y(\cdot;U(h,z)) for the post-update predictive, and QYQ_Y for the expected future predictive, so that QY=EPZ[PYZ]Q_Y=\mathbb{E}_{P_Z}[P_Y^Z] and QY∣z≡PYzQ_{Y\mid z}\equiv P_Y^z.

Predictive coherence. A predictive procedure is coherent for the target YY, history hh and update block ZZ when the current predictive and expected future predictive agree, PY=QYP_Y=Q_Y. It extends the martingale condition of Fong et al. [ [3] to any target, history and update block. Coherence is exactly the statement that the current predictive is recovered as the YY-marginal of the update-induced joint. If coherence fails, no joint can simultaneously have PZP_Z as its ZZ-marginal, PYzP_Y^z as its conditional distribution given Z=zZ=z, and PYP_Y as its YY-marginal. Exact conditioning under a model-specified joint is sufficient for predictive coherence. The general definition, however, does not require the model to specify a joint law over (Y,Z)(Y,Z) independently of the update-induced joint QQ.

Local coherence. Predictive coherence is a local condition: it is defined relative to a realised history hh, a target YY, and a specified update setup comprising the ordered design dd and its corresponding update block ZZ. Coherence for one such setup does not imply coherence at another history, for another target, or under another update design. This locality is both a limitation and a feature. It limits the scope of any coherence guarantee, but focuses the requirement on the predictions and updates that are relevant to the problem under consideration. Incoherence outside this scope need not affect the intended use of the procedure. From a predictivist perspective, coherence concerns the compatibility of predictions about the observable quantities YY and ZZ across the histories reachable within this update setup. It therefore makes no general claim about the model’s latent parameters, posterior approximation, or global Bayesianity. We call a predictive procedure globally coherent only if the coherence condition holds for every admissible history, target, and update design.

Incoherence admits a Dutch book. Incoherence can have direct consequences for tasks that rely on a predictive procedure. The canonical demonstration is a diachronic Dutch book: a sequence of contracts that the procedure regards as individually fair at the histories where they are offered, but whose combined payoff is a sure loss. The classical construction of Lewis [ [8] shows qualitatively that such exploitation is possible whenever the procedure’s current valuation of some payoff differs from its expected valuation after updating. We quantify the maximum severity of this exploitation under the betting protocol and a bounded-liability constraint. Writing DTV:=TV⁡(PY,QY)D_{\mathrm{TV}}:=\operatorname{TV}(P_Y,Q_Y), the maximum fraction of initial wealth which can be extracted as a sure loss is DTV/(1+DTV)D_{\mathrm{TV}}/(1+D_{\mathrm{TV}}). This Dutch-book result concerns the internal consistency of the procedure’s valuations and it does not imply that either predictive distribution is empirically inaccurate. Appendix § 9.§ 1 gives the classical construction and derives the maximum sure-loss fraction.

Bayesian models and coherence

Exact Bayesian conditioning provides the canonical sufficient condition for predictive coherence, but it is often unavailable. The familiar prior-likelihood construction in Bayesian deep learning has an intractable parametric posterior which must be approximated. Practical approximations include sampling-based methods such as SG-MCMC [Ma et al., 2015], variational inference over a restricted family [19], such as the diagonal Gaussian used by IVON [20], and the linearised Laplace approximation [21]. Whilst each targets a coherent Bayesian model, the approximations can induce incoherence in the resulting predictive procedure.

Amortised approaches such as neural processes [22 b] and prior-data fitted networks [24] bypass explicit parametric inference and learn a direct map from an observed dataset to a predictive distribution. They approximate the posterior predictive defined by their training prior, and in practice they too fall short of exact coherence, as observed for TabPFN [13]. Similarly, deep ensembles [25] can be interpreted as approximating Bayesian model averaging [26], despite their members not generally being samples from the posterior of a specified BNN [27]. These constructions demonstrate that a procedure can be approximately Bayesian without an explicit target model. A procedure’s Bayesian provenance does not determine how severe its incoherence will be, and we therefore separate construction from empirical behaviour.

Cost of incoherence

Incoherence matters when an action is taken before an update but valued by the predictions the procedure will report after it. We consider a decision problem spanning a single update. An initial action a∈Aa \in\mathcal{A} is taken, a block ZaZ^a is observed, the procedure is updated, a terminal action b∈Bb \in\mathcal{B} is chosen from the post-update predictive, and a loss ℓ(a,b,y)\ell(a,b,y) is realised at the target. The design of the block may depend on the initial action, so we index by the action, writing the update-induced joint as Qa(dy,dz)=PYa,Z(dy)PZa(dz)Q^a(dy,dz) = P_Y^{a,Z}(dy)P_Z^a(dz). Prediction-oriented experimental design is a problem of this form, and problems spanning several updates, as in Bayesian optimisation [5], are covered when each action is chosen greedily. The terminal action depends on the procedure only through the post-update predictive, which is a conditional of QaQ^a. Of the predictions consumed in valuing the initial action, only the current predictive PYP_Y can therefore disagree with the update-induced joint QaQ^a.

Action coherence. We coherentise an objective by substituting QYaQ_Y^a for PYP_Y wherever it appears, writing JJ for the stated objective and J+J^+ for its coherentisation, and a∗a^* and a+a^+ for the actions they select. A procedure is action coherent at hh for (A,J)(\mathcal{A}, J) when coherentising leaves the decision unchanged, a∗=a+a^* = a^+, with ties broken identically. Predictive coherence for every block in {Za:a∈A}\{Z^a : a \in\mathcal{A}\} implies action coherence for every objective, but not conversely, so action coherence is a relaxation that registers only the incoherence that reaches the action. Objectives that coincide under coherence share the same coherentisation, so two such objectives selecting different actions is sufficient for at least one of them to be action incoherent. Coherentisation is internal to the procedure and is not a correction towards the truth, since QaQ^a may describe the true process poorly. It removes inconsistency, not inaccuracy.

Severity of incoherence. What reaches the action is not the size of the discrepancy between QYaQ_Y^a and PYP_Y, but how it varies across candidates relative to how sharply the objective separates them, and how the objective consumes PYP_Y. A large discrepancy that is near-constant across candidates barely moves the decision, so detecting predictive incoherence establishes that a decision may be affected without indicating by how much. Incoherence also decouples objectives that coincide under coherence, and no member of the decoupled family is the correct one, since the current predictive and the expected future predictive disagree and neither is guaranteed to be the more accurate. Which member selects better actions under the true law is an empirical question we address in § 6.

Objective splitting in experimental design

Bayesian experimental design (BED) [4] selects designs which maximise information about a downstream quantity. In prediction-oriented design, that quantity is the procedure’s prediction at deployment. Here A\mathcal{A} is a set of candidate designs, ZaZ^a is the observation acquired at the chosen design, and the terminal action reports a predictive for a target YY under the log score. The expected predictive information gain (EPIG) [6] acquires the design which most reduces predictive uncertainty at the target, EPIG⁡H(a)=H[PY]−EZ∼PZa[H[PYa,Z]]\operatorname{EPIG}_H(a) = H[P_Y] - \mathbb{E}_{Z \sim P_Z^a}[H[P_Y^{a,Z}]], which with the procedure’s own update is the expected uncertainty reduction of [28]. Read decision-theoretically, EPIG is a design-free constant minus the procedure’s expected log loss after the update, so maximising it minimises that loss. Read information-theoretically, it is the mutual information between the target and the acquired observation under the joint the procedure induces over them. Under incoherence that joint has YY-marginal QYaQ_Y^a rather than PYP_Y, and the mutual information becomes EMI⁡(a)=H[QYa]−EZ∼PZa[H[PYa,Z]]\operatorname{EMI}(a) = H[Q_Y^a] - \mathbb{E}_{Z \sim P_Z^a}[H[P_Y^{a,Z}]], the coherentisation of EPIG⁡H\operatorname{EPIG}_H. The two differ only in their baseline, which is constant in aa for EPIGH\mathrm{EPIG}_H but varies with aa once coherentised, and so can change the ranking. A third standard form is the expected divergence from the current predictive, and both it and EPIGH\mathrm{EPIG}_H depart from EMI by explicit terms (Appendix § 9.§ 2),

EPIGKL(a):=EZ∼PZa[KL⁡(PYa,Z∥PY)]=EMI(a)+KL⁡(QYa∥PY),(3)\mathrm{EPIG}_{\mathrm{KL}}(a) := \mathbb{E}_{Z\sim P_Z^a}\left[\operatorname{KL}\left(P_Y^{a,Z} \mathbin{\|} P_Y\right)\right] = \mathrm{EMI}(a) + \operatorname{KL}\left(Q_Y^a \mathbin{\|} P_Y\right), \tag*{(3)}
EPIGH(a)=EMI(a)+H[PY]−H[QYa].(4)\mathrm{EPIG}_H(a) = \mathrm{EMI}(a) + H[P_Y] - H[Q_Y^a]. \tag*{(4)}

All three forms coincide under coherence and coherentise to EMI, but under incoherence they are different acquisition rules, so even a choice ordinarily made for numerical convenience becomes a choice of acquisition rule. The KL form departs from EMI by a non-negative term that is second order in the discrepancy between QYaQ_Y^a and PYP_Y, approximately half their χ2\chi^2 divergence, whereas the entropy form departs by a signed term that is first order. For small incoherence the KL form therefore stays much closer to EMI, as § 6 confirms.

We make the split concrete with two-parameter Bayesian linear regression under a Gaussian likelihood and prior, comparing exact conditioning against mean-field variational inference, whose factorised posterior cannot represent the intercept–slope correlation induced by training inputs clustered away from the origin. All three forms are available in closed form, so any discrepancy is attributable to incoherence rather than estimation noise. Sweeping the candidate design (Figure 1), they coincide under exact conditioning, and under mean-field VI EPIGH\mathrm{EPIG}_H and EMI rank two candidates in opposite orders. § 6 measures how often this occurs, and what it costs, across model families.

Figure 1. Objective splitting under mean-field VI for Bayesian linear regression. (a) The mean-field model’s current predictive PYP_Y (dashed) and its expected future predictives QYAQ_Y^A and QYBQ_Y^B after acquiring candidates AA and BB. The inset magnifies the peak, where neither expected future predictive matches the current one, so the model is incoherent for both candidates. (b) The three forms of EPIG against the candidate design aa. Under exact Bayes they coincide. Under mean-field VI, EPIGH\mathrm{EPIG}_H prefers BB whereas EMI\mathrm{EMI} prefers AA, and EPIGKL\mathrm{EPIG}_{\mathrm{KL}} (dashed) stays within 0.002 nats of EMI.

Measuring incoherence

§ 3 showed that incoherence can reach decisions. Since few procedures update by exact conditioning, the question that usually matters in practice is how incoherent a procedure is, rather than whether it is exactly coherent, which we return to in § 5. This section develops a scalar measure of that severity, computed from the procedure’s own predictions and updates.

The incoherence ratio

Fix a history hh, a target YY and a design dd with block ZZ. Each draw from QYQ_Y costs an update, so updates are the budget, and we treat the procedure as a black box that can be updated and sampled. A round draws a block Zr∼PZZ_r \sim P_Z, updates the fitted procedure on it, drawing any randomness internal to the update afresh, and returns PYZrP_Y^{Z_r}, so rounds are independent and identically distributed. The subscript on ZrZ_r indexes rounds, not the components of the block. Pituk et al. [ [14] use such rounds to compare test metrics, not predictive laws. The Dutch book of § 2 prices incoherence by TV⁡(PY,QY)\operatorname{TV}(P_Y,Q_Y), but total variation is non-linear in the mixture QYQ_Y and cannot be estimated without bias from RR post-update predictives. We instead use kernel mean embeddings [29], which are linear in the mixture. For a kernel kk with feature map ϕ\phi and ∥ϕ(y)∥2≤κ\lVert\phi(y)\rVert^2 \le\kappa, write μP\mu_P, μQ\mu_Q and μr\mu_r for the embeddings of PYP_Y, QYQ_Y and PYZrP_Y^{Z_r} respectively, and δr:=μr−μP\delta_r := \mu_r - \mu_P for the displacement of round rr. Then Δ:=μQ−μP=E[δr]\Delta:= \mu_Q - \mu_P = \mathbb{E}[\delta_r] and MMD⁡2(PY,QY)=∥Δ∥2\operatorname{MMD}^2(P_Y,Q_Y) = \lVert\Delta\rVert^2. For a characteristic kernel [30], Δ=0\Delta= 0 if and only if the procedure is coherent. Since MMD⁡≤2κDTV\operatorname{MMD} \le2\sqrt{\kappa}D_{\mathrm{TV}}, however, ∥Δ∥\lVert\Delta\rVert bounds the sure loss only from below, and a small Δ\Delta certifies only that incoherence is small in the kernel’s features (Appendix § 10.§ 2).

The incoherence ratio. The squared discrepancy ∥Δ∥2\lVert\Delta\rVert^2 grows with how far a procedure’s updates move its predictive, so procedures that update by different amounts cannot be compared on it. The bias–variance identity E∥δr∥2=∥Δ∥2+V\mathbb{E}\lVert\delta_r\rVert^2 = \lVert\Delta\rVert^2 + V, with V:=E∥δr−Δ∥2V := \mathbb{E}\lVert\delta_r - \Delta\rVert^2, separates the systematic part of an update from its response to the block, and we define the incoherence ratio as

Ψ:=∥Δ∥2V=MMD⁡2(PY,QY)E[MMD⁡2(PYZr,QY)],(5)\Psi:= \frac{\lVert\Delta\rVert^2}{V} = \frac{\operatorname{MMD}^2(P_Y,Q_Y)}{\mathbb{E}\left[\operatorname{MMD}^2(P_Y^{Z_r},Q_Y)\right]}, \tag*{(5)}

the squared signal-to-noise ratio of the update. It is invariant to rescaling the displacements, and Ψ/(1+Ψ)\Psi/(1+\Psi) is the systematic share of the expected squared displacement, so Ψ=1\Psi= 1 means bias and spread contribute equally. For a characteristic kernel, Ψ\Psi vanishes exactly under coherence, and a shift that ignores the block, with V=0V = 0 and Δ≠0\Delta\ne0, has Ψ=∞\Psi= \infty (Appendix § 10.§ 2). Ψ\Psi is a self-consistency check that needs no test set, reference posterior or oracle. Blocks are drawn from PZP_Z rather than from data because coherence concerns the procedure’s beliefs about its own future, and measuring on observed blocks would mix inconsistency with inaccuracy. Like coherence, Ψ\Psi is a property of the procedure for the update problem (h,Y,d)(h,Y,d), including the size of the block, which should match the update the procedure will face. Its level depends on the kernel and the design, so it ranks procedures only within a problem. Since VV includes any randomness internal to the update, a noisier update has a lower Ψ\Psi for the same bias, and Appendix § 10.§ 2 separates the spread driven by the block.

Estimation. Ψ\Psi is estimated from RR rounds, which can be shared with the test of § 5. U-statistics over rounds estimate ∥Δ∥2\lVert\Delta\rVert^2 and VV without bias, reducing with exact embeddings to ∥Δ^∥2−V^/R\lVert\widehat{\Delta}\rVert^2 - \widehat{V}/R and V^\widehat{V}, and the ratio has an O(R−1)O(R^{-1}) bias and a leave-one-round-out standard error (Appendix § 10.§ 1). Near coherence the numerator is a degenerate U-statistic and estimates can be negative, which we report unclipped. Displacements at several evaluation points are pooled as a ratio of summed numerators to summed denominators, which keeps scale invariance. For finite targets we use the Dirac kernel, and for continuous targets random Fourier features [31], which make estimation cheap but are not characteristic. We set their lengthscale to the noise scale of the problem, and in § 6, Ψ\Psi is found to be robust to this choice.

Measuring action incoherence

Ψ\Psi is decision-agnostic. Whether incoherence reaches an action is defined by the coherentised objective J+J^+, which is rarely computable, since it consumes QYaQ_Y^a through a non-linear functional at every candidate. Because objectives that coincide under coherence share a coherentisation (§ 3), the rate at which two tractable objectives from this class disagree witnesses action incoherence, although agreement does not certify its absence. The two measures answer different questions. The disagreement rate concerns a particular pair of objectives and decision problem. Ψ\Psi needs neither, so it can compare procedures before a decision problem is fixed, audit an approximation choice, and attribute a failed sequential procedure to inconsistent updates rather than inaccurate predictions. The link runs in one direction only. With a characteristic kernel, if Ψ=0\Psi= 0 for every block in {Za:a∈A}\{Z^a : a \in\mathcal{A}\}, then the procedure is action coherent for every objective. A positive Ψ\Psi diagnoses susceptibility, the distortion of the objective across candidates. Whether that distortion changes a choice also depends on the margin between candidates, and what a changed choice costs is evaluated under the true law rather than the procedure’s own beliefs (§ 3), so neither is determined by Ψ\Psi. Coherentisation removes inconsistency rather than inaccuracy, so realised performance on an oracle or held-out set remains the decision-relevant comparison, and § 6 examines how susceptibility relates to it.

Testing coherence

The incoherence ratio grades how incoherent a procedure is, but its estimate is not a test of coherence. Near coherence its numerator is a degenerate U-statistic, for which valid intervals need a dedicated construction [32], and Ψ^±2\hat{\Psi} \pm2 SE is not a confidence interval. Whether a procedure is exactly coherent matters most when its incoherence ratio is near zero and its joint is implicit in its update rule, so that coherence cannot be settled by construction. We therefore test H0:PY=QYH_0 : P_Y = Q_Y, so that rejecting a heuristic procedure is evidence of incoherence with a controlled type I error. Earlier checks of this null calibrate by bootstrap against a reference Bayesian model [12] or on test-set metrics [14], whereas ours needs neither. A batch MMD test [29] would fix the number of updates in advance, and its permutation calibration fails when a round’s samples share a block. We instead adopt testing by betting [33], whose anytime validity [34] lets the test stop once the evidence is decisive.

Sequential kernel test of coherence

Our test is the kernel-MMD betting test of Shekhar and Ramdas [ [35] with its plug-in witness and the one-sided online Newton step stakes of Podkopaev et al. [ [36], applied to update rounds rather than sample pairs. With the kernel, embeddings and displacements of § 4.1, coherence implies the kernel null H0k:E[δr]=0H_0^k : \mathbb{E}[\delta_r] = 0 for any kernel, so the test is valid whatever the kernel. The kernel determines only which departures can be detected. A quadratic kernel, for example, tests coherence only in mean and variance.

Each round draws Zr∼PZZ_r \sim P_Z, updates the fitted procedure, and estimates δr\delta_r by δ^r\hat{\delta}_r, the difference between the embeddings of PYZrP_Y^{Z_r} and PYP_Y, computed exactly or from samples (Appendix § 10.§ 3). Unlike the estimator of Ψ\Psi, the test draws fresh samples of PYP_Y in every round. Rounds are then independent given the fitted procedure, so E[δ^r∣Fr−1]=Δ\mathbb{E}[\hat{\delta}_r \mid\mathcal{F}_{r-1}] = \Delta, where Fr−1\mathcal{F}_{r-1} is everything observed before round rr. A pool shared across rounds would couple the bets through its Monte Carlo error and break this property. Before round rr, the test forms the running mean Dr−1:=1r−1∑i<rδ^iD_{r-1} := \frac{1}{r-1}\sum_{i<r}\hat{\delta}_i and the plug-in witness ur:=Dr−1/∥Dr−1∥u_r := D_{r-1}/\lVert D_{r-1}\rVert, or ur:=0u_r := 0 if ∥Dr−1∥≤ε\lVert D_{r-1}\rVert\le\varepsilon for a fixed noise floor ε\varepsilon. It then bets on the projection of the new displacement onto the witness,

br:=⟨ur,δ^r⟩2κ,Er:=Er−1(1+λrbr),E0:=1,(6)b_r := \frac{\langle u_r,\hat{\delta}_r\rangle}{2\sqrt{\kappa}}, \qquad E_r := E_{r-1}(1+\lambda_r b_r), \qquad E_0 := 1, \tag*{(6)}

where br∈[−1,1]b_r \in[-1,1] since ∥δ^r∥≤2κ\lVert\hat{\delta}_r\rVert\le2\sqrt{\kappa}, and the stake λr∈[0,1/2]\lambda_r \in[0,1/2] is set from earlier rounds by the online Newton step [37] betting strategy of [38]. Since uru_r and λr\lambda_r depend only on Fr−1\mathcal{F}_{r-1}, E[br∣Fr−1]=⟨ur,Δ⟩/(2κ)\mathbb{E}[b_r \mid\mathcal{F}_{r-1}] = \langle u_r,\Delta\rangle/(2\sqrt{\kappa}), which is zero under the null. The wealth (ErE_r) is then a non-negative martingale with E[Er]=1\mathbb{E}[E_r] = 1, and Ville’s inequality [39] gives PH0(sup⁡rEr≥1/α)≤α\mathbb{P}_{H_0}(\sup_r E_r \ge1/\alpha) \le\alpha. Rejecting the first time Er≥1/αE_r \ge1/\alpha therefore controls the type I error at every stopping time [40], and no update is spent after the evidence is decisive.

Under an alternative with ∥Δ∥≥ε\lVert\Delta\rVert\ge\varepsilon, the witness converges to Δ/∥Δ∥\Delta/\lVert\Delta\rVert and log⁡Er\log E_r grows linearly in rr, so the test is consistent above the noise floor, by the argument that Shekhar and Ramdas [ [35] and Podkopaev et al. [ [36] give for their tests (Appendix § 10.§ 3). Applied to rounds, their bounds on the growth rate show that it is set mainly by the absolute bias ∥Δ∥/(2κ)\lVert\Delta\rVert/(2\sqrt{\kappa}) rather than by Ψ\Psi, because bets are scaled by the worst-case bound 2κ2\sqrt{\kappa}, which keeps the test valid without assumptions on the procedure but makes it conservative when updates are small (Appendix § 10.§ 4). Failing to reject within a budget of updates does not certify coherence, since incoherence below the noise floor, outside the features of the kernel, or too small to accumulate evidence within the budget goes undetected.

Testing at multiple evaluation points. Coherence is local, but a procedure is used at many evaluation points, and incoherence at any of them matters. We test the joint null that PY=QYP_Y = Q_Y at every evaluation point l=1,…,Ll = 1,\ldots,L. A single update gives a displacement at every point, so testing at LL points costs no more updates than testing at one. Running the construction separately at each point gives wealths El,rE_{l,r}, which share the same updates and are dependent, so their product is not a valid test martingale. Under dependence they are instead combined by averaging [41]: an average of martingales with respect to a common filtration is itself a martingale, and we reject the first time Eˉr:=1L∑lEl,r≥1/α\bar{E}_r := \frac{1}{L}\sum_l E_{l,r} \ge1/\alpha. Since Eˉr≥max⁡lEl,r/L\bar{E}_r \ge\max_l E_{l,r}/L, incoherence confined to a single point costs at most log⁡L\log L in log-evidence relative to testing at that point alone (Appendix § 10.§ 3).

Experiments

The first two experiments ask whether incoherence measures what it should and what it reveals about models, in a controlled sparse GP setting and across models on real tabular datasets. The last two ask whether incoherence reaches decisions and what it costs, within mean-field Bayesian linear regression and across models on a synthetic neural-network testbed. Throughout, Ψ\Psi is pooled over evaluation points, and tests of coherence use level α=0.05\alpha= 0.05. Full experimental details are in Appendix § 11.

Incoherence within sparse GPs. In a controlled sparse GP setting, the number of inducing points MM controls the resolution of the approximation. Functions are drawn from a GP prior, and each task observes n=24n = 24 points before being updated on a block of K=8K = 8 additional points, with incoherence measured at 64 evaluation points. We draw 20 functions (environments) and form 10 tasks from each by subsampling the observed, block and evaluation points, reporting the mean over tasks. Sparse GPs use the variational approximation of Titsias [ [42], with the kernel hyperparameters fixed at their true values. Inducing points are chosen deterministically by greedy variance selection [43], and we compare an update procedure in which the inducing points are chosen before the update and held fixed with one in which they are re-selected during the update. The two make identical initial predictions, so they differ only in how they update. We also include an incremental transformer neural process (incTNP) [44] trained on the same GP prior, which updates by extending its context (Appendix § 11.§ 1). Within each update procedure, increasing the resolution of the approximation reduces incoherence and improves pre-update accuracy (Figure 2a). With fixed inducing points, incoherence plateaus since even when M=nM = n, the update cannot adequately represent the new data. Re-selection avoids this, but since with a fixed budget it must give up coverage of the existing data to accommodate the block, so its update departs from the current approximation, making it less coherent when MM is small and more coherent when MM is large. Re-selection is nevertheless no less accurate after the update, even where it is the less coherent procedure (Figure 2b), so across update procedures incoherence does not order post-update accuracy. Rather, it registers that current and future predictions disagree, not which of them is wrong. The incTNP is as coherent as the fixed sparse GPs with nearly all the inducing points while fitting less accurately, and after the update it is more accurate than all of them, lying close to the re-selected curve, consistent with an approximation that adapts to new data. The test never rejects the exact GP, and within the fixed procedure rejects more coherent approximations more slowly (Figure 2c). Across families its speed to reject is not set by Ψ\Psi alone, as the incTNP is rejected faster than fixed sparse GPs of similar Ψ\Psi.

Three-panel plots showing pre-update and post-update incoherence ratios and fraction rejected

Figure 2. Sparse GPs with fixed (blue) or re-selected (green) inducing points, and an incTNP. (a, b) Ψ\Psi against KL from the exact GP before and after the update. Vertical lines join procedures with the same number of inducing points, and bands show ±2\pm2 standard errors. (c) Fraction of tasks rejected at α=0.05\alpha= 0.05. The exact GP is never rejected.

Incoherence across models on tabular data. We measure incoherence for models of several kinds on three regression and five classification datasets, over ten random splits (one for TabPFN), updating every model on random blocks of further inputs with outputs drawn from the model itself (Appendix § 11.§ 2). Exact methods have Ψ\Psi indistinguishable from zero and are never rejected. The estimates are robust to the kernel: on the regression datasets, halving or doubling the lengthscale swaps only models whose Ψ\Psi is within ∼15%\sim15\% of each other. Ψ\Psi responds to how a model updates, even when its current predictions are unchanged. Refitting a GP’s hyperparameters at each update leaves its current predictions identical to the exact GP’s but introduces a small incoherence, as Mlodozeniec et al. [2024] find for exchangeability. Refitting TabPFN’s preprocessing on the updated context changes its Ψ\Psi by up to a factor of six at similar log loss, although the two variants also differ before the update. An approximately Bayesian provenance is also no guarantee of coherence. For example, the update in BART reruns a sampler that mixes poorly within a run, and its median Ψ\Psi of 0.71 is comparable to the regression medians of nearest neighbours (0.47) and CatBoost (0.90). Plug-in logistic regression, by contrast, is about as coherent as the Laplace approximations, with a median Ψ\Psi below 0.01. The ordering of models is largely shared across datasets, with rank correlations of at least 0.88 between them, but the level of Ψ\Psi is not (kernel ridge ranges from 0.008 to 0.12), so Ψ\Psi ranks models only within a problem. Finally, accuracy does not imply coherence. TabPFN, which departs from the martingale condition [13], has the lowest log loss on seven of the eight datasets in one variant or the other, yet on every classification dataset its Ψ\Psi is an order of magnitude or more above that of the full Laplace approximation, which is nearly coherent but generally less accurate (Table 4).

Regression
ModelΨ\PsiRangeRankRej.
GP0.0000.000–0.0003.50.00
Sparse GP0.0070.004–0.01890.13
GP, refitted0.0020.001–0.0023.50.03
BART0.7100.632–0.86371.00
XBART0.3780.271–0.78751.00
Kernel ridge0.0130.008–0.11980.37
Nearest neighbours0.4700.461–0.701101.00
CatBoost0.9000.596–0.91271.00
TabPFN, network0.1020.089–0.1952–
TabPFN, preproc.0.1180.083–0.2221–
Classification
ModelΨ\PsiRangeRankRej.
Logistic, exact0.0000.000–0.00040.00
Full Laplace0.0040.001–0.00740.02
Diagonal Laplace0.0080.002–0.01250.00
Logistic, plug-in0.0070.001–0.01630.10
Nearest neighbours0.8830.734–0.92781.00
CatBoost0.2730.145–0.33961.00
TabPFN, network0.0710.046–0.1902–
TabPFN, preproc.0.1260.054–0.4252–

Table 4. Summary over datasets. Ψ\Psi is the median over datasets of the model’s mean over splits, with its range. Rank is the median over datasets of the model’s rank by test log loss among the models listed for its task (1 is best, ties averaged). Rej. is the share of the 30 (regression) or 50 (classification) splits rejected within 500 rounds. TabPFN, never rejected on its one split within 128 rounds, is shown as – since this is not comparable.

Incoherence and acquisition within mean-field BLR. Incoherence means that a model’s current and future predictions disagree, and we now ask whether that disagreement reaches decisions. To isolate its effect, we hold the model and its predictions fixed and vary only the coherence of what the acquisition function consumes. We use Bayesian linear regression on random Fourier features and compare exact Bayes with mean-field variational inference, which we compute analytically so that any incoherence comes from the approximation rather than from optimisation error. Starting from 25 points, each model acquires 100 further points one at a time greedily from a pool, choosing each with either the entropy form of EPIG or EMI, and we average over 100 environments. With EPIGH\mathrm{EPIG}_{H}, the mean-field model acquires worse data than random acquisition (Figure 3a). With EMI, the same model nearly matches exact Bayes, with an excess of 0.008 nats, so the difference lies entirely in the objective. EMI is tractable here, but in general it requires the entropy of a mixture, so we also consider making the predictive itself coherent, building a ladder in which the mean-field predictive is made coherent with its own refit over a growing random share ρ\rho of the pool (Appendix § 11.§ 3). Excess loss falls monotonically with Ψ\Psi along the ladder, and coherentising over a small random subset already recovers most of the gap to exact Bayes (Figure 3b). This suggests that coherence need only hold over a representative selection of the pool rather than over every point the model might acquire.

Figure 3. Acquisition with mean-field Bayesian linear regression. Every curve is the test log loss of the mean-field model on the data it acquired. (a) Test log loss by acquisition step. (b) Excess log loss over acquisition with the exact predictive, averaged over steps, against Ψ\Psi. Points are labelled by the share ρ\rho of the pool over which the acquisition predictive is made coherent, with ρ=0%\rho= 0\% the mean-field model with EPIGH\mathrm{EPIG}_{H}. EMI changes only the objective, so it is plotted at the mean-field model’s Ψ\Psi. Bands and error bars show ±2\pm2 standard errors.

Incoherence and acquisition across models. We now ask how incoherence affects acquisition across different approximate models in a more realistic setting. We use a synthetic testbed modelled on the Neural Testbed of Osband et al. [ [16], with classification tasks generated by random networks, in which choosing one of several candidate batches to acquire is small enough that every labelling can be enumerated (Appendix § 11.§ 4). We consider eight approximate models (Laplace approximations, variational methods, deep ensembles, MC dropout and an incTNP) over 1000 environments. We measure the cost of a disagreement between objectives as the excess loss of EPIGHEPIG_H’s choice over EMI’s, and the distortion of the objective as the spread of EPIGHEPIG_H − EMI across candidates. Exact Bayes over a large discrete bank of networks, included as a coherent baseline, never separates the objectives, whereas for every approximate model the entropy form of EPIG disagrees with EMI on 26–80% of decisions and has a higher loss on average. The KL form of EPIG disagrees with EMI on only 1–20% of decisions, as the identity of § 3.1 predicts, making it a potentially tractable substitute for EMI when a model is incoherent. Ψ\Psi tracks how far incoherence distorts the objective (Figure 4a), and within an environment the ranking of models by Ψ\Psi has a rank correlation of 0.73 with their distortion. This makes Ψ\Psi a measure of how susceptible a model’s decisions are to incoherence. It does not tell us whether a choice will flip, which also depends on the margin between the best candidates, or what a flip will cost, which is uncorrelated with Ψ\Psi and instead tracks the model’s accuracy (Figure 4b, Appendix § 11.§ 4). Bayes by backprop and the incTNP, for example, are about equally incoherent, yet the entropy form of EPIG disagrees with EMI on 80% of Bayes by backprop’s decisions against 26% of the incTNP’s, and a disagreement costs 0.030 against 0.0035 nats.

Figure 4. Acquisition on the neural testbed for eight approximate models. Exact Bayes is omitted, as its objectives never disagree. (a) Distortion of the acquisition objective against Ψ\Psi. (b) Cost of a disagreement, the excess loss of EPIGHEPIG_H’s choice over EMI’s, against the model’s KL from the truth before the update. Curves are local linear fits, markers are model means, and bands show the 10th–90th percentiles in (a) and ±2 standard errors in (b). BBB is Bayes by backprop.

Discussion

Many sequential decision procedures act as though a model’s current and future beliefs are consistent. Measuring incoherence quantifies how far approximate models violate this, and flags when a procedure’s decisions are exposed to it. Our experiments show this is more than a theoretical concern, as acquisition objectives that should coincide frequently disagreed and chose different data. Incoherence indicates when this disagreement can occur, but not what it costs, which depends on accuracy. Susceptibility can also depend on the objective. On the neural testbed, the KL form of EPIG stayed much closer to EMI than the entropy form, suggesting it as a cheap way to mitigate some of the effect of incoherence in BED. Whilst our experiments are restricted to BED, our framework applies to any single-update sequential decision problem. Understanding which properties of a problem determine how much incoherence matters would show where these diagnostics are most needed. Extending the framework to procedures that plan through a sequence of actions and updates would cover longer-horizon problems, where the effects of incoherence may compound. Beyond the choice of decision problem, the main limitations of measuring incoherence are the computational cost of many model updates and the scope of what it can tell us. The best models are both accurate and coherent, but coherence alone guarantees little, since exact conditioning in a misspecified model is perfectly coherent and can still be badly wrong. Measuring incoherence can reveal a potential problem, and help in explaining and addressing it, but it cannot substitute for held-out evaluation when choosing a model. Once an accurate model is chosen, making it exactly coherent is currently intractable, and approximate coherentisation over a representative subset of the pool has so far been shown to help only in a simple setting. It nonetheless points to a promising direction. Models that excel at static prediction are rarely designed for sequential decisions, and approaches that account for incoherence could help them perform well there too.

Acknowledgments and Disclosure of Funding

JR was supported by the EPSRC Centre for Doctoral Training in Statistics and Machine Learning (StatML), grant EP/Y034813/1. VS was supported by the EPSRC Centre for Doctoral Training in Modern Statistics and Statistical Machine Learning (StatML), grant EP/S023151/1. The authors declare no competing interests.

Theoretical foundations

Diachronic Dutch book

We follow the diachronic Dutch book of Teller [1973] and Lewis [ [8], in the notation of § 2. Fix the history hh, the target YY and the update design dd with block ZZ. The procedure acts as an agent who will buy or sell any contract at its fair price, the contract’s expected payoff under the agent’s predictive at the history where it is offered. For a payoff f:Y→[0,1]f : \mathcal{Y} \to[0,1], paid once YY is observed, the fair prices before and after the update are

V0(f)=EPY[f(Y)],Vz(f)=EPYz[f(Y)].\mathcal{V}_0(f) = \mathbb{E}_{P_Y}[f(Y)], \qquad\mathcal{V}_z(f) = \mathbb{E}_{P_Y^z}[f(Y)].

Before the update, a contract paying the future price VZ(f)\mathcal{V}_Z(f) once ZZ is revealed has fair price

V‾(f)=EPZ[VZ(f)]=EQY[f(Y)],\overline{\mathcal{V}}(f) = \mathbb{E}_{P_Z}[\mathcal{V}_Z(f)] = \mathbb{E}_{Q_Y}[f(Y)],

where the second equality follows from (2).

Construction. Suppose Δ(f):=V0(f)−V‾(f)>0\Delta(f) := \mathcal{V}_0(f) - \overline{\mathcal{V}}(f) > 0. If instead Δ(f)<0\Delta(f) < 0, replace ff by 1−f1-f, which reverses the sign. At a stake α>0\alpha> 0, a bookmaker executes three trades, each at the agent’s price.

  • (A) At t=0t = 0, sell the agent a contract paying αf(Y)\alpha f(Y) at t=2t = 2, for αV0(f)\alpha\mathcal{V}_0(f).

  • (B) At t=0t = 0, buy from the agent a contract paying αVZ(f)\alpha\mathcal{V}_Z(f) at t=1t = 1, for αV‾(f)\alpha\overline{\mathcal{V}}(f).

  • (C) At t=1t = 1, after Z=zZ = z is observed, buy from the agent a contract paying αf(Y)\alpha f(Y) at t=2t = 2, for αVz(f)\alpha\mathcal{V}_z(f).

Starting from wealth 1, the agent holds W0=1−αΔ(f)W_0 = 1 - \alpha\Delta(f) after the trades at t=0t = 0. At t=1t = 1 it pays αVz(f)\alpha\mathcal{V}_z(f) to settle B and receives the same amount for C, so W1=W0W_1 = W_0. At t=2t = 2 the payoffs of A and C cancel, so W2=1−αΔ(f)W_2 = 1 - \alpha\Delta(f) whatever the values of ZZ and YY, a sure loss of αΔ(f)\alpha\Delta(f).

Bounded liability. At every time, the agent’s wealth must cover the sum of the largest possible payments on the contracts it has sold, without offsetting the contracts it holds. In the construction the agent has sold only B after t=0t = 0 and only C after t=1t = 1, each paying at most α\alpha, so the constraint holds whenever 1−αΔ(f)≥α1 - \alpha\Delta(f) \ge\alpha, that is, α≤1/(1+Δ(f))\alpha\le1/(1 + \Delta(f)). At α=1/(1+Δ(f))\alpha= 1/(1 + \Delta(f)) the sure loss is

1−W2=Δ(f)1+Δ(f).(7)1 - W_2 = \frac{\Delta(f)}{1 + \Delta(f)}. \tag*{(7)}

Maximal sure loss. The loss (7) increases with Δ(f)\Delta(f), whose supremum over ff is DTV:=TV⁡(PY,QY)D_{\mathrm{TV}} := \operatorname{TV}(P_Y,Q_Y). The construction therefore extracts a fraction DTV/(1+DTV)D_{\mathrm{TV}}/(1 + D_{\mathrm{TV}}) of initial wealth, and no book does better. Consider any book of trades at the agent’s prices, in contracts on YY at t=0t = 0 and t=1t = 1 and on ZZ at t=0t = 0, and let W1W_1 be the agent’s wealth after the trades at t=1t = 1. Write f0f_0 and fzf_z for the agent’s net payoffs from the contracts on YY traded at t=0t = 0 and at t=1t = 1 after Z=zZ = z, so that W2=W1+f0(Y)+fz(Y)W_2 = W_1 + f_0(Y) + f_z(Y). Split them into contracts held and sold, f0=u0−s0f_0 = u_0 - s_0 and fz=uz−szf_z = u_z - s_z with u0,uz,s0,sz≥0u_0, u_z, s_0, s_z \ge0. Every trade is fair, so averaging W2W_2 over PYP_Y and then over Z∼PZZ \sim P_Z gives 1−EPY[f0]+EQY[f0]1 - \mathbb{E}_{P_Y}[f_0] + \mathbb{E}_{Q_Y}[f_0]. A sure loss LL, with W2≤1−LW_2 \le1 - L whatever the values of ZZ and YY, therefore satisfies

L≤EPY[f0]−EQY[f0]≤DTV(sup⁡f0−inf⁡f0).L \le\mathbb{E}_{P_Y}[f_0] - \mathbb{E}_{Q_Y}[f_0] \le D_{\mathrm{TV}}(\sup f_0 - \inf f_0).

Bounded liability after the trades at t=1t = 1 requires W1≥sup⁡s0+sup⁡szW_1 \ge\sup s_0 + \sup s_z. Since W2≤1−LW_2 \le1 - L and fz≤szf_z \le s_z, this gives f0=W2−W1−fz≤1−L−W1+sup⁡sz≤1−L−sup⁡s0f_0 = W_2 - W_1 - f_z \le1 - L - W_1 + \sup s_z \le1 - L - \sup s_0, while f0≥−sup⁡s0f_0 \ge-\sup s_0. Hence sup⁡f0−inf⁡f0≤1−L\sup f_0 - \inf f_0 \le1 - L, and L≤DTV(1−L)L \le D_{\mathrm{TV}}(1 - L) gives L≤DTV/(1+DTV)L \le D_{\mathrm{TV}}/(1 + D_{\mathrm{TV}}). The maximal sure loss vanishes if and only if PY=QYP_Y = Q_Y, that is, exactly when the procedure is coherent for YY, hh and ZZ.

Forms of EPIG

This section derives the identities (3) and (4) relating the three forms of EPIG, and shows why the KL form stays closer to EMI than the entropy form. Fix the history hh and the candidate design aa, and write pYp_Y and qYaq_Y^a for the densities of PYP_Y and QYaQ_Y^a. We assume that PYa,z≪PYP_Y^{a,z} \ll P_Y for PZaP_Z^a-almost every zz, and that all entropies and cross-entropies below are finite. Relating the three forms. Since QYaQ_Y^a averages the post-update predictives, QYa=EZ∼PZa[PYa,Z]Q_Y^a = \mathbb{E}_{Z \sim P_Z^a}[P_Y^{a,Z}], Fubini’s theorem gives EZ∼PZaEPYa,Z[log⁡pY]=EQYa[log⁡pY]\mathbb{E}_{Z \sim P_Z^a}\mathbb{E}_{P_Y^{a,Z}}[\log p_Y] = \mathbb{E}_{Q_Y^a}[\log p_Y]. Expanding the KL divergence inside EPIGKL_{\mathrm{KL}} therefore gives

EPIG⁡KL(a)=−EZ∼PZa[H[PYa,Z]]−EQYa[log⁡pY].\operatorname{EPIG}_{\mathrm{KL}}(a)=-\mathbb{E}_{Z\sim P_Z^a}[H[P_Y^{a,Z}]]-\mathbb{E}_{Q_Y^a}[\log p_Y].

Substituting −EQYa[log⁡pY]=H[QYa]+KL⁡(QYa∥PY)-\mathbb{E}_{Q_Y^a}[\log p_Y] = H[Q_Y^a] + \operatorname{KL}(Q_Y^a \mathbin{\|} P_Y) yields (3). Identity (4) is immediate, since EPIGH_H and EMI share the term EZ∼PZa[H[PYa,Z]]\mathbb{E}_{Z \sim P_Z^a}[H[P_Y^{a,Z}]]. Under coherence, QYa=PYQ_Y^a = P_Y, the extra terms in both identities vanish, and all three forms equal EMI. Substituting QYaQ_Y^a for PYP_Y in EPIGH_H or EPIGKL_{\mathrm{KL}} gives EMI, which does not involve PYP_Y, so all three forms coherentise to EMI.

Size of the departures. Write qYa=pY(1+ηga)q_Y^a = p_Y(1+\eta g_a), where gag_a is bounded with EPY[ga]=0\mathbb{E}_{P_Y}[g_a]=0, so that η\eta sets the size of the incoherence. Expanding (1+x)log⁡(1+x)=x+x2/2+O(x3)(1+x)\log(1+x)=x+x^2/2+O(x^3) gives

KL⁡(QYa∥PY)=η22EPY[ga2]+O(η3)=12χ2(QYa∥PY)+O(η3),\operatorname{KL}(Q_Y^a\mathbin{\|}P_Y)=\frac{\eta^2}{2}\mathbb{E}_{P_Y}[g_a^2]+O(\eta^3)=\frac{1}{2}\chi^2(Q_Y^a\mathbin{\|}P_Y)+O(\eta^3),

so the KL form departs from EMI only at second order. The entropy form departs from EMI by H[PY]−H[QYa]=KL⁡(QYa∥PY)+EQYa[log⁡pY]−EPY[log⁡pY]H[P_Y]-H[Q_Y^a]=\operatorname{KL}(Q_Y^a\mathbin{\|}P_Y)+\mathbb{E}_{Q_Y^a}[\log p_Y]-\mathbb{E}_{P_Y}[\log p_Y], whose last two terms equal ηEPY[galog⁡pY]\eta\mathbb{E}_{P_Y}[g_a\log p_Y], so it is first order and can take either sign.

Consequence for rankings. Only differences between candidates affect which design is chosen. For a finite candidate set, let a+a^+ maximise EMI, and let m:=min⁡a≠a+{EMI⁡(a+)−EMI⁡(a)}m:=\min_{a\ne a^+}\{\operatorname{EMI}(a^+)-\operatorname{EMI}(a)\} be its margin over the next best candidate. Since KL divergence is non-negative, identity (3) gives

EPIG⁡KL(a+)−EPIG⁡KL(a)≥m−KL⁡(QYa∥PY)\operatorname{EPIG}_{\mathrm{KL}}(a^+)-\operatorname{EPIG}_{\mathrm{KL}}(a)\ge m-\operatorname{KL}(Q_Y^a\mathbin{\|}P_Y)

for every a≠a+a\ne a^+, so the KL form selects the same design as EMI whenever m>max⁡a≠a+KL⁡(QYa∥PY)m>\max_{a\ne a^+}\operatorname{KL}(Q_Y^a\mathbin{\|}P_Y), a second-order quantity. By identity (4), the same argument guarantees agreement for the entropy form only when m>max⁡a≠a+{H[QYa+]−H[QYa]}m>\max_{a\ne a^+}\{H[Q_Y^{a^+}]-H[Q_Y^a]\}, which is first order.

Measuring and testing incoherence

We use the notation of § 4.1, with a kernel kk on Y\mathcal{Y} whose feature map ϕ\phi has ∥ϕ(y)∥2≤κ\lVert\phi(y)\rVert^2\le\kappa. Round rr draws Zr∼PZZ_r\sim P_Z and returns the post-update predictive PYZrP_Y^{Z_r}, whose mean embedding [46] we write μr\mu_r, and

δr:=μr−μP,\delta_r:=\mu_r-\mu_P,
Δ:=μQ−μP=E[δr],\Delta:=\mu_Q-\mu_P=\mathbb{E}[\delta_r],
V:=E[∥δr−Δ∥2]=E[∥μr−μQ∥2].V:=\mathbb{E}\left[\lVert\delta_r-\Delta\rVert^2\right]=\mathbb{E}\left[\lVert\mu_r-\mu_Q\rVert^2\right].

Each round updates the same fitted procedure, drawing the block and any randomness internal to the update afresh. Since QYQ_Y averages over both, μ1,…,μR\mu_1,\ldots,\mu_R are independent and identically distributed with mean μQ\mu_Q.

Estimation

Every estimator below is a U-statistic [47], an average over pairs of distinct rounds and, when embeddings are estimated from samples, of distinct samples. As for the unbiased MMD estimator of Gretton et al. [ [29], excluding self-pairs is what makes each one unbiased.

Exact embeddings. The numerator and denominator are order-two U-statistics over rounds,

N^:=1R(R−1)∑r≠r′⟨δr,δr′⟩,\widehat{N}:=\frac{1}{R(R-1)}\sum_{r\ne r'}\langle\delta_r,\delta_{r'}\rangle,
V^:=12R(R−1)∑r≠r′∥μr−μr′∥2,\widehat{V}:=\frac{1}{2R(R-1)}\sum_{r\ne r'}\lVert\mu_r-\mu_{r'}\rVert^2,

and Ψ^:=N^/V^\hat{\Psi} := \hat{N}/\hat{V}. For r≠r′r \ne r', independence of the rounds gives E[⟨δr,δr′⟩]=∥Δ∥2\mathbb{E}[\langle\delta_r, \delta_{r'} \rangle] = \lVert\Delta\rVert^2 and E[12∥μr−μr′∥2]=V\mathbb{E}\left[\frac{1}{2}\lVert\mu_r-\mu_{r'}\rVert^2\right] = V, so both are unbiased. With Δ^:=1R∑rδr\hat{\Delta} := \frac{1}{R}\sum_r \delta_r, they equal

V^=1R−1∑r=1R∥δr−Δ^∥2,\hat{V} = \frac{1}{R-1}\sum_{r=1}^{R}\lVert\delta_r-\hat{\Delta}\rVert^2,
N^=∥Δ^∥2−V^/R,\hat{N} = \lVert\hat{\Delta}\rVert^2-\hat{V}/R,

the sample variance of the displacements and a plug-in corrected for E[∥Δ^∥2]=∥Δ∥2+V/R\mathbb{E}[\lVert\hat{\Delta}\rVert^2] = \lVert\Delta\rVert^2+V/R, since the covariance of Δ^\hat{\Delta} has trace V/RV/R.

Sampled embeddings. Round rr provides SS independent samples yr,1,…,yr,Sy_{r,1},\ldots,y_{r,S} from PYZrP_{Y}^{Z_r}, independently across rounds, and every round shares one reference sample of TT independent draws y1′,…,yT′y'_1,\ldots,y'_T from PYP_Y. Define

Krr′:=1S2∑s,s′k(yr,s,yr′,s′)(r≠r′),K_{rr'} := \frac{1}{S^2}\sum_{s,s'} k(y_{r,s},y_{r',s'}) \qquad(r\ne r'),
K~r:=1S(S−1)∑s≠s′k(yr,s,yr,s′),\widetilde{K}_r := \frac{1}{S(S-1)}\sum_{s\ne s'} k(y_{r,s},y_{r,s'}),
KrP:=1ST∑s,tk(yr,s,yt′),K_{rP} := \frac{1}{ST}\sum_{s,t} k(y_{r,s},y'_t),
KP:=1T(T−1)∑t≠t′k(yt′,yt′′).K_P := \frac{1}{T(T-1)}\sum_{t\ne t'} k(y'_t,y'_{t'}).

None of these pairs a sample with itself, so, given PYZ1,…,PYZRP_Y^{Z_1},\ldots,P_Y^{Z_R}, they are unbiased for ⟨μr,μr′⟩\langle\mu_r,\mu_{r'}\rangle, ∥μr∥2\lVert\mu_r\rVert^2, ⟨μr,μP⟩\langle\mu_r,\mu_P\rangle and ∥μP∥2\lVert\mu_P\rVert^2 respectively. Replacing each inner product in the exact-embedding estimators by the corresponding kernel average gives

N^:=1R(R−1)∑r≠r′Krr′−2R∑rKrP+KP,\hat{N} := \frac{1}{R(R-1)}\sum_{r\ne r'}K_{rr'}-\frac{2}{R}\sum_r K_{rP}+K_P,
V^:=1R∑rK~r−1R(R−1)∑r≠r′Krr′,\hat{V} := \frac{1}{R}\sum_r\widetilde{K}_r-\frac{1}{R(R-1)}\sum_{r\ne r'}K_{rr'},
Ψ^:=N^/V^.\hat{\Psi} := \hat{N}/\hat{V}.

Given the predictives, N^\hat{N} and V^\hat{V} are therefore unbiased for their exact-embedding counterparts, and hence unbiased. The sampled V^\hat{V} can be negative, and its Monte Carlo error includes a term of order κ/(SR)\kappa/(S\sqrt{R}), which must be small relative to VV for Ψ^\hat{\Psi} to be reliable.

Reusing the reference sample. The reference sample enters only N^\hat{N}, through KrPK_{rP} and KPK_P. Sharing it across rounds correlates the terms of N^\hat{N} but introduces no bias. Its Monte Carlo error, however, does not shrink with RR: as R→∞R\to\infty, N^\hat{N} still fluctuates about ∥Δ∥2\lVert\Delta\rVert^2 by order ∥Δ∥T−1/2+T−1\lVert\Delta\rVert T^{-1/2}+T^{-1}, against ∥Δ∥(RS)−1/2\lVert\Delta\rVert(RS)^{-1/2} from the round samples. The reference sample should therefore be large relative to RSRS, which is cheap because sampling PYP_Y needs no update, and comparing N^\hat{N} across its two halves checks this.

Cost. With an explicit feature map of dimension DD, such as the Dirac kernel or random Fourier features [31], each round reduces to its embedding in RD\mathbb{R}^D, exact or averaged over SS samples. The point estimates then cost O(RD)O(RD), or O((RS+T)D)O((RS+T)D) with samples, in memory independent of RR, although the standard error below needs all RR embeddings. With only a kernel, they compare every pair of rounds, at cost O(R2)O(R^2) with exact embeddings and O((RS+T)2)O((RS+T)^2) with samples.

Pooling over evaluation points. With estimators N^l\widehat{N}_l and V^l\widehat{V}_l at evaluation points l=1,…,Ll=1,\ldots,L, we report

Ψ^:=∑lN^l∑lV^l.\widehat{\Psi}:=\frac{\sum_l\widehat{N}_l}{\sum_l\widehat{V}_l}.

which estimates Ψ\Psi under the direct-sum kernel over points and so keeps its scale invariance. This pooled Ψ\Psi is the VlV_l-weighted average of the per-point ratios Ψl:=∥Δl∥2/Vl\Psi_l:=\lVert\Delta_l\rVert^2/V_l, whose estimates N^l/V^l\widehat{N}_l/\widehat{V}_l localise incoherence.

Standard errors. With Ψ^(−r)\widehat{\Psi}_{(-r)} the estimate without round rr, and Ψ‾\overline{\Psi} the mean of these RR values, the leave-one-round-out standard error

SE⁡:=(R−1R∑r=1R(Ψ^(−r)−Ψ‾)2)1/2.\operatorname{SE}:=\left(\frac{R-1}{R}\sum_{r=1}^{R}\left(\widehat{\Psi}_{(-r)}-\overline{\Psi}\right)^2\right)^{1/2}.

captures the variability of numerator and denominator jointly. With sampled embeddings it holds the reference sample fixed, so it omits that sample’s Monte Carlo error, which is negligible when T≫RST\gg RS. Since N^\widehat{N} and V^\widehat{V} are unbiased, the ratio has an O(R−1)O(R^{-1}) bias by the delta method [48].

Behaviour near coherence. When Δ=0\Delta=0, E[⟨δr,δr′⟩∣δr]=⟨δr,Δ⟩\mathbb{E}[\langle\delta_r,\delta_{r'}\rangle\mid\delta_r]=\langle\delta_r,\Delta\rangle vanishes, so N^\widehat{N} is a degenerate U-statistic [49]. It is then of order R−1R^{-1} rather than R−1/2R^{-1/2}: RN^R\widehat{N} converges in distribution to a centred weighted sum of χ12\chi_1^2 variables, and N^\widehat{N} is negative with appreciable probability. We report N^\widehat{N} and Ψ^\widehat{\Psi} unclipped, so that averaging over tasks does not bias them upwards, and read negative values as indistinguishable from zero. The standard error then also falls as R−1R^{-1} (Table 2), but it is not consistent: the leave-one-round-out variance of the degenerate N^\widehat{N} has expectation 2(R−1)/(R−2)2(R-1)/(R-2) times its true variance, so the standard error overstates the spread of Ψ^\widehat{\Psi} by a factor of about 2\sqrt{2}. It remains a useful, conservative scale, but, as § 5 notes, Ψ^±2SE⁡\widehat{\Psi}\pm2\operatorname{SE} is not a confidence interval for a procedure that may be exactly coherent.

Table 2: Estimates of Ψ\Psi after RR rounds, as the mean over tasks ± the root mean square of the per-task leave-one-round-out standard errors. The estimates for different RR are prefixes of the same rounds.

ModelClosed formR=10R = 10R=100R = 100R=1000R = 1000
Exact GP0−0.005 ± 0.0900.000 ± 0.0080.0000 ± 0.0007
incTNP–0.017 ± 0.0930.014 ± 0.0100.013 ± 0.002
Fixed, M=10M = 100.1240.127 ± 0.1410.123 ± 0.0190.124 ± 0.005
Fixed, M=24M = 240.0120.016 ± 0.1110.012 ± 0.0110.012 ± 0.002
Re-selected, M=20M = 200.00220.001 ± 0.0930.003 ± 0.0080.0022 ± 0.0008

Table 2.

Properties

Scale invariance and the systematic share. Rescaling every displacement, δr↦c δr\delta_r\mapsto c\,\delta_r with c>0c>0, multiplies ∥Δ∥2\lVert\Delta\rVert^2 and VV by c2c^2 and so leaves Ψ\Psi unchanged. Since E[∥δr∥2]=∥Δ∥2+V\mathbb{E}[\lVert\delta_r\rVert^2]=\lVert\Delta\rVert^2+V,

Ψ1+Ψ=∥Δ∥2∥Δ∥2+V=∥Δ∥2E[∥δr∥2]∈[0,1]\frac{\Psi}{1+\Psi}=\frac{\lVert\Delta\rVert^2}{\lVert\Delta\rVert^2+V}=\frac{\lVert\Delta\rVert^2}{\mathbb{E}[\lVert\delta_r\rVert^2]}\in[0,1]

is the systematic share of the expected squared displacement.

The limit V→0V\to0. If V=0V=0, then μr=μQ\mu_r=\mu_Q almost surely, so for a characteristic kernel [30] PYZr=QYP_Y^{Z_r}=Q_Y almost surely. If Δ≠0\Delta\ne0, the update is a shift that ignores the block, which is wholly systematic, so Ψ=∞\Psi=\infty. If Δ=0\Delta=0, the update leaves the predictive unchanged, which is trivially coherent, and we set Ψ:=0\Psi:=0, so that, for a characteristic kernel, Ψ=0\Psi=0 if and only if the procedure is coherent. For small positive VV, Ψ\Psi still reports whether what little movement there is tracks the block or drifts systematically, and N^\widehat{N} and V^\widehat{V} give the absolute sizes of the bias and the spread.

Randomised updates. Let ξr\xi_r be the randomness internal to the update in round rr, so that δr=δ(Zr,ξr)\delta_r=\delta(Z_r,\xi_r), and let δ‾(z):=E[δ(z,ξ)]\overline{\delta}(z):=\mathbb{E}[\delta(z,\xi)] be the mean displacement for block zz. The law of total variance gives V=VZ+VξV=V_Z+V_\xi, where

VZ:=E[∥δ‾(Zr)−Δ∥2](spread driven by the block),V_Z:=\mathbb{E}\left[\left\lVert\overline{\delta}(Z_r)-\Delta\right\rVert^2\right]\qquad\text{(spread driven by the block),}
Vξ:=E[∥δr−δ‾(Zr)∥2](update noise).V_\xi:=\mathbb{E}\left[\left\lVert\delta_r-\overline{\delta}(Z_r)\right\rVert^2\right]\qquad\text{(update noise).}

Randomising the update without changing its average for each block leaves QYQ_Y, and hence Δ\Delta and the Dutch book, unchanged, while increasing VV and lowering Ψ\Psi. By convexity of −log⁡-\log, it also cannot improve the expected log score, so update noise is not a free way to appear coherent. To exclude update noise, use the block-driven ratio

ΨZ:=∥Δ∥2VZ≥Ψ,\Psi_Z := \frac{\|\Delta\|^2}{V_Z} \ge\Psi,

which equals Ψ\Psi for deterministic updates. It costs a second update per block: for two updates δr(1)\delta_r^{(1)} and δr(2)\delta_r^{(2)} on the same block with independent internal randomness, E[⟨δr(1),δr(2)⟩]=∥Δ∥2+VZ\mathbb{E}[\langle\delta_r^{(1)},\delta_r^{(2)}\rangle] = \|\Delta\|^2 + V_Z, so

V^Z:=1R∑r=1R⟨δr(1),δr(2)⟩−N^\widehat{V}_Z := \frac{1}{R}\sum_{r=1}^{R}\langle\delta_r^{(1)},\delta_r^{(2)}\rangle-\widehat{N}

is unbiased for VZV_Z, with N^\widehat{N} computed from the δr(1)\delta_r^{(1)}.

Embedding discrepancy and total variation. Every ff in the unit ball of the RKHS satisfies ∣f(y)∣=∣⟨f,ϕ(y)⟩∣≤κ|f(y)| = |\langle f,\phi(y)\rangle| \le\sqrt{\kappa}, and ∣EPY[f]−EQY[f]∣≤2∥f∥∞DTV|\mathbb{E}_{P_Y}[f]-\mathbb{E}_{Q_Y}[f]| \le2\|f\|_\infty D_{\mathrm{TV}}, so

∥Δ∥=MMD⁡(PY,QY)=sup⁡∥f∥≤1∣EPY[f]−EQY[f]∣≤2κDTV.\|\Delta\| = \operatorname{MMD}(P_Y,Q_Y) = \sup_{\|f\|\le1}|\mathbb{E}_{P_Y}[f]-\mathbb{E}_{Q_Y}[f]| \le2\sqrt{\kappa}D_{\mathrm{TV}}.

The sure loss DTV/(1+DTV)D_{\mathrm{TV}}/(1+D_{\mathrm{TV}}) of Appendix § 9.§ 1 increases with DTVD_{\mathrm{TV}}, so the embedding discrepancy bounds it from below, by ∥Δ∥/(2κ+∥Δ∥)\|\Delta\|/(2\sqrt{\kappa}+\|\Delta\|), but gives no upper bound in general. For a finite Y\mathcal{Y} with the Dirac kernel, ϕ(y)=ey\phi(y)=e_y, so ∥Δ∥=∥q−p∥2\|\Delta\|=\|q-p\|_2 and DTV=12∥q−p∥1D_{\mathrm{TV}}=\frac{1}{2}\|q-p\|_1, where pp and qq are the probability vectors of PYP_Y and QYQ_Y. Since q−pq-p sums to zero, its positive and negative parts each have ℓ1\ell_1-norm DTVD_{\mathrm{TV}}, so ∥q−p∥22≤2DTV2\|q-p\|_2^2\le2D_{\mathrm{TV}}^2. With ∥q−p∥2≥∥q−p∥1/∣Y∣\|q-p\|_2\ge\|q-p\|_1/\sqrt{|\mathcal{Y}|}, this gives

2DTV∣Y∣≤∥Δ∥≤2DTV,\frac{2D_{\mathrm{TV}}}{\sqrt{|\mathcal{Y}|}} \le\|\Delta\| \le\sqrt{2}D_{\mathrm{TV}},

so in the classification experiments the two agree up to a factor of ∣Y∣/2\sqrt{|\mathcal{Y}|/2}.

The sequential test

The difference between the features of a pair of samples in the test of Shekhar and Ramdas [ [35] becomes the displacement estimate δ^r\widehat{\delta}_r, and their witness of norm 1/21/2 under a kernel bounded by one becomes ur/(2κ)u_r/(2\sqrt{\kappa}).

Displacement estimates. With exact embeddings, δ^r:=δr\widehat{\delta}_r := \delta_r. Otherwise, with an explicit feature map, round rr draws SS samples yr,1,…,yr,Sy_{r,1},\ldots,y_{r,S} from PYZrP_Y^{Z_r} and a fresh set of TT samples yr,1′,…,yr,T′y'_{r,1},\ldots,y'_{r,T} from PYP_Y, and sets

δ^r:=1S∑s=1Sϕ(yr,s)−1T∑t=1Tϕ(yr,t′).\widehat{\delta}_r := \frac{1}{S}\sum_{s=1}^{S}\phi(y_{r,s})-\frac{1}{T}\sum_{t=1}^{T}\phi(y'_{r,t}).

Each embedding or average of features has norm at most κ\sqrt{\kappa}, so ∥δ^r∥≤2κ\|\widehat{\delta}_r\|\le2\sqrt{\kappa} and every bet lies in [−1,1][-1,1]. All our experiments use exact embeddings with κ=1\kappa=1: the Dirac kernel has ϕ(y)=ey\phi(y)=e_y, and random Fourier features are normalised so that ∥ϕ(y)∥2=1\|\phi(y)\|^2=1.

Why the test needs fresh samples. Given the fitted procedure, each round draws the block, the update’s internal randomness and both sets of samples afresh, so δ^r\widehat{\delta}_r is independent of Fr−1\mathcal{F}_{r-1} and E[δ^r∣Fr−1]=Δ\mathbb{E}[\widehat{\delta}_r\mid\mathcal{F}_{r-1}]=\Delta. If every round instead reused one reference sample, its estimate μ^P\widehat{\mu}_P would be fixed by the past, so E[δ^r∣Fr−1]=μQ−μ^P\mathbb{E}[\widehat{\delta}_r\mid\mathcal{F}_{r-1}]=\mu_Q-\widehat{\mu}_P, which under the null is the Monte Carlo error of μ^P\widehat{\mu}_P rather than zero. The witness would learn this drift, and the test would eventually reject a coherent procedure. The batch estimator of Appendix § 10.§ 1 needs only unconditional unbiasedness, so it can share one reference sample, although that sample’s error then does not average out.

Witness and noise floor. The witness uru_r of § 5.1 is the plug-in estimate that Shekhar and Ramdas [ [35] describe for their test and Podkopaev et al. [ [36] use for independence testing, plus our noise floor. We use ε=10−8\varepsilon=10^{-8}, which guards against normalising a numerically zero vector, and D0:=0D_0:=0, so the first round never bets. Any rule for uru_r that depends only on Fr−1\mathcal{F}_{r-1} and has ∥ur∥≤1\|u_r\|\le1 preserves validity, so the floor affects power but not the type I error.

Stakes. The stake is chosen by online Newton steps on the log-wealth [37], proposed for betting by Cutkosky and Orabona [ [38], in the one-sided form of Podkopaev et al. [ [36]. Each evaluation point keeps its own stake and curvature, initialised at λ1:=0\lambda_1 := 0 and A0:=1A_0 := 1. After round rr, with brb_r observed and the wealth already updated using λr\lambda_r,

zr:=br1+λrbr,z_r := \frac{b_r}{1+\lambda_r b_r},
Ar:=Ar−1+zr2,A_r := A_{r-1}+z_r^2,
λr+1:=clip⁡[0,1/2](λr+czrAr),c:=22−log⁡3≈2.22,\lambda_{r+1} := \operatorname{clip}_{[0,1/2]}\left(\lambda_r+c\frac{z_r}{A_r}\right), \qquad c := \frac{2}{2-\log3}\approx2.22,

where zrz_r is the derivative of log⁡(1+λbr)\log(1+\lambda b_r) in λ\lambda at λr\lambda_r. In one dimension, clipping to [0,1/2][0,1/2] coincides with projection in the curvature norm. The stake λr+1\lambda_{r+1} depends only on rounds up to rr, so it is predictable, and the cap at 1/21/2 keeps 1+λrbr≥1/21+\lambda_r b_r\ge1/2, as the logarithmic regret guarantee of online Newton steps against the best constant stake requires.

Validity. Fix an evaluation point. Under the null, E[δ^r∣Fr−1]=0\mathbb{E}[\widehat{\delta}_r\mid\mathcal{F}_{r-1}]=0, and uru_r and λr\lambda_r are Fr−1\mathcal{F}_{r-1}-measurable, so the bounded bets brb_r have conditional mean zero [Waudby-Smith and Ramdas, 2024] and

E[Er∣Fr−1]=Er−1(1+λrE[br∣Fr−1])=Er−1.\mathbb{E}[E_r\mid\mathcal{F}_{r-1}]=E_{r-1}\left(1+\lambda_r\mathbb{E}[b_r\mid\mathcal{F}_{r-1}]\right)=E_{r-1}.

Since 1+λrbr≥1/21+\lambda_r b_r\ge1/2, (Er)(E_r) is a positive martingale with E0=1E_0=1. Ville’s inequality, PH0(sup⁡rEr≥1/α)≤α\mathbb{P}_{H_0}(\sup_r E_r\ge1/\alpha)\le\alpha, then bounds the type I error by α\alpha at any stopping time.

Multiple evaluation points. Let Fr\mathcal{F}_r contain everything observed at every evaluation point up to round rr. By the argument above, each per-point wealth (El,r)(E_{l,r}) is a non-negative martingale with respect to this common filtration, whatever the dependence between points, so their average Eˉr\bar{E}_r [41] is one too, with Eˉ0=1\bar{E}_0=1, so Ville’s inequality applies to it. Equivalently, EˉR\bar{E}_R is a product of predictable convex combinations of the per-point factors,

EˉR=∏r=1R∑l=1Lwl,r(1+λl,rbl,r),wl,r:=El,r−1∑l′=1LEl′,r−1,\bar{E}_R=\prod_{r=1}^{R}\sum_{l=1}^{L}w_{l,r}(1+\lambda_{l,r}b_{l,r}),\qquad w_{l,r}:=\frac{E_{l,r-1}}{\sum_{l'=1}^{L}E_{l',r-1}},

since each factor equals ∑lEl,r/∑lEl,r−1\sum_l E_{l,r}/\sum_l E_{l,r-1} and the product telescopes. Since every El,rE_{l,r} is positive, Eˉr≥max⁡lEl,r/L\bar{E}_r\ge\max_l E_{l,r}/L, so log⁡Eˉr≥max⁡llog⁡El,r−log⁡L\log\bar{E}_r\ge\max_l\log E_{l,r}-\log L. We compute log⁡Eˉr\log\bar{E}_r with a log-sum-exp over points.

Consistency. Following the consistency proofs of Shekhar and Ramdas [ [35] and Podkopaev et al. [ [36], suppose ∥Δ∥>ϵ\|\Delta\|>\epsilon at some point. The running mean Dr−1D_{r-1} converges to Δ\Delta by the strong law of large numbers, so the witness converges to Δ/∥Δ∥\Delta/\|\Delta\| and the conditional mean of the bets to ∥Δ∥/(2κ)>0\|\Delta\|/(2\sqrt{\kappa})>0. The log-wealth at that point then grows at the positive rate g⋆g^\star of Appendix B.4, log⁡Eˉr\log\bar{E}_r grows at least as fast by the bound above, and the test rejects with probability one.

Severity and detectability

The oracle bet. Above the noise floor, the witness converges to u:=Δ/∥Δ∥u:=\Delta/\|\Delta\|. With this witness fixed, the bets

br:=⟨u,δ^r⟩2κ∈[−1,1]b_r:=\frac{\langle u,\widehat{\delta}_r\rangle}{2\sqrt{\kappa}}\in[-1,1]

are independent and identically distributed, with mean and variance

m:=∥Δ∥2κ,s2:=Var⁡(⟨u,δr⟩)4κ,m:=\frac{\|\Delta\|}{2\sqrt{\kappa}},\qquad s^2:=\frac{\operatorname{Var}(\langle u,\delta_r\rangle)}{4\kappa},

where s2s^2 includes any Monte Carlo error in δ^r\widehat{\delta}_r. Write ρ:=m2/s2\rho:=m^2/s^2 for the squared signal-to-noise ratio along the bias. With exact embeddings, the Cauchy–Schwarz inequality gives $\operatorname{Var}(\langle u,\delta_r\rangle)\le E[∥δr−Δ∥2]=V\mathbb{E}[\lVert\delta_r-\Delta\rVert^2]=V, so ρ≥Ψ\rho\ge\Psi, with equality only when the spread lies entirely along the bias. By the logarithmic regret of the stakes (Appendix § 10.§ 3), the log-wealth grows asymptotically at rate

g⋆:=max⁡λ∈[0,1/2]E[log⁡(1+λbr)],g^\star:= \max_{\lambda\in[0,1/2]} \mathbb{E}[\log(1+\lambda b_r)],

which plays the role of power for such tests [51], and the test rejects after roughly log⁡(1/α)/g⋆\log(1/\alpha)/g^\star updates.

Two regimes. Up to constants, the bounds below are those of Podkopaev et al. [ [36] in our notation, which Shekhar and Ramdas [ [35] also give for two-sample tests. For ∣x∣≤1/2|x|\le1/2,

x−x2≤log⁡(1+x)≤x−38x2.x-x^2\le\log(1+x)\le x-\frac{3}{8}x^2.

Apply these with x=λbrx=\lambda b_r, using E[br2]=m2+s2\mathbb{E}[b_r^2]=m^2+s^2. The lower bound gives E[log⁡(1+λbr)]≥λm−λ2(m2+s2)\mathbb{E}[\log(1+\lambda b_r)]\ge\lambda m-\lambda^2(m^2+s^2). The stake λ=m/(2(m2+s2))\lambda=m/(2(m^2+s^2)) attains 14ρ/(1+ρ)\frac{1}{4}\rho/(1+\rho) when it is at most 1/21/2, and otherwise λ=1/2\lambda=1/2 attains more than m/4m/4. The upper bound gives E[log⁡(1+λbr)]≤λm−38λ2(m2+s2)\mathbb{E}[\log(1+\lambda b_r)]\le\lambda m-\frac{3}{8}\lambda^2(m^2+s^2), whose maximum over λ\lambda is 23ρ/(1+ρ)\frac{2}{3}\rho/(1+\rho), and log⁡(1+x)≤x\log(1+x)\le x gives E[log⁡(1+λbr)]≤m/2\mathbb{E}[\log(1+\lambda b_r)]\le m/2. Together,

g⋆≥14min⁡{ρ1+ρ,m},g^\star\ge\frac{1}{4}\min\left\{\frac{\rho}{1+\rho},m\right\},
g⋆≤min⁡{23ρ1+ρ,m2}.g^\star\le\min\left\{\frac{2}{3}\frac{\rho}{1+\rho},\frac{m}{2}\right\}.

The growth rate is thus governed, up to constants, by the smaller of ρ/(1+ρ)\rho/(1+\rho), the systematic share along the bias, and m=∥Δ∥/(2κ)m=\lVert\Delta\rVert/(2\sqrt{\kappa}), the bias relative to the kernel bound. The latter enters because bets are scaled by the worst-case bound ∥δ^r∥≤2κ\lVert\hat{\delta}_r\rVert\le2\sqrt{\kappa}, which keeps the wealth non-negative whatever the procedure does. The stake cap then stops the test betting more heavily when bets are small. A single update typically moves the embedding far less than 2κ2\sqrt{\kappa}, so mm is small and, unless ρ/(1+ρ)<m\rho/(1+\rho)<m, the rate lies between m/4m/4 and m/2m/2, scaling with the absolute bias and barely depending on the spread. This is the low-variance regime of Shekhar and Ramdas [ [35], in which the rate improves from order m2m^2 to order mm, as Podkopaev et al. [ [36] also show. Since Ψ\Psi is scale invariant, procedures with the same Ψ\Psi but different absolute bias are rejected at different speeds, as for the incTNP and the fixed sparse GPs of similar Ψ\Psi in Figure 2c. With exact embeddings, ρ≥Ψ\rho\ge\Psi gives g⋆≥14min⁡{Ψ/(1+Ψ),m}g^\star\ge\frac{1}{4}\min\{\Psi/(1+\Psi),m\}, so Ψ\Psi alone bounds the rate from below only when Ψ/(1+Ψ)≤m\Psi/(1+\Psi)\le m, as in the low signal-to-noise regime.

Conservativeness. Scaling bets by an estimated displacement size would add power, but bets that could then exceed one in magnitude would need clipping, which breaks the mean-zero property under the null unless the displacements are symmetric. For a non-negative kernel, such as the Dirac kernel, ∥δ^r∥2≤2κ\lVert\hat{\delta}_r\rVert^2\le2\kappa, so bets could be scaled by 2κ\sqrt{2\kappa} without loss of validity. A tighter almost-sure bound on δ^r\hat{\delta}_r would need assumptions on how far an update can move the predictive, and exact embeddings, since Monte Carlo error alone can make δ^r\hat{\delta}_r as large as the kernel allows. Asymptotic anytime-valid inference [52] avoids the range bound and adapts to the variance of the bets, but is valid only asymptotically. We keep the non-asymptotic test because it is meant to audit exact coherence, where validity matters more than speed, and grade severity with Ψ\Psi.

Experimental details

This appendix gives the details needed to reproduce each experiment in § 6, one subsection per experiment, and the code is available at https://github.com/jreaso/predictive-coherence.

Compute. The sparse GP, tabular and Bayesian linear regression experiments used about 155 CPU core-hours and 4 GPU-hours on one laptop, and the neural testbed about 8 hours on one NVIDIA H100 GPU. Preliminary runs used roughly as much again.

Sparse GPs

The data are drawn from the GP prior, so the exact GP is the Bayesian ground truth for the sparse GPs and the incTNP.

Environments and tasks. Each of 20 environments is a function drawn from a GP prior (squared-exponential kernel, lengthscale 0.25, unit variance), observed with noise of standard deviation 0.1 at 512 inputs drawn uniformly from [−2,2][-2, 2]. Each environment is split at random 10 times into 24 context points, a block of 8 and 64 evaluation points.

Sparse GPs. Inducing points are a subset of the context inputs, chosen by greedy variance selection [43], and q(u)q(u) is the optimum of Titsias [ [42]. Both update procedures start from this fit. With fixed inducing points, the update keeps them and updates q(u)q(u), whereas re-selection chooses up to MM afresh from all 32 inputs and refits. The predictive includes the variance that the inducing points leave unexplained, but the update absorbs new data through the projected likelihood, which omits it. This mismatch is the source of the incoherence. Figure 2 and Table 1 show M=10,12,…,24M = 10, 12, \ldots, 24, although re-selection at M=24M = 24, with Ψ=5×10−6\Psi= 5 \times10^{-6}, lies below the figure’s axes. Near M=24M = 24, selection stops once the context is explained to numerical precision, so some tasks use fewer than MM inducing points.

Table 1: Results of the sparse GP experiment over 200 tasks. Ψ\Psi is the mean over tasks with one standard error over environments, in closed form for the GP-based models and estimated from 1000 rounds for the incTNP. KL is the geometric mean over tasks, in nats per evaluation point, before and after the update, with values below 10−1010^{-10} clamped before averaging. Rejected is the share of tasks rejected within 1000 rounds at level 0.05.

ModelMMΨ\PsiKL beforeKL afterRejected
Exact GP–0000.00
incTNP–0.0130 ± 0.00083.1×10−33.1 \times10^{-3}2.9×10−32.9 \times10^{-3}0.99
Fixed100.124 ± 0.0060.470.660.99
Fixed120.093 ± 0.0050.180.350.99
Fixed140.066 ± 0.0054.3×10−24.3 \times10^{-2}0.170.95
Fixed160.040 ± 0.0044.5×10−34.5 \times10^{-3}7.0×10−27.0 \times10^{-2}0.83
Fixed180.026 ± 0.0031.6×10−41.6 \times10^{-4}3.1×10−23.1 \times10^{-2}0.66
Fixed200.016 ± 0.0032.1×10−62.1 \times10^{-6}1.4×10−21.4 \times10^{-2}0.46
Fixed220.013 ± 0.0026.0×10−96.0 \times10^{-9}7.1×10−37.1 \times10^{-3}0.40
Fixed240.012 ± 0.0023.5×10−103.5 \times10^{-10}5.3×10−35.3 \times10^{-3}0.38
Re-selected101.16 ± 0.080.470.661.00
Re-selected120.93 ± 0.080.180.341.00
Re-selected140.45 ± 0.044.3×10−24.3 \times10^{-2}0.120.99
Re-selected160.153 ± 0.0214.5×10−34.5 \times10^{-3}2.9×10−22.9 \times10^{-2}0.99
Re-selected180.021 ± 0.0031.6×10−41.6 \times10^{-4}3.3×10−33.3 \times10^{-3}0.83
Re-selected200.0022 ± 0.00032.1×10−62.1 \times10^{-6}2.3×10−42.3 \times10^{-4}0.29
Re-selected22(1.4±0.3)×10−4(1.4 \pm0.3) \times10^{-4}6.0×10−96.0 \times10^{-9}9.6×10−69.6 \times10^{-6}0.02
Re-selected245×10−65 \times10^{-6}3.5×10−103.5 \times10^{-10}1.9×10−71.9 \times10^{-7}0.00

Table 1.

incTNP. The incTNP uses the sequential architecture of Mortimer et al. [ [44], which updates by extending a key–value cache rather than re-encoding the context. It has five layers of width 128 and eight attention heads, and a Gaussian output for each target. We train it on sequences of 49 points from the same prior, predicting each point from those before it, for 400 000 steps with batches of 32. Training uses Muon [53] on hidden weight matrices and AdamW [54] elsewhere, at learning rate 5×10−45 \times10^{-4} and weight decay 0.01, with 40 000 warm-up steps followed by cosine decay.

Measuring and testing incoherence. Each round draws the block’s outputs from the model’s own joint predictive, autoregressively for the incTNP. The kernel on outputs is Gaussian with lengthscale 0.1, the noise standard deviation, and is approximated by 256 random Fourier features. Every predictive is Gaussian, so the embeddings are exact. We run 1000 rounds for each task and model, shared by the estimator and the test of Appendix § 10.§ 3. For the exact and sparse GPs, Ψ\Psi is also available in closed form, which Figure 2 and Table 1 report.

Accuracy. Accuracy is the KL divergence from the exact GP’s predictive to the model’s, computed between their marginals at each evaluation point and averaged over the points. After the update, both are conditioned on the block’s observed outputs. Figure 2 reports its geometric mean over tasks. Ψ\Psi is the same in panels (a) and (b), since it describes a single update of the fitted model.

Results. Table 1 gives the values behind Figure 2, and Table 2 the estimates of Ψ\Psi and their leave-one-round-out standard errors after 10, 100 and 1000 rounds. The estimates agree with the closed form within their standard errors, and match it from 100 rounds. Between 100 and 1000 rounds, the standard error falls roughly as R−1/2R^{-1/2} for clearly incoherent models and roughly as R−1R^{-1} near coherence, where the U-statistic is degenerate.

Tabular data

Data and splits. We use three regression and five classification datasets from OpenML (Table 3). The seven UCI datasets are under CC BY 4.0, and california is listed as public on OpenML. We drop rows with missing values, one-hot encode categorical inputs, and standardise inputs and regression outputs with the training set’s mean and standard deviation. A split divides a dataset at random into a training set (half the rows, at most 1000), a test set (15%, at most 100) and a pool (the rest, at most 2000). Each round draws block inputs, a quarter as many as the training set, without replacement from the pool. The design is therefore random, and QYQ_Y and VV average over it as well as over the block’s outputs. Coherence for every design implies coherence in this sense, but not conversely. Every model is fitted on the training set and updated on the training set and block. All models use the same ten splits per dataset, block inputs and 500 rounds per split, except TabPFN, which because of its cost is run on the first split only, for 128 rounds.

DatasetOpenML idRowsInputsClassesTrainingTestPoolBlock
Regression
concrete4495910308–515100415128
california44977206408–10001002000250
yacht423703086–1544610838
Classification
australian4098169042234510024586
cleveland1942972551484410537
german311000612500100400125
vehicle54846184423100323105
wdbc15105693022848520071

Table 3. Datasets and the sizes of each part of a split. Inputs counts features after one-hot encoding. Sources: concrete [55 b], california [57], yacht [58], australian [59], cleveland [60, 61], german [62], vehicle [63] and wdbc [64], all fetched from OpenML [65].

Exact models. The GP (ARD squared-exponential kernel, hyperparameters maximising the marginal likelihood) conditions on the block. Exact logistic regression, a softmax model with a Gaussian prior on its weights, represents its posterior by 8192 HMC draws [66] and updates by reweighting them, which is exact Bayes with the draws as a discrete prior.

Approximately Bayesian models. These recompute their approximation to update. The sparse GP takes the GP’s hyperparameters and the optimal q(u)q(u) of Titsias [ [42], and updates q(u)q(u) with its inducing points fixed (Appendix § 11.§ 1). Greedy variance selection picks these from the training inputs until the unexplained prior variance per input falls below the noise variance, giving 29 to 100 points. The refitted GP refits the GP’s hyperparameters on the training set and block before conditioning. The full and diagonal Laplace approximations [67] share exact logistic regression’s prior and move to the new mode, taking the full precision or its diagonal there. BART [68] and XBART [69] (stochtree 0.4.5 [70], default priors, 200 trees) predict with the mixture over 100 MCMC draws from one chain (BART) or the last 25 of 40 grow-from-root sweeps (XBART), and update by rerunning the sampler with a fresh seed.

Plug-in models. These refit to update. For regression, kernel ridge and nearest neighbours predict a Gaussian at the fitted function or at the mean of the kk nearest outputs, with the mean squared leave-one-out residual as variance. For classification, plug-in logistic regression predicts at the posterior mode of exact logistic regression, and nearest neighbours the class frequencies among the kk neighbours. CatBoost [71] predicts a mean and variance (RMSEWithUncertainty loss) or class probabilities, with its seed fixed at zero. A plug-in model’s joint is by default the product of its predictives. Kernel ridge, plug-in logistic regression and nearest neighbours are also run with the prequential joint of § 2, in random order, which leaves the current predictive, and so its accuracy, unchanged.

TabPFN. TabPFN [72] (v2.5 weights [73], TabPFN-2.5 licence, one estimator, softmax temperature one, not the default 0.9) conditions on the training set and updates by appending the block to its context. It draws the block’s outputs autoregressively, in the block’s order. The network variant switches off the package’s input and target transforms, fingerprint feature, and feature and class shifts, keeping only the clipping of outlying inputs for classification. The preprocessing variant uses the default configuration, whose transforms are refitted whenever the context changes.

Hyperparameters. Kernel ridge’s penalty and bandwidth, the number of neighbours, CatBoost’s depth and number of trees, and the standard deviation of the logistic regression prior are chosen once per dataset by five-fold cross-validated log likelihood on the first split’s training set. This set contains about half of each other split’s test set, except on california. A regression dataset’s noise scale is the smallest cross-validated RMSE among the tuned models and the GP, BART and XBART, which are scored only for this, and is 0.309 on concrete, 0.499 on california and 0.042 on yacht.

Evaluation. Regression uses a Gaussian kernel with the noise scale as lengthscale, approximated by 256 random Fourier features [31] shared by every model, and classification the Dirac kernel, both giving every predictive an exact embedding. The evaluation points are the test inputs, and the test is that of Appendix § 10.§ 3, sharing the rounds of the Ψ\Psi estimate. Accuracy is the current predictive’s test log loss, with class probabilities clamped below at 10−1610^{-16}, which matters only for nearest neighbours, whose predictive can give a label probability zero.

Results. Table 4 summarises the results over datasets, and Tables 5 and 6 give Ψ\Psi and test log loss on each. The exact models have ∣Ψ∣<5×10−4|\Psi| < 5 \times10^{-4} on every dataset. Exact logistic regression on cleveland is 4.5 standard errors below zero because near coherence the estimate is right-skewed (Appendix § 10.§ 1), so most splits fall below its mean, and rerunning one split 60 times gives an estimate centred on zero. For the inexact models, the standard deviation of Ψ\Psi over the ten splits is 1.8 to 15 times its within-split standard error (median over datasets), so TabPFN’s standard errors, from its one split, understate its variation across splits. Across the models of Table 4, the Spearman correlation of Ψ\Psi between datasets is 0.98 for every regression pair and at least 0.88 for classification, lowest for australian and german.

Robustness. Halving or doubling the lengthscale reorders models only on yacht, swapping BART and CatBoost, XBART and nearest neighbours, and the TabPFN variants, each pair within 15% (Table 7). A lengthscale of one, 2 to 24 times the noise scale, shrinks small values of Ψ\Psi and swaps models up to 23% apart. The prequential joints also change Ψ\Psi, for example from 0.12 to 0.19 for kernel ridge on yacht.

Monte Carlo error in BART. BART’s incoherence is largely Monte Carlo error. On the first split of each regression dataset, rerunning the sampler with fresh seeds on the training set alone gives a squared discrepancy ∥Δ∥2\lVert\Delta\rVert^2 which is 1.1 to 1.2 times that of an update. A run’s 100 draws are worth about two independent draws on concrete and california, and fewer than one on yacht.

Modelconcretecaliforniayacht
GP (×10−4\times10^{-4})−0.6±1.4-0.6 \pm1.42.3±2.22.3 \pm2.22.2±2.42.2 \pm2.4
Sparse GP0.007±0.0010.007 \pm0.0010.004±0.0000.004 \pm0.0000.018±0.0040.018 \pm0.004
GP, refitted0.002±0.0010.002 \pm0.0010.001±0.0000.001 \pm0.0000.002±0.0010.002 \pm0.001
BART0.632±0.0350.632 \pm0.0350.710±0.0510.710 \pm0.0510.863±0.0730.863 \pm0.073
XBART0.378±0.0340.378 \pm0.0340.271±0.0180.271 \pm0.0180.787±0.0750.787 \pm0.075
Kernel ridge0.013±0.0020.013 \pm0.0020.008±0.0010.008 \pm0.0010.119±0.0120.119 \pm0.012
Nearest neighbours0.461±0.0200.461 \pm0.0200.470±0.0170.470 \pm0.0170.701±0.0670.701 \pm0.067
CatBoost0.900±0.0480.900 \pm0.0480.596±0.0450.596 \pm0.0450.912±0.2390.912 \pm0.239
TabPFN, network0.089±0.0080.089 \pm0.0080.102±0.0080.102 \pm0.0080.195±0.0190.195 \pm0.019
TabPFN, preproc.0.083±0.0060.083 \pm0.0060.118±0.0090.118 \pm0.0090.222±0.0180.222 \pm0.018
Modelaustralianclevelandgermanvehiclewdbc
Logistic, exact (×10−4\times10^{-4})2.3±2.02.3 \pm2.0−4.5±1.0-4.5 \pm1.00.0±1.70.0 \pm1.70.9±2.40.9 \pm2.4−1.2±2.0-1.2 \pm2.0
Full Laplace0.003±0.0010.003 \pm0.0010.004±0.0010.004 \pm0.0010.001±0.0000.001 \pm0.0000.007±0.0020.007 \pm0.0020.004±0.0020.004 \pm0.002
Diagonal Laplace0.008±0.0010.008 \pm0.0010.006±0.0010.006 \pm0.0010.002±0.0000.002 \pm0.0000.009±0.0010.009 \pm0.0010.012±0.0020.012 \pm0.002
Logistic, plug-in0.007±0.0030.007 \pm0.0030.003±0.0010.003 \pm0.0010.001±0.0000.001 \pm0.0000.012±0.0080.012 \pm0.0080.016±0.0080.016 \pm0.008
Nearest neighbours0.897±0.0380.897 \pm0.0380.849±0.0360.849 \pm0.0360.734±0.0420.734 \pm0.0420.883±0.0260.883 \pm0.0260.927±0.0590.927 \pm0.059
CatBoost0.145±0.0110.145 \pm0.0110.339±0.0230.339 \pm0.0230.211±0.0120.211 \pm0.0120.327±0.0120.327 \pm0.0120.273±0.0430.273 \pm0.043
TabPFN, network0.190±0.0340.190 \pm0.0340.046±0.0100.046 \pm0.0100.071±0.0130.071 \pm0.0130.142±0.0200.142 \pm0.0200.048±0.0130.048 \pm0.013
TabPFN, preproc.0.098±0.0120.098 \pm0.0120.054±0.0080.054 \pm0.0080.425±0.0340.425 \pm0.0340.248±0.0220.248 \pm0.0220.126±0.0250.126 \pm0.025

Table 5. Ψ\Psi by dataset: the mean over ten splits ±\pm their standard deviation divided by 10\sqrt{10}, or for TabPFN its one split ±\pm the leave-one-round-out standard error.

Detectability. Small incoherence often goes unrejected within the budget. The sparse GP’s Ψ\Psi exceeds twice its standard error on all 30 splits, but only 4 are rejected within 500 rounds, against a predicted median of about 1500. The predicted round, min⁡{1/Ψ^l+4log⁡(L/α)/∥Δ^l∥}\min\{1/\widehat{\Psi}_l + 4\log(L/\alpha)/\lVert\widehat{\Delta}_l\rVert\} over test inputs ll, is a heuristic for the rounds the witness needs to align plus those needed to reach log⁡(L/α)\log(L/\alpha) at the upper bound m/2m/2 on the growth rate (Appendix § 10.§ 4), so it is optimistic. TabPFN’s predicted rounds, 127 to 585, mostly exceed its budget of 128.

Table 6: Test log loss in nats per output, on the standardised outputs for regression: the mean over ten splits, or TabPFN’s one split. The lowest on each dataset before rounding is in bold.

Modelconcretecaliforniayacht
GP0.2830.738−1.803
Sparse GP0.4570.889−1.749
GP, refitted0.2830.738−1.803
BART0.1510.773−1.540
XBART0.0890.738−0.128
Kernel ridge0.4120.762−0.231
Nearest neighbours0.8920.9561.045
CatBoost0.4000.877−1.601
TabPFN, network−0.2540.291−3.021
TabPFN, preproc.−0.3080.237−2.959
Modelaustralianclevelandgermanvehiclewdbc
Logistic, exact0.3551.0560.4880.4340.088
Full Laplace0.3521.0430.4900.4490.094
Diagonal Laplace0.3561.0430.4930.4430.107
Logistic, plug-in0.3561.0740.4880.4280.086
Nearest neighbours0.4091.4210.5980.8400.225
CatBoost0.3231.1900.4900.5000.097
TabPFN, network0.2110.9460.4970.3400.049
TabPFN, preproc.0.2180.9160.5120.3480.034

Table 6.

Table 7: Ψ\Psi with the kernel lengthscale at the noise scale / half it / twice it / one (the training outputs’ standard deviation). The GP, with ∣Ψ∣<5×10−4\lvert\Psi\rvert< 5 \times10^{-4} at every lengthscale, is omitted.

Modelconcretecaliforniayacht
Sparse GP0.007 / 0.008 / 0.005 / 0.0030.004 / 0.004 / 0.002 / 0.0020.018 / 0.020 / 0.015 / 0.001
GP, refitted0.002 / 0.002 / 0.002 / 0.0020.001 / 0.001 / 0.001 / 0.0010.002 / 0.002 / 0.001 / 0.001
BART0.632 / 0.641 / 0.617 / 0.6100.710 / 0.710 / 0.708 / 0.7080.863 / 0.941 / 0.783 / 0.674
XBART0.378 / 0.389 / 0.364 / 0.3570.271 / 0.277 / 0.261 / 0.2610.787 / 0.860 / 0.699 / 0.540
Kernel ridge0.013 / 0.015 / 0.008 / 0.0050.008 / 0.010 / 0.005 / 0.0050.119 / 0.118 / 0.112 / 0.020
Nearest neighbours0.461 / 0.468 / 0.452 / 0.4410.470 / 0.473 / 0.463 / 0.4630.701 / 0.704 / 0.703 / 0.683
CatBoost0.900 / 0.913 / 0.876 / 0.8600.596 / 0.601 / 0.591 / 0.5910.912 / 0.920 / 0.840 / 0.823
TabPFN, network0.089 / 0.095 / 0.084 / 0.0800.102 / 0.099 / 0.105 / 0.1050.195 / 0.167 / 0.246 / 0.269
TabPFN, preproc.0.083 / 0.092 / 0.078 / 0.0750.118 / 0.121 / 0.115 / 0.1150.222 / 0.274 / 0.192 / 0.153

Table 7.

Mean-field Bayesian linear regression

The two-parameter example (Figure 1). The model is y=w0+w1xy = w_0 + w_1 x plus unit-variance Gaussian noise, with prior w∼N(0,4I)w \sim\mathcal{N}(0, 4I), fitted to five inputs drawn from U[4,6]U[4, 6] with outputs 0.2x0.2x plus standard Gaussian noise. The exact posterior correlation between intercept and slope is −0.97-0.97, and the mean-field approximation is formed as in the experiment below. The target is x∗=3.5x_* = 3.5, and the candidate design ranges over [−10,30][-10, 30] in steps of 0.10.1, of which the figure shows [0,18][0, 18] and marks A=2A = 2 and B=16B = 16. Under mean-field VI, EPIGH\mathrm{EPIG}_H and EMI both prefer AA (0.081 against 0.068 nats). Under mean-field VI, EPIGH\mathrm{EPIG}_H prefers BB (0.037 against 0.013), whereas EMI prefers AA (0.045 against 0.014), as does EPIGKL\mathrm{EPIG}_{\mathrm{KL}} (0.046 against 0.015), which stays within 0.0012 nats of EMI over the range shown.

Environments. The acquisition experiment is entirely closed form, so there is no estimation noise beyond the choice of environments. The model is Bayesian linear regression on 100 random Fourier features [31] of a Gaussian kernel with lengthscale 0.1 on a scalar input, with weights w∼N(0,3I)w \sim\mathcal{N}(0, 3I) and noise variance 0.8. The truth is drawn from this prior, so the model is well specified. Each of 100 environments, shared by every arm, draws its features, the true weights, 25 initial inputs from U[−1,1]U[-1, 1], a pool of 1000 inputs and 100 target inputs from U[−10,10]U[-10, 10], noisy labels for the initial and pool inputs, and a random order of the pool. The reported log loss is the learner’s expected log loss for a noisy observation from the truth at each target input, so there is no test set.

Learner and acquisition. The learner keeps the exact conjugate posterior. Its mean-field approximation, the optimum of KL⁡(q∥p)\operatorname{KL}(q \mathbin\Vert p) over factorised Gaussians, takes the exact posterior means and the inverse diagonal of the exact posterior precision as variances. It is refitted after each acquisition rather than updated in a streaming fashion. Each arm acquires 100 points one at a time, maximising its score over the remaining pool. The refit’s variance does not depend on the outcome and its mean is affine in it, so QYQ_Y is Gaussian and EPIGH_H and EMI, averaged uniformly over the targets, are closed form. Exact Bayes acquires with the exact posterior, and random acquisition takes the pool in its random order.

The coherentised predictive. Rung ρ\rho of the ladder scores candidates by EPIGH_H under a predictive made coherent for the mean-field refit on a reference set SS, the first round(ρN)(\rho N) of the NN remaining pool points in the environment’s fixed order. Coherentising an objective (§ 3) would instead substitute QYaQ_Y^a for PYP_Y at each candidate aa. Every predictive here is N(ϕ(x)Tμ,ϕ(x)TΩϕ(x)+σ2)\mathcal{N}(\phi(x)^\mathsf{T}\mu,\phi(x)^\mathsf{T}\Omega\phi(x)+\sigma^2) at xx, where μ\mu is the exact posterior mean and only the weight covariance Ω\Omega varies. With Σ+\Sigma_+ and Ω+\Omega_+ the exact and mean-field covariances after observing SS, B=σ−2Σ+ΦSTB=\sigma^{-2}\Sigma_+\Phi_S^\mathsf{T} and G=BΦSG=B\Phi_S, coherence requires

Ω=Ω++σ2BBT+GΩGT.\Omega=\Omega_+ + \sigma^2BB^\mathsf{T}+G\Omega G^\mathsf{T}.

As GG has spectral radius below one, this discrete Lyapunov equation has the unique solution

∑nGn(Ω++σ2BBT)(GT)n,\sum_n G^n(\Omega_+ + \sigma^2BB^\mathsf{T})(G^\mathsf{T})^n,

which we sum by repeated doubling. In EPIGH(c)_H(c), the coherentised predictive supplies H[PY]H[P_Y] and the law of the outcome zz at cc. The post-update predictive PYc,zP_Y^{c,z} is the mean-field refit on S∪{c}S\cup\{c\}, averaged over outcomes at SS drawn from the coherentised predictive given zz. A candidate inside SS is therefore counted both as observed and as part of the block. The reference set is recomputed at every step as the pool shrinks. The shares are ρ∈{0,1,2.5,5,10,100}%\rho\in\{0,1,2.5,5,10,100\}\%, and ρ=0\rho=0 gives mean-field EPIGH_H.

Measuring incoherence. Ψ\Psi is computed before any acquisition and pooled over the 100 targets, with a Gaussian kernel of lengthscale 0.8\sqrt{0.8}, the noise standard deviation. For each rung, the update is the mean-field refit on the whole pool, with outcomes drawn from the rung’s predictive. We use this block, larger than § 4.1 would call for, because the ladder varies coherence with refits on shares of the pool. Figure 3b plots its mean over environments. For the mean-field model, the pool-scale Ψ\Psi is 1.05×10−31.05\times10^{-3}, whereas its Ψ\Psi for an update on a single candidate, the block the acquisition faces, is 0.0460.046, pooled over candidates within each environment and averaged over environments.

Results. Table 8 gives the values behind Figure 3. Every adjacent pair of rungs differs in loss by at least four standard errors of the paired difference over environments. Coherentising over 1%, 2.5%, 5% and 10% of the pool recovers 32%, 51%, 80% and 95% of mean-field EPIGH_H’s excess loss. Mean-field EPIGH_H and EMI choose different points at the first step in every environment, and their 100 acquired points overlap by 13 on average.

AcquisitionExcess loss (nats)Ψ\Psi (×10−3\times10^{-3})
Mean-field, EPIGH_H (ρ=0\rho= 0)0.270 ±\pm 0.0111.05 ±\pm 0.05
Coherentised, ρ=1%\rho= 1\%0.184 ±\pm 0.0090.80 ±\pm 0.04
Coherentised, ρ=2.5%\rho= 2.5\%0.133 ±\pm 0.0080.55 ±\pm 0.03
Coherentised, ρ=5%\rho= 5\%0.054 ±\pm 0.0050.32 ±\pm 0.02
Coherentised, ρ=10%\rho= 10\%0.014 ±\pm 0.0030.13 ±\pm 0.01
Coherentised, ρ=100%\rho= 100\%0.002 ±\pm 0.0020
Mean-field, EMI0.008 ±\pm 0.002–
Random0.156 ±\pm 0.008–

Table 8. Results of the Bayesian linear regression experiment over 100 environments. Excess loss is the mean-field learner’s log loss less that after acquiring with the exact predictive, averaged over the 100 steps, with one standard error over environments. Ψ\Psi is measured before acquisition for an update on the whole pool.

Robustness. We rerun the experiment with noise variance 0.1 or 3, with targets from U[4,8]U[4, 8], away from most of the pool, and with a truth on independent random features, so that the model is misspecified (Table 9). Mean-field EPIGH_H acquires worse data than random in every setting. With low noise the mean-field variances are badly wrong: before acquisition, the mean-field log loss exceeds the exact one by 1.31 nats, against 0.045 in the baseline. EMI, which still consumes these variances, then does no better than random, whereas the coherentised predictive, which changes them, still comes close to exact Bayes.

SettingMean-field EPIGH_HMean-field EMIRandomρ=5%\rho= 5\%ρ=100%\rho= 100\%
Baseline0.270 ±\pm 0.0110.008 ±\pm 0.0020.156 ±\pm 0.0080.054 ±\pm 0.0050.002 ±\pm 0.002
Low noise1.966 ±\pm 0.0700.920 ±\pm 0.0270.929 ±\pm 0.0370.033 ±\pm 0.0070.009 ±\pm 0.004
High noise0.065 ±\pm 0.0040.003 ±\pm 0.0010.042 ±\pm 0.0030.017 ±\pm 0.0030.001 ±\pm 0.002
Distant targets0.385 ±\pm 0.0190.006 ±\pm 0.0020.270 ±\pm 0.0130.017 ±\pm 0.003−0.001±0.002-0.001 \pm0.002
Misspecified0.537 ±\pm 0.0180.016 ±\pm 0.0030.454 ±\pm 0.0160.204 ±\pm 0.0090.008 ±\pm 0.002

Table 9. Excess loss as in Table 8, with one standard error over environments, under changes to the baseline. Low and high noise set the noise variance to 0.1 and 3, against 0.8 in the baseline.

Neural testbed

Our testbed is modelled on the Neural Testbed Osband et al., [16], with batch acquisition in place of its joint-prediction benchmark.

Environments. The truth is a random ReLU network ff (two inputs, two hidden layers of 32 units, three outputs) with Xavier-normal weights Glorot and Bengio, [74] and biases uniform on ±1/fan-in\pm1/\sqrt{\text{fan-in}}. Its class probabilities are softmax⁡((f(x)−o)/0.35)\operatorname{softmax}((f(x)-o)/0.35), with offsets oo giving each class average probability 1/31/3 over a grid on [−6,6]2[-6, 6]^2. Inputs come from four fixed Gaussian clusters centred at (±3,±3)(\pm3, \pm3), each input’s cluster uniform over those allowed. The 24 training inputs use three clusters and standard deviation 0.5. The pool of 512 uses all four, and the 64 targets and 4096 test inputs two, one with training data and one without, all with standard deviation 1.5. Environments are kept if every class has mean true probability at least 0.1 over the test inputs, leaving 1000 of 1160.

The decision. Each environment has ten candidate batches of five pool inputs, shared by all models. Eight are designed from per-input BALD (label–parameter mutual information) under a linearised diagonal Laplace approximation fitted to the training data. They take the most informative inputs from the pool, target clusters or training clusters (greedily, with a minimum spacing), an informative input and its nearest neighbours, inputs spread over the pool or target clusters, inputs near the decision boundary, or a mix of target and training inputs. The other two, and replacements for duplicated designed batches, are sampled with random cluster proportions, about three per environment.

Models. Network models share the truth’s architecture without its temperature or offsets, and are fitted to the 24 training points. Exact Bayes condition 8192 networks drawn per environment from the generative prior, with temperature and offsets. The full, Kronecker-factored Martens and Grosse, [75]; Ritter et al. [76] and diagonal linearised Laplace approximations Immer et al., [21] share a MAP fit (Adam, Kingma and Ba, 2015), learning rate 0.05, 1000 steps, prior scale four times the generative one) and use generalised Gauss–Newton curvature Schraudolph, [78]. They condition the linearised model on the batch via a Laplace approximation in its logits (16 damped Newton steps). Bayes by backprop Blundell et al., [79] (Adam, learning rate 0.003, 8 samples) and IVON Shen et al., [20] (learning rate 0.03, 4 samples) fit factorised Gaussians under the generative prior for 2000 steps, with KL weight 0.1. They update by variational continual learning Nguyen et al., [80], with the current posterior as prior and the five new points as data (same KL weight, 100 steps, learning rates 3×10−43 \times10^{-4} and 10−310^{-3} respectively). The deep ensemble Lakshminarayanan et al., [25] (8 members, Adam, learning rate 0.05, 1000 steps, weight penalty 3) and MC dropout Gal and Ghahramani, [81] (rate 0.2, learning rate 0.05, 500 steps, weight penalty 1) update by continuing training on all 29 points, for 1000 and 50 steps respectively. The Laplace and variational predictives mix 128 parameter draws, parameterised by each posterior from standard-normal base draws fixed once per environment, and MC dropout’s mixes 128 masks fixed once and shared by every predictive. The incTNP Mortimer et al., [44] (eight layers of width 256, eight attention heads) updates by appending the batch to its context. It is trained on generative tasks to predict each training point from its predecessors and each target from the training points and part of a batch, with Muon Jordan et al., [53] on hidden weight matrices and AdamW Loshchilov and Hutter, [54] elsewhere (learning rate 5×10−45 \times10^{-4}, cosine decay, 90 000 steps of 128 tasks). Hyperparameters were tuned on 64 separate environments by the KL from the truth before and after updating. Only IVON’s update learning rate and the ensemble’s and MC dropout’s update steps lie at the edge of a grid of over two values.

Objectives, incoherence and cost. For each candidate aa, the model is updated on all 35=2433^{5}=243 labellings of the batch, so the objectives of § 3.1 are exact sums weighted by its joint PZaP^{a}_{Z}, averaged over the 64 targets. Mixtures label each input independently given a component, and the incTNP labels them autoregressively. Ψ\Psi and ∥Δ∥2\lVert\Delta\rVert^{2} use the Dirac kernel, pooled over the targets, and an environment’s value is the median over its candidates’ updates. The random updates of Bayes by backprop, IVON and MC dropout are drawn twice from independent streams, and the objectives and losses use the first. ∥Δ∥2\lVert\Delta\rVert^{2} is estimated without bias by the inner product of the two draws’ PZaP^{a}_{Z}-weighted mean displacements, and VV by their mean squared displacement less that estimate. These estimates are negative in 1, 6 and 36 of the 1000 environments respectively, and are kept. The distortion is the standard deviation of EPIG⁡H−EMI⁡\operatorname{EPIG}_{H}-\operatorname{EMI} over the ten candidates. A candidate’s loss is the KL from the truth to the updated predictive at the test inputs, averaged over the batch’s labellings under the truth, and a decision’s cost is the loss of EPIGH_{H}’s choice less EMI’s. Accuracy is the KL from the truth to the fitted predictive, averaged over the test inputs.

Figure 4. The local linear fits use a Gaussian kernel with bandwidth 0.16 in log⁡10Ψ\log_{10}\Psi in (a), leaving out environments with Ψ≤0\Psi\le0, and 0.06 nats in (b). The percentiles in (a) are kernel-weighted and the standard errors in (b) approximate. Markers are means over all environments in (a), unlike the medians of Table 10, and over those where the objectives disagree in (b).

Disagreement
ModelΨ\Psi∥Δ∥\lVert\Delta\rVertEPIG⁡H\operatorname{EPIG}_{H}EPIG⁡KL\operatorname{EPIG}_{KL}CostKL
Exact Bayes000.000.0000.171 ±\pm 0.003
IVON0.00580.0410.280.010.0084 ±\pm 0.00180.366 ±\pm 0.007
Laplace, KFAC0.00670.0590.400.010.0101 ±\pm 0.00200.310 ±\pm 0.007
Bayes by backprop0.01020.0940.800.100.0245 ±\pm 0.00270.362 ±\pm 0.007
incTNP0.01030.0970.260.020.0009 ±\pm 0.00050.168 ±\pm 0.003
Ensemble0.01810.1180.520.080.0195 ±\pm 0.00330.406 ±\pm 0.009
Laplace, full0.01910.1710.330.040.0071 ±\pm 0.00190.270 ±\pm 0.004
Laplace, diagonal0.04740.3180.380.080.0104 ±\pm 0.00200.269 ±\pm 0.004
MC dropout0.1170.4250.540.200.0300 ±\pm 0.00690.478 ±\pm 0.012

Table 10. Results on the neural testbed. Ψ\Psi and ∥Δ∥\lVert\Delta\rVert are medians over environments. Disagreement is the share of decisions where each form of EPIG differs from EMI. Cost is averaged over all decisions, with agreements counting as zero, so the cost of a disagreement is Cost divided by the EPIG⁡H\operatorname{EPIG}_{H} disagreement. KL measures accuracy before the update. Both are in nats with one standard error.

Results. The per-environment Spearman correlation between the eight models’ Ψ\Psi and distortion has mean 0.73 and median 0.79 over environments, and is positive in 996 of 1000. EPIG⁡H\operatorname{EPIG}_{H} has a higher mean loss than EMI for every approximate model, by over two standard errors except for the incTNP (0.0009 ±\pm 0.0005). Yet EPIG⁡H\operatorname{EPIG}_{H} chooses better in 37–49% of disagreements. Disagreement depends on the margin between EMI’s two best candidates relative to the distortion. Pooled over models, its rate falls from 0.74 in the lowest quintile of this ratio to 0.06 in the highest. Over all models’ disagreements, the cost of a disagreement has rank correlation 0.01 with Ψ\Psi and 0.21 with KL. Across the eight models, its mean has rank correlation 0.24 with Ψ\Psi and 0.90 with the KL of Table 10.

References

  1. [1]David J. C. MacKay. Information-based objective functions for active data selection. Neural Computation, 4(4):590–604, 1992a.DOI
  2. [2]David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. Active learning with statistical models. Journal of Artificial Intelligence Research, 4:129–145, 1996.
  3. [3]Edwin Fong, Chris Holmes, and Stephen G. Walker. Martingale posterior distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(5):1357–1391, 2023.arxiv.org/abs/2103.15671
  4. [4]Tom Rainforth, Adam Foster, Desi Ivanova, and Freddie Bickford Smith. Modern Bayesian experimental design. Statistical Science, 39(1):100–114, 2024.arxiv.org/abs/2302.14545
  5. [5]Roman Garnett. Bayesian Optimization. Cambridge University Press, 2023.DOI
  6. [6]Freddie Bickford Smith, Andreas Kirsch, Sebastian Farquhar, Yarin Gal, Adam Foster, and Tom Rainforth. Prediction-oriented Bayesian active learning. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 7331–7348. PMLR, 2023.arxiv.org/abs/2304.08151
  7. [7]Bas C. van Fraassen. Belief and the will. The Journal of Philosophy, 81(5):235–256, 1984.
  8. [8]David Lewis. Why conditionalize? In Papers in Metaphysics and Epistemology, Cambridge Studies in Philosophy, pages 403–407. Cambridge University Press, 1999.DOI
  9. [9]Michael Goldstein. The prevision of a prevision. Journal of the American Statistical Association, 78(384):817–819, 1983.DOI
  10. [11]Bruno Kacper Mlodozieniec, David Krueger, and Richard E. Turner. Implicitly Bayesian prediction rules in deep learning. In Proceedings of the 6th Symposium on Advances in Approximate Bayesian Inference, volume 253 of Proceedings of Machine Learning Research, pages 79–110. PMLR, 2024.
  11. [12]Fabian Falck, Ziyu Wang, and Christopher C. Holmes. Is in-context learning in large language models Bayesian? A martingale perspective. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 12784–12805. PMLR, 2024.arxiv.org/abs/2406.00793
  12. [13]Kenyon Ng, Edwin Fong, David T. Frazier, Jeremias Knoblauch, and Susan Wei. TabMGP: Martingale posterior with TabPFN. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research. PMLR, 2026.arxiv.org/abs/2510.25154
  13. [14]Gábor Pituk, Vik Shirvaikar, and Tom Rainforth. Do Bayesian neural networks actually behave like Bayesian models? In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 49420–49458. PMLR, 2025.
  14. [15]Zheng Wen, Ian Osband, Chao Qin, Xiuyuan Lu, Morteza Ibrahimi, Vikranth Dwaracherla, Mohammad Asghari, and Benjamin Van Roy. From predictions to decisions: The importance of joint predictive distributions. arXiv preprint arXiv:2107.09224, 2022.arxiv.org/abs/2107.09224
  15. [16]Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Xiuyuan Lu, Morteza Ibrahimi, Dieterich Lawson, Botao Hao, Brendan O’Donoghue, and Benjamin Van Roy. The neural testbed: Evaluating joint predictions. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2110.04629
  16. [17]A. P. Dawid. Present position and potential developments: Some personal views: Statistical theory: The prequential approach. Journal of the Royal Statistical Society. Series A (General), 147(2): 278–292, 1984.DOI
  17. [19]David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859–877, 2017.
  18. [20]Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Bazan Clement Emile Marcel Raoul, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, and Thomas Möllenhoff. Variational learning is effective for large deep networks. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 44665–44686. PMLR, 2024.arxiv.org/abs/2402.17641
  19. [21]Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of Bayesian neural nets via local linearization. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 703–711. PMLR, 2021.arxiv.org/abs/2008.08400
  20. [22]Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes. In ICML Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018a.arxiv.org/abs/1807.01622
  21. [24]Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do Bayesian inference. In International Conference on Learning Representations, 2022.arxiv.org/abs/2112.10510
  22. [25]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, pages 6402–6413, 2017.
  23. [26]Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. In Advances in Neural Information Processing Systems, volume 33, pages 4697–4708, 2020.arxiv.org/abs/2002.08791
  24. [27]Veit David Wild, Sahra Ghalebikesabi, Dino Sejdinovic, and Jeremias Knoblauch. A rigorous link between deep ensembles and (variational) Bayesian methods. In Advances in Neural Information Processing Systems, volume 36, pages 39782–39811, 2023.
  25. [28]Freddie Bickford Smith, Jannik Kossen, Eleanor Trollope, Mark van der Wilk, Adam Foster, and Tom Rainforth. Rethinking aleatoric and epistemic uncertainty. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 4345–4359. PMLR, 2025.
  26. [29]Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723–773, 2012.
  27. [30]Bharath K. Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert R. G. Lanckriet. Hilbert space embeddings and metrics on probability measures. Journal of Machine Learning Research, 11(50):1517–1561, 2010.
  28. [31]Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, volume 20, 2007.
  29. [32]Tudor Manole and Aaditya Ramdas. Martingale methods for sequential estimation of convex functionals and divergences. IEEE Transactions on Information Theory, 69(7):4641–4658, 2023.
  30. [33]Glenn Shafer. Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society Series A: Statistics in Society, 184(2):407–431, 2021.DOI
  31. [34]Aaditya Ramdas, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4):576–601, 2023.arxiv.org/abs/2210.01948
  32. [35]Shubhanshu Shekhar and Aaditya Ramdas. Nonparametric two-sample testing by betting. IEEE Transactions on Information Theory, 70(2):1178–1203, 2024.arxiv.org/abs/2112.09162
  33. [36]Aleksandr Podkopaev, Patrick Blöbaum, Shiva Kasiviswanathan, and Aaditya Ramdas. Sequential kernelized independence testing. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 27957–27993. PMLR, 2023.arxiv.org/abs/2212.07383
  34. [37]Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning, 69(2–3):169–192, 2007.
  35. [38]Ashok Cutkosky and Francesco Orabona. Black-box reductions for parameter-free online learning in Banach spaces. In Proceedings of the 31st Conference on Learning Theory, volume 75 of Proceedings of Machine Learning Research, pages 1493–1529. PMLR, 2018.arxiv.org/abs/1802.06293
  36. [39]Jean Ville. Étude critique de la notion de collectif. Gauthier-Villars, 1939.
  37. [40]Glenn Shafer, Alexander Shen, Nikolai Vereshchagin, and Vladimir Vovk. Test martingales, Bayes factors and p-values. Statistical Science, 26(1):84–101, 2011.
  38. [41]Vladimir Vovk and Ruodu Wang. E-values: Calibration, combination and applications. The Annals of Statistics, 49(3):1736–1754, 2021.DOI
  39. [42]Michalis Titsias. Variational learning of inducing variables in sparse Gaussian processes. In Proceedings of the 12th International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pages 567–574. PMLR, 2009.
  40. [43]David R. Burt, Carl Edward Rasmussen, and Mark van der Wilk. Convergence of sparse variational inference in Gaussian processes regression. Journal of Machine Learning Research, 21(131):1–63, 2020.arxiv.org/abs/2008.00323
  41. [44]Philip Mortimer, Cristiana Diaconu, Tommy Rochussen, Bruno Mlodozeniec, and Richard E. Turner. Incremental transformer neural processes. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research. PMLR, 2026.arxiv.org/abs/2602.18955
  42. [46]Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Bernhard Schölkopf. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning, 10(1–2):1–141, 2017.DOI
  43. [47]Wassily Hoeffding. A class of statistics with asymptotically normal distribution. The Annals of Mathematical Statistics, 19(3):293–325, 1948.
  44. [48]A. W. van der Vaart. Asymptotic Statistics. Cambridge University Press, 1998.
  45. [49]Robert J. Serfling. Approximation Theorems of Mathematical Statistics. John Wiley & Sons, 1980.
  46. [51]Peter Grünwald, Rianne de Heide, and Wouter M. Koolen. Safe testing. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(5):1091–1128, 2024.
  47. [52]Ian Waudby-Smith, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya Ramdas. Time-uniform central limit theory and asymptotic confidence sequences. The Annals of Statistics, 52(6):2613–2640, 2024.
  48. [53]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024.
  49. [54]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
  50. [55]I-Cheng Yeh. Concrete Compressive Strength. UCI Machine Learning Repository, 1998a. DOI: https://doi.org/10.24432/C5PK67.
  51. [57]R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33 (3):291–297, 1997.DOI
  52. [58]J. Gerritsma, R. Onnink, and A. Versluis. Yacht Hydrodynamics. UCI Machine Learning Repository, 1981. DOI: https://doi.org/10.24432/C5XG7R.
  53. [59]Ross Quinlan. Statlog (Australian Credit Approval). UCI Machine Learning Repository, 1987. DOI: https://doi.org/10.24432/C59012.
  54. [60]Andras Janosi, William Steinbrunn, Matthias Pfisterer, and Robert Detrano. Heart Disease. UCI Machine Learning Repository, 1989. DOI: https://doi.org/10.24432/C52P4X.
  55. [61]Robert Detrano, Andras Janosi, Walter Steinbrunn, Matthias Pfisterer, Johann-Jakob Schmid, Sarbjit Sandhu, Kern H. Guppy, Stella Lee, and Victor Froelicher. International application of a new probability algorithm for the diagnosis of coronary artery disease. The American Journal of Cardiology, 64(5):304–310, 1989.pubmed.ncbi.nlm.nih.gov/2756873
  56. [62]Hans Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository, 1994. DOI: https://doi.org/10.24432/C5NC7T.
  57. [63]Pete Mowforth and Barry Shepherd. Statlog (Vehicle Silhouettes). UCI Machine Learning Repository, 1987. DOI: https://doi.org/10.24432/C5HG6N.
  58. [64]William Wolberg, Olvi Mangasarian, Nick Street, and W Street. Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository, 1993. DOI: https://doi.org/10.24432/C5DW2B.
  59. [65]Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. OpenML: Networked science in machine learning. SIGKDD Explorations, 15(2):49–60, 2013.DOI
  60. [66]Radford M. Neal. MCMC using Hamiltonian dynamics. In Steve Brooks, Andrew Gelman, Galin L. Jones, and Xiao-Li Meng, editors, Handbook of Markov Chain Monte Carlo, pages 113–162. Chapman and Hall/CRC, 2011.DOI
  61. [67]David J. C. MacKay. The evidence framework applied to classification networks. Neural Computation, 4(5):720–736, 1992b.DOI
  62. [68]Hugh A. Chipman, Edward I. George, and Robert E. McCulloch. BART: Bayesian additive regression trees. The Annals of Applied Statistics, 4(1):266–298, 2010.
  63. [69]Jingyu He, Saar Yalov, and P. Richard Hahn. XBART: Accelerated Bayesian additive regression trees. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 1130–1138. PMLR, 2019.arxiv.org/abs/1810.02215
  64. [70]Andrew Herren, P. Richard Hahn, Jared Murray, and Carlos Carvalho. StochTree: BART-based modeling in R and Python. arXiv preprint arXiv:2512.12051, 2025.arxiv.org/abs/2512.12051
  65. [71]Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, volume 31, pages 6638–6648, 2018.arxiv.org/abs/1706.09516
  66. [72]Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, 2025.DOI
  67. [73]Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Benjamin Jäger, Dominik Safaric, Simone Alessi, Adrian Hayler, Mihir Manum, Rosen Yu, Felix Jablonski, Shi Bin Hoo, Anurag Garg, Jake Robertson, Magnus Bühler, Vladyslav Moroshan, Lennart Purucker, Clara Cornu, Lilly Charlotte Wehrhahn, Alessandro Bonetto, Bernhard Schölkopf, Sauraj Gambhir, Noah Hollmann, and Frank Hutter. TabPFN-2.5: Advancing the state of the art in tabular foundation models. arXiv preprint arXiv:2511.08667, 2025.
  68. [74]Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249–256. PMLR, 2010.
  69. [75]James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2408–2417. PMLR, 2015.arxiv.org/abs/1503.05671
  70. [76]Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable Laplace approximation for neural networks. In International Conference on Learning Representations, 2018.
  71. [78]Nicol N. Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural Computation, 14(7):1723–1738, 2002.pubmed.ncbi.nlm.nih.gov/12079553
  72. [79]Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1613–1622. PMLR, 2015.
  73. [80]Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In International Conference on Learning Representations, 2018.arxiv.org/abs/1710.10628
  74. [81]Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050–1059. PMLR, 2016.arxiv.org/abs/1506.02142

Paper details

vlnc:W_48hv779g
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents