WorksMathematics
Measuring Coherence in Predictive Models
Abstract
Many sequential decision-making procedures, such as active learning, implicitly assume that a model's current predictions agree with those it expects to make after updating on new data, a property we call coherence. Most models in use do not update by exact conditioning and are not guaranteed to be coherent, which can lead to worse decisions. Yet there is currently no general way to measure incoherence or to tell when it matters. We develop tools to measure incoherence and characterise when it affects decisions. We introduce the incoherence ratio, an MMD-based measure of how far any updating procedure departs from coherence, computed only from the procedure's own predictions and updates, and accompany it with an anytime-valid test of coherence. In Bayesian experimental design, acquisition objectives that are equivalent for coherent models choose different data under every approximate model we study, and the incoherence ratio tracks how far incoherence distorts them. We also introduce action coherence, which captures only the incoherence that changes a decision. What a resulting disagreement costs, however, is determined primarily by predictive accuracy. Sequential decisions depend on how a model updates as well as on what it predicts, so coherence, like accuracy, should be checked rather than assumed.
Introduction
In sequential decision-making problems, actions are often chosen using a model that is updated as new data arrive. Active learning [1, 2], for example, selects the data whose acquisition will most improve the model’s predictions after updating. Making this choice requires the model to anticipate how its own predictions will change, which relies on its current predictions agreeing with those it expects to make after updating. Following Fong et al. [ [3], we call such a model coherent. Many methods for sequential decision-making are derived assuming exact Bayesian conditioning [4, 5], which guarantees coherence by construction. The models they are applied to, however, rarely update by exact conditioning, and so need not be coherent. Rather than assuming coherence, we measure how severely it is violated, and identify when and how incoherence affects decisions.
Bayesian experimental design shows how incoherence can reach a decision. The expected predictive information gain (EPIG) [6] selects the design whose outcome most reduces uncertainty in the model’s predictions at a target of interest. It can be written in several standard forms: a reduction in predictive entropy, a mutual information, or an expected divergence from the current predictive, and these are used interchangeably because they agree under coherence. For an incoherent model, however, they become different acquisition rules that can rank candidate designs in opposite orders (§ 3). Empirically, the entropy form acquires worse data on average than the mutual-information form for every approximate model we study (§ 6).
Coherence has a long history in both philosophy [7, 8] and statistics [9], where it corresponds to the martingale property of predictive distributions [3] and underpins the predictive approach to Bayesian inference [de Finetti, 1937]. Most models central to machine learning do not update by explicit conditioning, and a growing body of work asks whether their updates nonetheless behave like a Bayesian update. Prediction rules have been measured against exchangeability [11], in-context learning in large language models and tabular foundation models has been shown to depart from the martingale property [12, 13], and approximate Bayesian neural networks have been found to update incoherently [14]. These works establish that coherence can fail in important models, using diagnostics specific to each setting, most of which need data from the task or a reference Bayesian model. Separately, work on sequential decisions has shown that decision quality depends on a model’s joint predictive distribution, rather than its marginal predictions [15, 16]. A model’s update rule implicitly defines such a joint, and coherence is the requirement that it agree with the model’s current predictions. Because coherence concerns how predictions update rather than how accurate they are, a model can predict accurately at every input and still be incoherent. What is still missing is a general way to measure the incoherence of an arbitrary updating procedure using only the procedure itself, and an account of when incoherence matters for the decisions these models are used to make.
We address both. We first set out what coherence means for any procedure that predicts and updates, without reference to a prior, likelihood or parameter posterior (§ 2). The requirement is local to the history, target and update relevant to the problem, and, in keeping with a predictivist view, concerns only predictions of observable quantities. To measure incoherence, we introduce the incoherence ratio , a scale-invariant measure of how far a procedure’s updates depart from conditioning, estimated from its own predictions and updates alone (§ 4). We also give an anytime-valid sequential test of coherence (§ 5). To determine when incoherence matters, we introduce action coherence, a relaxation that asks only whether incoherence changes the action a procedure selects, and so isolates exactly what reaches a decision (§ 3).
Our experiments show what incoherence reveals about models and when it reaches decisions (§ 6). Our measure of incoherence falls as an approximation is refined and ranks models consistently across datasets. For every approximate model we study, objectives that agree under coherence select different data on a substantial share of decisions, so incoherence reaches the action. The measure tracks how susceptible a model’s decisions are to this disagreement, while accuracy determines what it costs. Incoherence is therefore a warning rather than a verdict. Sequential decisions rely on how a model updates, not only on what it predicts, and coherence is the property that captures this.
Predictive coherence
A predictive procedure comprises a probabilistic model and its prescribed update rule. At a realised history , the procedure returns predictive laws for the variables required by the prediction and update problem. We write for the prediction target and for the future observations that will be incorporated before the procedure is queried for again. We call the update block, which may contain one or more observations. An ordered update design specifies which observations comprise the block and their order. The update is written and maps a history and an observed update block to an updated history. In supervised learning, for example, the initial history may be the observed dataset , and the target the response at a fixed test input . An update design specifies a collection of additional inputs, with corresponding update block . Here the components of are responses of the same kind as . More generally, however, the construction assumes neither that the observations are exchangeable nor that the components of take values in the same space as .
We write and for the procedure’s current predictive laws for the target and the complete update block, respectively. When the procedure does not supply a joint law for , we construct one sequentially [17] in the order specified by the design , as in recursive resampling [3]: , where and is the updated history after observing . We demonstrate throughout that updating on the whole block agrees with updating component-wise along , so that . Throughout, a semicolon indicates that a predictive law is indexed by a history, and we reserve the vertical bar for probabilistic conditioning within a specified joint distribution. Subscripts name the variable whose law is described and superscripts carry the indices on which the law depends, namely the update design and its realisation. We suppress any index that is held fixed in the surrounding discussion, and reinstate it where it varies. In this section the design is fixed and suppressed.
Update-induced joint. The model’s predictive law for the update block and its post-update target predictions together define the update-induced joint,
By construction, and for -almost every . Thus is the unique joint distribution whose -marginal represents the model’s current beliefs about the update block and whose conditional distributions represent the predictions that the procedure will actually return after updating. The target marginal of this joint is
This is the model’s post-update predictive averaged under its current beliefs about the update block, which we call the expected future predictive. The update-induced joint is the operationally relevant distribution for a procedure which first observes and then . When the history is fixed, we suppress it, writing and for the current predictives, for the post-update predictive, and for the expected future predictive, so that and .
Predictive coherence. A predictive procedure is coherent for the target , history and update block when the current predictive and expected future predictive agree, . It extends the martingale condition of Fong et al. [ [3] to any target, history and update block. Coherence is exactly the statement that the current predictive is recovered as the -marginal of the update-induced joint. If coherence fails, no joint can simultaneously have as its -marginal, as its conditional distribution given , and as its -marginal. Exact conditioning under a model-specified joint is sufficient for predictive coherence. The general definition, however, does not require the model to specify a joint law over independently of the update-induced joint .
Local coherence. Predictive coherence is a local condition: it is defined relative to a realised history , a target , and a specified update setup comprising the ordered design and its corresponding update block . Coherence for one such setup does not imply coherence at another history, for another target, or under another update design. This locality is both a limitation and a feature. It limits the scope of any coherence guarantee, but focuses the requirement on the predictions and updates that are relevant to the problem under consideration. Incoherence outside this scope need not affect the intended use of the procedure. From a predictivist perspective, coherence concerns the compatibility of predictions about the observable quantities and across the histories reachable within this update setup. It therefore makes no general claim about the model’s latent parameters, posterior approximation, or global Bayesianity. We call a predictive procedure globally coherent only if the coherence condition holds for every admissible history, target, and update design.
Incoherence admits a Dutch book. Incoherence can have direct consequences for tasks that rely on a predictive procedure. The canonical demonstration is a diachronic Dutch book: a sequence of contracts that the procedure regards as individually fair at the histories where they are offered, but whose combined payoff is a sure loss. The classical construction of Lewis [ [8] shows qualitatively that such exploitation is possible whenever the procedure’s current valuation of some payoff differs from its expected valuation after updating. We quantify the maximum severity of this exploitation under the betting protocol and a bounded-liability constraint. Writing , the maximum fraction of initial wealth which can be extracted as a sure loss is . This Dutch-book result concerns the internal consistency of the procedure’s valuations and it does not imply that either predictive distribution is empirically inaccurate. Appendix § 9.§ 1 gives the classical construction and derives the maximum sure-loss fraction.
Bayesian models and coherence
Exact Bayesian conditioning provides the canonical sufficient condition for predictive coherence, but it is often unavailable. The familiar prior-likelihood construction in Bayesian deep learning has an intractable parametric posterior which must be approximated. Practical approximations include sampling-based methods such as SG-MCMC [Ma et al., 2015], variational inference over a restricted family [19], such as the diagonal Gaussian used by IVON [20], and the linearised Laplace approximation [21]. Whilst each targets a coherent Bayesian model, the approximations can induce incoherence in the resulting predictive procedure.
Amortised approaches such as neural processes [22 b] and prior-data fitted networks [24] bypass explicit parametric inference and learn a direct map from an observed dataset to a predictive distribution. They approximate the posterior predictive defined by their training prior, and in practice they too fall short of exact coherence, as observed for TabPFN [13]. Similarly, deep ensembles [25] can be interpreted as approximating Bayesian model averaging [26], despite their members not generally being samples from the posterior of a specified BNN [27]. These constructions demonstrate that a procedure can be approximately Bayesian without an explicit target model. A procedure’s Bayesian provenance does not determine how severe its incoherence will be, and we therefore separate construction from empirical behaviour.
Cost of incoherence
Incoherence matters when an action is taken before an update but valued by the predictions the procedure will report after it. We consider a decision problem spanning a single update. An initial action is taken, a block is observed, the procedure is updated, a terminal action is chosen from the post-update predictive, and a loss is realised at the target. The design of the block may depend on the initial action, so we index by the action, writing the update-induced joint as . Prediction-oriented experimental design is a problem of this form, and problems spanning several updates, as in Bayesian optimisation [5], are covered when each action is chosen greedily. The terminal action depends on the procedure only through the post-update predictive, which is a conditional of . Of the predictions consumed in valuing the initial action, only the current predictive can therefore disagree with the update-induced joint .
Action coherence. We coherentise an objective by substituting for wherever it appears, writing for the stated objective and for its coherentisation, and and for the actions they select. A procedure is action coherent at for when coherentising leaves the decision unchanged, , with ties broken identically. Predictive coherence for every block in implies action coherence for every objective, but not conversely, so action coherence is a relaxation that registers only the incoherence that reaches the action. Objectives that coincide under coherence share the same coherentisation, so two such objectives selecting different actions is sufficient for at least one of them to be action incoherent. Coherentisation is internal to the procedure and is not a correction towards the truth, since may describe the true process poorly. It removes inconsistency, not inaccuracy.
Severity of incoherence. What reaches the action is not the size of the discrepancy between and , but how it varies across candidates relative to how sharply the objective separates them, and how the objective consumes . A large discrepancy that is near-constant across candidates barely moves the decision, so detecting predictive incoherence establishes that a decision may be affected without indicating by how much. Incoherence also decouples objectives that coincide under coherence, and no member of the decoupled family is the correct one, since the current predictive and the expected future predictive disagree and neither is guaranteed to be the more accurate. Which member selects better actions under the true law is an empirical question we address in § 6.
Objective splitting in experimental design
Bayesian experimental design (BED) [4] selects designs which maximise information about a downstream quantity. In prediction-oriented design, that quantity is the procedure’s prediction at deployment. Here is a set of candidate designs, is the observation acquired at the chosen design, and the terminal action reports a predictive for a target under the log score. The expected predictive information gain (EPIG) [6] acquires the design which most reduces predictive uncertainty at the target, , which with the procedure’s own update is the expected uncertainty reduction of [28]. Read decision-theoretically, EPIG is a design-free constant minus the procedure’s expected log loss after the update, so maximising it minimises that loss. Read information-theoretically, it is the mutual information between the target and the acquired observation under the joint the procedure induces over them. Under incoherence that joint has -marginal rather than , and the mutual information becomes , the coherentisation of . The two differ only in their baseline, which is constant in for but varies with once coherentised, and so can change the ranking. A third standard form is the expected divergence from the current predictive, and both it and depart from EMI by explicit terms (Appendix § 9.§ 2),
All three forms coincide under coherence and coherentise to EMI, but under incoherence they are different acquisition rules, so even a choice ordinarily made for numerical convenience becomes a choice of acquisition rule. The KL form departs from EMI by a non-negative term that is second order in the discrepancy between and , approximately half their divergence, whereas the entropy form departs by a signed term that is first order. For small incoherence the KL form therefore stays much closer to EMI, as § 6 confirms.
We make the split concrete with two-parameter Bayesian linear regression under a Gaussian likelihood and prior, comparing exact conditioning against mean-field variational inference, whose factorised posterior cannot represent the intercept–slope correlation induced by training inputs clustered away from the origin. All three forms are available in closed form, so any discrepancy is attributable to incoherence rather than estimation noise. Sweeping the candidate design (Figure 1), they coincide under exact conditioning, and under mean-field VI and EMI rank two candidates in opposite orders. § 6 measures how often this occurs, and what it costs, across model families.
Figure 1. Objective splitting under mean-field VI for Bayesian linear regression. (a) The mean-field model’s current predictive (dashed) and its expected future predictives and after acquiring candidates and . The inset magnifies the peak, where neither expected future predictive matches the current one, so the model is incoherent for both candidates. (b) The three forms of EPIG against the candidate design . Under exact Bayes they coincide. Under mean-field VI, prefers whereas prefers , and (dashed) stays within 0.002 nats of EMI.
Measuring incoherence
§ 3 showed that incoherence can reach decisions. Since few procedures update by exact conditioning, the question that usually matters in practice is how incoherent a procedure is, rather than whether it is exactly coherent, which we return to in § 5. This section develops a scalar measure of that severity, computed from the procedure’s own predictions and updates.
The incoherence ratio
Fix a history , a target and a design with block . Each draw from costs an update, so updates are the budget, and we treat the procedure as a black box that can be updated and sampled. A round draws a block , updates the fitted procedure on it, drawing any randomness internal to the update afresh, and returns , so rounds are independent and identically distributed. The subscript on indexes rounds, not the components of the block. Pituk et al. [ [14] use such rounds to compare test metrics, not predictive laws. The Dutch book of § 2 prices incoherence by , but total variation is non-linear in the mixture and cannot be estimated without bias from post-update predictives. We instead use kernel mean embeddings [29], which are linear in the mixture. For a kernel with feature map and , write , and for the embeddings of , and respectively, and for the displacement of round . Then and . For a characteristic kernel [30], if and only if the procedure is coherent. Since , however, bounds the sure loss only from below, and a small certifies only that incoherence is small in the kernel’s features (Appendix § 10.§ 2).
The incoherence ratio. The squared discrepancy grows with how far a procedure’s updates move its predictive, so procedures that update by different amounts cannot be compared on it. The bias–variance identity , with , separates the systematic part of an update from its response to the block, and we define the incoherence ratio as
the squared signal-to-noise ratio of the update. It is invariant to rescaling the displacements, and is the systematic share of the expected squared displacement, so means bias and spread contribute equally. For a characteristic kernel, vanishes exactly under coherence, and a shift that ignores the block, with and , has (Appendix § 10.§ 2). is a self-consistency check that needs no test set, reference posterior or oracle. Blocks are drawn from rather than from data because coherence concerns the procedure’s beliefs about its own future, and measuring on observed blocks would mix inconsistency with inaccuracy. Like coherence, is a property of the procedure for the update problem , including the size of the block, which should match the update the procedure will face. Its level depends on the kernel and the design, so it ranks procedures only within a problem. Since includes any randomness internal to the update, a noisier update has a lower for the same bias, and Appendix § 10.§ 2 separates the spread driven by the block.
Estimation. is estimated from rounds, which can be shared with the test of § 5. U-statistics over rounds estimate and without bias, reducing with exact embeddings to and , and the ratio has an bias and a leave-one-round-out standard error (Appendix § 10.§ 1). Near coherence the numerator is a degenerate U-statistic and estimates can be negative, which we report unclipped. Displacements at several evaluation points are pooled as a ratio of summed numerators to summed denominators, which keeps scale invariance. For finite targets we use the Dirac kernel, and for continuous targets random Fourier features [31], which make estimation cheap but are not characteristic. We set their lengthscale to the noise scale of the problem, and in § 6, is found to be robust to this choice.
Measuring action incoherence
is decision-agnostic. Whether incoherence reaches an action is defined by the coherentised objective , which is rarely computable, since it consumes through a non-linear functional at every candidate. Because objectives that coincide under coherence share a coherentisation (§ 3), the rate at which two tractable objectives from this class disagree witnesses action incoherence, although agreement does not certify its absence. The two measures answer different questions. The disagreement rate concerns a particular pair of objectives and decision problem. needs neither, so it can compare procedures before a decision problem is fixed, audit an approximation choice, and attribute a failed sequential procedure to inconsistent updates rather than inaccurate predictions. The link runs in one direction only. With a characteristic kernel, if for every block in , then the procedure is action coherent for every objective. A positive diagnoses susceptibility, the distortion of the objective across candidates. Whether that distortion changes a choice also depends on the margin between candidates, and what a changed choice costs is evaluated under the true law rather than the procedure’s own beliefs (§ 3), so neither is determined by . Coherentisation removes inconsistency rather than inaccuracy, so realised performance on an oracle or held-out set remains the decision-relevant comparison, and § 6 examines how susceptibility relates to it.
Testing coherence
The incoherence ratio grades how incoherent a procedure is, but its estimate is not a test of coherence. Near coherence its numerator is a degenerate U-statistic, for which valid intervals need a dedicated construction [32], and SE is not a confidence interval. Whether a procedure is exactly coherent matters most when its incoherence ratio is near zero and its joint is implicit in its update rule, so that coherence cannot be settled by construction. We therefore test , so that rejecting a heuristic procedure is evidence of incoherence with a controlled type I error. Earlier checks of this null calibrate by bootstrap against a reference Bayesian model [12] or on test-set metrics [14], whereas ours needs neither. A batch MMD test [29] would fix the number of updates in advance, and its permutation calibration fails when a round’s samples share a block. We instead adopt testing by betting [33], whose anytime validity [34] lets the test stop once the evidence is decisive.
Sequential kernel test of coherence
Our test is the kernel-MMD betting test of Shekhar and Ramdas [ [35] with its plug-in witness and the one-sided online Newton step stakes of Podkopaev et al. [ [36], applied to update rounds rather than sample pairs. With the kernel, embeddings and displacements of § 4.1, coherence implies the kernel null for any kernel, so the test is valid whatever the kernel. The kernel determines only which departures can be detected. A quadratic kernel, for example, tests coherence only in mean and variance.
Each round draws , updates the fitted procedure, and estimates by , the difference between the embeddings of and , computed exactly or from samples (Appendix § 10.§ 3). Unlike the estimator of , the test draws fresh samples of in every round. Rounds are then independent given the fitted procedure, so , where is everything observed before round . A pool shared across rounds would couple the bets through its Monte Carlo error and break this property. Before round , the test forms the running mean and the plug-in witness , or if for a fixed noise floor . It then bets on the projection of the new displacement onto the witness,
where since , and the stake is set from earlier rounds by the online Newton step [37] betting strategy of [38]. Since and depend only on , , which is zero under the null. The wealth () is then a non-negative martingale with , and Ville’s inequality [39] gives . Rejecting the first time therefore controls the type I error at every stopping time [40], and no update is spent after the evidence is decisive.
Under an alternative with , the witness converges to and grows linearly in , so the test is consistent above the noise floor, by the argument that Shekhar and Ramdas [ [35] and Podkopaev et al. [ [36] give for their tests (Appendix § 10.§ 3). Applied to rounds, their bounds on the growth rate show that it is set mainly by the absolute bias rather than by , because bets are scaled by the worst-case bound , which keeps the test valid without assumptions on the procedure but makes it conservative when updates are small (Appendix § 10.§ 4). Failing to reject within a budget of updates does not certify coherence, since incoherence below the noise floor, outside the features of the kernel, or too small to accumulate evidence within the budget goes undetected.
Testing at multiple evaluation points. Coherence is local, but a procedure is used at many evaluation points, and incoherence at any of them matters. We test the joint null that at every evaluation point . A single update gives a displacement at every point, so testing at points costs no more updates than testing at one. Running the construction separately at each point gives wealths , which share the same updates and are dependent, so their product is not a valid test martingale. Under dependence they are instead combined by averaging [41]: an average of martingales with respect to a common filtration is itself a martingale, and we reject the first time . Since , incoherence confined to a single point costs at most in log-evidence relative to testing at that point alone (Appendix § 10.§ 3).
Experiments
The first two experiments ask whether incoherence measures what it should and what it reveals about models, in a controlled sparse GP setting and across models on real tabular datasets. The last two ask whether incoherence reaches decisions and what it costs, within mean-field Bayesian linear regression and across models on a synthetic neural-network testbed. Throughout, is pooled over evaluation points, and tests of coherence use level . Full experimental details are in Appendix § 11.
Incoherence within sparse GPs. In a controlled sparse GP setting, the number of inducing points controls the resolution of the approximation. Functions are drawn from a GP prior, and each task observes points before being updated on a block of additional points, with incoherence measured at 64 evaluation points. We draw 20 functions (environments) and form 10 tasks from each by subsampling the observed, block and evaluation points, reporting the mean over tasks. Sparse GPs use the variational approximation of Titsias [ [42], with the kernel hyperparameters fixed at their true values. Inducing points are chosen deterministically by greedy variance selection [43], and we compare an update procedure in which the inducing points are chosen before the update and held fixed with one in which they are re-selected during the update. The two make identical initial predictions, so they differ only in how they update. We also include an incremental transformer neural process (incTNP) [44] trained on the same GP prior, which updates by extending its context (Appendix § 11.§ 1). Within each update procedure, increasing the resolution of the approximation reduces incoherence and improves pre-update accuracy (Figure 2a). With fixed inducing points, incoherence plateaus since even when , the update cannot adequately represent the new data. Re-selection avoids this, but since with a fixed budget it must give up coverage of the existing data to accommodate the block, so its update departs from the current approximation, making it less coherent when is small and more coherent when is large. Re-selection is nevertheless no less accurate after the update, even where it is the less coherent procedure (Figure 2b), so across update procedures incoherence does not order post-update accuracy. Rather, it registers that current and future predictions disagree, not which of them is wrong. The incTNP is as coherent as the fixed sparse GPs with nearly all the inducing points while fitting less accurately, and after the update it is more accurate than all of them, lying close to the re-selected curve, consistent with an approximation that adapts to new data. The test never rejects the exact GP, and within the fixed procedure rejects more coherent approximations more slowly (Figure 2c). Across families its speed to reject is not set by alone, as the incTNP is rejected faster than fixed sparse GPs of similar .

Figure 2. Sparse GPs with fixed (blue) or re-selected (green) inducing points, and an incTNP. (a, b) against KL from the exact GP before and after the update. Vertical lines join procedures with the same number of inducing points, and bands show standard errors. (c) Fraction of tasks rejected at . The exact GP is never rejected.
Incoherence across models on tabular data. We measure incoherence for models of several kinds on three regression and five classification datasets, over ten random splits (one for TabPFN), updating every model on random blocks of further inputs with outputs drawn from the model itself (Appendix § 11.§ 2). Exact methods have indistinguishable from zero and are never rejected. The estimates are robust to the kernel: on the regression datasets, halving or doubling the lengthscale swaps only models whose is within of each other. responds to how a model updates, even when its current predictions are unchanged. Refitting a GP’s hyperparameters at each update leaves its current predictions identical to the exact GP’s but introduces a small incoherence, as Mlodozeniec et al. [2024] find for exchangeability. Refitting TabPFN’s preprocessing on the updated context changes its by up to a factor of six at similar log loss, although the two variants also differ before the update. An approximately Bayesian provenance is also no guarantee of coherence. For example, the update in BART reruns a sampler that mixes poorly within a run, and its median of 0.71 is comparable to the regression medians of nearest neighbours (0.47) and CatBoost (0.90). Plug-in logistic regression, by contrast, is about as coherent as the Laplace approximations, with a median below 0.01. The ordering of models is largely shared across datasets, with rank correlations of at least 0.88 between them, but the level of is not (kernel ridge ranges from 0.008 to 0.12), so ranks models only within a problem. Finally, accuracy does not imply coherence. TabPFN, which departs from the martingale condition [13], has the lowest log loss on seven of the eight datasets in one variant or the other, yet on every classification dataset its is an order of magnitude or more above that of the full Laplace approximation, which is nearly coherent but generally less accurate (Table 4).
| Regression | ||||
| Model | Range | Rank | Rej. | |
| GP | 0.000 | 0.000–0.000 | 3.5 | 0.00 |
| Sparse GP | 0.007 | 0.004–0.018 | 9 | 0.13 |
| GP, refitted | 0.002 | 0.001–0.002 | 3.5 | 0.03 |
| BART | 0.710 | 0.632–0.863 | 7 | 1.00 |
| XBART | 0.378 | 0.271–0.787 | 5 | 1.00 |
| Kernel ridge | 0.013 | 0.008–0.119 | 8 | 0.37 |
| Nearest neighbours | 0.470 | 0.461–0.701 | 10 | 1.00 |
| CatBoost | 0.900 | 0.596–0.912 | 7 | 1.00 |
| TabPFN, network | 0.102 | 0.089–0.195 | 2 | – |
| TabPFN, preproc. | 0.118 | 0.083–0.222 | 1 | – |
| Classification | ||||
| Model | Range | Rank | Rej. | |
| Logistic, exact | 0.000 | 0.000–0.000 | 4 | 0.00 |
| Full Laplace | 0.004 | 0.001–0.007 | 4 | 0.02 |
| Diagonal Laplace | 0.008 | 0.002–0.012 | 5 | 0.00 |
| Logistic, plug-in | 0.007 | 0.001–0.016 | 3 | 0.10 |
| Nearest neighbours | 0.883 | 0.734–0.927 | 8 | 1.00 |
| CatBoost | 0.273 | 0.145–0.339 | 6 | 1.00 |
| TabPFN, network | 0.071 | 0.046–0.190 | 2 | – |
| TabPFN, preproc. | 0.126 | 0.054–0.425 | 2 | – |
Table 4. Summary over datasets. is the median over datasets of the model’s mean over splits, with its range. Rank is the median over datasets of the model’s rank by test log loss among the models listed for its task (1 is best, ties averaged). Rej. is the share of the 30 (regression) or 50 (classification) splits rejected within 500 rounds. TabPFN, never rejected on its one split within 128 rounds, is shown as – since this is not comparable.
Incoherence and acquisition within mean-field BLR. Incoherence means that a model’s current and future predictions disagree, and we now ask whether that disagreement reaches decisions. To isolate its effect, we hold the model and its predictions fixed and vary only the coherence of what the acquisition function consumes. We use Bayesian linear regression on random Fourier features and compare exact Bayes with mean-field variational inference, which we compute analytically so that any incoherence comes from the approximation rather than from optimisation error. Starting from 25 points, each model acquires 100 further points one at a time greedily from a pool, choosing each with either the entropy form of EPIG or EMI, and we average over 100 environments. With , the mean-field model acquires worse data than random acquisition (Figure 3a). With EMI, the same model nearly matches exact Bayes, with an excess of 0.008 nats, so the difference lies entirely in the objective. EMI is tractable here, but in general it requires the entropy of a mixture, so we also consider making the predictive itself coherent, building a ladder in which the mean-field predictive is made coherent with its own refit over a growing random share of the pool (Appendix § 11.§ 3). Excess loss falls monotonically with along the ladder, and coherentising over a small random subset already recovers most of the gap to exact Bayes (Figure 3b). This suggests that coherence need only hold over a representative selection of the pool rather than over every point the model might acquire.
Figure 3. Acquisition with mean-field Bayesian linear regression. Every curve is the test log loss of the mean-field model on the data it acquired. (a) Test log loss by acquisition step. (b) Excess log loss over acquisition with the exact predictive, averaged over steps, against . Points are labelled by the share of the pool over which the acquisition predictive is made coherent, with the mean-field model with . EMI changes only the objective, so it is plotted at the mean-field model’s . Bands and error bars show standard errors.
Incoherence and acquisition across models. We now ask how incoherence affects acquisition across different approximate models in a more realistic setting. We use a synthetic testbed modelled on the Neural Testbed of Osband et al. [ [16], with classification tasks generated by random networks, in which choosing one of several candidate batches to acquire is small enough that every labelling can be enumerated (Appendix § 11.§ 4). We consider eight approximate models (Laplace approximations, variational methods, deep ensembles, MC dropout and an incTNP) over 1000 environments. We measure the cost of a disagreement between objectives as the excess loss of ’s choice over EMI’s, and the distortion of the objective as the spread of − EMI across candidates. Exact Bayes over a large discrete bank of networks, included as a coherent baseline, never separates the objectives, whereas for every approximate model the entropy form of EPIG disagrees with EMI on 26–80% of decisions and has a higher loss on average. The KL form of EPIG disagrees with EMI on only 1–20% of decisions, as the identity of § 3.1 predicts, making it a potentially tractable substitute for EMI when a model is incoherent. tracks how far incoherence distorts the objective (Figure 4a), and within an environment the ranking of models by has a rank correlation of 0.73 with their distortion. This makes a measure of how susceptible a model’s decisions are to incoherence. It does not tell us whether a choice will flip, which also depends on the margin between the best candidates, or what a flip will cost, which is uncorrelated with and instead tracks the model’s accuracy (Figure 4b, Appendix § 11.§ 4). Bayes by backprop and the incTNP, for example, are about equally incoherent, yet the entropy form of EPIG disagrees with EMI on 80% of Bayes by backprop’s decisions against 26% of the incTNP’s, and a disagreement costs 0.030 against 0.0035 nats.
Figure 4. Acquisition on the neural testbed for eight approximate models. Exact Bayes is omitted, as its objectives never disagree. (a) Distortion of the acquisition objective against . (b) Cost of a disagreement, the excess loss of ’s choice over EMI’s, against the model’s KL from the truth before the update. Curves are local linear fits, markers are model means, and bands show the 10th–90th percentiles in (a) and ±2 standard errors in (b). BBB is Bayes by backprop.
Discussion
Many sequential decision procedures act as though a model’s current and future beliefs are consistent. Measuring incoherence quantifies how far approximate models violate this, and flags when a procedure’s decisions are exposed to it. Our experiments show this is more than a theoretical concern, as acquisition objectives that should coincide frequently disagreed and chose different data. Incoherence indicates when this disagreement can occur, but not what it costs, which depends on accuracy. Susceptibility can also depend on the objective. On the neural testbed, the KL form of EPIG stayed much closer to EMI than the entropy form, suggesting it as a cheap way to mitigate some of the effect of incoherence in BED. Whilst our experiments are restricted to BED, our framework applies to any single-update sequential decision problem. Understanding which properties of a problem determine how much incoherence matters would show where these diagnostics are most needed. Extending the framework to procedures that plan through a sequence of actions and updates would cover longer-horizon problems, where the effects of incoherence may compound. Beyond the choice of decision problem, the main limitations of measuring incoherence are the computational cost of many model updates and the scope of what it can tell us. The best models are both accurate and coherent, but coherence alone guarantees little, since exact conditioning in a misspecified model is perfectly coherent and can still be badly wrong. Measuring incoherence can reveal a potential problem, and help in explaining and addressing it, but it cannot substitute for held-out evaluation when choosing a model. Once an accurate model is chosen, making it exactly coherent is currently intractable, and approximate coherentisation over a representative subset of the pool has so far been shown to help only in a simple setting. It nonetheless points to a promising direction. Models that excel at static prediction are rarely designed for sequential decisions, and approaches that account for incoherence could help them perform well there too.
Acknowledgments and Disclosure of Funding
JR was supported by the EPSRC Centre for Doctoral Training in Statistics and Machine Learning (StatML), grant EP/Y034813/1. VS was supported by the EPSRC Centre for Doctoral Training in Modern Statistics and Statistical Machine Learning (StatML), grant EP/S023151/1. The authors declare no competing interests.
Theoretical foundations
Diachronic Dutch book
We follow the diachronic Dutch book of Teller [1973] and Lewis [ [8], in the notation of § 2. Fix the history , the target and the update design with block . The procedure acts as an agent who will buy or sell any contract at its fair price, the contract’s expected payoff under the agent’s predictive at the history where it is offered. For a payoff , paid once is observed, the fair prices before and after the update are
Before the update, a contract paying the future price once is revealed has fair price
where the second equality follows from (2).
Construction. Suppose . If instead , replace by , which reverses the sign. At a stake , a bookmaker executes three trades, each at the agent’s price.
(A) At , sell the agent a contract paying at , for .
(B) At , buy from the agent a contract paying at , for .
(C) At , after is observed, buy from the agent a contract paying at , for .
Starting from wealth 1, the agent holds after the trades at . At it pays to settle B and receives the same amount for C, so . At the payoffs of A and C cancel, so whatever the values of and , a sure loss of .
Bounded liability. At every time, the agent’s wealth must cover the sum of the largest possible payments on the contracts it has sold, without offsetting the contracts it holds. In the construction the agent has sold only B after and only C after , each paying at most , so the constraint holds whenever , that is, . At the sure loss is
Maximal sure loss. The loss (7) increases with , whose supremum over is . The construction therefore extracts a fraction of initial wealth, and no book does better. Consider any book of trades at the agent’s prices, in contracts on at and and on at , and let be the agent’s wealth after the trades at . Write and for the agent’s net payoffs from the contracts on traded at and at after , so that . Split them into contracts held and sold, and with . Every trade is fair, so averaging over and then over gives . A sure loss , with whatever the values of and , therefore satisfies
Bounded liability after the trades at requires . Since and , this gives , while . Hence , and gives . The maximal sure loss vanishes if and only if , that is, exactly when the procedure is coherent for , and .
Forms of EPIG
This section derives the identities (3) and (4) relating the three forms of EPIG, and shows why the KL form stays closer to EMI than the entropy form. Fix the history and the candidate design , and write and for the densities of and . We assume that for -almost every , and that all entropies and cross-entropies below are finite. Relating the three forms. Since averages the post-update predictives, , Fubini’s theorem gives . Expanding the KL divergence inside EPIG therefore gives
Substituting yields (3). Identity (4) is immediate, since EPIG and EMI share the term . Under coherence, , the extra terms in both identities vanish, and all three forms equal EMI. Substituting for in EPIG or EPIG gives EMI, which does not involve , so all three forms coherentise to EMI.
Size of the departures. Write , where is bounded with , so that sets the size of the incoherence. Expanding gives
so the KL form departs from EMI only at second order. The entropy form departs from EMI by , whose last two terms equal , so it is first order and can take either sign.
Consequence for rankings. Only differences between candidates affect which design is chosen. For a finite candidate set, let maximise EMI, and let be its margin over the next best candidate. Since KL divergence is non-negative, identity (3) gives
for every , so the KL form selects the same design as EMI whenever , a second-order quantity. By identity (4), the same argument guarantees agreement for the entropy form only when , which is first order.
Measuring and testing incoherence
We use the notation of § 4.1, with a kernel on whose feature map has . Round draws and returns the post-update predictive , whose mean embedding [46] we write , and
Each round updates the same fitted procedure, drawing the block and any randomness internal to the update afresh. Since averages over both, are independent and identically distributed with mean .
Estimation
Every estimator below is a U-statistic [47], an average over pairs of distinct rounds and, when embeddings are estimated from samples, of distinct samples. As for the unbiased MMD estimator of Gretton et al. [ [29], excluding self-pairs is what makes each one unbiased.
Exact embeddings. The numerator and denominator are order-two U-statistics over rounds,
and . For , independence of the rounds gives and , so both are unbiased. With , they equal
the sample variance of the displacements and a plug-in corrected for , since the covariance of has trace .
Sampled embeddings. Round provides independent samples from , independently across rounds, and every round shares one reference sample of independent draws from . Define
None of these pairs a sample with itself, so, given , they are unbiased for , , and respectively. Replacing each inner product in the exact-embedding estimators by the corresponding kernel average gives
Given the predictives, and are therefore unbiased for their exact-embedding counterparts, and hence unbiased. The sampled can be negative, and its Monte Carlo error includes a term of order , which must be small relative to for to be reliable.
Reusing the reference sample. The reference sample enters only , through and . Sharing it across rounds correlates the terms of but introduces no bias. Its Monte Carlo error, however, does not shrink with : as , still fluctuates about by order , against from the round samples. The reference sample should therefore be large relative to , which is cheap because sampling needs no update, and comparing across its two halves checks this.
Cost. With an explicit feature map of dimension , such as the Dirac kernel or random Fourier features [31], each round reduces to its embedding in , exact or averaged over samples. The point estimates then cost , or with samples, in memory independent of , although the standard error below needs all embeddings. With only a kernel, they compare every pair of rounds, at cost with exact embeddings and with samples.
Pooling over evaluation points. With estimators and at evaluation points , we report
which estimates under the direct-sum kernel over points and so keeps its scale invariance. This pooled is the -weighted average of the per-point ratios , whose estimates localise incoherence.
Standard errors. With the estimate without round , and the mean of these values, the leave-one-round-out standard error
captures the variability of numerator and denominator jointly. With sampled embeddings it holds the reference sample fixed, so it omits that sample’s Monte Carlo error, which is negligible when . Since and are unbiased, the ratio has an bias by the delta method [48].
Behaviour near coherence. When , vanishes, so is a degenerate U-statistic [49]. It is then of order rather than : converges in distribution to a centred weighted sum of variables, and is negative with appreciable probability. We report and unclipped, so that averaging over tasks does not bias them upwards, and read negative values as indistinguishable from zero. The standard error then also falls as (Table 2), but it is not consistent: the leave-one-round-out variance of the degenerate has expectation times its true variance, so the standard error overstates the spread of by a factor of about . It remains a useful, conservative scale, but, as § 5 notes, is not a confidence interval for a procedure that may be exactly coherent.
Table 2: Estimates of after rounds, as the mean over tasks ± the root mean square of the per-task leave-one-round-out standard errors. The estimates for different are prefixes of the same rounds.
| Model | Closed form | |||
| Exact GP | 0 | −0.005 ± 0.090 | 0.000 ± 0.008 | 0.0000 ± 0.0007 |
| incTNP | – | 0.017 ± 0.093 | 0.014 ± 0.010 | 0.013 ± 0.002 |
| Fixed, | 0.124 | 0.127 ± 0.141 | 0.123 ± 0.019 | 0.124 ± 0.005 |
| Fixed, | 0.012 | 0.016 ± 0.111 | 0.012 ± 0.011 | 0.012 ± 0.002 |
| Re-selected, | 0.0022 | 0.001 ± 0.093 | 0.003 ± 0.008 | 0.0022 ± 0.0008 |
Table 2.
Properties
Scale invariance and the systematic share. Rescaling every displacement, with , multiplies and by and so leaves unchanged. Since ,
is the systematic share of the expected squared displacement.
The limit . If , then almost surely, so for a characteristic kernel [30] almost surely. If , the update is a shift that ignores the block, which is wholly systematic, so . If , the update leaves the predictive unchanged, which is trivially coherent, and we set , so that, for a characteristic kernel, if and only if the procedure is coherent. For small positive , still reports whether what little movement there is tracks the block or drifts systematically, and and give the absolute sizes of the bias and the spread.
Randomised updates. Let be the randomness internal to the update in round , so that , and let be the mean displacement for block . The law of total variance gives , where
Randomising the update without changing its average for each block leaves , and hence and the Dutch book, unchanged, while increasing and lowering . By convexity of , it also cannot improve the expected log score, so update noise is not a free way to appear coherent. To exclude update noise, use the block-driven ratio
which equals for deterministic updates. It costs a second update per block: for two updates and on the same block with independent internal randomness, , so
is unbiased for , with computed from the .
Embedding discrepancy and total variation. Every in the unit ball of the RKHS satisfies , and , so
The sure loss of Appendix § 9.§ 1 increases with , so the embedding discrepancy bounds it from below, by , but gives no upper bound in general. For a finite with the Dirac kernel, , so and , where and are the probability vectors of and . Since sums to zero, its positive and negative parts each have -norm , so . With , this gives
so in the classification experiments the two agree up to a factor of .
The sequential test
The difference between the features of a pair of samples in the test of Shekhar and Ramdas [ [35] becomes the displacement estimate , and their witness of norm under a kernel bounded by one becomes .
Displacement estimates. With exact embeddings, . Otherwise, with an explicit feature map, round draws samples from and a fresh set of samples from , and sets
Each embedding or average of features has norm at most , so and every bet lies in . All our experiments use exact embeddings with : the Dirac kernel has , and random Fourier features are normalised so that .
Why the test needs fresh samples. Given the fitted procedure, each round draws the block, the update’s internal randomness and both sets of samples afresh, so is independent of and . If every round instead reused one reference sample, its estimate would be fixed by the past, so , which under the null is the Monte Carlo error of rather than zero. The witness would learn this drift, and the test would eventually reject a coherent procedure. The batch estimator of Appendix § 10.§ 1 needs only unconditional unbiasedness, so it can share one reference sample, although that sample’s error then does not average out.
Witness and noise floor. The witness of § 5.1 is the plug-in estimate that Shekhar and Ramdas [ [35] describe for their test and Podkopaev et al. [ [36] use for independence testing, plus our noise floor. We use , which guards against normalising a numerically zero vector, and , so the first round never bets. Any rule for that depends only on and has preserves validity, so the floor affects power but not the type I error.
Stakes. The stake is chosen by online Newton steps on the log-wealth [37], proposed for betting by Cutkosky and Orabona [ [38], in the one-sided form of Podkopaev et al. [ [36]. Each evaluation point keeps its own stake and curvature, initialised at and . After round , with observed and the wealth already updated using ,
where is the derivative of in at . In one dimension, clipping to coincides with projection in the curvature norm. The stake depends only on rounds up to , so it is predictable, and the cap at keeps , as the logarithmic regret guarantee of online Newton steps against the best constant stake requires.
Validity. Fix an evaluation point. Under the null, , and and are -measurable, so the bounded bets have conditional mean zero [Waudby-Smith and Ramdas, 2024] and
Since , is a positive martingale with . Ville’s inequality, , then bounds the type I error by at any stopping time.
Multiple evaluation points. Let contain everything observed at every evaluation point up to round . By the argument above, each per-point wealth is a non-negative martingale with respect to this common filtration, whatever the dependence between points, so their average [41] is one too, with , so Ville’s inequality applies to it. Equivalently, is a product of predictable convex combinations of the per-point factors,
since each factor equals and the product telescopes. Since every is positive, , so . We compute with a log-sum-exp over points.
Consistency. Following the consistency proofs of Shekhar and Ramdas [ [35] and Podkopaev et al. [ [36], suppose at some point. The running mean converges to by the strong law of large numbers, so the witness converges to and the conditional mean of the bets to . The log-wealth at that point then grows at the positive rate of Appendix B.4, grows at least as fast by the bound above, and the test rejects with probability one.
Severity and detectability
The oracle bet. Above the noise floor, the witness converges to . With this witness fixed, the bets
are independent and identically distributed, with mean and variance
where includes any Monte Carlo error in . Write for the squared signal-to-noise ratio along the bias. With exact embeddings, the Cauchy–Schwarz inequality gives $\operatorname{Var}(\langle u,\delta_r\rangle)\le , so , with equality only when the spread lies entirely along the bias. By the logarithmic regret of the stakes (Appendix § 10.§ 3), the log-wealth grows asymptotically at rate
which plays the role of power for such tests [51], and the test rejects after roughly updates.
Two regimes. Up to constants, the bounds below are those of Podkopaev et al. [ [36] in our notation, which Shekhar and Ramdas [ [35] also give for two-sample tests. For ,
Apply these with , using . The lower bound gives . The stake attains when it is at most , and otherwise attains more than . The upper bound gives , whose maximum over is , and gives . Together,
The growth rate is thus governed, up to constants, by the smaller of , the systematic share along the bias, and , the bias relative to the kernel bound. The latter enters because bets are scaled by the worst-case bound , which keeps the wealth non-negative whatever the procedure does. The stake cap then stops the test betting more heavily when bets are small. A single update typically moves the embedding far less than , so is small and, unless , the rate lies between and , scaling with the absolute bias and barely depending on the spread. This is the low-variance regime of Shekhar and Ramdas [ [35], in which the rate improves from order to order , as Podkopaev et al. [ [36] also show. Since is scale invariant, procedures with the same but different absolute bias are rejected at different speeds, as for the incTNP and the fixed sparse GPs of similar in Figure 2c. With exact embeddings, gives , so alone bounds the rate from below only when , as in the low signal-to-noise regime.
Conservativeness. Scaling bets by an estimated displacement size would add power, but bets that could then exceed one in magnitude would need clipping, which breaks the mean-zero property under the null unless the displacements are symmetric. For a non-negative kernel, such as the Dirac kernel, , so bets could be scaled by without loss of validity. A tighter almost-sure bound on would need assumptions on how far an update can move the predictive, and exact embeddings, since Monte Carlo error alone can make as large as the kernel allows. Asymptotic anytime-valid inference [52] avoids the range bound and adapts to the variance of the bets, but is valid only asymptotically. We keep the non-asymptotic test because it is meant to audit exact coherence, where validity matters more than speed, and grade severity with .
Experimental details
This appendix gives the details needed to reproduce each experiment in § 6, one subsection per experiment, and the code is available at https://github.com/jreaso/predictive-coherence.
Compute. The sparse GP, tabular and Bayesian linear regression experiments used about 155 CPU core-hours and 4 GPU-hours on one laptop, and the neural testbed about 8 hours on one NVIDIA H100 GPU. Preliminary runs used roughly as much again.
Sparse GPs
The data are drawn from the GP prior, so the exact GP is the Bayesian ground truth for the sparse GPs and the incTNP.
Environments and tasks. Each of 20 environments is a function drawn from a GP prior (squared-exponential kernel, lengthscale 0.25, unit variance), observed with noise of standard deviation 0.1 at 512 inputs drawn uniformly from . Each environment is split at random 10 times into 24 context points, a block of 8 and 64 evaluation points.
Sparse GPs. Inducing points are a subset of the context inputs, chosen by greedy variance selection [43], and is the optimum of Titsias [ [42]. Both update procedures start from this fit. With fixed inducing points, the update keeps them and updates , whereas re-selection chooses up to afresh from all 32 inputs and refits. The predictive includes the variance that the inducing points leave unexplained, but the update absorbs new data through the projected likelihood, which omits it. This mismatch is the source of the incoherence. Figure 2 and Table 1 show , although re-selection at , with , lies below the figure’s axes. Near , selection stops once the context is explained to numerical precision, so some tasks use fewer than inducing points.
Table 1: Results of the sparse GP experiment over 200 tasks. is the mean over tasks with one standard error over environments, in closed form for the GP-based models and estimated from 1000 rounds for the incTNP. KL is the geometric mean over tasks, in nats per evaluation point, before and after the update, with values below clamped before averaging. Rejected is the share of tasks rejected within 1000 rounds at level 0.05.
| Model | KL before | KL after | Rejected | ||
| Exact GP | – | 0 | 0 | 0 | 0.00 |
| incTNP | – | 0.0130 ± 0.0008 | 0.99 | ||
| Fixed | 10 | 0.124 ± 0.006 | 0.47 | 0.66 | 0.99 |
| Fixed | 12 | 0.093 ± 0.005 | 0.18 | 0.35 | 0.99 |
| Fixed | 14 | 0.066 ± 0.005 | 0.17 | 0.95 | |
| Fixed | 16 | 0.040 ± 0.004 | 0.83 | ||
| Fixed | 18 | 0.026 ± 0.003 | 0.66 | ||
| Fixed | 20 | 0.016 ± 0.003 | 0.46 | ||
| Fixed | 22 | 0.013 ± 0.002 | 0.40 | ||
| Fixed | 24 | 0.012 ± 0.002 | 0.38 | ||
| Re-selected | 10 | 1.16 ± 0.08 | 0.47 | 0.66 | 1.00 |
| Re-selected | 12 | 0.93 ± 0.08 | 0.18 | 0.34 | 1.00 |
| Re-selected | 14 | 0.45 ± 0.04 | 0.12 | 0.99 | |
| Re-selected | 16 | 0.153 ± 0.021 | 0.99 | ||
| Re-selected | 18 | 0.021 ± 0.003 | 0.83 | ||
| Re-selected | 20 | 0.0022 ± 0.0003 | 0.29 | ||
| Re-selected | 22 | 0.02 | |||
| Re-selected | 24 | 0.00 |
Table 1.
incTNP. The incTNP uses the sequential architecture of Mortimer et al. [ [44], which updates by extending a key–value cache rather than re-encoding the context. It has five layers of width 128 and eight attention heads, and a Gaussian output for each target. We train it on sequences of 49 points from the same prior, predicting each point from those before it, for 400 000 steps with batches of 32. Training uses Muon [53] on hidden weight matrices and AdamW [54] elsewhere, at learning rate and weight decay 0.01, with 40 000 warm-up steps followed by cosine decay.
Measuring and testing incoherence. Each round draws the block’s outputs from the model’s own joint predictive, autoregressively for the incTNP. The kernel on outputs is Gaussian with lengthscale 0.1, the noise standard deviation, and is approximated by 256 random Fourier features. Every predictive is Gaussian, so the embeddings are exact. We run 1000 rounds for each task and model, shared by the estimator and the test of Appendix § 10.§ 3. For the exact and sparse GPs, is also available in closed form, which Figure 2 and Table 1 report.
Accuracy. Accuracy is the KL divergence from the exact GP’s predictive to the model’s, computed between their marginals at each evaluation point and averaged over the points. After the update, both are conditioned on the block’s observed outputs. Figure 2 reports its geometric mean over tasks. is the same in panels (a) and (b), since it describes a single update of the fitted model.
Results. Table 1 gives the values behind Figure 2, and Table 2 the estimates of and their leave-one-round-out standard errors after 10, 100 and 1000 rounds. The estimates agree with the closed form within their standard errors, and match it from 100 rounds. Between 100 and 1000 rounds, the standard error falls roughly as for clearly incoherent models and roughly as near coherence, where the U-statistic is degenerate.
Tabular data
Data and splits. We use three regression and five classification datasets from OpenML (Table 3). The seven UCI datasets are under CC BY 4.0, and california is listed as public on OpenML. We drop rows with missing values, one-hot encode categorical inputs, and standardise inputs and regression outputs with the training set’s mean and standard deviation. A split divides a dataset at random into a training set (half the rows, at most 1000), a test set (15%, at most 100) and a pool (the rest, at most 2000). Each round draws block inputs, a quarter as many as the training set, without replacement from the pool. The design is therefore random, and and average over it as well as over the block’s outputs. Coherence for every design implies coherence in this sense, but not conversely. Every model is fitted on the training set and updated on the training set and block. All models use the same ten splits per dataset, block inputs and 500 rounds per split, except TabPFN, which because of its cost is run on the first split only, for 128 rounds.
| Dataset | OpenML id | Rows | Inputs | Classes | Training | Test | Pool | Block |
| Regression | ||||||||
| concrete | 44959 | 1030 | 8 | – | 515 | 100 | 415 | 128 |
| california | 44977 | 20640 | 8 | – | 1000 | 100 | 2000 | 250 |
| yacht | 42370 | 308 | 6 | – | 154 | 46 | 108 | 38 |
| Classification | ||||||||
| australian | 40981 | 690 | 42 | 2 | 345 | 100 | 245 | 86 |
| cleveland | 194 | 297 | 25 | 5 | 148 | 44 | 105 | 37 |
| german | 31 | 1000 | 61 | 2 | 500 | 100 | 400 | 125 |
| vehicle | 54 | 846 | 18 | 4 | 423 | 100 | 323 | 105 |
| wdbc | 1510 | 569 | 30 | 2 | 284 | 85 | 200 | 71 |
Table 3. Datasets and the sizes of each part of a split. Inputs counts features after one-hot encoding. Sources: concrete [55 b], california [57], yacht [58], australian [59], cleveland [60, 61], german [62], vehicle [63] and wdbc [64], all fetched from OpenML [65].
Exact models. The GP (ARD squared-exponential kernel, hyperparameters maximising the marginal likelihood) conditions on the block. Exact logistic regression, a softmax model with a Gaussian prior on its weights, represents its posterior by 8192 HMC draws [66] and updates by reweighting them, which is exact Bayes with the draws as a discrete prior.
Approximately Bayesian models. These recompute their approximation to update. The sparse GP takes the GP’s hyperparameters and the optimal of Titsias [ [42], and updates with its inducing points fixed (Appendix § 11.§ 1). Greedy variance selection picks these from the training inputs until the unexplained prior variance per input falls below the noise variance, giving 29 to 100 points. The refitted GP refits the GP’s hyperparameters on the training set and block before conditioning. The full and diagonal Laplace approximations [67] share exact logistic regression’s prior and move to the new mode, taking the full precision or its diagonal there. BART [68] and XBART [69] (stochtree 0.4.5 [70], default priors, 200 trees) predict with the mixture over 100 MCMC draws from one chain (BART) or the last 25 of 40 grow-from-root sweeps (XBART), and update by rerunning the sampler with a fresh seed.
Plug-in models. These refit to update. For regression, kernel ridge and nearest neighbours predict a Gaussian at the fitted function or at the mean of the nearest outputs, with the mean squared leave-one-out residual as variance. For classification, plug-in logistic regression predicts at the posterior mode of exact logistic regression, and nearest neighbours the class frequencies among the neighbours. CatBoost [71] predicts a mean and variance (RMSEWithUncertainty loss) or class probabilities, with its seed fixed at zero. A plug-in model’s joint is by default the product of its predictives. Kernel ridge, plug-in logistic regression and nearest neighbours are also run with the prequential joint of § 2, in random order, which leaves the current predictive, and so its accuracy, unchanged.
TabPFN. TabPFN [72] (v2.5 weights [73], TabPFN-2.5 licence, one estimator, softmax temperature one, not the default 0.9) conditions on the training set and updates by appending the block to its context. It draws the block’s outputs autoregressively, in the block’s order. The network variant switches off the package’s input and target transforms, fingerprint feature, and feature and class shifts, keeping only the clipping of outlying inputs for classification. The preprocessing variant uses the default configuration, whose transforms are refitted whenever the context changes.
Hyperparameters. Kernel ridge’s penalty and bandwidth, the number of neighbours, CatBoost’s depth and number of trees, and the standard deviation of the logistic regression prior are chosen once per dataset by five-fold cross-validated log likelihood on the first split’s training set. This set contains about half of each other split’s test set, except on california. A regression dataset’s noise scale is the smallest cross-validated RMSE among the tuned models and the GP, BART and XBART, which are scored only for this, and is 0.309 on concrete, 0.499 on california and 0.042 on yacht.
Evaluation. Regression uses a Gaussian kernel with the noise scale as lengthscale, approximated by 256 random Fourier features [31] shared by every model, and classification the Dirac kernel, both giving every predictive an exact embedding. The evaluation points are the test inputs, and the test is that of Appendix § 10.§ 3, sharing the rounds of the estimate. Accuracy is the current predictive’s test log loss, with class probabilities clamped below at , which matters only for nearest neighbours, whose predictive can give a label probability zero.
Results. Table 4 summarises the results over datasets, and Tables 5 and 6 give and test log loss on each. The exact models have on every dataset. Exact logistic regression on cleveland is 4.5 standard errors below zero because near coherence the estimate is right-skewed (Appendix § 10.§ 1), so most splits fall below its mean, and rerunning one split 60 times gives an estimate centred on zero. For the inexact models, the standard deviation of over the ten splits is 1.8 to 15 times its within-split standard error (median over datasets), so TabPFN’s standard errors, from its one split, understate its variation across splits. Across the models of Table 4, the Spearman correlation of between datasets is 0.98 for every regression pair and at least 0.88 for classification, lowest for australian and german.
Robustness. Halving or doubling the lengthscale reorders models only on yacht, swapping BART and CatBoost, XBART and nearest neighbours, and the TabPFN variants, each pair within 15% (Table 7). A lengthscale of one, 2 to 24 times the noise scale, shrinks small values of and swaps models up to 23% apart. The prequential joints also change , for example from 0.12 to 0.19 for kernel ridge on yacht.
Monte Carlo error in BART. BART’s incoherence is largely Monte Carlo error. On the first split of each regression dataset, rerunning the sampler with fresh seeds on the training set alone gives a squared discrepancy which is 1.1 to 1.2 times that of an update. A run’s 100 draws are worth about two independent draws on concrete and california, and fewer than one on yacht.
| Model | concrete | california | yacht | ||
| GP () | |||||
| Sparse GP | |||||
| GP, refitted | |||||
| BART | |||||
| XBART | |||||
| Kernel ridge | |||||
| Nearest neighbours | |||||
| CatBoost | |||||
| TabPFN, network | |||||
| TabPFN, preproc. | |||||
| Model | australian | cleveland | german | vehicle | wdbc |
| Logistic, exact () | |||||
| Full Laplace | |||||
| Diagonal Laplace | |||||
| Logistic, plug-in | |||||
| Nearest neighbours | |||||
| CatBoost | |||||
| TabPFN, network | |||||
| TabPFN, preproc. |
Table 5. by dataset: the mean over ten splits their standard deviation divided by , or for TabPFN its one split the leave-one-round-out standard error.
Detectability. Small incoherence often goes unrejected within the budget. The sparse GP’s exceeds twice its standard error on all 30 splits, but only 4 are rejected within 500 rounds, against a predicted median of about 1500. The predicted round, over test inputs , is a heuristic for the rounds the witness needs to align plus those needed to reach at the upper bound on the growth rate (Appendix § 10.§ 4), so it is optimistic. TabPFN’s predicted rounds, 127 to 585, mostly exceed its budget of 128.
Table 6: Test log loss in nats per output, on the standardised outputs for regression: the mean over ten splits, or TabPFN’s one split. The lowest on each dataset before rounding is in bold.
| Model | concrete | california | yacht | ||
| GP | 0.283 | 0.738 | −1.803 | ||
| Sparse GP | 0.457 | 0.889 | −1.749 | ||
| GP, refitted | 0.283 | 0.738 | −1.803 | ||
| BART | 0.151 | 0.773 | −1.540 | ||
| XBART | 0.089 | 0.738 | −0.128 | ||
| Kernel ridge | 0.412 | 0.762 | −0.231 | ||
| Nearest neighbours | 0.892 | 0.956 | 1.045 | ||
| CatBoost | 0.400 | 0.877 | −1.601 | ||
| TabPFN, network | −0.254 | 0.291 | −3.021 | ||
| TabPFN, preproc. | −0.308 | 0.237 | −2.959 | ||
| Model | australian | cleveland | german | vehicle | wdbc |
| Logistic, exact | 0.355 | 1.056 | 0.488 | 0.434 | 0.088 |
| Full Laplace | 0.352 | 1.043 | 0.490 | 0.449 | 0.094 |
| Diagonal Laplace | 0.356 | 1.043 | 0.493 | 0.443 | 0.107 |
| Logistic, plug-in | 0.356 | 1.074 | 0.488 | 0.428 | 0.086 |
| Nearest neighbours | 0.409 | 1.421 | 0.598 | 0.840 | 0.225 |
| CatBoost | 0.323 | 1.190 | 0.490 | 0.500 | 0.097 |
| TabPFN, network | 0.211 | 0.946 | 0.497 | 0.340 | 0.049 |
| TabPFN, preproc. | 0.218 | 0.916 | 0.512 | 0.348 | 0.034 |
Table 6.
Table 7: with the kernel lengthscale at the noise scale / half it / twice it / one (the training outputs’ standard deviation). The GP, with at every lengthscale, is omitted.
| Model | concrete | california | yacht |
| Sparse GP | 0.007 / 0.008 / 0.005 / 0.003 | 0.004 / 0.004 / 0.002 / 0.002 | 0.018 / 0.020 / 0.015 / 0.001 |
| GP, refitted | 0.002 / 0.002 / 0.002 / 0.002 | 0.001 / 0.001 / 0.001 / 0.001 | 0.002 / 0.002 / 0.001 / 0.001 |
| BART | 0.632 / 0.641 / 0.617 / 0.610 | 0.710 / 0.710 / 0.708 / 0.708 | 0.863 / 0.941 / 0.783 / 0.674 |
| XBART | 0.378 / 0.389 / 0.364 / 0.357 | 0.271 / 0.277 / 0.261 / 0.261 | 0.787 / 0.860 / 0.699 / 0.540 |
| Kernel ridge | 0.013 / 0.015 / 0.008 / 0.005 | 0.008 / 0.010 / 0.005 / 0.005 | 0.119 / 0.118 / 0.112 / 0.020 |
| Nearest neighbours | 0.461 / 0.468 / 0.452 / 0.441 | 0.470 / 0.473 / 0.463 / 0.463 | 0.701 / 0.704 / 0.703 / 0.683 |
| CatBoost | 0.900 / 0.913 / 0.876 / 0.860 | 0.596 / 0.601 / 0.591 / 0.591 | 0.912 / 0.920 / 0.840 / 0.823 |
| TabPFN, network | 0.089 / 0.095 / 0.084 / 0.080 | 0.102 / 0.099 / 0.105 / 0.105 | 0.195 / 0.167 / 0.246 / 0.269 |
| TabPFN, preproc. | 0.083 / 0.092 / 0.078 / 0.075 | 0.118 / 0.121 / 0.115 / 0.115 | 0.222 / 0.274 / 0.192 / 0.153 |
Table 7.
Mean-field Bayesian linear regression
The two-parameter example (Figure 1). The model is plus unit-variance Gaussian noise, with prior , fitted to five inputs drawn from with outputs plus standard Gaussian noise. The exact posterior correlation between intercept and slope is , and the mean-field approximation is formed as in the experiment below. The target is , and the candidate design ranges over in steps of , of which the figure shows and marks and . Under mean-field VI, and EMI both prefer (0.081 against 0.068 nats). Under mean-field VI, prefers (0.037 against 0.013), whereas EMI prefers (0.045 against 0.014), as does (0.046 against 0.015), which stays within 0.0012 nats of EMI over the range shown.
Environments. The acquisition experiment is entirely closed form, so there is no estimation noise beyond the choice of environments. The model is Bayesian linear regression on 100 random Fourier features [31] of a Gaussian kernel with lengthscale 0.1 on a scalar input, with weights and noise variance 0.8. The truth is drawn from this prior, so the model is well specified. Each of 100 environments, shared by every arm, draws its features, the true weights, 25 initial inputs from , a pool of 1000 inputs and 100 target inputs from , noisy labels for the initial and pool inputs, and a random order of the pool. The reported log loss is the learner’s expected log loss for a noisy observation from the truth at each target input, so there is no test set.
Learner and acquisition. The learner keeps the exact conjugate posterior. Its mean-field approximation, the optimum of over factorised Gaussians, takes the exact posterior means and the inverse diagonal of the exact posterior precision as variances. It is refitted after each acquisition rather than updated in a streaming fashion. Each arm acquires 100 points one at a time, maximising its score over the remaining pool. The refit’s variance does not depend on the outcome and its mean is affine in it, so is Gaussian and EPIG and EMI, averaged uniformly over the targets, are closed form. Exact Bayes acquires with the exact posterior, and random acquisition takes the pool in its random order.
The coherentised predictive. Rung of the ladder scores candidates by EPIG under a predictive made coherent for the mean-field refit on a reference set , the first round of the remaining pool points in the environment’s fixed order. Coherentising an objective (§ 3) would instead substitute for at each candidate . Every predictive here is at , where is the exact posterior mean and only the weight covariance varies. With and the exact and mean-field covariances after observing , and , coherence requires
As has spectral radius below one, this discrete Lyapunov equation has the unique solution
which we sum by repeated doubling. In EPIG, the coherentised predictive supplies and the law of the outcome at . The post-update predictive is the mean-field refit on , averaged over outcomes at drawn from the coherentised predictive given . A candidate inside is therefore counted both as observed and as part of the block. The reference set is recomputed at every step as the pool shrinks. The shares are , and gives mean-field EPIG.
Measuring incoherence. is computed before any acquisition and pooled over the 100 targets, with a Gaussian kernel of lengthscale , the noise standard deviation. For each rung, the update is the mean-field refit on the whole pool, with outcomes drawn from the rung’s predictive. We use this block, larger than § 4.1 would call for, because the ladder varies coherence with refits on shares of the pool. Figure 3b plots its mean over environments. For the mean-field model, the pool-scale is , whereas its for an update on a single candidate, the block the acquisition faces, is , pooled over candidates within each environment and averaged over environments.
Results. Table 8 gives the values behind Figure 3. Every adjacent pair of rungs differs in loss by at least four standard errors of the paired difference over environments. Coherentising over 1%, 2.5%, 5% and 10% of the pool recovers 32%, 51%, 80% and 95% of mean-field EPIG’s excess loss. Mean-field EPIG and EMI choose different points at the first step in every environment, and their 100 acquired points overlap by 13 on average.
| Acquisition | Excess loss (nats) | () |
| Mean-field, EPIG () | 0.270 0.011 | 1.05 0.05 |
| Coherentised, | 0.184 0.009 | 0.80 0.04 |
| Coherentised, | 0.133 0.008 | 0.55 0.03 |
| Coherentised, | 0.054 0.005 | 0.32 0.02 |
| Coherentised, | 0.014 0.003 | 0.13 0.01 |
| Coherentised, | 0.002 0.002 | 0 |
| Mean-field, EMI | 0.008 0.002 | – |
| Random | 0.156 0.008 | – |
Table 8. Results of the Bayesian linear regression experiment over 100 environments. Excess loss is the mean-field learner’s log loss less that after acquiring with the exact predictive, averaged over the 100 steps, with one standard error over environments. is measured before acquisition for an update on the whole pool.
Robustness. We rerun the experiment with noise variance 0.1 or 3, with targets from , away from most of the pool, and with a truth on independent random features, so that the model is misspecified (Table 9). Mean-field EPIG acquires worse data than random in every setting. With low noise the mean-field variances are badly wrong: before acquisition, the mean-field log loss exceeds the exact one by 1.31 nats, against 0.045 in the baseline. EMI, which still consumes these variances, then does no better than random, whereas the coherentised predictive, which changes them, still comes close to exact Bayes.
| Setting | Mean-field EPIG | Mean-field EMI | Random | ||
| Baseline | 0.270 0.011 | 0.008 0.002 | 0.156 0.008 | 0.054 0.005 | 0.002 0.002 |
| Low noise | 1.966 0.070 | 0.920 0.027 | 0.929 0.037 | 0.033 0.007 | 0.009 0.004 |
| High noise | 0.065 0.004 | 0.003 0.001 | 0.042 0.003 | 0.017 0.003 | 0.001 0.002 |
| Distant targets | 0.385 0.019 | 0.006 0.002 | 0.270 0.013 | 0.017 0.003 | |
| Misspecified | 0.537 0.018 | 0.016 0.003 | 0.454 0.016 | 0.204 0.009 | 0.008 0.002 |
Table 9. Excess loss as in Table 8, with one standard error over environments, under changes to the baseline. Low and high noise set the noise variance to 0.1 and 3, against 0.8 in the baseline.
Neural testbed
Our testbed is modelled on the Neural Testbed Osband et al., [16], with batch acquisition in place of its joint-prediction benchmark.
Environments. The truth is a random ReLU network (two inputs, two hidden layers of 32 units, three outputs) with Xavier-normal weights Glorot and Bengio, [74] and biases uniform on . Its class probabilities are , with offsets giving each class average probability over a grid on . Inputs come from four fixed Gaussian clusters centred at , each input’s cluster uniform over those allowed. The 24 training inputs use three clusters and standard deviation 0.5. The pool of 512 uses all four, and the 64 targets and 4096 test inputs two, one with training data and one without, all with standard deviation 1.5. Environments are kept if every class has mean true probability at least 0.1 over the test inputs, leaving 1000 of 1160.
The decision. Each environment has ten candidate batches of five pool inputs, shared by all models. Eight are designed from per-input BALD (label–parameter mutual information) under a linearised diagonal Laplace approximation fitted to the training data. They take the most informative inputs from the pool, target clusters or training clusters (greedily, with a minimum spacing), an informative input and its nearest neighbours, inputs spread over the pool or target clusters, inputs near the decision boundary, or a mix of target and training inputs. The other two, and replacements for duplicated designed batches, are sampled with random cluster proportions, about three per environment.
Models. Network models share the truth’s architecture without its temperature or offsets, and are fitted to the 24 training points. Exact Bayes condition 8192 networks drawn per environment from the generative prior, with temperature and offsets. The full, Kronecker-factored Martens and Grosse, [75]; Ritter et al. [76] and diagonal linearised Laplace approximations Immer et al., [21] share a MAP fit (Adam, Kingma and Ba, 2015), learning rate 0.05, 1000 steps, prior scale four times the generative one) and use generalised Gauss–Newton curvature Schraudolph, [78]. They condition the linearised model on the batch via a Laplace approximation in its logits (16 damped Newton steps). Bayes by backprop Blundell et al., [79] (Adam, learning rate 0.003, 8 samples) and IVON Shen et al., [20] (learning rate 0.03, 4 samples) fit factorised Gaussians under the generative prior for 2000 steps, with KL weight 0.1. They update by variational continual learning Nguyen et al., [80], with the current posterior as prior and the five new points as data (same KL weight, 100 steps, learning rates and respectively). The deep ensemble Lakshminarayanan et al., [25] (8 members, Adam, learning rate 0.05, 1000 steps, weight penalty 3) and MC dropout Gal and Ghahramani, [81] (rate 0.2, learning rate 0.05, 500 steps, weight penalty 1) update by continuing training on all 29 points, for 1000 and 50 steps respectively. The Laplace and variational predictives mix 128 parameter draws, parameterised by each posterior from standard-normal base draws fixed once per environment, and MC dropout’s mixes 128 masks fixed once and shared by every predictive. The incTNP Mortimer et al., [44] (eight layers of width 256, eight attention heads) updates by appending the batch to its context. It is trained on generative tasks to predict each training point from its predecessors and each target from the training points and part of a batch, with Muon Jordan et al., [53] on hidden weight matrices and AdamW Loshchilov and Hutter, [54] elsewhere (learning rate , cosine decay, 90 000 steps of 128 tasks). Hyperparameters were tuned on 64 separate environments by the KL from the truth before and after updating. Only IVON’s update learning rate and the ensemble’s and MC dropout’s update steps lie at the edge of a grid of over two values.
Objectives, incoherence and cost. For each candidate , the model is updated on all labellings of the batch, so the objectives of § 3.1 are exact sums weighted by its joint , averaged over the 64 targets. Mixtures label each input independently given a component, and the incTNP labels them autoregressively. and use the Dirac kernel, pooled over the targets, and an environment’s value is the median over its candidates’ updates. The random updates of Bayes by backprop, IVON and MC dropout are drawn twice from independent streams, and the objectives and losses use the first. is estimated without bias by the inner product of the two draws’ -weighted mean displacements, and by their mean squared displacement less that estimate. These estimates are negative in 1, 6 and 36 of the 1000 environments respectively, and are kept. The distortion is the standard deviation of over the ten candidates. A candidate’s loss is the KL from the truth to the updated predictive at the test inputs, averaged over the batch’s labellings under the truth, and a decision’s cost is the loss of EPIG’s choice less EMI’s. Accuracy is the KL from the truth to the fitted predictive, averaged over the test inputs.
Figure 4. The local linear fits use a Gaussian kernel with bandwidth 0.16 in in (a), leaving out environments with , and 0.06 nats in (b). The percentiles in (a) are kernel-weighted and the standard errors in (b) approximate. Markers are means over all environments in (a), unlike the medians of Table 10, and over those where the objectives disagree in (b).
| Disagreement | ||||||
| Model | Cost | KL | ||||
| Exact Bayes | 0 | 0 | 0.00 | 0.00 | 0 | 0.171 0.003 |
| IVON | 0.0058 | 0.041 | 0.28 | 0.01 | 0.0084 0.0018 | 0.366 0.007 |
| Laplace, KFAC | 0.0067 | 0.059 | 0.40 | 0.01 | 0.0101 0.0020 | 0.310 0.007 |
| Bayes by backprop | 0.0102 | 0.094 | 0.80 | 0.10 | 0.0245 0.0027 | 0.362 0.007 |
| incTNP | 0.0103 | 0.097 | 0.26 | 0.02 | 0.0009 0.0005 | 0.168 0.003 |
| Ensemble | 0.0181 | 0.118 | 0.52 | 0.08 | 0.0195 0.0033 | 0.406 0.009 |
| Laplace, full | 0.0191 | 0.171 | 0.33 | 0.04 | 0.0071 0.0019 | 0.270 0.004 |
| Laplace, diagonal | 0.0474 | 0.318 | 0.38 | 0.08 | 0.0104 0.0020 | 0.269 0.004 |
| MC dropout | 0.117 | 0.425 | 0.54 | 0.20 | 0.0300 0.0069 | 0.478 0.012 |
Table 10. Results on the neural testbed. and are medians over environments. Disagreement is the share of decisions where each form of EPIG differs from EMI. Cost is averaged over all decisions, with agreements counting as zero, so the cost of a disagreement is Cost divided by the disagreement. KL measures accuracy before the update. Both are in nats with one standard error.
Results. The per-environment Spearman correlation between the eight models’ and distortion has mean 0.73 and median 0.79 over environments, and is positive in 996 of 1000. has a higher mean loss than EMI for every approximate model, by over two standard errors except for the incTNP (0.0009 0.0005). Yet chooses better in 37–49% of disagreements. Disagreement depends on the margin between EMI’s two best candidates relative to the distortion. Pooled over models, its rate falls from 0.74 in the lowest quintile of this ratio to 0.06 in the highest. Over all models’ disagreements, the cost of a disagreement has rank correlation 0.01 with and 0.21 with KL. Across the eight models, its mean has rank correlation 0.24 with and 0.90 with the KL of Table 10.
References
- [1]David J. C. MacKay. Information-based objective functions for active data selection. Neural Computation, 4(4):590–604, 1992a.DOI
- [2]David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. Active learning with statistical models. Journal of Artificial Intelligence Research, 4:129–145, 1996.
- [3]Edwin Fong, Chris Holmes, and Stephen G. Walker. Martingale posterior distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(5):1357–1391, 2023.arxiv.org/abs/2103.15671
- [4]Tom Rainforth, Adam Foster, Desi Ivanova, and Freddie Bickford Smith. Modern Bayesian experimental design. Statistical Science, 39(1):100–114, 2024.arxiv.org/abs/2302.14545
- [5]Roman Garnett. Bayesian Optimization. Cambridge University Press, 2023.DOI
- [6]Freddie Bickford Smith, Andreas Kirsch, Sebastian Farquhar, Yarin Gal, Adam Foster, and Tom Rainforth. Prediction-oriented Bayesian active learning. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 7331–7348. PMLR, 2023.arxiv.org/abs/2304.08151
- [7]Bas C. van Fraassen. Belief and the will. The Journal of Philosophy, 81(5):235–256, 1984.
- [8]David Lewis. Why conditionalize? In Papers in Metaphysics and Epistemology, Cambridge Studies in Philosophy, pages 403–407. Cambridge University Press, 1999.DOI
- [9]Michael Goldstein. The prevision of a prevision. Journal of the American Statistical Association, 78(384):817–819, 1983.DOI
- [11]Bruno Kacper Mlodozieniec, David Krueger, and Richard E. Turner. Implicitly Bayesian prediction rules in deep learning. In Proceedings of the 6th Symposium on Advances in Approximate Bayesian Inference, volume 253 of Proceedings of Machine Learning Research, pages 79–110. PMLR, 2024.
- [12]Fabian Falck, Ziyu Wang, and Christopher C. Holmes. Is in-context learning in large language models Bayesian? A martingale perspective. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 12784–12805. PMLR, 2024.arxiv.org/abs/2406.00793
- [13]Kenyon Ng, Edwin Fong, David T. Frazier, Jeremias Knoblauch, and Susan Wei. TabMGP: Martingale posterior with TabPFN. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research. PMLR, 2026.arxiv.org/abs/2510.25154
- [14]Gábor Pituk, Vik Shirvaikar, and Tom Rainforth. Do Bayesian neural networks actually behave like Bayesian models? In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 49420–49458. PMLR, 2025.
- [15]Zheng Wen, Ian Osband, Chao Qin, Xiuyuan Lu, Morteza Ibrahimi, Vikranth Dwaracherla, Mohammad Asghari, and Benjamin Van Roy. From predictions to decisions: The importance of joint predictive distributions. arXiv preprint arXiv:2107.09224, 2022.arxiv.org/abs/2107.09224
- [16]Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Xiuyuan Lu, Morteza Ibrahimi, Dieterich Lawson, Botao Hao, Brendan O’Donoghue, and Benjamin Van Roy. The neural testbed: Evaluating joint predictions. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2110.04629
- [17]A. P. Dawid. Present position and potential developments: Some personal views: Statistical theory: The prequential approach. Journal of the Royal Statistical Society. Series A (General), 147(2): 278–292, 1984.DOI
- [19]David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859–877, 2017.
- [20]Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Bazan Clement Emile Marcel Raoul, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, and Thomas Möllenhoff. Variational learning is effective for large deep networks. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 44665–44686. PMLR, 2024.arxiv.org/abs/2402.17641
- [21]Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of Bayesian neural nets via local linearization. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 703–711. PMLR, 2021.arxiv.org/abs/2008.08400
- [22]Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes. In ICML Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018a.arxiv.org/abs/1807.01622
- [24]Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do Bayesian inference. In International Conference on Learning Representations, 2022.arxiv.org/abs/2112.10510
- [25]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, pages 6402–6413, 2017.
- [26]Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. In Advances in Neural Information Processing Systems, volume 33, pages 4697–4708, 2020.arxiv.org/abs/2002.08791
- [27]Veit David Wild, Sahra Ghalebikesabi, Dino Sejdinovic, and Jeremias Knoblauch. A rigorous link between deep ensembles and (variational) Bayesian methods. In Advances in Neural Information Processing Systems, volume 36, pages 39782–39811, 2023.
- [28]Freddie Bickford Smith, Jannik Kossen, Eleanor Trollope, Mark van der Wilk, Adam Foster, and Tom Rainforth. Rethinking aleatoric and epistemic uncertainty. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 4345–4359. PMLR, 2025.
- [29]Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723–773, 2012.
- [30]Bharath K. Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert R. G. Lanckriet. Hilbert space embeddings and metrics on probability measures. Journal of Machine Learning Research, 11(50):1517–1561, 2010.
- [31]Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, volume 20, 2007.
- [32]Tudor Manole and Aaditya Ramdas. Martingale methods for sequential estimation of convex functionals and divergences. IEEE Transactions on Information Theory, 69(7):4641–4658, 2023.
- [33]Glenn Shafer. Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society Series A: Statistics in Society, 184(2):407–431, 2021.DOI
- [34]Aaditya Ramdas, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4):576–601, 2023.arxiv.org/abs/2210.01948
- [35]Shubhanshu Shekhar and Aaditya Ramdas. Nonparametric two-sample testing by betting. IEEE Transactions on Information Theory, 70(2):1178–1203, 2024.arxiv.org/abs/2112.09162
- [36]Aleksandr Podkopaev, Patrick Blöbaum, Shiva Kasiviswanathan, and Aaditya Ramdas. Sequential kernelized independence testing. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 27957–27993. PMLR, 2023.arxiv.org/abs/2212.07383
- [37]Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning, 69(2–3):169–192, 2007.
- [38]Ashok Cutkosky and Francesco Orabona. Black-box reductions for parameter-free online learning in Banach spaces. In Proceedings of the 31st Conference on Learning Theory, volume 75 of Proceedings of Machine Learning Research, pages 1493–1529. PMLR, 2018.arxiv.org/abs/1802.06293
- [39]Jean Ville. Étude critique de la notion de collectif. Gauthier-Villars, 1939.
- [40]Glenn Shafer, Alexander Shen, Nikolai Vereshchagin, and Vladimir Vovk. Test martingales, Bayes factors and p-values. Statistical Science, 26(1):84–101, 2011.
- [41]Vladimir Vovk and Ruodu Wang. E-values: Calibration, combination and applications. The Annals of Statistics, 49(3):1736–1754, 2021.DOI
- [42]Michalis Titsias. Variational learning of inducing variables in sparse Gaussian processes. In Proceedings of the 12th International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pages 567–574. PMLR, 2009.
- [43]David R. Burt, Carl Edward Rasmussen, and Mark van der Wilk. Convergence of sparse variational inference in Gaussian processes regression. Journal of Machine Learning Research, 21(131):1–63, 2020.arxiv.org/abs/2008.00323
- [44]Philip Mortimer, Cristiana Diaconu, Tommy Rochussen, Bruno Mlodozeniec, and Richard E. Turner. Incremental transformer neural processes. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research. PMLR, 2026.arxiv.org/abs/2602.18955
- [46]Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Bernhard Schölkopf. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning, 10(1–2):1–141, 2017.DOI
- [47]Wassily Hoeffding. A class of statistics with asymptotically normal distribution. The Annals of Mathematical Statistics, 19(3):293–325, 1948.
- [48]A. W. van der Vaart. Asymptotic Statistics. Cambridge University Press, 1998.
- [49]Robert J. Serfling. Approximation Theorems of Mathematical Statistics. John Wiley & Sons, 1980.
- [51]Peter Grünwald, Rianne de Heide, and Wouter M. Koolen. Safe testing. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(5):1091–1128, 2024.
- [52]Ian Waudby-Smith, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya Ramdas. Time-uniform central limit theory and asymptotic confidence sequences. The Annals of Statistics, 52(6):2613–2640, 2024.
- [53]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024.
- [54]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
- [55]I-Cheng Yeh. Concrete Compressive Strength. UCI Machine Learning Repository, 1998a. DOI: https://doi.org/10.24432/C5PK67.
- [57]R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33 (3):291–297, 1997.DOI
- [58]J. Gerritsma, R. Onnink, and A. Versluis. Yacht Hydrodynamics. UCI Machine Learning Repository, 1981. DOI: https://doi.org/10.24432/C5XG7R.
- [59]Ross Quinlan. Statlog (Australian Credit Approval). UCI Machine Learning Repository, 1987. DOI: https://doi.org/10.24432/C59012.
- [60]Andras Janosi, William Steinbrunn, Matthias Pfisterer, and Robert Detrano. Heart Disease. UCI Machine Learning Repository, 1989. DOI: https://doi.org/10.24432/C52P4X.
- [61]Robert Detrano, Andras Janosi, Walter Steinbrunn, Matthias Pfisterer, Johann-Jakob Schmid, Sarbjit Sandhu, Kern H. Guppy, Stella Lee, and Victor Froelicher. International application of a new probability algorithm for the diagnosis of coronary artery disease. The American Journal of Cardiology, 64(5):304–310, 1989.pubmed.ncbi.nlm.nih.gov/2756873
- [62]Hans Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository, 1994. DOI: https://doi.org/10.24432/C5NC7T.
- [63]Pete Mowforth and Barry Shepherd. Statlog (Vehicle Silhouettes). UCI Machine Learning Repository, 1987. DOI: https://doi.org/10.24432/C5HG6N.
- [64]William Wolberg, Olvi Mangasarian, Nick Street, and W Street. Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository, 1993. DOI: https://doi.org/10.24432/C5DW2B.
- [65]Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. OpenML: Networked science in machine learning. SIGKDD Explorations, 15(2):49–60, 2013.DOI
- [66]Radford M. Neal. MCMC using Hamiltonian dynamics. In Steve Brooks, Andrew Gelman, Galin L. Jones, and Xiao-Li Meng, editors, Handbook of Markov Chain Monte Carlo, pages 113–162. Chapman and Hall/CRC, 2011.DOI
- [67]David J. C. MacKay. The evidence framework applied to classification networks. Neural Computation, 4(5):720–736, 1992b.DOI
- [68]Hugh A. Chipman, Edward I. George, and Robert E. McCulloch. BART: Bayesian additive regression trees. The Annals of Applied Statistics, 4(1):266–298, 2010.
- [69]Jingyu He, Saar Yalov, and P. Richard Hahn. XBART: Accelerated Bayesian additive regression trees. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 1130–1138. PMLR, 2019.arxiv.org/abs/1810.02215
- [70]Andrew Herren, P. Richard Hahn, Jared Murray, and Carlos Carvalho. StochTree: BART-based modeling in R and Python. arXiv preprint arXiv:2512.12051, 2025.arxiv.org/abs/2512.12051
- [71]Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, volume 31, pages 6638–6648, 2018.arxiv.org/abs/1706.09516
- [72]Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, 2025.DOI
- [73]Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Benjamin Jäger, Dominik Safaric, Simone Alessi, Adrian Hayler, Mihir Manum, Rosen Yu, Felix Jablonski, Shi Bin Hoo, Anurag Garg, Jake Robertson, Magnus Bühler, Vladyslav Moroshan, Lennart Purucker, Clara Cornu, Lilly Charlotte Wehrhahn, Alessandro Bonetto, Bernhard Schölkopf, Sauraj Gambhir, Noah Hollmann, and Frank Hutter. TabPFN-2.5: Advancing the state of the art in tabular foundation models. arXiv preprint arXiv:2511.08667, 2025.
- [74]Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249–256. PMLR, 2010.
- [75]James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2408–2417. PMLR, 2015.arxiv.org/abs/1503.05671
- [76]Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable Laplace approximation for neural networks. In International Conference on Learning Representations, 2018.
- [78]Nicol N. Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural Computation, 14(7):1723–1738, 2002.pubmed.ncbi.nlm.nih.gov/12079553
- [79]Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1613–1622. PMLR, 2015.
- [80]Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In International Conference on Learning Representations, 2018.arxiv.org/abs/1710.10628
- [81]Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050–1059. PMLR, 2016.arxiv.org/abs/1506.02142


