betaFeedback

WorksComputer Science

What "Form'' of Uncertainty Does Conformal Prediction Quantify?

Abstract

Conformal prediction (CP) is a post-hoc calibration method that returns prediction sets with distribution-free coverage guarantees, and the size of the output set is often interpreted as a notion of uncertainty. We show that the difference in set size between full and split CP is directly connected to Bayesian posterior variance, which is the epistemic uncertainty of the model. Essentially, this difference in set size is an aggregation of two opposite effects: (1) better model accuracy and therefore smaller set size achieved by training over extra calibration points, and (2) the "formability" of the model at test point towards every possible label. The latter effect vanishes with stronger models or more calibration data (as epistemic uncertainty is known for that behavior). It turns out that the model's formability itself (without the quantization effect of prediction sets, and through estimating the posterior variance) is a useful tool to quantify the epistemic uncertainty. In our empirical evaluation it performs on par with the state of the art in many setups.

Introduction

A model’s prediction can be uncertain, either since the data is hard to predict (aleatoric), or because the model is not confident in its prediction due to lack of information (epistemic). Conformal prediction (CP) is a widely used approach in risk-critical applications where quantifying total predictive uncertainty is crucial. Given any model f(⋅;θ)f(\cdot;\theta), CP returns a prediction set C(xn+1)C(x_{n+1}) for a test point xn+1x_{n+1} that is guaranteed to include the true label with user-specified probability 1−α1-\alpha. This guarantee is distribution-free; namely, it holds regardless of the underlying data distribution, or the model family [3]. CP requires an exchangeable set of calibration points, and can be applied in two ways: (i) fine-tuning the model jointly on the calibration set and the test point for every possible pseudo-label (full CP), or (ii) only evaluating the model on the calibration points (split CP). Full CP is known to be more data-efficient, which is particularly appealing in cases with label scarcity [32] (e.g. semi-supervised graph node classification). The argument is that in these settings the data is not divided into separate training and calibration sets, which potentially results in better model performance, and smaller sets.

In contrast, Zong & Xu [46] show that full CP doesn’t always outperform split CP without specifying a definitive reason, leaving this question open: when and by how much full CP can be more efficient, and what can be concluded from their performance gap? It turns out (as shown in Figure 1, and Figure 2), the performance gap (set size difference) reduces with a larger dataset (calibration, or training), and a lower feature dimension. Intuitively, while in most cases a larger dataset (combination of training and calibration) improves the model’s performance, fine-tuning on the test point with a pseudo-label deforms the model towards that label. The strength of this deformation is governed by the calibration density, and the confidence from pretraining both as resisting factors. Strong deformation includes wrong pseudo-labels in the set, resulting in worse overall performance.

Figure 1. Comparison of full (blue, solid) vs. split (orange, dashed) CP, against the adaptation floor ∣CB∣|C_B| (see § 3), over ncaln_{\mathrm{cal}} and feature dimension KK, and for [Left] CoraML (transductive node-classification, GCN), and [Right] CIFAR-10. The approximation is with m=50m=50 samples (see § 4).

Figure 2. Performance gap between ∣Cfull∣\lvert C_{\mathrm{full}}\rvert and ∣Csplit∣\lvert C_{\mathrm{split}}\rvert over a grid of training and calibration set sizes. This is an extension of the result shown in Figure 1.

In particular, we define formability at a test point, as the difference between the predictive probability of any label between two models: one fine-tuned on a calibration set, and the other on the union of the calibration set and the pseudo-labeled test point (on that label). Under Bayesian training, where the epistemic uncertainty is quantified as the disagreement among the plausible (under the posterior) parameters, we show that formability is directly related to the epistemic uncertainty of the predictive distribution at the test point; namely we bound the posterior variance, by a function of formability. This also shows the reason why by increasing calibration set size the difference between full and split CP disappears: the formability (epistemic uncertainty) is known to reduce with more data.

Background

Let f(⋅;θ)f(\cdot; \theta) be a pretrained model from the family F\mathcal{F} specified by the parameters θ\theta. We use pθ(y∣x)p_{\theta}(y \mid x) for the predictive label distribution of ff at the input xx. Let Dn={(xi,yi)}i=1nD_n = \{(x_i, y_i)\}_{i=1}^{n} be a holdout calibration set jointly exchangeable with the test point (xn+1,yn+1)(x_{n+1}, y_{n+1}).

Split CP (Vovk et al. [43]) (SCP). With riθ=pθ(yi∣xi)r_i^\theta= p_\theta(y_i \mid x_i), define the threshold

qθ=Qα,n(r1:n):=Qα(∑i=1n1n+1δriθ+1n+1δ∞).(1)q_\theta= Q_{\alpha,n}(r_{1:n}) := Q_\alpha\left(\sum_{i=1}^{n}\frac{1}{n+1}\delta_{r_i^\theta} + \frac{1}{n+1}\delta_\infty\right). \tag*{(1)}

and the prediction set CθS(xn+1)={c∈Y:pθ(c∣xn+1)≥qθ}C_\theta^S(x_{n+1}) = \{c \in\mathcal{Y}: p_\theta(c \mid x_{n+1}) \ge q_\theta\}. This set has a distribution-free coverage guarantee Pr⁡[yn+1∈CθS(xn+1)]≥1−α\Pr[y_{n+1} \in C_\theta^S(x_{n+1})] \ge1-\alpha. Other scores are discussed in § C.1.6.

Full CP. We define the augmented dataset Dn+1(c)=Dn∪{(xn+1,c)}D_{n+1}^{(c)} = D_n \cup\{(x_{n+1},c)\}, for a pseudo-label cc. Full conformal prediction (Vovk et al. [43]) (FCP) defines the prediction set CθF(xn+1)={c∈Y:pθ(c)(c∣xn+1)≥Qα,n(r1:n(c))}C_\theta^F(x_{n+1}) = \{c \in\mathcal{Y}: p_\theta^{(c)}(c \mid x_{n+1}) \ge Q_{\alpha,n}(r_{1:n}^{(c)})\} for ri(c)=pθ(c)(yi∣xi)r_i^{(c)} = p_\theta^{(c)}(y_i \mid x_i), where pθ(c)p_\theta^{(c)} is the predictive distribution of the model after fine-tuning on Dn+1(c)D_{n+1}^{(c)}. This prediction set also comes with the same distribution-free guarantee. In both cases, the guarantee comes from the fact that the score pθ(xi,yi)p_\theta(x_i,y_i) are exchangeable for i∈[n+1]i \in[n+1]. The difference is that FCP fine-tunes a single model per test point, per label. The potential use case of this computationally expensive approach is when the labeled dataset is small, and the model can benefit from training on the entire dataset (training and calibration).

Bayesian deep learning. Instead of a point estimate θ\theta (deterministic value for parameters), Bayesian deep learning (Neal [34]; ); Wilson & Izmailov [44]) defines a distribution π(θ)\pi(\theta) over the model parameters. As a result, the predictive distribution is also defined as pπ(y∣x):=Eθ∼π[pθ(y∣x)]p_\pi(y \mid x) := \mathbb{E}_{\theta\sim\pi}[p_\theta(y \mid x)] instead of a single p(y∣x;θ)p(y \mid x;\theta). Training a Bayesian model is equivalent to finding the posterior parameter distribution π(θ∣D)\pi(\theta\mid D) given the dataset DD as evidence, and the prior π0(θ)\pi_0(\theta) over the parameters (e.g. N(0,σI)\mathcal{N}(0,\sigma I) for initialization). Under this definition, the epistemic uncertainty is quantified as the disagreement among parameters that are plausible under the posterior – measured as the variance Var⁡θ∼π[pθ(c∣x)]\operatorname{Var}_{\theta\sim\pi}[p_\theta(c \mid x)] (Sale et al. [40]). Notably, as Jiménez et al. [25] argue, one cannot quantify the aleatoric uncertainty unless the epistemic uncertainty is sufficiently low.

Basin decomposition. The support of a probability distribution (e.g. π(θ)\pi(\theta)) can be decomposed into disjoint basins of attraction B1,…,BK\mathcal{B}_1,\ldots,\mathcal{B}_K with ⋃k=1KBk=Support⁡(π)\bigcup_{k=1}^{K}\mathcal{B}_k = \operatorname{Support}(\pi), and Bk∩Bk′=∅\mathcal{B}_k \cap\mathcal{B}_{k'} = \varnothing for k≠k′k \ne k'. Each basin is a connected region of the parameter space that contains a local optimum θ^k\hat{\theta}_k of the loss function. In § 3.3, we discuss that the posterior predictive distribution can be decomposed into a mixture of distributions over each basin. A better approximation of the variance considers basins other than only the one containing the MAP as well.

Formability

Motivating Experiment

First we evaluate the claim that “in label scarcity, full CP outperforms split CP” by comparing full and split conformal prediction on two simple setups with images (CIFAR-10; Krizhevsky, [29]), and graphs (CoraML; Bojchevski & Gunnemann, [9]). It is already mentioned in a couple of works that in some cases full CP is not better than split CP in set size (e.g., [6, 46]). As shown in Figure 1, however, by increasing the calibration set size, or reducing the feature dimension (using PCA), the performance gap is reduced, until the two methods cross over. These two observations suggest that the performance gap is not governed by the amount of data alone, but by how strongly the model is already determined around the test point. Notably, the argument of “full CP leverages a better model” still serves as a lower bound on the set size, which full CP converges to, by increasing the size of the calibration set.

An intuitive explanation for such behaviour is that when the training and calibration data is scarce, the model can still be deformed towards any label at the test point. Therefore, it is very easy for every (true or false) label of xn+1x_{n+1} to pull its own score above the threshold. To observe this mechanism in isolation, we show a synthesized setup where every score is visible on a 2-dim landscape (Figure 3, details are in § C.1.1). We compare a test point inside a dense calibration neighborhood to one in a sparse support. In all cases, fine-tuning on the pseudo-label deforms the predictive distribution towards the pseudo-label. However, for the one in the dense region, the calibration density resists this deformation, and the self-pull remains small. In the sparse region there are no proximate data to resist. Hence, every pseudo-label pulls its own score above the threshold easily. As a result, the wrong label enters the full set exactly where the model was free to deform. Therefore the performance gap in CPs is governed by the significance of the new test data point w.r.t. the model: how much the model can deform at that point (which is inversely related to the size of DnD_n, and the strength of the pretrained model as a resisting factor). Namely, this deformation is connected to the epistemic uncertainty – a model with more knowledge (training, and calibration data) is harder to form towards wrong labels.

Formability by fine-tuning on the pseudo-label at a test point

Figure 3. The formability pn,c(c∣x)−pn(c∣x)p_{n,c}(c \mid x)-p_n(c \mid x) by fine-tuning on the pseudo-label cc at the test point (star), in a dense [left pair], and a sparse [right pair] (calibration) neighborhood. Pairs are a correct, and a wrong label cc. Calibration points are colored by the shift of their true score (negligible at many points), and the bars report An,cA_{n,c}, Φn,c\Phi_{n,c}, and Tn,cT_{n,c} from Eq. (2).

This direct connection between formability and epistemic uncertainty is particularly important when it compares with methods like softmax entropy. Naturally, the softmax entropy, when evaluated on a single predictor can not distinguish between the two types of uncertainty. Figure 4 shows that by replicating the setup from Figure 3 while moving the generative Gaussians close to each other. At the center, the epistemic uncertainty decrease, while aleatoric uncertainty increase. On a single label distribution however, it is presented as [1/k]k[1/k]_k the entire time. Therefore the softmax entropy does not react to this change, while the formability variance does.

Softmax entropy versus formability variance across blob separation and position

Figure 4. Softmax entropy vs formability variance. The setup is similar to Figure 3 (see § C.1.2). [Left] Data, and direction of movements. Circles show the mean and standard deviation (radius). We move xn+1x_{n+1} from the center to between two blobs (alea ↑\uparrow, and epis ↓\downarrow), and from the other side towards the center of the other blob (both ↓\downarrow). [b] Uncertainty at center (star) when blobs move closer (epis ↓\downarrow, and alea ↑\uparrow), only formability reacts to it. [c], and [d] show it when moving towards the center of one blob (both ↓\downarrow), and between two blobs (epis ↓\downarrow, alea ↑\uparrow).

Theory

Let m∘(x,y):=p∘(y∣x)−q∘m_{\circ}(x,y) := p_{\circ}(y \mid x) - q_{\circ} be the margin of the label yy at xx in any of these predictive regimes: (i) split CP with the pretrained predictive distribution pθp_{\theta} (call it m0m_0), (ii) full CP fine-tuned on the augmented set Dn+1D_{n+1} with predictive distribution pn,cp_{n,c} (call it mn,cm_{n,c}), and (iii) CP fine-tuned only on the calibration set DnD_n with predictive distribution pnp_n (call it mnm_n). Note that (iii) is not a valid CP, as the calibration and test points are not exchangeable, but it represents the effect of “better model from more data”. In all cases, a label is in the prediction set if its margin is non-negative. We decompose the full margin and define three new variables An,cA_{n,c}, Φn,c\Phi_{n,c}, and Tn,cT_{n,c} as

mn,c(x,c)=m0(x,c)+(mn(x,c)−m0(x,c))+(mn,c(x,c)−mn(x,c))=m0(x,c)+[pθ(n)(c∣xn+1)−pθ(c∣xn+1)]−(qθ(n)−qθ)⏟An,c+pθ(c)(c∣xn+1)−pθ(n)(c∣xn+1)⏟Φn,c−(qθ(c)−qθ(n))⏟Tn,c.(2)\begin{aligned} m_{n,c}(x,c) &= m_0(x,c) + \left(m_n(x,c)-m_0(x,c)\right) + \left(m_{n,c}(x,c)-m_n(x,c)\right) \\ &= m_0(x,c) + \underbrace{\left[p_{\theta}^{(n)}(c \mid x_{n+1})-p_{\theta}(c \mid x_{n+1})\right]-\left(q_{\theta}^{(n)}-q_{\theta}\right)}_{A_{n,c}} \\ &\quad+ \underbrace{p_{\theta}^{(c)}(c \mid x_{n+1})-p_{\theta}^{(n)}(c \mid x_{n+1})}_{\Phi_{n,c}}-\underbrace{\left(q_{\theta}^{(c)}-q_{\theta}^{(n)}\right)}_{T_{n,c}}. \tag*{(2)} \end{aligned}

The above decomposition is trivial, but each of the introduced terms has an interpretation: An,cA_{n,c} captures the adaptation-driven efficiency which is the result of fine-tuning on the calibration set. It simulates the effect from which we expected full CP to be more efficient. Φn,c\Phi_{n,c} and Tn,cT_{n,c} are the effects of training on the additional pseudo-labeled test point (we call the total Φn,c:=Φn,c−Tn,c\Phi_{n,c} := \Phi_{n,c} - T_{n,c} the net formability). Adding (xn+1,c)(x_{n+1},c) to the training set pulls the model towards that label (increasing the margin), but at the same time, it affects the other calibration points, which can change the quantile threshold. Ultimately, the full and split CP set size is only a quantization of the margins; i.e.

∣CF(xn+1)∣−∣CS(xn+1)∣=∑c∈Y[1{m0+An,c+Φn,c−Tn,c≥0}−1{m0≥0}].\left|\mathcal{C}^{F}(x_{n+1})\right|-\left|\mathcal{C}^{S}(x_{n+1})\right|=\sum_{c\in\mathcal{Y}}\left[\mathbb{1}\{m_{0}+A_{n,c}+\Phi_{n,c}-T_{n,c}\geq0\}-\mathbb{1}\{m_{0}\geq0\}\right].

Under Bayesian Learning. Discussing this decomposition under Bayesian learning has two advantages: (i) Bayesian learning comes with a well-defined notion of epistemic uncertainty; i.e. the variance of the posterior predictive distribution, and (ii) in the Bayesian framework, we can write fine-tunings as a sequence of Bayesian updates, which decouples the effect of the additional pseudo-label from the calibration data. Let π0(θ)\pi_{0}(\theta) be the prior parameter distribution centered around θ0\theta_{0} (initial or pretrained model), πn(θ)\pi_{n}(\theta) be the posterior distribution after observing the calibration set Dn\mathcal{D}_{n} (we call it “calibration posterior”), and πn,c(θ)\pi_{n,c}(\theta) be the posterior distribution with Dn+1(c)=Dn∪(xn+1,c)\mathcal{D}_{n+1}^{(c)}=\mathcal{D}_{n}\cup(x_{n+1},c) as evidence (we call it “augmented posterior”). By sequential Bayesian updating (Lemma B.1), the posterior distribution πn,c\pi_{n,c} can be computed using Bayes’ rule with {(xn+1,c)}\{(x_{n+1},c)\} as a single evidence, and πn\pi_{n} as the prior. For ease of notation, let

wn,c:=pπn(c∣xn+1)=Eθ∼πn[pθ(c∣xn+1)],(3)w_{n,c}:=p_{\pi_{n}}(c\mid x_{n+1})=\mathbb{E}_{\theta\sim\pi_{n}}[p_{\theta}(c\mid x_{n+1})], \tag*{(3)}

and assume wn,c>0w_{n,c}>0. In Bayesian learning, we define the formability on (xn+1,c)(x_{n+1},c) as

Φn,c(xn+1)=Eθ∼πn,c[pθ(c∣xn+1)]−Eθ∼πn[pθ(c∣xn+1)].(4)\Phi_{n,c}(x_{n+1})=\mathbb{E}_{\theta\sim\pi_{n,c}}[p_{\theta}(c\mid x_{n+1})]-\mathbb{E}_{\theta\sim\pi_{n}}[p_{\theta}(c\mid x_{n+1})]. \tag*{(4)}

Lemma 3.1. The (per-label) variance of the calibration posterior predictive distribution at xn+1x_{n+1} is proportional to the formability at xn+1x_{n+1}:

Var⁡θ∼πn[pθ(c∣xn+1)]=wn,cΦn,c(xn+1),and(5)\operatorname{Var}_{\theta\sim\pi_{n}}[p_{\theta}(c\mid x_{n+1})]=w_{n,c}\Phi_{n,c}(x_{n+1}), \qquad\text{and} \tag*{(5)}

therefore, 0≤Var⁡θ∼πn[pθ(c∣xn+1)]≤Φn,c(xn+1)(1−Φn,c(xn+1))0\leq\operatorname{Var}_{\theta\sim\pi_{n}}[p_{\theta}(c\mid x_{n+1})]\leq\Phi_{n,c}(x_{n+1})(1-\Phi_{n,c}(x_{n+1})).

As a result, for a finite label set, the total posterior variance (epistemic uncertainty) satisfies

Eθ∼πn[∥pθ(⋅∣xn+1)−pπn(⋅∣xn+1)∥2]=∑c∈Ywn,cΦn,c(xn+1)\mathbb{E}_{\theta\sim\pi_{n}}\left[\left\|p_{\theta}(\mathord{\cdot}\mid x_{n+1})-p_{\pi_{n}}(\mathord{\cdot}\mid x_{n+1})\right\|^{2}\right]=\sum_{c\in\mathcal{Y}}w_{n,c}\Phi_{n,c}(x_{n+1})
≤∑c∈YΦn,c(xn+1)(1−Φn,c(xn+1)).(6)\leq\sum_{c\in\mathcal{Y}}\Phi_{n,c}(x_{n+1})(1-\Phi_{n,c}(x_{n+1})). \tag*{(6)}

The two terms Φn,c\Phi_{n,c} and Tn,cT_{n,c} in Eq. (2) are governed by the same posterior on the pseudo-label. As both variance, and wn,cw_{n,c} are positive, the formability Φn,c\Phi_{n,c} is positive. This means that training on the pseudo-label necessarily increases its score. At the same time, it also moves every calibration score, and therefore the threshold. To study all of the above, we define the forming kernel as the covariance kernel over the posterior distribution:

Kn(z,z′):=Cov⁡θ∼πn(pθ(y∣x),pθ(y′∣x′)),z=(x,y), z′=(x′,y′),(7)K_{n}(z,z^{\prime}):=\operatorname{Cov}_{\theta\sim\pi_{n}}\left(p_{\theta}(y\mid x),p_{\theta}(y^{\prime}\mid x^{\prime})\right), \qquad z=(x,y),\ z^{\prime}=(x^{\prime},y^{\prime}), \tag*{(7)}

and use the shorthands Ci,c:=Kn((xi,yi),(xn+1,c))C_{i,c}:=K_{n}((x_{i},y_{i}),(x_{n+1},c)) for the kernel between calibration points and the pseudo-label hypothesis, Vi:=Kn((xi,yi),(xi,yi))=Var⁡θ∼πn[pθ(yi∣xi)]V_{i}:=K_{n}((x_{i},y_{i}),(x_{i},y_{i}))=\operatorname{Var}_{\theta\sim\pi_{n}}[p_{\theta}(y_{i}\mid x_{i})], and the calibration-posterior variance at the test point, Vn,c:=Var⁡θ∼πn[pθ(c∣xn+1)]=Φn,c(xn+1)wn,cV_{n,c}:=\operatorname{Var}_{\theta\sim\pi_{n}}[p_{\theta}(c\mid x_{n+1})]=\Phi_{n,c}(x_{n+1})w_{n,c} (Lemma 3.1).

Theorem 3.2. Under Bayesian training, with scores ri(n)=pπn(yi∣xi)r_{i}^{(n)}=p_{\pi_{n}}(y_{i}\mid x_{i}), ri(c)=pπn,c(yi∣xi)r_{i}^{(c)}=p_{\pi_{n,c}}(y_{i}\mid x_{i}) and qθ(n)=Qα,n(r1:n(n))q_{\theta}^{(n)}=Q_{\alpha,n}(r_{1:n}^{(n)}), qθ(c)=Qα,n(r1:n(c))q_{\theta}^{(c)}=Q_{\alpha,n}(r_{1:n}^{(c)}), the following statements hold: (i) the update in each calibration score rir_{i} is exactly ri(c)−ri(n)=Ci,cwn,cr_{i}^{(c)}-r_{i}^{(n)}=\frac{C_{i,c}}{w_{n,c}} and bounded in absolute value by ViVn,cwn,c\frac{\sqrt{V_{i}V_{n,c}}}{w_{n,c}}. As a result, with [u±v][u\pm v] denoting the closed interval [u−v,u+v][u-v,u+v], we have (ii - Decomposition):

mn,c∈m0+An,c+[Vn,c−max⁡iCi,cwn,c,Vn,c−min⁡iCi,cwn,c](8)m_{n,c}\in m_{0}+A_{n,c}+\left[\frac{V_{n,c}-\max_{i}C_{i,c}}{w_{n,c}},\frac{V_{n,c}-\min_{i}C_{i,c}}{w_{n,c}}\right] \tag*{(8)}
∈m0+An,c+[Vn,c(Vn,c±max⁡iVi)wn,c].(9)\in m_{0}+A_{n,c}+\left[\frac{\sqrt{V_{n,c}}\left(\sqrt{V_{n,c}}\pm\max_{i}\sqrt{V_{i}}\right)}{w_{n,c}}\right]. \tag*{(9)}

Thm. 3.2 shows that under Bayesian training, the set size for full CP can be bounded using only π0\pi_0, and πn\pi_n without recomputing πn,c\pi_{n,c} over every label cc. Figure 5 evaluates this section exactly on a conjugate Bayesian linear regression (setup in § C.1.3), where all three posteriors are Gaussian and every term of Eq. 9 is a closed-form moment: the identities of Lemma 3.1 and Thm. 3.2(i) hold to machine precision, the full margin computed from the actual augmented posterior stays inside the band for every candidate, and the band is tight exactly where the calibration neighborhood is dense.

Multi-panel illustration of Bayesian linear regression and deterministic fine-tuning

Figure 5. [a - e] Thm. 3.2 for Bayesian linear regression: [a] data, training covariate, and calibration points; [b] split and full sets along xx; [c,d] the margins m∘m_{\circ} from Thm. 3.2 in dense and sparse area; [e] posterior variance, and its upper bound from Eq. 6. Details are in § C.1.3. [f-h] Illustration of Prop. 3.3, and Prop. 3.4: [f] same Laplace approximation on two different distributions: πn\pi_n (solid), and a second distribution πn′\pi_n^{\prime} (dashed); [g] the variance difference by increasing the impact of πres\pi_{\mathrm{res}}. [h] The refit can jump to a different basin, i.e. πn,c\pi_{n,c} can be outside the basin of the original MAP θ^n\hat{\theta}_n.

Consistent Bayesian learner. By increasing nn (calibration set size) the posterior πn\pi_n concentrates. Lemma 3.1, and Thm. 3.2 tie formability, and quantile change to exactly the disagreement between plausible parameters from πn\pi_n. The concentration shrinks Φn,c\Phi_{n,c} and Tn,cT_{n,c} and the full margin mn,cm_{n,c} converges to the calibration-posterior margin mn,c→m0+An,cm_{n,c} \to m_0 + A_{n,c} which is also observed in Figure 1.

Deterministic Fine-Tuning

Thm. 3.2, and Lemma 3.1 are written for Bayesian learning (given posterior πn\pi_n), which is intractable to compute for deep models. A common regime for deterministic fine-tuning finds the maximizer of the (log-) posterior, i.e., θ^n:=arg⁡max⁡θL(θ)\hat{\theta}_n := \mathop{\arg\max}_{\theta} \mathcal{L}(\theta), for L(θ):=log⁡π0(θ)+∑i=1nlog⁡pθ(yi∣xi)\mathcal{L}(\theta) := \log\pi_0(\theta) + \sum_{i=1}^{n} \log p_{\theta}(y_i \mid x_i); e.g. the common ℓ2\ell_2-regularized objective assigns a Gaussian prior π0(θ)=N(θ0,τ2I)\pi_0(\theta) = \mathcal{N}(\theta_0,\tau^2 I). Daxberger et al. [13] approximate πn\pi_n with a Gaussian distribution N(θ^n,Σ^n)\mathcal{N}(\hat{\theta}_n,\hat{\Sigma}_n) centered at the MAP θ^n\hat{\theta}_n, and covariance given by the inverse Hessian of the log-posterior; i.e. Σ=−∇θ2L(θ^n)−1∣θ^n\Sigma= -\nabla_{\theta}^{2}\mathcal{L}(\hat{\theta}_n)^{-1}\big|_{\hat{\theta}_n}, which approximates the variance of the predictive distribution as Vn,c≈V^L:=∇θpθ(c∣xn+1)TΣ^n∇θpθ(c∣xn+1)V_{n,c} \approx\hat{V}^{\mathrm{L}} := \nabla_{\theta}p_{\theta}(c \mid x_{n+1})^{\mathsf{T}}\hat{\Sigma}_n\nabla_{\theta}p_{\theta}(c \mid x_{n+1}). This approximation with a Gaussian distribution is the result of a second-order Taylor expansion of L\mathcal{L} around θ^n\hat{\theta}_n. By construction, a shortcoming is that this approximation is always uni-modal (Gaussian), regardless of the shape of the true posterior (which can be multi-modal, skewed, etc.). In other words, there are dissimilar distributions (with different variances) that are all approximated by the same Gaussian distribution under this approach. Aware of this approximation scheme, Prop. 3.3 designs such dissimilar distributions.

Proposition 3.3. Let πn\pi_n be twice differentiable at θ^n\hat{\theta}_n, and UU be an open neighborhood of θ^n\hat{\theta}_n. Define η:=πn(θ^n)−sup⁡θ∉Uπn(θ)>0\eta:= \pi_n(\hat{\theta}_n) - \sup_{\theta\notin U}\pi_n(\theta) > 0. Let πres\pi_{\mathrm{res}} be a probability density that vanishes on UU, and for δ∈(0,1)\delta\in(0,1) satisfy δ∥πres∥∞<(1−δ)η\delta\lVert\pi_{\mathrm{res}}\rVert_{\infty} < (1-\delta)\eta, and set πn′:=(1−δ)πn+δπres\pi_n^{\prime} := (1-\delta)\pi_n + \delta\pi_{\mathrm{res}}. Then:

(i) Every maximizer of πn′\pi_n' lies in UU and is a maximizer of πn\pi_n. In particular θ^n\hat{\theta}_n maximizes πn′\pi_n', and V^L(πn′)=V^L(πn)\hat{V}^{\mathrm{L}}(\pi_n') = \hat{V}^{\mathrm{L}}(\pi_n).

(ii) With wres,c:=Eθ∼πres[pθ(c∣xn+1)]w_{\mathrm{res},c} := \mathbb{E}_{\theta\sim\pi_{\mathrm{res}}}[p_\theta(c \mid x_{n+1})] and Vres,c:=Var⁡θ∼πres[pθ(c∣xn+1)]V_{\mathrm{res},c} := \operatorname{Var}_{\theta\sim\pi_{\mathrm{res}}}[p_\theta(c \mid x_{n+1})],

Vn,c(πn′)−Vn,c(πn)≥δ(1−δ)(wres,c−wn,c)2−Vn,c(πn).(10)V_{n,c}(\pi_n') - V_{n,c}(\pi_n) \ge\frac{\delta}{(1-\delta)(w_{\mathrm{res},c}-w_{n,c})^2 - V_{n,c}(\pi_n)}. \tag*{(10)}

Part (i) shows that the Laplace approximation of the different distributions remains the same while (ii) the variance of the predictive distribution can differ. This is well known, and it is what motivates mixing several Laplace fits [16], ensembling across basins [44], or searching directly for plausible parameters that disagree about the prediction [41]. Basically, the issue stems from the fact that the Bayesian posterior distribution can have multiple basins, and basins other than the one with θ^n\hat{\theta}_n can contribute meaningfully to the predictive variance. Formability retrains the model over each candidate label, meaning that it can jump to the basin that is more likely to predict the candidate label. The following proposition shows that the refit can result in a maximum a posteriori estimate outside of the basin of the original MAP.

Proposition 3.4. Let B0B_0 be a basin containing θ^n\hat{\theta}_n, and suppose that for some r≥0r \ge0 and 0<u<v≤10 < u < v \le1, and some θ′∈B∖B0\theta' \in\mathcal{B} \setminus B_0 outside of the basin containing the MAP estimate, we have

sup⁡θ∈B0pθ(c∣xn+1)≤u<v≤pθ′(c∣xn+1),L(θ^n)−L(θ′)≤r.\sup_{\theta\in B_0} p_\theta(c \mid x_{n+1}) \le u < v \le p_{\theta'}(c \mid x_{n+1}), \qquad\mathcal{L}(\hat{\theta}_n) - \mathcal{L}(\theta') \le r.

Then, if log⁡(v/u)>r\log(v/u) > r, the augmented MAP θ^n,c∉B0\hat{\theta}_{n,c} \notin B_0. In either case pθ^n,c(c∣xn+1)≥e−rvp_{\hat{\theta}_{n,c}}(c \mid x_{n+1}) \ge e^{-r}v.

Experiments

With our experiments we aim to show that (i) the performance gap between full and split conformal prediction is governed by the size of the calibration set, training set, and the dimension of the feature space; and that (ii) it stems from the model’s formability which is directly connected to epistemic uncertainty (synthetic experiments). (iii) The formability itself can serve as a tool to quantify epistemic uncertainty, through the variance as computed in Eq. 6, V^n,c=w^n,cΦ^n,c\hat{V}_{n,c} = \hat{w}_{n,c}\hat{\Phi}_{n,c}. We compare formability against several existing uncertainty quantification methods: softmax entropy [21], deep ensembles [31], MC-dropout [17], Laplace approximation [13], and DeepKNN [36]. The closest methods to formability are pNML regret [8], and the set size gap, against which we specifically compare, together with the best of the aforementioned methods in Figure 7. Reporting against the best does not mean that there is a single best method at every level, but rather every method has its own sweet spot. Overall relative performance is reported in Figure 6, and detailed reports are in § C.

Figure 6. Formability against families of uncertainty measures, for in-distribution data, on graphs, images and language. For each modality the relative performance is gathered from different datasets and models. Each family of methods includes several variations (e.g. Laplace is last-layer and all-parameter linearised, last-layer sampled; self-influence). A detailed comparison is in Figure 13.

Figure 7. Comparison of formability, regret and conformal set size gap over image benchmarks.

For language models, refitting together over a calibration set and test point for each pseudo-label becomes unrealistically expensive. Therefore we approximate the refit with a single step of LoRA [23] fine-tuning. Our updates are done through damped curvature-informed steps, and for calibration points we cache the gradients. We compare methods on six four-option question-answering setups (MMLU, MedQA on small Qwen, and Llama models). The specifics of the setup are in § C.2. Note that the language modeling setup is defined in a very simplistic way. The model is simply asked to choose one of the A–D answers and the score is the softmax over those specific tokens (see § C.2). The model updates are over a LoRA that is applied to the query and value matrices of the attention layers. These are specifically chosen for the computation to fit on a single GPU. The results are also not comparable to the image and graph setups. Defining a clear formability signal for language models is an open question, and we leave it for future work.

Throughout this paper we assumed the TPS score function (Sadinle et al., 2019) s(x,y)=pθ(y∣x)s(x, y) = p_{\theta}(y \mid x), and 1−α=0.91-\alpha= 0.9 unless otherwise specified. While CP is commonly phrased with non-conformity (disagreement) scores, we use agreement as it simplifies the derivations, and interpretation. Setups are equivalent through a change in sign. For uncertainty quantification, we use two main metrics: (i) the misclassification AUROC [21] which is the area under the curve of true positive rate vs false positive rate over rejection according to the uncertainty score function; 0.5 is chance and 1.0 is the perfect score. (ii) The selective E-AURC [18] is the difference in AURC between the uncertainty score, and the ideal uncertainty score (the oracle). In this score lower is better, and 0 is the best score.

Full-vs-Split CP. In Figure 1 we compare full and split conformal prediction on node and image-classification tasks. For dimensionality reduction, we apply PCA on input features. Specifically on CIFAR-10, the features are the embeddings from an ImageNet-pretrained ResNet-18 [20] backbone, (training is done on the two-layer MLP as classification head). Since full CP is computationally expensive to examine, we report these results on a random subset of the test set containing 300 ∼ 400 instances. Prior to observing test examples, we can approximate the full CP set size by a small (e.g. m=50m = 50) number of calibration points over leave-one-out refits (keeping n−1n-1 points as calibration set and the left out point as test set). We run ablation study and show in Figure 2, that by increasing calibration size, training size, or decreasing the dimension of the feature space, the full CP outperforms split CP, and it converges to the adaptation lower bound ∣CB∣|C_B| (consistent with Thm. 3.2). With a larger training set, the effect of calibration set shows a diminishing return.

Synthetic experiment. In Figure 3 we visualize the concept discussed in § 3. If a point is far from the calibration set it can deform the model more easily, its score can increase more, and full conformal set size would also inflate. It also shows the three variables defined in Eq. (2). To synthesize the data, we sample three classes from overlapping Gaussians, and use a multinomial logistic regression over fixed RBF features as the model. The pretrained model θ0\theta_0 is trained on only 3 points per class, and every fine-tuning minimizes ℓ2\ell_2-regularized cross-entropy towards θ0\theta_0 (i.e. the MAP objective with the prior N(θ0,τ2I)\mathcal{N}(\theta_0,\tau^2 I)). We fine-tune once on the calibration set (n=42n = 42) to obtain pnp_n, and once per candidate label cc on Dn+1c\mathcal{D}_{n+1}^{c} for pn,cp_{n,c}. The complete anatomy for every candidate label is in § C.1.1. An important extension of this experiment is shown in Figure 4, where we move the calibration set generators towards the center which reduces epistemic uncertainty, and increases aleatoric uncertainty at the center. Here the softmax entropy remains steady, while the formability variance decreases. This shows that formability directly distinguishes the epistemic uncertainty, while the softmax entropy is agnostic to this decomposition.

In Figure 5 we evaluate Thm. 3.2 in a setup where every posterior can be computed exactly. We sample from y=sin⁡(1.5x)+0.3x+ϵy = \sin(1.5x) + 0.3x + \epsilon, with Gaussian noise of standard deviation 0.5, and use Bayesian linear regression over fixed RBF features. The model is pretrained on 40 points in [−4,−0.5][-4, -0.5], while the 60 calibration points concentrate around x=0x = 0. The full sets are narrower than split, especially where calibration data is dense. In sparse regions, the posterior variance and formability are larger, and the full set expands relative to the model fitted on calibration data alone. The full margins remain within the theoretical bounds at both test points. Details are in § C.1.3.

Formability as a quantification of epistemic uncertainty. While refitting on every label as an uncertainty quantification is not an entirely new concept [8] define regret based on refitting to every candidate label), we connect the refit through formability to Bayesian posterior variance, and the uncertainty captured in full conformal prediction. While both we and Bibas et al. [8] take information from models fitted on pseudo-labels, they take the logarithm-sum of the candidate probabilities, namely, log⁡(1+∑c∈YΦn,c)\log(1+\sum_{c\in\mathcal{Y}}\Phi_{n,c}), while we directly estimate the posterior variance Φn,c⋅wn,c\Phi_{n,c}\cdot w_{n,c}. This is derived directly from CP’s uncertainty. Interestingly, as shown in Figure 7, this minor change in the definition can result in a different evaluation on the data. Formability performs significantly better on in-domain error detection, while the regret-based uncertainty performs better on out-of-distribution data. We compare all methods briefly in Figure 6. The CP set size gap (full - split) itself is not a good uncertainty quantification; mainly because it quantizes all the information captured in Eq. 6, which is a continuous value, into a binary decision per label.

Conclusion

We investigated the performance gap between full, and split conformal prediction. We showed that the gap is the resultant of two opposite effects: adapting the model on the calibration set (more data resulting in better model, and hence smaller sets), and forming it towards each pseudo-label at the test point. Under Bayesian learning, the second effect (formability) is directly tied to posterior predictive variance, and fades as the calibration posterior concentrates. This explains why full CP can become more efficient when the model is already well determined around the test point, and in sparse regions it performs poorly. However, this gap suggest to use formability as a measure of epistemic uncertainty that carries much more information compared to set size (due to the 0/10/1 binarization). Our experiments on images and graphs show its value for detecting prediction errors; and to some degree, ood detection. In many benchmarks, formability is on par with the best of the existing uncertainty quantification methods, and in some cases it outperforms them.

Limitations. The most important limitation is the computational cost of refitting the model on every candidate label. Additionally our analysis, and definition of formability is based on the softmax score function. CP itself is agnostic to the choice of score function, and scores like APS also can lead to meaningful interpretations of the full-vs-split gap. We leave this for future work. Additionally, conformal prediction assumes exchangeability of the data, which is not always satisfied in practice, specifically for uncertainty quantification use cases. Formability implicitly assume that refits are able to reach the MAP estimate. This could challenge the quality in overparameterized models, or in wrong early stopping scenarios. Potentially our language model experiments are affected by this.

AI USE STATEMENT

We used generative AI tools for implementing the experiments (we designed the experimental setup ourselves and asked AI to implement and run them. We also inspected the results for potential issues). We also used AI for rephrasing parts of the paper, from grammar and choice of words, to better presenting the proofs. The proofs are ultimately rewritten by the authors, but AI was used to assist in the process. The research idea was originally proposed by the authors, including the main idea, theory, and the primary design of the experiments.

ETHICS STATEMENT

This work does not raise any ethical concerns, as it is a general study, and a theoretical contribution to uncertainty quantification. The experiments are conducted on publicly available datasets, and the models are trained on these datasets. The authors have no conflicts of interest to disclose.

REPRODUCIBILITY STATEMENT

All datasets, and models used in the experiments are publicly available. The code for reproducing the experiments is publicly available at https://github.com/soroushzargar/Formability.

REFERENCES

Related Works

Conformal prediction. CP [43, 3] is considered a distribution-free uncertainty quantification technique, where the prediction sets are guaranteed to cover the true label with high probability. With that, many consider the size of the prediction set to represent how uncertain the model is on a specific input. While the method works with any model, and any score function, there are many works designing various scores to improve the set size efficiency, adaptivity, and approximation of conditional coverage [38, 4, 45]. We use the basic threshold prediction sets (Sadinle et al., 2019) since it directly connects to the predictive distribution of the model which can connect perfectly with our decomposition in § 3, and through which we bound the epistemic uncertainty. In § C.1.6 we repeat our experiments under an adaptive (rank-based) score, where the full–split gap is wider, and never closes. While full and split conformal prediction are two endpoints of a computational cost spectrum, there are other variants such as Jackknife+ [5] in between. There are techniques that approximate full CP with lower computational cost [7, 2], through influence functions [28]. Both directions are outside of our focus, as we address the fundamental difference between full and split CP.

CP and epistemic uncertainty. Javanmardi et al. (2026); Cruz Cabezas et al. [12] incorporate a second-order distribution (which contains explicit epistemic uncertainty) into conformal prediction. In the reverse direction, Chau et al. [10]; Correia et al. [11] provide interpretations of split conformal prediction in uncertainty quantification; one through connecting split CP to imprecise integral probability metrics, and the other through information theory. In contrast, we provide a direct connection between the difference of full and split CP set size and the epistemic uncertainty, by bounding the posterior variance of the predictive distribution.

Our work connects three arguments: (i) When does full CP outperform split CP? Inconsistencies in comparison are mentioned by Barber et al. [6] and extensively studied by Zong & Xu [46]. However, neither work discusses what the underlying factor is that governs this difference. We show that the unexpected inefficiency in full CP set size stems from the formability of the model at the test point. (ii) We show that, in a Bayesian framework, the formability is directly related to the posterior variance (epistemic uncertainty). The concept of deterministic formability itself is not entirely new. Bibas et al. [8] define a learnability measure (regret) as the log sum of the predictive probabilities θ^n,c\hat{\theta}_{n,c} normalized by the probabilities from θ^n\hat{\theta}_n. However, its connection with Bayesian posterior variance, comparison with deterministic Laplace approximation through basin decomposition [13], or its influence on full conformal prediction do not follow automatically. (iii) We show that on larger models and various tasks this notion of formability still can be used as a practical tool to quantify epistemic uncertainty. Our work also connects with many other uncertainty quantification approaches. While the posterior distribution of parameters is a canonical way to quantify epistemic uncertainty, on larger models it is computationally near-impossible to compute. Laplace Redux [13] approximates this distribution as a Gaussian around the MAP estimate. However, as we show in Prop. 3.3 it is agnostic to other basins of attraction, and therefore miscalculates the epistemic uncertainty where other highly plausible basins disagree with the current predictive distribution. A remedy to that is MC-dropout [17], and ensemble training [31], which are parameter centric approximations. Formability however is a data-centric approximation. On computation, the comparison depends on whether multiple model inference or test-time training is more favorable. Schweighofer et al. [41] instead searches for adversarial models with high disagreement to the baseline, while attaining high posterior plausibility. In comparison, formability is a local measure, searching for such models through retraining. The comparison results as shown in Figure 17 are very close. Our ultimate goal is also not to come up with a new notion of uncertainty, but to showcase what is the underlying uncertainty quantification in conformal prediction.

Figure 17. Cost against quality, one marker per measure, one column per dataset. [Horizontal] Wall-clock seconds per evaluation point (log), timed with CUDA synchronization. The softmax controls are a forward pass that is already made, and are drawn hollow at the left edge. [Vertical] Misclassification AUROC [Top, higher is better], and selective E-AURC [Bottom, lower is better]. Means over calibration splits, bars ±1 standard error on both axes. QUAM (Schweighofer et al., [41]) is drawn at two budgets – the two most probable classes attacked for 50 steps, and every class for 100 – joined by a line from the cheap budget to the full one.

Supplementary to Theory

We first prove the sequential updating property used in § 3.2.

Lemma B.1 (Sequential Bayesian updating). *Let πn\pi_n be the posterior from the prior π0\pi_0 and evidence Dn\mathcal{D}_n, and πn,c\pi_{n,c} be the posterior from the prior π0\pi_0 and evidence Dn+1(c)\mathcal{D}_{n+1}^{(c)}. For a fixed xn+1x_{n+1}, assume its

label to be independent of DnD_n given θ\theta, and wn,c=Eθ∼πn[pθ(c∣xn+1)]>0w_{n,c} = \mathbb{E}_{\theta\sim\pi_n}[p_\theta(c \mid x_{n+1})] > 0 (recalling Eq. (3)). Then πn,c\pi_{n,c} equals the posterior computed from the prior πn\pi_n on the single observation {(xn+1,c)}\{(x_{n+1},c)\}; i.e.,

πn,c(θ)=pθ(c∣xn+1)πn(θ)Eθ∼πn[pθ(c∣xn+1)]=pθ(c∣xn+1)πn(θ)wn,c.(11)\pi_{n,c}(\theta) = \frac{p_\theta(c \mid x_{n+1})\pi_n(\theta)}{\mathbb{E}_{\theta\sim\pi_n}[p_\theta(c \mid x_{n+1})]} = \frac{p_\theta(c \mid x_{n+1})\pi_n(\theta)}{w_{n,c}}. \tag*{(11)}

Proof. For two sets of evidence E1E_1 and E2E_2, Bayes’ rule with the joint evidence gives

p(θ∣E1,E2)=p(E1,E2∣θ)π0(θ)p(E1,E2)=p(E2∣θ,E1)p(E1∣θ)π0(θ)p(E2∣E1)p(E1),p(\theta\mid E_1,E_2) = \frac{p(E_1,E_2 \mid\theta)\pi_0(\theta)}{p(E_1,E_2)} = \frac{p(E_2 \mid\theta,E_1)p(E_1 \mid\theta)\pi_0(\theta)}{p(E_2 \mid E_1)p(E_1)},

Bayes’ rule on E1E_1 alone yields p(E1∣θ)π0(θ)=p(θ∣E1)p(E1)p(E_1 \mid\theta)\pi_0(\theta) = p(\theta\mid E_1)p(E_1), and the law of total probability expands p(E2∣E1)=∫p(E2∣ϑ,E1)p(ϑ∣E1) dϑp(E_2 \mid E_1) = \int p(E_2 \mid\vartheta,E_1)p(\vartheta\mid E_1)\,\mathrm{d}\vartheta. Substituting both,

p(θ∣E1,E2)=p(E2∣θ,E1)p(θ∣E1)∫p(E2∣ϑ,E1)p(ϑ∣E1) dϑ.(12)p(\theta\mid E_1,E_2) = \frac{p(E_2 \mid\theta,E_1)p(\theta\mid E_1)}{\int p(E_2 \mid\vartheta,E_1)p(\vartheta\mid E_1)\,\mathrm{d}\vartheta}. \tag*{(12)}

This means that the posterior after observing E1E_1 can be used as the prior when conditioning on E2E_2, and it results in the same posterior as conditioning jointly on both sets of evidence.

In our case, E1=DnE_1 = D_n and E2={(xn+1,c)}E_2 = \{(x_{n+1},c)\}, therefore p(θ∣E1)=πn(θ)p(\theta\mid E_1) = \pi_n(\theta) and p(θ∣E1,E2)=πn,c(θ)p(\theta\mid E_1,E_2) = \pi_{n,c}(\theta). With xn+1x_{n+1} fixed, we have p(E2∣θ,E1)=pθ(c∣xn+1)p(E_2 \mid\theta,E_1) = p_\theta(c \mid x_{n+1}), and the denominator of Eq. (12) becomes Eθ∼πn[pθ(c∣xn+1)]=wn,c\mathbb{E}_{\theta\sim\pi_n}[p_\theta(c \mid x_{n+1})] = w_{n,c}, which is positive by assumption.

Lemma 3.1. The (per-label) variance of the calibration posterior predictive distribution at xn+1x_{n+1} is proportional to the formability at xn+1x_{n+1}:

Var⁡θ∼πn[pθ(c∣xn+1)]=wn,cΦn,c(xn+1),\operatorname{Var}_{\theta\sim\pi_n}[p_\theta(c \mid x_{n+1})] = w_{n,c}\Phi_{n,c}(x_{n+1}),

and

0≤Var⁡θ∼πn[pθ(c∣xn+1)]≤Φn,c(xn+1)(1−Φn,c(xn+1)).0 \le\operatorname{Var}_{\theta\sim\pi_n}[p_\theta(c \mid x_{n+1})] \le\Phi_{n,c}(x_{n+1})(1-\Phi_{n,c}(x_{n+1})).

As a result, for a finite label set, the total posterior variance (epistemic uncertainty) satisfies

Eθ∼πn[∥pθ(⋅∣xn+1)−pπn(⋅∣xn+1)∥22]=∑c∈Ywn,cΦn,c(xn+1)\mathbb{E}_{\theta\sim\pi_n}\left[\left\|p_\theta(\cdot\mid x_{n+1})-p_{\pi_n}(\cdot\mid x_{n+1})\right\|_2^2\right] = \sum_{c\in\mathcal{Y}} w_{n,c}\Phi_{n,c}(x_{n+1})
≤∑c∈YΦn,c(xn+1)(1−Φn,c(xn+1)).(13)\le\sum_{c\in\mathcal{Y}} \Phi_{n,c}(x_{n+1})(1-\Phi_{n,c}(x_{n+1})). \tag*{(13)}

Proof. Using Eq. (11),

Eθ∼πn,c[pθ(c∣xn+1)]=∫pθ(c∣xn+1)πn,c(θ) dθ=∫pθ(c∣xn+1)pθ(c∣xn+1)πn(θ)wn,c dθ=Eθ∼πn[pθ(c∣xn+1)2]wn,c.\begin{aligned} \mathbb{E}_{\theta\sim\pi_{n,c}}[p_\theta(c \mid x_{n+1})] = \int p_\theta(c \mid x_{n+1})\pi_{n,c}(\theta)\,\mathrm{d}\theta\\ &= \int p_\theta(c \mid x_{n+1})\frac{p_\theta(c \mid x_{n+1})\pi_n(\theta)}{w_{n,c}}\,\mathrm{d}\theta= \frac{\mathbb{E}_{\theta\sim\pi_n}[p_\theta(c \mid x_{n+1})^2]}{w_{n,c}}. \end{aligned}

Substituting it into Eq. (4) yields

Φn,c(xn+1)=Eθ∼πn[pθ(c∣xn+1)2]wn,c−wn,c=Eθ∼πn[pθ(c∣xn+1)2]−wn,c2wn,c=Var⁡θ∼πn[pθ(c∣xn+1)]wn,c.\begin{aligned} \Phi_{n,c}(x_{n+1}) = \frac{\mathbb{E}_{\theta\sim\pi_n}[p_\theta(c \mid x_{n+1})^2]}{w_{n,c}} - w_{n,c} \\ &= \frac{\mathbb{E}_{\theta\sim\pi_n}[p_\theta(c \mid x_{n+1})^2]-w_{n,c}^2}{w_{n,c}} = \frac{\operatorname{Var}_{\theta\sim\pi_n}[p_\theta(c \mid x_{n+1})]}{w_{n,c}}. \end{aligned}

Multiplying both sides by wn,cw_{n,c} results in Eq. (5).

For the remainder of the proof, we will use the following rule: for any X∈[0,1]X \in[0,1] we have

Var⁡(X)=E[X2]−(E[X])2≤E[X]−(E[X])2=E[X](1−E[X]).\operatorname{Var}(X) = \mathbb{E}[X^2]-(\mathbb{E}[X])^2 \le\mathbb{E}[X]-(\mathbb{E}[X])^2 = \mathbb{E}[X](1-\mathbb{E}[X]).

As 0≤pθ(c∣xn+1)≤10 \le p_\theta(c \mid x_{n+1}) \le1, its posterior variance is bounded by

Var⁡θ∼πn[pθ(c∣xn+1)]≤wn,c(1−wn,c).\operatorname{Var}_{\theta\sim\pi_n}[p_\theta(c \mid x_{n+1})] \le w_{n,c}(1-w_{n,c}).

Replacing the variance in Eq. (5) with this inequality gives Φn,c(xn+1)≤1−wn,c\Phi_{n,c}(x_{n+1}) \le1-w_{n,c}, and hence wn,c≤1−Φn,c(xn+1)w_{n,c} \le1-\Phi_{n,c}(x_{n+1}). Therefore,

Var⁡θ∼πn[pθ(c∣xn+1)]=wn,cΦn,c(xn+1)≤Φn,c(xn+1)(1−Φn,c(xn+1)).\operatorname{Var}_{\theta\sim\pi_n}[p_\theta(c\mid x_{n+1})] = w_{n,c}\Phi_{n,c}(x_{n+1}) \le\Phi_{n,c}(x_{n+1})(1-\Phi_{n,c}(x_{n+1})).

For a finite number of labels, the expected squared Euclidean norm is

Eθ∼πn[∥pθ(⋅∣xn+1)−pπn(⋅∣xn+1)∥22]=∑c∈YEθ∼πn[(pθ(c∣xn+1)−wn,c)2]=∑c∈YVar⁡θ∼πn[pθ(c∣xn+1)]=∑c∈Ywn,cΦn,c(xn+1)≤∑c∈YΦn,c(xn+1)(1−Φn,c(xn+1)),\begin{aligned} \mathbb{E}_{\theta\sim\pi_n}\left[\left\|p_\theta(\cdot\mid x_{n+1})-p_{\pi_n}(\cdot\mid x_{n+1})\right\|_2^2\right] &= \sum_{c\in\mathcal{Y}}\mathbb{E}_{\theta\sim\pi_n}\left[(p_\theta(c\mid x_{n+1})-w_{n,c})^2\right] \\ &= \sum_{c\in\mathcal{Y}}\operatorname{Var}_{\theta\sim\pi_n}[p_\theta(c\mid x_{n+1})] \\ &= \sum_{c\in\mathcal{Y}}w_{n,c}\Phi_{n,c}(x_{n+1}) \\ &\le\sum_{c\in\mathcal{Y}}\Phi_{n,c}(x_{n+1})(1-\Phi_{n,c}(x_{n+1})), \end{aligned}

which proves Eq. 6.

Proof of Thm. 3.2

Before proving Thm. 3.2, we first prove two lemmas: (i) Lemma B.2 shows how conditioning on the pseudo-labeled test point moves the posterior expectation w.r.t. the expectation under the prior. In our case the posterior is the augmented posterior, and the prior is the calibration posterior. (ii) Lemma B.3 records the stability of the conformal quantile under coordinatewise shifts of the scores. The proof relies on two auxiliary lemmas: the first shows how conditioning on the pseudo-labeled test point moves any posterior expectation, and the second records the stability of the conformal quantile under coordinatewise shifts of the scores.

Lemma B.2. For any function h:Θ→Rh:\Theta\to\mathbb{R} with Eθ∼πn∣h(θ)∣<∞\mathbb{E}_{\theta\sim\pi_n}|h(\theta)|<\infty,

Eθ∼πn,c[h(θ)]=Eθ∼πn[h(θ)]+Cov⁡θ∼πn(h(θ),pθ(c∣xn+1))wn,c.(14)\mathbb{E}_{\theta\sim\pi_{n,c}}[h(\theta)] = \mathbb{E}_{\theta\sim\pi_n}[h(\theta)] + \frac{\operatorname{Cov}_{\theta\sim\pi_n}(h(\theta),p_\theta(c\mid x_{n+1}))}{w_{n,c}}. \tag*{(14)}

Proof. Using Eq. (11),

Eθ∼πn,c[h(θ)]=∫h(θ)pθ(c∣xn+1)πn(θ)wn,c dθ=Eθ∼πn[h(θ)pθ(c∣xn+1)]wn,c.\mathbb{E}_{\theta\sim\pi_{n,c}}[h(\theta)] = \int h(\theta)\frac{p_\theta(c\mid x_{n+1})\pi_n(\theta)}{w_{n,c}}\,\mathrm{d}\theta= \frac{\mathbb{E}_{\theta\sim\pi_n}[h(\theta)p_\theta(c\mid x_{n+1})]}{w_{n,c}}.

By the definition of covariance,

Eθ∼πn[h(θ)pθ(c∣xn+1)]=Eθ∼πn[h(θ)]Eθ∼πn[pθ(c∣xn+1)]+Cov⁡θ∼πn(h(θ),pθ(c∣xn+1)),\mathbb{E}_{\theta\sim\pi_n}[h(\theta)p_\theta(c\mid x_{n+1})] = \mathbb{E}_{\theta\sim\pi_n}[h(\theta)]\mathbb{E}_{\theta\sim\pi_n}[p_\theta(c\mid x_{n+1})] + \operatorname{Cov}_{\theta\sim\pi_n}(h(\theta),p_\theta(c\mid x_{n+1})),

and Eθ∼πn[pθ(c∣xn+1)]=wn,c\mathbb{E}_{\theta\sim\pi_n}[p_\theta(c\mid x_{n+1})]=w_{n,c}. Dividing by wn,cw_{n,c} completes the proof.

Note that Eq. (5) in Lemma 3.1 is the special case h(θ)=pθ(c∣xn+1)h(\theta)=p_\theta(c\mid x_{n+1}), since the covariance of a variable with itself is its variance.

Lemma B.3 (Quantile stability under score shifts). The map r1:n↦Qα,n(r1:n)r_{1:n}\mapsto Q_{\alpha,n}(r_{1:n}) defined in Eq. (1) is non-decreasing in every coordinate, and translation equivariant whenever the quantile is finite, i.e., Qα,n(r1:n+t1)=Qα,n(r1:n)+tQ_{\alpha,n}(r_{1:n}+t\mathbf{1})=Q_{\alpha,n}(r_{1:n})+t for every t∈Rt\in\mathbb{R}. For any shift vector δ1:n\delta_{1:n},

Qα,n(r1:n)+min⁡iδi≤Qα,n(r1:n+δ1:n)≤Qα,n(r1:n)+max⁡iδi.(15)Q_{\alpha,n}(r_{1:n})+\min_i\delta_i\le Q_{\alpha,n}(r_{1:n}+\delta_{1:n})\le Q_{\alpha,n}(r_{1:n})+\max_i\delta_i. \tag*{(15)}

Proof. Qα,n(r1:n)Q_{\alpha,n}(r_{1:n}) is the α\alpha-quantile of the empirical distribution over the multiset {r1,…,rn,∞}\{r_1,\ldots,r_n,\infty\}, and hence returns the kk-th smallest element of this multiset, where the rank kk is fixed by α\alpha and nn (with Qα,n=∞Q_{\alpha,n}=\infty when k=n+1k=n+1; we assume α\alpha and nn are such that the quantile is finite). Increasing any coordinate cannot decrease any order statistic, which gives monotonicity, and adding the same tt to every score adds tt to every finite order statistic, which gives translation equivariance. For the sandwich, note that coordinatewise ri+min⁡jδj≤ri+δi≤ri+max⁡jδjr_i+\min_j\delta_j\le r_i+\delta_i\le r_i+\max_j\delta_j. Applying monotonicity and then translation equivariance to both sides yields Eq. (15).

Theorem 3.2. Under Bayesian training, with scores ri(n)=pπn(yi∣xi)r_i^{(n)}=p_{\pi_n}(y_i\mid x_i), ri(c)=pπn,c(yi∣xi)r_i^{(c)}=p_{\pi_{n,c}}(y_i\mid x_i) and qθ(n)=Qα,n(r1:n(n))q_\theta^{(n)}=Q_{\alpha,n}(r_{1:n}^{(n)}), qθ(c)=Qα,n(r1:n(c))q_\theta^{(c)}=Q_{\alpha,n}(r_{1:n}^{(c)}), the following statements hold: (i) the update in each calibration score rir_i is exactly ri(c)−ri(n)=Ci,cwn,cr_i^{(c)}-r_i^{(n)}=\frac{C_{i,c}}{w_{n,c}} and bounded in absolute value by ViVn,cwn,c\frac{\sqrt{V_iV_{n,c}}}{w_{n,c}}. As a result, with [u±v][u\pm v] denoting the closed interval [u−v,u+v][u-v,u+v], we have (ii - Decomposition):

mn,c∈m0+An,c+[Vn,c−max⁡iCi,cwn,c,Vn,c−min⁡iCi,cwn,c](16)m_{n,c}\in m_0+A_{n,c}+\left[\frac{V_{n,c}-\max_i C_{i,c}}{w_{n,c}},\frac{V_{n,c}-\min_i C_{i,c}}{w_{n,c}}\right] \tag*{(16)}
∈m0+An,c+[Vn,c(Vn,c±max⁡iVi)wn,c](17)\in m_0+A_{n,c}+\left[\frac{\sqrt{V_{n,c}}\left(\sqrt{V_{n,c}}\pm\max_i\sqrt{V_i}\right)}{w_{n,c}}\right] \tag*{(17)}

Proof. Part (i). Under Bayesian training, the calibration scores are posterior predictive probabilities: ri(n)=pπn(yi∣xi)=Eθ∼πn[pθ(yi∣xi)]r_i^{(n)}=p_{\pi_n}(y_i\mid x_i)=\mathbb{E}_{\theta\sim\pi_n}[p_\theta(y_i\mid x_i)] and ri(c)=pπn,c(yi∣xi)=Eθ∼πn,c[pθ(yi∣xi)]r_i^{(c)}=p_{\pi_{n,c}}(y_i\mid x_i)=\mathbb{E}_{\theta\sim\pi_{n,c}}[p_\theta(y_i\mid x_i)]. Applying Lemma B.2 with h(θ)=pθ(yi∣xi)h(\theta)=p_\theta(y_i\mid x_i) gives

ri(c)−ri(n)=Cov⁡θ∼πn(pθ(yi∣xi),pθ(c∣xn+1))wn,c=Ci,cwn,c.r_i^{(c)}-r_i^{(n)}=\frac{\operatorname{Cov}_{\theta\sim\pi_n}\left(p_\theta(y_i\mid x_i),p_\theta(c\mid x_{n+1})\right)}{w_{n,c}}=\frac{C_{i,c}}{w_{n,c}}.

The bound ∣Ci,c∣≤ViVn,c|C_{i,c}|\le\sqrt{V_iV_{n,c}} is the Cauchy–Schwarz inequality for the covariance.

Part (ii). We first bound the threshold drift. By definition, Tn,c=qθ(c)−qθ(n)T_{n,c}=q_\theta^{(c)}-q_\theta^{(n)} with qθ(c)=Qα,n(r1:n(c))q_\theta^{(c)}=Q_{\alpha,n}(r_{1:n}^{(c)}) and qθ(n)=Qα,n(r1:n(n))q_\theta^{(n)}=Q_{\alpha,n}(r_{1:n}^{(n)}). Defining δi\delta_i as the difference in the scores after the update, we have r1:n(c)=r1:n(n)+δ1:nr_{1:n}^{(c)}=r_{1:n}^{(n)}+\delta_{1:n} with δi=Ci,c/wn,c\delta_i=C_{i,c}/w_{n,c}, so Lemma B.3 gives

min⁡iCi,cwn,c≤Tn,c≤max⁡iCi,cwn,c,\frac{\min_i C_{i,c}}{w_{n,c}}\le T_{n,c}\le\frac{\max_i C_{i,c}}{w_{n,c}},

hence

∣Tn,c∣≤max⁡i∣Ci,c∣wn,c≤Vn,cmax⁡iViwn,c.(18)|T_{n,c}|\le\frac{\max_i|C_{i,c}|}{w_{n,c}}\le\frac{\sqrt{V_{n,c}}\max_i\sqrt{V_i}}{w_{n,c}}. \tag*{(18)}

where the last step is the Cauchy–Schwarz bound of part (i).

Next, Eq. (2) is an algebraic identity that holds for any three fits: mn,c=m0+An,c+Φn,c−Tn,cm_{n,c}=m_0+A_{n,c}+\Phi_{n,c}-T_{n,c}. Under Bayesian refitting where the formability is defined as Eq. (4), Φn,c=pπn,c(c∣xn+1)−pπn(c∣xn+1)\Phi_{n,c}=p_{\pi_{n,c}}(c\mid x_{n+1})-p_{\pi_n}(c\mid x_{n+1}), we write Φn,c−Tn,c=Vn,cwn,c−Tn,c\Phi_{n,c}-T_{n,c}=\frac{V_{n,c}}{w_{n,c}}-T_{n,c} (given Φn,c=Vn,c/wn,c\Phi_{n,c}=V_{n,c}/w_{n,c}). Therefore, mn,c=m0+An,c+Vn,c/wn,c−Tn,cm_{n,c}=m_0+A_{n,c}+V_{n,c}/w_{n,c}-T_{n,c}. By Eq. (18), we have

Vn,cwn,c−Tn,c∈[Vn,c−Vn,cmax⁡iViwn,c,Vn,c+Vn,cmax⁡iViwn,c]=[Vn,c(Vn,c±max⁡iVi)wn,c]=Φ‾n,c,\begin{aligned} \frac{V_{n,c}}{w_{n,c}}-T_{n,c}\in\left[\frac{V_{n,c}-\sqrt{V_{n,c}}\max_i\sqrt{V_i}}{w_{n,c}},\frac{V_{n,c}+\sqrt{V_{n,c}}\max_i\sqrt{V_i}}{w_{n,c}}\right] \\ &=\left[\frac{\sqrt{V_{n,c}}\left(\sqrt{V_{n,c}}\pm\max_i\sqrt{V_i}\right)}{w_{n,c}}\right]=\overline{\Phi}_{n,c}, \end{aligned}

which proves Eq. 9.

Proof of Prop. 3.3

Proposition 3.3. Let πn\pi_n be twice differentiable at θ^n\hat{\theta}_n, and UU be an open neighborhood of θ^n\hat{\theta}_n. Define η:=πn(θ^n)−sup⁡θ∉Uπn(θ)>0\eta:=\pi_n(\hat{\theta}_n)-\sup_{\theta\notin U}\pi_n(\theta)>0. Let πres\pi_{\mathrm{res}} be a probability density that vanishes on UU, and for δ∈(0,1)\delta\in(0,1) satisfy δ∥πres∥∞<(1−δ)η\delta\|\pi_{\mathrm{res}}\|_\infty<(1-\delta)\eta, and set πn′:=(1−δ)πn+δπres\pi_n':=(1-\delta)\pi_n+\delta\pi_{\mathrm{res}}. Then:

(i) Every maximizer of πn′∗liesin∗\pi_n' *lies in*U$ and is a maximizer of πn\pi_n. In particular θ^n\hat{\theta}_n maximizes πn′\pi_n', and V^L(πn′)=V^L(πn)\hat{V}^{L}(\pi_n')=\hat{V}^{L}(\pi_n).

(ii) With wres,c:=Eθ∼πres[pθ(c∣xn+1)]w_{\mathrm{res},c}:=\mathbb{E}_{\theta\sim\pi_{\mathrm{res}}}[p_\theta(c\mid x_{n+1})] and Vres,c:=Var⁡θ∼πres[pθ(c∣xn+1)]V_{\mathrm{res},c}:=\operatorname{Var}_{\theta\sim\pi_{\mathrm{res}}}[p_\theta(c\mid x_{n+1})],

Vn,c(πn′)−Vn,c(πn)≥δ[(1−δ)(wres,c−wn,c)2−Vn,c(πn)].(19)V_{n,c}(\pi_n')-V_{n,c}(\pi_n)\ge\delta\left[(1-\delta)(w_{\mathrm{res},c}-w_{n,c})^2-V_{n,c}(\pi_n)\right]. \tag*{(19)}

Proof. (i) On UU, πres=0\pi_{\mathrm{res}}=0, so πn′=(1−δ)πn≤(1−δ)πn(θ^n)=πn′(θ^n)\pi_n'=(1-\delta)\pi_n\le(1-\delta)\pi_n(\hat{\theta}_n)=\pi_n'(\hat{\theta}_n), which means that πn′\pi_n' is maximized at the same points as πn\pi_n in UU. Outside UU,

πn′(θ)≤(1−δ)(πn(θ^n)−η)+δ∥πres∥∞<(1−δ)πn(θ^n)=πn′(θ^n).\pi_n'(\theta)\le(1-\delta)\left(\pi_n(\hat{\theta}_n)-\eta\right)+\delta\|\pi_{\mathrm{res}}\|_\infty<(1-\delta)\pi_n(\hat{\theta}_n)=\pi_n'(\hat{\theta}_n).

Hence, the maximizers of πn′\pi_n' and πn\pi_n are the same in UU, and θ^n\hat{\theta}_n is one of them. Since log⁡πn′=log⁡(1−δ)+log⁡πn\log\pi_n'=\log(1-\delta)+\log\pi_n on the open set UU (meaning that the distribution is a scaled version of πn\pi_n in that region), the two log-densities have the same Hessian at θ^n\hat{\theta}_n, hence the same Σ^n\hat{\Sigma}_n (the log⁡(1−δ)\log(1-\delta) disappears when differentiating). The gradient ∇θpθ^n(c∣xn+1)\nabla_\theta p_{\hat{\theta}_n}(c\mid x_{n+1}) depends only on the model and on θ^n\hat{\theta}_n. Therefore V^L(πn′)=V^L(πn)\hat{V}^{L}(\pi_n')=\hat{V}^{L}(\pi_n).

(ii) Write f(θ):=pθ(c∣xn+1)f(\theta):=p_\theta(c\mid x_{n+1}). Expectations are linear in the density, so

Eπn′[f]=(1−δ)wn,c+δwres,c,Eπn′[f2]=(1−δ)(Vn,c(πn)+wn,c2)+δ(Vres,c+wres,c2).\mathbb{E}_{\pi_n'}[f]=(1-\delta)w_{n,c}+\delta w_{\mathrm{res},c},\qquad\mathbb{E}_{\pi_n'}[f^2]=(1-\delta)\left(V_{n,c}(\pi_n)+w_{n,c}^2\right)+\delta\left(V_{\mathrm{res},c}+w_{\mathrm{res},c}^2\right).

Subtracting the square of the first from the second,

Vn,c(πn′)=(1−δ)Vn,c(πn)+δVres,c+(1−δ)wn,c2+δwres,c2−((1−δ)wn,c+δwres,c)2,=(1−δ)Vn,c(πn)+δVres,c+δ(1−δ)(wres,c−wn,c)2.\begin{aligned} V_{n,c}(\pi_n')=(1-\delta)V_{n,c}(\pi_n)+\delta V_{\mathrm{res},c}+(1-\delta)w_{n,c}^2+\delta w_{\mathrm{res},c}^2-\left((1-\delta)w_{n,c}+\delta w_{\mathrm{res},c}\right)^2, \\ &=(1-\delta)V_{n,c}(\pi_n)+\delta V_{\mathrm{res},c}+\delta(1-\delta)(w_{\mathrm{res},c}-w_{n,c})^2. \end{aligned}

where the last three terms equal δ(1−δ)wn,c2+δ(1−δ)wres,c2−2δ(1−δ)wn,cwres,c=δ(1−δ)(wres,c−wn,c)2\delta(1-\delta)w_{n,c}^2+\delta(1-\delta)w_{\mathrm{res},c}^2-2\delta(1-\delta)w_{n,c}w_{\mathrm{res},c}=\delta(1-\delta)(w_{\mathrm{res},c}-w_{n,c})^2. Subtracting Vn,c(πn)V_{n,c}(\pi_n) from both sides results in Eq. 10, and the inequality follows from Vres,c≥0V_{\mathrm{res},c}\ge0.

Proof of Prop. 3.4

Proposition 3.4. Let B0B_0 be a basin containing θ^n\hat{\theta}_n, and suppose that for some r≥0r\ge0 and 0<u<v≤10<u<v\le1, and some θ′∉B0\theta'\notin B_0 outside of the basin containing the MAP estimate, we have

sup⁡θ∈B0pθ(c∣xn+1)≤u<v≤pθ′(c∣xn+1),andL(θ^n)−L(θ′)≤r.\sup_{\theta\in B_0}p_\theta(c\mid x_{n+1})\le u<v\le p_{\theta'}(c\mid x_{n+1}),\qquad\text{and}\qquad\mathcal{L}(\hat{\theta}_n)-\mathcal{L}(\theta')\le r.

Then, if log⁡(v/u)>r\log(v/u)>r, the augmented MAP θ^n,c∉B0\hat{\theta}_{n,c}\notin B_0. In either case pθ^n,c(c∣xn+1)≥e−rvp_{\hat{\theta}_{n,c}}(c\mid x_{n+1})\ge e^{-r}v.

Proof. Adding the candidate (xn+1,c)(x_{n+1},c) to the calibration set adds log⁡pθ(c∣xn+1)\log p_\theta(c\mid x_{n+1}) to the log posterior. Thus the augmented MAP θ^n,c\hat{\theta}_{n,c} globally maximizes L(θ)+log⁡pθ(c∣xn+1)\mathcal{L}(\theta)+\log p_\theta(c\mid x_{n+1}). With L(θ′)≥L(θ^n)−r\mathcal{L}(\theta')\ge\mathcal{L}(\hat{\theta}_n)-r, and log⁡pθ′(c∣xn+1)≥log⁡v\log p_{\theta'}(c\mid x_{n+1})\ge\log v, we have:

L(θ^n,c)+log⁡pθ^n,c(c∣xn+1)≥L(θ′)+log⁡pθ′(c∣xn+1)≥L(θ^n)−r+log⁡v.\begin{aligned} \mathcal{L}(\hat{\theta}_{n,c})+\log p_{\hat{\theta}_{n,c}}(c\mid x_{n+1})\ge\mathcal{L}(\theta')+\log p_{\theta'}(c\mid x_{n+1}) \\ &\ge\mathcal{L}(\hat{\theta}_n)-r+\log v. \end{aligned}

Since θ^n\hat{\theta}_n maximizes the calibration log posterior, L(θ^n,c)≤L(θ^n)\mathcal{L}(\hat{\theta}_{n,c})\le\mathcal{L}(\hat{\theta}_n). Subtracting L(θ^n,c)\mathcal{L}(\hat{\theta}_{n,c}) from the preceding inequality therefore yields log⁡pθ^n,c(c∣xn+1)≥log⁡v−r\log p_{\hat{\theta}_{n,c}}(c\mid x_{n+1})\ge\log v-r, or pθ^n,c(c∣xn+1)≥e−rvp_{\hat{\theta}_{n,c}}(c\mid x_{n+1})\ge e^{-r}v. This bound holds whether or not log⁡(v/u)>r\log(v/u)>r. If that condition holds, then e−rv>ue^{-r}v>u, whereas every model in B0B_0 assigns cc probability at most uu. Hence θ^n,c∉B0\hat{\theta}_{n,c}\notin B_0.

Supplementary Results, and Experiments

Synthetic Experiments

Anatomy of the Split-to-Full CP

This experiment replicates the transition from split to full conformal prediction in a two-dimensional setup where every quantity of Eq. (2) is directly visible. Figure 3 in § 3.1 shows a compact presentation of the result (one correct, and one wrong candidate per test point). Figure 8 reports the complete anatomy for every candidate label at both test points, in addition to the score function at each stage.

Figure 8. Complete anatomy of the split-to-full margin.

Setup. We sample three classes from overlapping Gaussians (means on a triangle with pairwise distance 4, standard deviation 1.5). The model is a multinomial logistic regression over fixed RBF

features, so that every refit is an exact convex MAP problem. The pretrained model θ0\theta_0 is trained on 3 points per class (with a prior centered at zero), and every fine-tuning minimizes the cross-entropy plus the L2 pull 12∥θ−θ0∥2\frac{1}{2}\lVert\theta-\theta_0\rVert^2 with τ2=1\tau^2 = 1; i.e. the MAP objective under the prior N(θ0,τ2I)\mathcal{N}(\theta_0,\tau^2I). The calibration set has n=42n = 42 points, the score is the predictive probability, and 1−α=0.91 - \alpha= 0.9. We fine-tune once on DnD_n (the base model pnp_n), and once per candidate on Dn+1(c)D_{n+1}^{(c)} (the full models pn,cp_{n,c}). We examine two test points that differ only in their local calibration density: one is placed inside the class-0 cluster, the other away from every calibration point.

Description for Figure 8. Rows iterate over the candidate labels c∈{0,1,2}c \in\{0,1,2\}. There are two blocks, one for the test point in the dense area (top), and the other (bottom) the sparse one. Columns from left to right are (i) the base predictive field pn(c∣x)p_n(c \mid x) together with its level set at the base threshold qθ(n)q_{\theta}^{(n)} (dashed): the region inside the contour is where the label cc would enter the prediction set under the base model. (ii) The same field after fine-tuning on the pseudo-label, pn,c(c∣x)p_{n,c}(c \mid x), with the level set at qθ(n,c)q_{\theta}^{(n,c)} (solid). (iii) Deformation pn,c(c∣x)−pn(c∣x)p_{n,c}(c \mid x)-p_n(c \mid x). The calibration markers are colored by the shift of their own score, ri(c)−ri(n)=Ci,c/wn,cr_i^{(c)}-r_i^{(n)}=C_{i,c}/w_{n,c} from Thm. 3.2. The figure also shows the quantities An,cA_{n,c}, Φn,c\Phi_{n,c}, and Tn,cT_{n,c} on one scale shared across all plots. (iv) Shows every score through the three regimes (split, base, full): thin lines are the calibration scores, the dashed line is the threshold, the shaded area is where the label is included in the set, and the bold line is the candidate itself, with filled markers indicating membership at that stage.

As expected, at the dense region the self-pull is resisted by the surrounding calibration points (and the training) with Φn,c≤0.13\Phi_{n,c} \leq0.13 for the wrong candidates. In other words, the deformation is small, and sharply localized. The threshold also barely moves, and the two level sets remain relatively steady. Conversely, at the sparse test point where there is much less resistance from other points, fine-tuning on any pseudo-label bends the model freely (Φn,c≈0.3\Phi_{n,c} \approx0.3–0.40.4 for every candidate), no calibration score reacts (Ci,c≈0C_{i,c} \approx0, hence Tn,c≈0T_{n,c} \approx0), and as a result, every candidate – the wrong ones included – flips into the full set. The takeaway is that the full CP loses exactly where the formability Φn,c=Vn,c/wn,c\Phi_{n,c}=V_{n,c}/w_{n,c} is large.

Aleatoric against Epistemic Uncertainty

Figure 4 in § 3.1 reads two uncertainty measures on the setup of § C.1.1 while the aleatoric part of the uncertainty is moved on purpose. The generating law is known, so the aleatoric part is known in closed form: with μk\mu_k the class means (at pairwise distance dd), and the common standard deviation 1.5, the Bayes posterior at xx is ηk(x)∝exp⁡(−∥x−μk∥2/(2⋅1.52))\eta_k(x) \propto\exp\left(-\lVert x - \mu_k\rVert^2/(2 \cdot1.5^2)\right), and we report 1−KK−1TV⁡(η(x),1K)1 - \frac{K}{K-1}\operatorname{TV}\left(\eta(x), \frac{1}{K}\right), with TV the total variation distance to the uniform law over the K=3K = 3 classes. It is 1 where the label is a coin flip among the three classes, 1/21/2 where it is a coin flip between two of them, 0 where it is determined, and it depends neither on the model nor on the sample. The two measures are read from the model fine-tuned on DnD_n: the normalized softmax entropy H(pn(⋅∣xn+1))/log⁡KH(p_n(\cdot\mid x_{n+1}))/\log K, and the formability variance ∑cV^n,c=∑cw^n,cΦ^n,c\sum_c \widehat{V}_{n,c} = \sum_c \widehat{w}_{n,c}\widehat{\Phi}_{n,c} of Eq. 6, with w^n,c=pn(c∣xn+1)\widehat{w}_{n,c} = p_n(c \mid x_{n+1}), and Φ^n,c=pn,c(c∣xn+1)−pn(c∣xn+1)\widehat{\Phi}_{n,c} = p_{n,c}(c \mid x_{n+1}) - p_n(c \mid x_{n+1}) from one MAP refit per pseudo-label, exactly as in § C.1.1.

Setup. Model, pretraining (3 points per class), calibration (n=42n = 42), prior, and α\alpha are those of § C.1.1. The grid of RBF centers is the one of that experiment extended by the same spacing (to 15×1415 \times14 centers), so that it covers the wider configurations, and the longer paths below. The data are drawn once as standard normal offsets ziz_i, and placed as xi=μyi(d)+1.5zix_i = \mu_{y_i}(d) + 1.5z_i, so that the sample moves continuously with dd. The means μk(d)\mu_k(d) are the means of § C.1.1 scaled about their centroid μˉ\bar{\mu} to pairwise distance dd (d=4d = 4 is the configuration of Figure 3), and the pretrained model, and pnp_n are refitted at every dd. In Figure 4(b), dd runs from 8 to 0, and xn+1=μˉx_{n+1} = \bar{\mu}. In (c), and (d), d=4d = 4, and xn+1(t)=μˉ+t∥μ0−μˉ∥ux_{n+1}(t) = \bar{\mu} + t\lVert\mu_0 - \bar{\mu}\rVert u for t∈[0,3.5]t \in[0, 3.5] along a unit direction uu: in (c) towards the center of class 0, which the path reaches at t=1t = 1; in (d) towards the midpoint μˉ12\bar{\mu}_{12} of classes 1 and 2, which it reaches at t=0.5t = 0.5. The path of tt lies, up to 0.01, on the perpendicular bisector of μ1\mu_1 and μ2\mu_2, so the Bayes posterior stays a two-way coin flip between the two classes along the rest of it. Every curve is the mean over 5 draws of the data, and the band is their range. The aleatoric reference does not depend on the draw.

Reading. In (b), the label at xn+1x_{n+1} is a three-way coin flip at every dd (the centroid is, up to 0.02, equidistant from the three means), so the aleatoric reference is at least 0.976 throughout. The softmax entropy reads this: its mean stays between 0.82 and 0.92 (the range over the draws is 0.59–0.98). The labeled points, however, pile up around xn+1x_{n+1} as the classes converge – the Gaussian kernel density of Dn\mathcal{D}_n at xn+1x_{n+1} (bandwidth 1.1) grows from 0.002 to 0.046 – and the formability variance reads this: it falls in every draw, from 0.276 to 0.061 on average (∑cΦ^n,c\sum_c \widehat{\Phi}_{n,c} from 0.94 to 0.20). The two measures move independently: the entropy is flat, while the formability variance falls by a factor of 4.5. In (c), the aleatoric reference falls from 0.994 to 0.079 at μ0\mu_0, and to 0.002 at t=2t = 2. Both measures fall towards μ0\mu_0, the entropy from 0.92 to 0.32, and the formability variance from 0.166 to 0.024, as they must: with pn(0∣xn+1)p_n(0 \mid x_{n+1}) near one every plausible model agrees, and Vn,c≤wn,c(1−wn,c)V_{n,c} \leq w_{n,c}(1-w_{n,c}) (Lemma 3.1) leaves no room for disagreement. Both reach their minimum at t=1.4t = 1.4 (0.22, and 0.012), where the labeled points start to thin out (the density is 0.018 there, 0.009 at t=2t = 2, 0.003 at t=2.5t = 2.5, and below 10−310^{-3} from t=3t = 3 on), and then both rise: the formability variance to 0.37, since a model that no data resist is free to deform towards any label (∑cΦ^n,c=1.1\sum_c \widehat{\Phi}_{n,c} = 1.1; every refit lifts its pseudo-label to about 0.7), and the entropy to 0.99, since the RBF logits decay to the bias away from the data, and the softmax returns to the near-uniform bias. The entropy therefore reads the same value, its maximum, at the centroid and at the far end of the path, where the aleatoric reference is 0.994, and 0: the first reading is aleatoric, the second is entirely epistemic, and the entropy cannot tell the two apart. In (d), the aleatoric reference falls from 0.994 to 0.62 at the midpoint μˉ12\bar{\mu}_{12}, and then settles at 0.50: the label remains a coin flip between classes 1, and 2 along the rest of the path. The entropy reads 0.74 at the midpoint, its minimum 0.69 at t=0.9t = 0.9, and then climbs back to 0.99. The formability variance reads 0.12 at the midpoint, and rises to 0.37 as the density falls from 0.014 to 0. Between t=1t = 1 and t=3.5t = 3.5 the aleatoric part is constant, so the rise of the entropy from 0.69 to 0.99 is entirely epistemic; the entropy does not say so, and the formability variance does. Wherever the entropy is high, then, the formability variance separates the two cases: it is small where the entropy is high because the labels are mixed and the labeled points are many (the centroid at small dd, the neighborhood of μˉ12\bar{\mu}_{12}), and large where the entropy is high because nothing constrains the model (the centroid at d=8d = 8, the far ends of both paths).

Exact Evaluation on Bayesian Linear Regression

The margin decomposition for full - split CP in § 3.2 is general. However, using the Bayesian learning regime we can show that Φn,c\Phi_{n,c} is a function of epistemic uncertainty. Such a setup is unrealistic for large-scale deep learning, for which we discuss the same concept under deterministic fine-tuning in § 3.3. Meanwhile, the Bayesian setup is exactly realizable in a small Gaussian-linear model, where every quantity of § 3.2 can be evaluated in closed form. The synthetic dataset is drawn from f∗(x)=sin⁡(1.5x)+0.3xf^*(x) = \sin(1.5x) + 0.3x plus noise: 40 pretraining points uniform on [−4,−0.5][-4,-0.5], and the calibration set is drawn from the mixture x∼34N(0,0.52)+14U[−4,4]x \sim\frac{3}{4}\mathcal{N}(0,0.5^2) + \frac{1}{4}\mathcal{U}[-4,4], therefore its density is high around x=0x = 0 and low elsewhere (n=60n = 60). The model is y=θTψ(x)+ϵy = \boldsymbol{\theta}^{\mathsf{T}}\psi(x) + \epsilon with ϵ∼N(0,σ2)\epsilon\sim\mathcal{N}(0,\sigma^2), σ=0.5\sigma= 0.5, and ψ(x)∈R3\psi(x) \in\mathbb{R}^{3} is a bias plus 12 Gaussian bumps from −0.7-0.7 centered on a grid over [−4,4][-4,4]. The score is the likelihood pθ(y∣x)=N(y;θTψ(x),σ2)p_\theta(y \mid x) = \mathcal{N}(y;\boldsymbol{\theta}^{\mathsf{T}}\psi(x),\sigma^2), which takes values in [0,1/(σ2π)]=[0,0.8][0,1/(\sigma\sqrt{2\pi})] = [0,0.8]. The bound Vn,c≤Φn,c(1−Φn,c)V_{n,c} \leq\Phi_{n,c}(1-\Phi_{n,c}) of Lemma 3.1 therefore applies to it as it does to a class probability, and the candidates cc range over a grid on [−3,3][-3,3]. The pretrained model θ0\theta_0 is trained with prior N(0,I)\mathcal{N}(0,I), and 1−α=0.91-\alpha= 0.9.

Closed forms. For θ∼N(μ,S)\theta\sim\mathcal{N}(\mu,S) write m(x):=μTψ(x)m(x) := \mu^{\mathsf{T}}\psi(x) and s2(x):=ψ(x)TSψ(x)s^2(x) := \psi(x)^{\mathsf{T}}S\psi(x). The calibration posterior is πn=N(μn,Sn)\pi_n = \mathcal{N}(\mu_n,S_n) with Sn−1=I+σ−2∑i=1nψ(xi)ψ(xi)TS_n^{-1} = I + \sigma^{-2}\sum_{i=1}^{n}\psi(x_i)\psi(x_i)^{\mathsf{T}} and μn=Sn(θ0+σ−2∑i=1nyiψ(xi))\mu_n = S_n(\theta_0 + \sigma^{-2}\sum_{i=1}^{n}y_i\psi(x_i)), and πn,c\pi_{n,c} is the same update with the pair (xn+1,c)(x_{n+1},c) added. This update is the refit that produces mn,c,Φn,cm_{n,c},\Phi_{n,c}, and Tn,cT_{n,c} in Figure 5. The moments that enter Lemma 3.1 and Thm. 3.2 are, with Ψ=[ψ(x),ψ(x′)]\Psi= [\psi(x),\psi(x')],

Eθ[pθ(y∣x)]=N(y;m(x),σ2+s2(x)),\mathbb{E}_{\theta}[p_{\theta}(y \mid x)] = \mathcal{N}(y;m(x),\sigma^2+s^2(x)),
Eθ[pθ(y∣x)2]=12σπN(y;m(x),σ2/2+s2(x)),\mathbb{E}_{\theta}[p_{\theta}(y \mid x)^2] = \frac{1}{2\sigma\sqrt{\pi}}\mathcal{N}(y;m(x),\sigma^2/2+s^2(x)),
Eθ[pθ(y∣x)pθ(y′∣x′)]=N((y,y′);(m(x),m(x′)),σ2I+ΨTSΨ),\mathbb{E}_{\theta}[p_{\theta}(y \mid x)p_{\theta}(y' \mid x')] = \mathcal{N}((y,y');(m(x),m(x')),\sigma^2I+\Psi^{\mathsf{T}}S\Psi),

which are used to compute the scores rir_i and the candidate mass wn,cw_{n,c}, the variances ViV_i and Vn,cV_{n,c}, and the forming kernel Ci,cC_{i,c} of Eq. (7). Each identity of § 3.2 is then compared against the quantity it predicts, computed independently from the refit πn,c\pi_{n,c}.

Conclusion. Our sanity check (which is also shown in Figure 5) succeeds as over all 481×3481 \times3 candidate–test pairs, the two independent computations are close up to the numerical instability; i.e.

max⁡i,c∣ri(c)−ri(n)−Ci,c/wn,c∣=7⋅10−15\max_{i,c}|r_i^{(c)}-r_i^{(n)}-C_{i,c}/w_{n,c}|=7\cdot10^{-15}, max⁡c∣Vn,c−wn,cΦn,c∣=3⋅10−16\max_c|V_{n,c}-w_{n,c}\Phi_{n,c}|=3\cdot10^{-16}, ∣Ci,c∣≤ViVn,c|C_{i,c}|\le\sqrt{V_iV_{n,c}} throughout, and mn,cm_{n,c} lies inside the band of Eq. 9 with a smallest slack of 6.5⋅10−36.5\cdot10^{-3} (inside the kernel band of Eq. (18), 2.4⋅10−42.4\cdot10^{-4}). The formability is the dominant term wherever the neighborhood is sparse (max⁡cΦn,c=0.24\max_c\Phi_{n,c}=0.24 against max⁡c∣Tn,c∣=0.03\max_c|T_{n,c}|=0.03 at the sparse test point), which is why the full set inflates there and not at the dense point, where max⁡cΦn,c=0.02\max_c\Phi_{n,c}=0.02.

The Frozen Image Backbones

Every image experiment in this paper trains a small head on top of a frozen ImageNet ([14])-pretrained backbone, and the feature space from which the formability is defined is the output of this backbone (this especially matters for CIFAR-10 experiments with feature reduction). The ImageNet output layer is removed, the remaining network embeds each image once, and only the head is trained, or refitted. The backbone itself is never updated, which helps us to reduce the cost of per-candidate refit from the entire model to only the head. Table 1 lists the specifications of the ResNet ([20]), ViT ([15]), and ConvNeXt ([33]) backbones. ViT-H/14 uses the SWAG weights of Singh et al. [42].

Table 1: The frozen ImageNet-pretrained backbones over which the image experiments are run. Weights is the torchvision checkpoint. Original counts the parameters of the classifier as loaded, and fixed ones that remain once its ImageNet output layer is removed, which is the form in which we load it. dd is the dimension of the embedding the head sees, and it is also the dimension the PCA reduction of § C.1.5 starts from. The top-1 accuracy is the reference number reported for the same checkpoint on ImageNet-1k.

BackboneWeightsOriginalFixeddim. ddtop-1 (%)
ResNet-18IMAGENET1K_V111.7M11.2M51269.8
ResNet-50IMAGENET1K_V225.6M23.5M204880.9
ViT-B/16IMAGENET1K_V186.6M85.8M76881.1
ConvNeXt-BIMAGENET1K_V188.6M87.6M102484.1
ViT-H/14IMAGENET1K_SWAG_E2E_V1633.5M632.2M128088.6

Table 1.

Full vs. Split CP over the Feature Dimension

Setup. We use a two-layer GCN ([27]) over the 2879-dimensional bag-of-words node features of CoraML, and a two-layer MLP over the 512-dimensional embedding of a frozen ImageNet-pretrained ResNet-18 for CIFAR-10. Full CP trains the model from initialization on the training set, the calibration set, and the pseudo-labeled test point. We reduce the features by PCA to K∈{32,16,8}K \in\{32, 16, 8\} for CoraML, and K∈{64,32,16}K \in\{64, 32, 16\} for CIFAR-10, and sweep ncaln_{\mathrm{cal}}, and ntrn_{\mathrm{tr}}. The score is the predictive probability of § 2, and 1−α=0.91-\alpha=0.9. The adaptation floor ∣CB∣\lvert C_B\rvert is the size of the posterior conformal set Cθn(xn+1)C_{\theta_n}(x_{n+1}) i.e. split CP on the model trained on the training, and the calibration set together, which needs a single fit per calibration set, and no per-candidate refit.

Results. In Figure 9, and Figure 10 the plots go from the full feature space to the smallest KK. The top row reports the average set size against ncaln_{\mathrm{cal}}, where the hollow markers are the approximate full CP with m=50m=50, and the bottom row reports the empirical coverage of the same sets against the nominal 1−α1-\alpha. Full, and split CP are at the nominal level in every panel which shows the sanity of the result. The adaptation floor is not (as expected): it under-covers by a wide margin in the full feature space (≈0.57\approx0.57 on CoraML at ncal=50n_{\mathrm{cal}}=50, and ≈0.49\approx0.49 on CIFAR-10 at ncal=100n_{\mathrm{cal}}=100), and it approaches 1−α1-\alpha as ncaln_{\mathrm{cal}} grows, or KK shrinks. This is expected, since CBC_B scores the calibration points it was trained on, and the amount of this optimism is again a formability – now read at the calibration points.

Figure 9. Full (blue, solid) vs. split (orange, dashed) CP on CoraML (transductive node-classification, GCN) over the calibration set size ncaln_{\mathrm{cal}}, with the 2879-d bag-of-words features [Left], and their PCA reduction to K=32K = 32, 16, and 8 [Middle Left to Right]. [Top] Average set size against the adaptation floor ∣CB∣|C_B| (dotted). Hollow markers are the approximate full CP with m=50m = 50. [Bottom] Empirical coverage of the same sets against the nominal 1−α=0.91 - \alpha= 0.9 (black, dashed). Bands are ±1\pm1 standard deviation across calibration splits.

Figure 10. Full (blue, solid) vs. split (orange, dashed) CP on CIFAR-10 (two-layer MLP over a frozen ResNet-18) over the calibration set size ncaln_{\mathrm{cal}}, with the 512-d embedding [Left], and its PCA reduction to K=64K = 64, 32, and 16 [Middle Left to Right]. [Top] Average set size against the adaptation floor ∣CB∣|C_B| (dotted). Hollow markers are the approximate full CP with m=50m = 50. [Bottom] Empirical coverage of the same sets against the nominal 1−α=0.91 - \alpha= 0.9 (black, dashed). Bands are ±1\pm1 standard deviation across calibration splits.

Similarly in set size, we observe that by increasing the calibration points, the difference between the two approaches decreases, until full CP crosses the split CP. Notably this effect is more visible in CIFAR-10 than in CoraML. Mainly this is since the training size of CIFAR-10 is larger than that of CoraML. We also observe that reducing the feature dimension closes the performance gap.

An interesting side observation is that the question of whether the full CP is more efficient than the split CP can be answered without the need to compute the full CP. One can simply estimate it with a small sample rate (m=50m=50) by computing a leave-one-out refit on the calibration set, and that point alone. This is a relatively cheap operation to check the performance gap, without the need for expensive computation. The method turns out to be a good approximation as shown in both figures (the hollow markers are the approximate full CP with m=50m=50).

We report the extended experiment on the gap ∣CF(xn+1)∣−∣CS(xn+1)∣\lvert C^F(x_{n+1})\rvert-\lvert C^S(x_{n+1})\rvert over a grid of ntrainn_{\mathrm{train}}, and ncaln_{\mathrm{cal}}, on CoraML (2879-dd), and on CIFAR-10 (the 512-dd embedding). The results are shown in Figure 11. This is also same as our expectation. Another takeaway from this result is that the training set size also decreases the performance, and as the training size increases, the effect of the calibration set size becomes less significant.

Figure 11. The gap ∣CF∣−∣CS∣|\mathcal{C}^{F}|-|\mathcal{C}^{S}| in average set size over the training set size ntrainn_{\mathrm{train}}, and the calibration set size ncaln_{\mathrm{cal}} (positive: split CP is smaller, negative: full CP is smaller), on CoraML with 2879-d features [Top], and CIFAR-10 with the full 512-d embedding [Bottom]. [Left] The gap over the grid. [Middle] The gap against ncaln_{\mathrm{cal}}, one line per ntrainn_{\mathrm{train}}. Hollow markers are the CoraML extension arm up to ncal=2800n_{\mathrm{cal}}=2800. [Right] The gap against ntrainn_{\mathrm{train}}, one line per ncaln_{\mathrm{cal}}. In the line panels, the shade encodes the size of the other set (light to dark = small to large), and bands are ±1\pm1 standard error of the paired difference across calibration splits. 1−α=0.91-\alpha=0.9.

Adaptive Prediction Sets

In this work we focus on threshold prediction sets as the score function ((Sadinle et al., 2019)): it directly sets the conformity score to the predictive probability of the candidate label; i.e. s(x,y)=pθ(y∣x)s(x,y)=p_{\theta}(y\mid x). That object is very easy to work with and it directly connects to the decomposition of Eq. (2) to the updates of the predictive distribution after refits. However, conformal prediction is agnostic to the choice of score, and the same margin-decomposition can apply on other score functions as well. It is an important question to ask what interpretation does the full-split gap (and in particular the margin gap) has under that different score.

We repeat the experiment shown in Figure 10, this time under the adaptive prediction sets (APS) score [38]. Result is shown in Figure 12. This ablation does not need to train a new model since we have already cached the refits from the previous experiment.

Figure 12. The grid of Figure 11 read under the randomized APS score [38], off the same refits: the gap ∣CF∣−∣CS∣|C^{F}|-|C^{S}| in average set size over the training set size ntrainn_{\mathrm{train}}, and the calibration set size ncaln_{\mathrm{cal}} (positive: split CP is smaller), on CoraML with 2879-d features [Top], and CIFAR-10 with the full 512-d embedding [Bottom]. [Left] The gap over the grid. [Middle] The gap against ncaln_{\mathrm{cal}}, one line per ntrainn_{\mathrm{train}}. Hollow markers are the CoraML extension arm up to ncal=2800n_{\mathrm{cal}}=2800. [Right] The gap against ntrainn_{\mathrm{train}}, one line per ncaln_{\mathrm{cal}}. In the line panels, the shade encodes the size of the other set (light to dark = small to large), and bands are ±1\pm1 standard error of the paired difference across calibration splits. The color scale is centered at zero as in Figure 11, but it spans a range about two and a half times wider. No cell of either grid is negative. 1−α=0.91-\alpha=0.9.

APS applies the same conformal prediction as shown in § 2, but instead on the following score function:

s(x,y)=1−∑y′:pθ(y′∣x)>pθ(y∣x)(pθ(y′∣x)−upθ(y∣x))s(x,y)=1-\sum_{\substack{y':\,p_{\theta}(y' \mid x)>p_{\theta}(y \mid x)}}\left(p_{\theta}(y' \mid x)-u p_{\theta}(y \mid x)\right)

and we use the randomized variant, with one u∼Unif⁡(0,1)u \sim\operatorname{Unif}(0,1) drawn per (point, rank) once per split, and reused by every refit. The score therefore remains a deterministic function of the predictive distribution, and the exchangeability argument is untouched. Everything else – the grid, the test points, the models, and 1−α=0.91-\alpha=0.9 – is as in Figure 11.

Result. Split CP is smaller in every cell of both grids, and by a wide margin, and by increasing the training points or calibration points the gap converges to zero. APS can be read as a fractional knapsack problem over the probability distribution: take the smallest number of labels (all with equal weight), until 1−β1-\beta cumulative probability is captured. Then conformalization (computing the quantile) is an answer to: what cumulative 1−β1-\beta in model’s predictive probability can capture the 1−α1-\alpha probability in the true distribution. The gap is therefore a measure of how much the model’s predictive distribution is misaligned with the true distribution, and the gap converges to zero as the model sees more labels and becomes better aligned with the true distribution. However deriving the same formability decomposition as in § 3.2 is not straightforward, specially since the score involves the ranking of the labels, and the refit can change this ranking. We leave this as an open question for future work.

Breakdown of Uncertainty Quantification

In Figure 13 we report the detailed comparison against other uncertainty measures. Formability is close to the best methods on graphs and images, while on language softmax entropy performs better. The full–split set-size gap performs poorly across all three modalities, as it quantizes the continuous formability signal into a binary signal.

Figure 13. Formability against families of uncertainty measures, for in-distribution data, on graphs, images and language. For each modality the relative performance is gathered from different datasets and models.

Figure 14 extends the image comparison of Figure 6, and Figure 7 across different backbones and distribution shifts. Formability stays close to the best uncertainty measures for ranking errors on clean images, while for shift detection the result depends on the type of shift and is weaker on Gaussian noise.

Figure 14. Formability against uncertainty measures across frozen image backbones and distribution shifts. [Top] Misclassification AUROC (higher is better) and selective E-AURC (lower is better, log scale) on clean CIFAR-10, CIFAR-100, ImageNet, and ImageNet-V2, with five backbones in each block. [Bottom] Shift-detection AUROC for CIFAR-10-C at three severities, Street View House Numbers (SVHN), CIFAR-100, held-out classes, and ImageNet-V2 against ImageNet validation images. The blue line is ∑cV^n,c\sum_c \widehat{V}_{n,c}. Bars are ±1 standard error over splits. Other colors distinguish Laplace, ensemble and MC-dropout, QUAM, feature-based, and softmax measures.

Figure 15 further extends the image comparison of Figure 6 by grouping CIFAR–10 images according to human-label disagreement (which can be considered a proxy for aleatoric uncertainty). Formability remains close to the best methods consistently within these groups, while ranking the errors generally becomes harder as the disagreement increases.

Figure 15. Error ranking within bins of human-label disagreement on CIFAR-10H. The 1000 CIFAR-10 test images are grouped by the entropy HH of about 51 human labels per image: unanimous (H=0H = 0), low (0<H<0.20 < H < 0.2), medium (0.2≤H<0.50.2 \le H < 0.5), and high (H≥0.5H \ge0.5). “All” is the pooled result. Each column is a frozen backbone. [Top] Misclassification AUROC (higher is better). [Bottom] Selective E-AURC (lower is better, log scale). Blue: ∑cV^n,c\sum_c \hat{V}_{n,c} with ±1\pm1 standard error over splits. Other lines: last-layer Laplace, ensemble BALD, QUAM, Mahalanobis, and softmax entropy.

Ablation on training set size. In addition we also stress-test the measures against the number of labels the model has seen during training. Figure 16 shows this comparison across various training budgets.

Figure 16. The epistemic measures of § 4 against the number of labels the model has seen, N=ntrain+ncalN = n_{\mathrm{train}} + n_{\mathrm{cal}} (log), at the end of training (epoch 300), on CoraML with the 2879-d node features [Left], and CIFAR-10 with the 512-d embedding [Right]. The accuracy of the base model at each rung is printed along the top. [Top] Misclassification AUROC (higher is better; dotted: chance). [Bottom] Selective E-AURC (lower is better). Every measure ranks the same base model on the same 200 evaluation points of a split, with ncal=20n_{\mathrm{cal}} = 20 held fixed. Blue circles: ∑cV^n,c\sum_c \hat{V}_{n,c} from the refits. Purple diamonds: the linearized last-layer Laplace (solid), the all-parameter empirical-Fisher form (dashed), and the sampled last-layer Laplace (dotted). Grey squares: ensemble BALD (solid), and MC-dropout BALD (dashed). Green triangles: deep kNN. Orange: softmax entropy. Red crosses: the set-size gap ∣CF∣−∣CS∣\lvert\mathcal{C}^{F}\rvert- \lvert\mathcal{C}^{S}\rvert. Lines are means over 8 calibration splits, bands ±1\pm1 standard error.

The models are the ones of § C.1.5. We vary the number of labels the model has seen, N=ntrain+ncalN=n_{\mathrm{train}}+n_{\mathrm{cal}}, from 50 to 1600 on CoRaML, and from 50 to 3000 on CIFAR–10 – with the calibration set held at ncal=20n_{\mathrm{cal}}=20. Note that here our main argument is not to show a powerful conformal prediction, but to show the quality of formability in capturing epistemic uncertainty. Therefore the small calibration size is justified for computational efficiency. Columns are the two datasets, with the accuracy of the base model at each rung along the top. [Top] The misclassification AUROC (higher is better; the dotted rule is chance). [Bottom] The selective E-AURC (lower is better). In all training budgets, formability remains among the best methods.

Cost against quality. The refits are the most expensive reading of the epistemic uncertainty by construction: one fine-tuning per pseudo-label, per test point. We evaluate the cost, and (AUROC, E-AURC) on different measures for five datasets: CoraML, CiteSeer, and PubMed with the two-layer GCN of § 4, and CIFAR-10, and ImageNet-100 (its validation images) with the two-layer MLP over the 512-d embedding of the frozen ResNet-18 (Table 1). The refits, the measures, and the two metrics are as in § 4. Every measure is timed per evaluation point with CUDA synchronization. The fit of the base model is charged to no one, since every measure needs it, and the softmax controls are a forward pass that is already made. The refits are fitted in a batch over the candidate labels, which is how full CPU is run throughout the paper. One at a time, they would cost one to two orders of magnitude more. QUAM is timed at two budgets: every class attacked for 100 steps, and the two most probable classes for 50.

Figure 17 shows the result: one column per dataset. The horizontal axis is the wall-clock seconds per evaluation point (log). The vertical axis is the misclassification AUROC [Top], and the selective E-AURC [Bottom]. Markers are means over the calibration, and bars are ±1 standard error on both axes. The two QUAM budgets are joined by a line, from the cheap one to the full one.

Formability on Language Models

For language model tasks, we use four-option question answering: each model assigns probabilities to the answer letters A–D, and the conformal set contains a subset of these letters. We do not evaluate free-form generations. Even keeping most of the parameters frozen, running the same pipeline as the images, and graphs can be significantly more expensive. Therefore we approximate formability by a local update of a LoRA adapter. In general we follow the full conformal approximation from Abad Martinez et al. [2], but this time over the LoRA adapter parameters, for the classification task.

Setup. We use six model–dataset combinations: Llama-1B, Llama-3B ([19]), Qwen-1.5B, and Qwen-3B (Qwen Team, 2024) on MMLU ([22]) humanities and business, and Llama-1B on MedQA ([26]) and MedMCQA ([35]). The pretrained language model is frozen, and a LoRA adapter is fitted on 2000 labeled training questions. The LoRA adapter is applied with rank r=8r = 8, and scaling α=16\alpha= 16, dropout 0, on query and value matrices of the attention layers. The same setup is also used for the ensemble model. From separate pools of 6000 calibration questions, and 800 evaluation questions, we vary the calibration set size from 50 to 2000, with 20 calibration draws. The score is the probability assigned to an answer letter. Every uncertainty measure ranks the same evaluation questions, using the misclassification AUROC, and selective E-AURC as mentioned in § 4. Softmax entropy, kNN, Mahalanobis and the ensemble are not computed from pθp_\theta. They come from a separate cached model fitted on the training questions plus one fixed set of calibration questions. Formability is however computed from damped influence steps that start at that same anchor adapter. The comparison is not entirely fair, but it is the best we can do given the expensive computational cost of the refits on LLMs.

Approximate refits. Let θ~n\tilde{\theta}_n denote the adapter fitted on the training questions, and updated by a single damped influence step over calibration points, and let θ~n,c\tilde{\theta}_{n,c} denote the adapter fitted on those same questions plus (xn+1,c)(x_{n+1}, c). These are the base, and refit whose difference defines formability. The pseudo-labeled test question is the only additional observation. We approximate the two fits by damped influence steps in the adapter parameters ([28, 2]), rather than optimizing a new adapter for every candidate. In particular, the additional step from the base has the form

θ~n,c−θ~n≈−1N+1(H+λI)−1gc,(20)\tilde{\theta}_{n,c} - \tilde{\theta}_n \approx-\frac{1}{N+1}(H+\lambda I)^{-1}g_c, \tag*{(20)}

where gc=∇ℓ(xn+1,c;θref)g_c = \nabla\ell(x_{n+1}, c; \theta_{\mathrm{ref}}), and θref\theta_{\mathrm{ref}} is the adapter at which the gradients are cached. Here HH is the empirical Fisher of the existing training questions (note that a preconditioner containing the calibration questions, but not the test question would break their exchangeability). Writing NN for the combined size of the training, and calibration questions, and gi=∇θℓ(xi,yi;θref)g_i = \nabla_\theta\ell(x_i, y_i; \theta_{\mathrm{ref}}), we use

H=1ntrain∑i=1ntraingigiT.(21)H = \frac{1}{n_{\mathrm{train}}} \sum_{i=1}^{n_{\mathrm{train}}} g_i g_i^{\mathsf{T}}. \tag*{(21)}

Each gig_i uses the question’s observed label yiy_i. This differs from the generalized Gauss–Newton matrix (the model Fisher for softmax cross-entropy), which averages gi,cgi,cTg_{i,c}g_{i,c}^{\mathsf{T}} over all four labels with weights p(c∣xi)p(c \mid x_i). This observed-label outer product is not, in general, a reliable approximation to the model Fisher or the Hessian [30]. Cached gradients, and their bilinear products give the changes of the candidate probability, and the calibration scores without running every refit in the main sweep. We vary the λ\lambda independently of the calibration size (every calibration size is run at each λ\lambda). Fig. 18 shows the results at λ/ref=1.5\lambda/\mathrm{ref} = 1.5, with ref=tr⁡(H)/ntrain\mathrm{ref} = \operatorname{tr}(H)/n_{\mathrm{train}}. With p~n\tilde{p}_n, and p~n,c\tilde{p}_{n,c} denoting the predictions of the two approximate fits, we compute w^n,c=p~n(c∣xn+1)\hat{w}_{n,c} = \tilde{p}_n(c \mid x_{n+1}), Φ^n,c=p~n,c(c∣xn+1)−p~n(c∣xn+1)\hat{\Phi}_{n,c} = \tilde{p}_{n,c}(c \mid x_{n+1}) - \tilde{p}_n(c \mid x_{n+1}), and ∑cV^n,c=∑cw^n,cΦ^n,c\sum_c \hat{V}_{n,c} = \sum_c \hat{w}_{n,c}\hat{\Phi}_{n,c}. The same approximate refit changes the calibration scores, and hence the full-conformal threshold. This is a local approximation to adapter refitting, and does not update the frozen language-model weights.

Result. Similar to the image and graph experiments, the approximate formability shows a performance that is competitive with the best method and performs better than many other approaches. However, the performance of the method is not as strong as the softmax entropy, and in general the results are not at all comparable with the image baseline. Our intuition is that with all the compromises we made to make the experiments feasible (and the training to fit on a single GPU), it is very much possible that the refit approximation is not as accurate as it should be. Finding a more efficient way to compute the refit, or a better approximation is an interesting direction for future work. The results are shown in Figure 18.

Figure 18. Formability on four-option language-model tasks. Each column ranks errors on the same evaluation questions. [Top] Misclassification AUROC (higher is better). [Bottom] Selective E-AURC (lower is better).

References

  1. [2]Javier Abad Martinez, Umang Bhatt, Adrian Weller, and Giovanni Cherubin. Approximating full conformal prediction at scale via influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 6631–6639, 2023. doi: 10.1609/aaai.v37i6.25814. URL https://ojs.aaai.org/index.php/AAAI/article/view/25814.DOI
  2. [3]Anastasios N Angelopoulos, Rina Foygel Barber, and Stephen Bates. Theoretical foundations of conformal prediction. arXiv preprint arXiv:2411.11824, 2024. URL https://arxiv.org/abs/2411.11824.
  3. [4]Anastasios Nikolas Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I. Jordan. Uncertainty sets for image classifiers using conformal prediction. International Conference on Learning Representations, 2021. URL https://api.semanticscholar.org/CorpusID: 221995507.arxiv.org/abs/2009.14193
  4. [5]Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. Predictive inference with the jackknife+. The Annals of Statistics, 49(1):486–507, 2021. doi: 10.1214/20-AOS1965. URL https://arxiv.org/abs/1905.02928.
  5. [6]Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816–845, 2023. doi: 10.1214/23-AOS2276. URL https://arxiv.org/abs/2202.13415.
  6. [7]Umang Bhatt, Adrian Weller, and Giovanni Cherubin. Fast conformal classification using influence functions. In Proceedings of the Tenth Symposium on Conformal and Probabilistic Prediction and Applications, volume 152 of Proceedings of Machine Learning Research, pp. 303–305, 2021. URL https://proceedings.mlr.press/v152/bhatt21a.html.
  7. [8]Koby Bibas, Yaniv Fogel, and Meir Feder. Deep pNML: Predictive normalized maximum likelihood for deep neural networks. arXiv preprint arXiv:1904.12286, 2019. URL https://arxiv.org/abs/1904.12286.
  8. [9]Aleksandar Bojchevski and Stephan Günnemann. Deep Gaussian embedding of graphs: Unsupervised inductive learning via ranking. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=r1ZdkJ-0W.
  9. [10]Siu Lun Chau, Soroush H. Zargarbashi, Yusuf Sale, and Michele Caprio. Quantifying epistemic predictive uncertainty in conformal prediction. arXiv preprint arXiv:2602.01667, 2026. URL https://arxiv.org/abs/2602.01667. Preprint. Under review.
  10. [11]Alvaro H. C. Correia, Fabio Valerio Massoli, Christos Louizos, and Arash Behboodi. An information theoretic perspective on conformal prediction. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://papers.nips.cc/paper_files/paper/2024/hash/b6fa3ed9624c184bd73e435123bd576a-Abstract-Conference.html.arxiv.org/abs/2405.02140
  11. [12]Luben Miguel Cruz Cabezas, Vagner Silva Santos, Thiago Ramos, and Rafael Izbicki. Epistemic uncertainty in conformal scores: A unified approach. In Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, volume 286 of Proceedings of Machine Learning Research, pp. 443–470. PMLR, 2025. URL https://proceedings.mlr.press/v286/cruz-cabezas25a.html.arxiv.org/abs/2502.06995
  12. [13]Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux – effortless Bayesian deep learning. In Advances in Neural Information Processing Systems, volume 34, 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/hash/a7c9585703d275249f30a088cebba0ad-Abstract.html.arxiv.org/abs/2106.14806
  13. [14]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009.
  14. [15]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mustafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
  15. [16]Runa Eschenhagen, Erik Daxberger, Philipp Hennig, and Agustinus Kristiadi. Mixtures of Laplace approximations for improved post-hoc uncertainty in deep learning. Bayesian Deep Learning Workshop, NeurIPS, 2021. URL https://arxiv.org/abs/2111.03577.
  16. [17]Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050–1059. PMLR, 2016. URL https://proceedings.mlr.press/v48/gal16.html.arxiv.org/abs/1506.02142
  17. [18]Yonatan Geifman, Guy Uziel, and Ran El-Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=SJfb5jCqKm.arxiv.org/abs/1805.08206
  18. [19]Aaron Grattafiori et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783.
  19. [20]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016.arxiv.org/abs/1512.03385
  20. [21]Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=HkG4Ti9xl.
  21. [22]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ.
  22. [23]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022. URL https://api.semanticscholar.org/CorpusID:235458009. Preprint. Under review.arxiv.org/abs/2106.09685
  23. [25]Sebastián Jiménez, Mira Jürgens, and Willem Waegeman. Why machine learning models fail to fully capture epistemic uncertainty. arXiv preprint arXiv:2505.23506, 2025. URL https://arxiv.org/abs/2505.23506v1.
  24. [26]Di Jin, Eileen Pan, Nassim Oufattolle, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
  25. [27]Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=SJU4ayYgl.arxiv.org/abs/1609.02907
  26. [28]Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885–1894, 2017. URL https://proceedings.mlr.press/v70/koh17a.html.
  27. [29]Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
  28. [30]Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical Fisher approximation for natural gradient descent. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/hash/46a558d97954d0692411c861cf78ef79-Abstract.html.arxiv.org/abs/1905.12558
  29. [31]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://arxiv.org/abs/1612.01474.
  30. [32]Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111, 2018. doi: 10.1080/01621459.2017.1307116. URL https://arxiv.org/abs/1604.04173.
  31. [33]Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976–11986, 2022.
  32. [34]Radford M. Neal. Bayesian Learning for Neural Networks, volume 118 of Lecture Notes in Statistics. Springer, 1996. URL https://link.springer.com/book/10.1007/978-1-4612-0745-0.DOI
  33. [35]Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarsubbu. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pp. 248–260. PMLR, 2022.arxiv.org/abs/2203.14371
  34. [36]Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.04765, 2018. URL https://arxiv.org/abs/1803.04765.
  35. [38]Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. In Advances in Neural Information Processing Systems, volume 33, 2020. URL https://papers.nips.cc/paper/2020/hash/244edd7e85dc81602b7615cd705545f5-Abstract.html. Preprint. Under review.arxiv.org/abs/2006.02544
  36. [40]Yusuf Sale, Paul Hofman, Timo Löhr, Lisa Wimmer, Thomas Nagler, and Eyke Hüllermeier. Label-wise aleatoric and epistemic uncertainty quantification. In Negar Kiyavash and Joris M. Mooij (eds.), Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, volume 244 of Proceedings of Machine Learning Research, pp. 3159–3179. PMLR, 15–19 Jul 2024. URL https://proceedings.mlr.press/v244/sale24a.html.arxiv.org/abs/2406.02354
  37. [41]Kajetan Schweighofer, Lukas Aichberger, Mykyta Ielanskyi, Günter Klambauer, and Sepp Hochreiter. Quantification of uncertainty with adversarial models. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://papers.nips.cc/paper_files/paper/2023/hash/3e0b96206965f5f05b0b4550c0e73ff0–Abstract-Conference.html.arxiv.org/abs/2307.03217
  38. [42]Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting weekly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 804–814, 2022.arxiv.org/abs/2201.08371
  39. [43]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005. URL https://alrw.net/.
  40. [44]Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. In Advances in Neural Information Processing Systems, volume 33, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/322f62469c5e3c7dc3e58f5a4d1ea399–Abstract.html.arxiv.org/abs/2002.08791
  41. [45]Soroush H. Zargarbashi, Simone Antonelli, and Aleksandar Bojchevski. Conformal prediction sets for graph neural networks. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 12292–12318, 2023. URL https://proceedings.mlr.press/v202/h-zargarbashi23a.html.
  42. [46]Yuwei Zong and Yanwen Xu. Comparative analysis of conformal prediction: Split, full, and adaptive approaches for statistical and neural network models. Proceedings of the Design Society, 5:981–990, 2025. doi: 10.1017/pds.2025.10112. URL https://doi.org/10.1017/pds.2025.10112.

Paper details

vlnc:W_u6u6jx4t
Version
v1 · latest
Published
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents