WorksComputer Science
What "Form'' of Uncertainty Does Conformal Prediction Quantify?
Abstract
Conformal prediction (CP) is a post-hoc calibration method that returns prediction sets with distribution-free coverage guarantees, and the size of the output set is often interpreted as a notion of uncertainty. We show that the difference in set size between full and split CP is directly connected to Bayesian posterior variance, which is the epistemic uncertainty of the model. Essentially, this difference in set size is an aggregation of two opposite effects: (1) better model accuracy and therefore smaller set size achieved by training over extra calibration points, and (2) the "formability" of the model at test point towards every possible label. The latter effect vanishes with stronger models or more calibration data (as epistemic uncertainty is known for that behavior). It turns out that the model's formability itself (without the quantization effect of prediction sets, and through estimating the posterior variance) is a useful tool to quantify the epistemic uncertainty. In our empirical evaluation it performs on par with the state of the art in many setups.
Introduction
A model’s prediction can be uncertain, either since the data is hard to predict (aleatoric), or because the model is not confident in its prediction due to lack of information (epistemic). Conformal prediction (CP) is a widely used approach in risk-critical applications where quantifying total predictive uncertainty is crucial. Given any model , CP returns a prediction set for a test point that is guaranteed to include the true label with user-specified probability . This guarantee is distribution-free; namely, it holds regardless of the underlying data distribution, or the model family [3]. CP requires an exchangeable set of calibration points, and can be applied in two ways: (i) fine-tuning the model jointly on the calibration set and the test point for every possible pseudo-label (full CP), or (ii) only evaluating the model on the calibration points (split CP). Full CP is known to be more data-efficient, which is particularly appealing in cases with label scarcity [32] (e.g. semi-supervised graph node classification). The argument is that in these settings the data is not divided into separate training and calibration sets, which potentially results in better model performance, and smaller sets.
In contrast, Zong & Xu [46] show that full CP doesn’t always outperform split CP without specifying a definitive reason, leaving this question open: when and by how much full CP can be more efficient, and what can be concluded from their performance gap? It turns out (as shown in Figure 1, and Figure 2), the performance gap (set size difference) reduces with a larger dataset (calibration, or training), and a lower feature dimension. Intuitively, while in most cases a larger dataset (combination of training and calibration) improves the model’s performance, fine-tuning on the test point with a pseudo-label deforms the model towards that label. The strength of this deformation is governed by the calibration density, and the confidence from pretraining both as resisting factors. Strong deformation includes wrong pseudo-labels in the set, resulting in worse overall performance.
Figure 1. Comparison of full (blue, solid) vs. split (orange, dashed) CP, against the adaptation floor (see § 3), over and feature dimension , and for [Left] CoraML (transductive node-classification, GCN), and [Right] CIFAR-10. The approximation is with samples (see § 4).
Figure 2. Performance gap between and over a grid of training and calibration set sizes. This is an extension of the result shown in Figure 1.
In particular, we define formability at a test point, as the difference between the predictive probability of any label between two models: one fine-tuned on a calibration set, and the other on the union of the calibration set and the pseudo-labeled test point (on that label). Under Bayesian training, where the epistemic uncertainty is quantified as the disagreement among the plausible (under the posterior) parameters, we show that formability is directly related to the epistemic uncertainty of the predictive distribution at the test point; namely we bound the posterior variance, by a function of formability. This also shows the reason why by increasing calibration set size the difference between full and split CP disappears: the formability (epistemic uncertainty) is known to reduce with more data.
Background
Let be a pretrained model from the family specified by the parameters . We use for the predictive label distribution of at the input . Let be a holdout calibration set jointly exchangeable with the test point .
Split CP (Vovk et al. [43]) (SCP). With , define the threshold
and the prediction set . This set has a distribution-free coverage guarantee . Other scores are discussed in § C.1.6.
Full CP. We define the augmented dataset , for a pseudo-label . Full conformal prediction (Vovk et al. [43]) (FCP) defines the prediction set for , where is the predictive distribution of the model after fine-tuning on . This prediction set also comes with the same distribution-free guarantee. In both cases, the guarantee comes from the fact that the score are exchangeable for . The difference is that FCP fine-tunes a single model per test point, per label. The potential use case of this computationally expensive approach is when the labeled dataset is small, and the model can benefit from training on the entire dataset (training and calibration).
Bayesian deep learning. Instead of a point estimate (deterministic value for parameters), Bayesian deep learning (Neal [34]; ); Wilson & Izmailov [44]) defines a distribution over the model parameters. As a result, the predictive distribution is also defined as instead of a single . Training a Bayesian model is equivalent to finding the posterior parameter distribution given the dataset as evidence, and the prior over the parameters (e.g. for initialization). Under this definition, the epistemic uncertainty is quantified as the disagreement among parameters that are plausible under the posterior – measured as the variance (Sale et al. [40]). Notably, as Jiménez et al. [25] argue, one cannot quantify the aleatoric uncertainty unless the epistemic uncertainty is sufficiently low.
Basin decomposition. The support of a probability distribution (e.g. ) can be decomposed into disjoint basins of attraction with , and for . Each basin is a connected region of the parameter space that contains a local optimum of the loss function. In § 3.3, we discuss that the posterior predictive distribution can be decomposed into a mixture of distributions over each basin. A better approximation of the variance considers basins other than only the one containing the MAP as well.
Formability
Motivating Experiment
First we evaluate the claim that “in label scarcity, full CP outperforms split CP” by comparing full and split conformal prediction on two simple setups with images (CIFAR-10; Krizhevsky, [29]), and graphs (CoraML; Bojchevski & Gunnemann, [9]). It is already mentioned in a couple of works that in some cases full CP is not better than split CP in set size (e.g., [6, 46]). As shown in Figure 1, however, by increasing the calibration set size, or reducing the feature dimension (using PCA), the performance gap is reduced, until the two methods cross over. These two observations suggest that the performance gap is not governed by the amount of data alone, but by how strongly the model is already determined around the test point. Notably, the argument of “full CP leverages a better model” still serves as a lower bound on the set size, which full CP converges to, by increasing the size of the calibration set.
An intuitive explanation for such behaviour is that when the training and calibration data is scarce, the model can still be deformed towards any label at the test point. Therefore, it is very easy for every (true or false) label of to pull its own score above the threshold. To observe this mechanism in isolation, we show a synthesized setup where every score is visible on a 2-dim landscape (Figure 3, details are in § C.1.1). We compare a test point inside a dense calibration neighborhood to one in a sparse support. In all cases, fine-tuning on the pseudo-label deforms the predictive distribution towards the pseudo-label. However, for the one in the dense region, the calibration density resists this deformation, and the self-pull remains small. In the sparse region there are no proximate data to resist. Hence, every pseudo-label pulls its own score above the threshold easily. As a result, the wrong label enters the full set exactly where the model was free to deform. Therefore the performance gap in CPs is governed by the significance of the new test data point w.r.t. the model: how much the model can deform at that point (which is inversely related to the size of , and the strength of the pretrained model as a resisting factor). Namely, this deformation is connected to the epistemic uncertainty – a model with more knowledge (training, and calibration data) is harder to form towards wrong labels.

Figure 3. The formability by fine-tuning on the pseudo-label at the test point (star), in a dense [left pair], and a sparse [right pair] (calibration) neighborhood. Pairs are a correct, and a wrong label . Calibration points are colored by the shift of their true score (negligible at many points), and the bars report , , and from Eq. (2).
This direct connection between formability and epistemic uncertainty is particularly important when it compares with methods like softmax entropy. Naturally, the softmax entropy, when evaluated on a single predictor can not distinguish between the two types of uncertainty. Figure 4 shows that by replicating the setup from Figure 3 while moving the generative Gaussians close to each other. At the center, the epistemic uncertainty decrease, while aleatoric uncertainty increase. On a single label distribution however, it is presented as the entire time. Therefore the softmax entropy does not react to this change, while the formability variance does.

Figure 4. Softmax entropy vs formability variance. The setup is similar to Figure 3 (see § C.1.2). [Left] Data, and direction of movements. Circles show the mean and standard deviation (radius). We move from the center to between two blobs (alea , and epis ), and from the other side towards the center of the other blob (both ). [b] Uncertainty at center (star) when blobs move closer (epis , and alea ), only formability reacts to it. [c], and [d] show it when moving towards the center of one blob (both ), and between two blobs (epis , alea ).
Theory
Let be the margin of the label at in any of these predictive regimes: (i) split CP with the pretrained predictive distribution (call it ), (ii) full CP fine-tuned on the augmented set with predictive distribution (call it ), and (iii) CP fine-tuned only on the calibration set with predictive distribution (call it ). Note that (iii) is not a valid CP, as the calibration and test points are not exchangeable, but it represents the effect of “better model from more data”. In all cases, a label is in the prediction set if its margin is non-negative. We decompose the full margin and define three new variables , , and as
The above decomposition is trivial, but each of the introduced terms has an interpretation: captures the adaptation-driven efficiency which is the result of fine-tuning on the calibration set. It simulates the effect from which we expected full CP to be more efficient. and are the effects of training on the additional pseudo-labeled test point (we call the total the net formability). Adding to the training set pulls the model towards that label (increasing the margin), but at the same time, it affects the other calibration points, which can change the quantile threshold. Ultimately, the full and split CP set size is only a quantization of the margins; i.e.
Under Bayesian Learning. Discussing this decomposition under Bayesian learning has two advantages: (i) Bayesian learning comes with a well-defined notion of epistemic uncertainty; i.e. the variance of the posterior predictive distribution, and (ii) in the Bayesian framework, we can write fine-tunings as a sequence of Bayesian updates, which decouples the effect of the additional pseudo-label from the calibration data. Let be the prior parameter distribution centered around (initial or pretrained model), be the posterior distribution after observing the calibration set (we call it “calibration posterior”), and be the posterior distribution with as evidence (we call it “augmented posterior”). By sequential Bayesian updating (Lemma B.1), the posterior distribution can be computed using Bayes’ rule with as a single evidence, and as the prior. For ease of notation, let
and assume . In Bayesian learning, we define the formability on as
Lemma 3.1. The (per-label) variance of the calibration posterior predictive distribution at is proportional to the formability at :
therefore, .
As a result, for a finite label set, the total posterior variance (epistemic uncertainty) satisfies
The two terms and in Eq. (2) are governed by the same posterior on the pseudo-label. As both variance, and are positive, the formability is positive. This means that training on the pseudo-label necessarily increases its score. At the same time, it also moves every calibration score, and therefore the threshold. To study all of the above, we define the forming kernel as the covariance kernel over the posterior distribution:
and use the shorthands for the kernel between calibration points and the pseudo-label hypothesis, , and the calibration-posterior variance at the test point, (Lemma 3.1).
Theorem 3.2. Under Bayesian training, with scores , and , , the following statements hold: (i) the update in each calibration score is exactly and bounded in absolute value by . As a result, with denoting the closed interval , we have (ii - Decomposition):
Thm. 3.2 shows that under Bayesian training, the set size for full CP can be bounded using only , and without recomputing over every label . Figure 5 evaluates this section exactly on a conjugate Bayesian linear regression (setup in § C.1.3), where all three posteriors are Gaussian and every term of Eq. 9 is a closed-form moment: the identities of Lemma 3.1 and Thm. 3.2(i) hold to machine precision, the full margin computed from the actual augmented posterior stays inside the band for every candidate, and the band is tight exactly where the calibration neighborhood is dense.

Figure 5. [a - e] Thm. 3.2 for Bayesian linear regression: [a] data, training covariate, and calibration points; [b] split and full sets along ; [c,d] the margins from Thm. 3.2 in dense and sparse area; [e] posterior variance, and its upper bound from Eq. 6. Details are in § C.1.3. [f-h] Illustration of Prop. 3.3, and Prop. 3.4: [f] same Laplace approximation on two different distributions: (solid), and a second distribution (dashed); [g] the variance difference by increasing the impact of . [h] The refit can jump to a different basin, i.e. can be outside the basin of the original MAP .
Consistent Bayesian learner. By increasing (calibration set size) the posterior concentrates. Lemma 3.1, and Thm. 3.2 tie formability, and quantile change to exactly the disagreement between plausible parameters from . The concentration shrinks and and the full margin converges to the calibration-posterior margin which is also observed in Figure 1.
Deterministic Fine-Tuning
Thm. 3.2, and Lemma 3.1 are written for Bayesian learning (given posterior ), which is intractable to compute for deep models. A common regime for deterministic fine-tuning finds the maximizer of the (log-) posterior, i.e., , for ; e.g. the common -regularized objective assigns a Gaussian prior . Daxberger et al. [13] approximate with a Gaussian distribution centered at the MAP , and covariance given by the inverse Hessian of the log-posterior; i.e. , which approximates the variance of the predictive distribution as . This approximation with a Gaussian distribution is the result of a second-order Taylor expansion of around . By construction, a shortcoming is that this approximation is always uni-modal (Gaussian), regardless of the shape of the true posterior (which can be multi-modal, skewed, etc.). In other words, there are dissimilar distributions (with different variances) that are all approximated by the same Gaussian distribution under this approach. Aware of this approximation scheme, Prop. 3.3 designs such dissimilar distributions.
Proposition 3.3. Let be twice differentiable at , and be an open neighborhood of . Define . Let be a probability density that vanishes on , and for satisfy , and set . Then:
(i) Every maximizer of lies in and is a maximizer of . In particular maximizes , and .
(ii) With and ,
Part (i) shows that the Laplace approximation of the different distributions remains the same while (ii) the variance of the predictive distribution can differ. This is well known, and it is what motivates mixing several Laplace fits [16], ensembling across basins [44], or searching directly for plausible parameters that disagree about the prediction [41]. Basically, the issue stems from the fact that the Bayesian posterior distribution can have multiple basins, and basins other than the one with can contribute meaningfully to the predictive variance. Formability retrains the model over each candidate label, meaning that it can jump to the basin that is more likely to predict the candidate label. The following proposition shows that the refit can result in a maximum a posteriori estimate outside of the basin of the original MAP.
Proposition 3.4. Let be a basin containing , and suppose that for some and , and some outside of the basin containing the MAP estimate, we have
Then, if , the augmented MAP . In either case .
Experiments
With our experiments we aim to show that (i) the performance gap between full and split conformal prediction is governed by the size of the calibration set, training set, and the dimension of the feature space; and that (ii) it stems from the model’s formability which is directly connected to epistemic uncertainty (synthetic experiments). (iii) The formability itself can serve as a tool to quantify epistemic uncertainty, through the variance as computed in Eq. 6, . We compare formability against several existing uncertainty quantification methods: softmax entropy [21], deep ensembles [31], MC-dropout [17], Laplace approximation [13], and DeepKNN [36]. The closest methods to formability are pNML regret [8], and the set size gap, against which we specifically compare, together with the best of the aforementioned methods in Figure 7. Reporting against the best does not mean that there is a single best method at every level, but rather every method has its own sweet spot. Overall relative performance is reported in Figure 6, and detailed reports are in § C.
Figure 6. Formability against families of uncertainty measures, for in-distribution data, on graphs, images and language. For each modality the relative performance is gathered from different datasets and models. Each family of methods includes several variations (e.g. Laplace is last-layer and all-parameter linearised, last-layer sampled; self-influence). A detailed comparison is in Figure 13.
Figure 7. Comparison of formability, regret and conformal set size gap over image benchmarks.
For language models, refitting together over a calibration set and test point for each pseudo-label becomes unrealistically expensive. Therefore we approximate the refit with a single step of LoRA [23] fine-tuning. Our updates are done through damped curvature-informed steps, and for calibration points we cache the gradients. We compare methods on six four-option question-answering setups (MMLU, MedQA on small Qwen, and Llama models). The specifics of the setup are in § C.2. Note that the language modeling setup is defined in a very simplistic way. The model is simply asked to choose one of the A–D answers and the score is the softmax over those specific tokens (see § C.2). The model updates are over a LoRA that is applied to the query and value matrices of the attention layers. These are specifically chosen for the computation to fit on a single GPU. The results are also not comparable to the image and graph setups. Defining a clear formability signal for language models is an open question, and we leave it for future work.
Throughout this paper we assumed the TPS score function (Sadinle et al., 2019) , and unless otherwise specified. While CP is commonly phrased with non-conformity (disagreement) scores, we use agreement as it simplifies the derivations, and interpretation. Setups are equivalent through a change in sign. For uncertainty quantification, we use two main metrics: (i) the misclassification AUROC [21] which is the area under the curve of true positive rate vs false positive rate over rejection according to the uncertainty score function; 0.5 is chance and 1.0 is the perfect score. (ii) The selective E-AURC [18] is the difference in AURC between the uncertainty score, and the ideal uncertainty score (the oracle). In this score lower is better, and 0 is the best score.
Full-vs-Split CP. In Figure 1 we compare full and split conformal prediction on node and image-classification tasks. For dimensionality reduction, we apply PCA on input features. Specifically on CIFAR-10, the features are the embeddings from an ImageNet-pretrained ResNet-18 [20] backbone, (training is done on the two-layer MLP as classification head). Since full CP is computationally expensive to examine, we report these results on a random subset of the test set containing 300 ∼ 400 instances. Prior to observing test examples, we can approximate the full CP set size by a small (e.g. ) number of calibration points over leave-one-out refits (keeping points as calibration set and the left out point as test set). We run ablation study and show in Figure 2, that by increasing calibration size, training size, or decreasing the dimension of the feature space, the full CP outperforms split CP, and it converges to the adaptation lower bound (consistent with Thm. 3.2). With a larger training set, the effect of calibration set shows a diminishing return.
Synthetic experiment. In Figure 3 we visualize the concept discussed in § 3. If a point is far from the calibration set it can deform the model more easily, its score can increase more, and full conformal set size would also inflate. It also shows the three variables defined in Eq. (2). To synthesize the data, we sample three classes from overlapping Gaussians, and use a multinomial logistic regression over fixed RBF features as the model. The pretrained model is trained on only 3 points per class, and every fine-tuning minimizes -regularized cross-entropy towards (i.e. the MAP objective with the prior ). We fine-tune once on the calibration set () to obtain , and once per candidate label on for . The complete anatomy for every candidate label is in § C.1.1. An important extension of this experiment is shown in Figure 4, where we move the calibration set generators towards the center which reduces epistemic uncertainty, and increases aleatoric uncertainty at the center. Here the softmax entropy remains steady, while the formability variance decreases. This shows that formability directly distinguishes the epistemic uncertainty, while the softmax entropy is agnostic to this decomposition.
In Figure 5 we evaluate Thm. 3.2 in a setup where every posterior can be computed exactly. We sample from , with Gaussian noise of standard deviation 0.5, and use Bayesian linear regression over fixed RBF features. The model is pretrained on 40 points in , while the 60 calibration points concentrate around . The full sets are narrower than split, especially where calibration data is dense. In sparse regions, the posterior variance and formability are larger, and the full set expands relative to the model fitted on calibration data alone. The full margins remain within the theoretical bounds at both test points. Details are in § C.1.3.
Formability as a quantification of epistemic uncertainty. While refitting on every label as an uncertainty quantification is not an entirely new concept [8] define regret based on refitting to every candidate label), we connect the refit through formability to Bayesian posterior variance, and the uncertainty captured in full conformal prediction. While both we and Bibas et al. [8] take information from models fitted on pseudo-labels, they take the logarithm-sum of the candidate probabilities, namely, , while we directly estimate the posterior variance . This is derived directly from CP’s uncertainty. Interestingly, as shown in Figure 7, this minor change in the definition can result in a different evaluation on the data. Formability performs significantly better on in-domain error detection, while the regret-based uncertainty performs better on out-of-distribution data. We compare all methods briefly in Figure 6. The CP set size gap (full - split) itself is not a good uncertainty quantification; mainly because it quantizes all the information captured in Eq. 6, which is a continuous value, into a binary decision per label.
Conclusion
We investigated the performance gap between full, and split conformal prediction. We showed that the gap is the resultant of two opposite effects: adapting the model on the calibration set (more data resulting in better model, and hence smaller sets), and forming it towards each pseudo-label at the test point. Under Bayesian learning, the second effect (formability) is directly tied to posterior predictive variance, and fades as the calibration posterior concentrates. This explains why full CP can become more efficient when the model is already well determined around the test point, and in sparse regions it performs poorly. However, this gap suggest to use formability as a measure of epistemic uncertainty that carries much more information compared to set size (due to the binarization). Our experiments on images and graphs show its value for detecting prediction errors; and to some degree, ood detection. In many benchmarks, formability is on par with the best of the existing uncertainty quantification methods, and in some cases it outperforms them.
Limitations. The most important limitation is the computational cost of refitting the model on every candidate label. Additionally our analysis, and definition of formability is based on the softmax score function. CP itself is agnostic to the choice of score function, and scores like APS also can lead to meaningful interpretations of the full-vs-split gap. We leave this for future work. Additionally, conformal prediction assumes exchangeability of the data, which is not always satisfied in practice, specifically for uncertainty quantification use cases. Formability implicitly assume that refits are able to reach the MAP estimate. This could challenge the quality in overparameterized models, or in wrong early stopping scenarios. Potentially our language model experiments are affected by this.
AI USE STATEMENT
We used generative AI tools for implementing the experiments (we designed the experimental setup ourselves and asked AI to implement and run them. We also inspected the results for potential issues). We also used AI for rephrasing parts of the paper, from grammar and choice of words, to better presenting the proofs. The proofs are ultimately rewritten by the authors, but AI was used to assist in the process. The research idea was originally proposed by the authors, including the main idea, theory, and the primary design of the experiments.
ETHICS STATEMENT
This work does not raise any ethical concerns, as it is a general study, and a theoretical contribution to uncertainty quantification. The experiments are conducted on publicly available datasets, and the models are trained on these datasets. The authors have no conflicts of interest to disclose.
REPRODUCIBILITY STATEMENT
All datasets, and models used in the experiments are publicly available. The code for reproducing the experiments is publicly available at https://github.com/soroushzargar/Formability.
REFERENCES
Related Works
Conformal prediction. CP [43, 3] is considered a distribution-free uncertainty quantification technique, where the prediction sets are guaranteed to cover the true label with high probability. With that, many consider the size of the prediction set to represent how uncertain the model is on a specific input. While the method works with any model, and any score function, there are many works designing various scores to improve the set size efficiency, adaptivity, and approximation of conditional coverage [38, 4, 45]. We use the basic threshold prediction sets (Sadinle et al., 2019) since it directly connects to the predictive distribution of the model which can connect perfectly with our decomposition in § 3, and through which we bound the epistemic uncertainty. In § C.1.6 we repeat our experiments under an adaptive (rank-based) score, where the full–split gap is wider, and never closes. While full and split conformal prediction are two endpoints of a computational cost spectrum, there are other variants such as Jackknife+ [5] in between. There are techniques that approximate full CP with lower computational cost [7, 2], through influence functions [28]. Both directions are outside of our focus, as we address the fundamental difference between full and split CP.
CP and epistemic uncertainty. Javanmardi et al. (2026); Cruz Cabezas et al. [12] incorporate a second-order distribution (which contains explicit epistemic uncertainty) into conformal prediction. In the reverse direction, Chau et al. [10]; Correia et al. [11] provide interpretations of split conformal prediction in uncertainty quantification; one through connecting split CP to imprecise integral probability metrics, and the other through information theory. In contrast, we provide a direct connection between the difference of full and split CP set size and the epistemic uncertainty, by bounding the posterior variance of the predictive distribution.
Our work connects three arguments: (i) When does full CP outperform split CP? Inconsistencies in comparison are mentioned by Barber et al. [6] and extensively studied by Zong & Xu [46]. However, neither work discusses what the underlying factor is that governs this difference. We show that the unexpected inefficiency in full CP set size stems from the formability of the model at the test point. (ii) We show that, in a Bayesian framework, the formability is directly related to the posterior variance (epistemic uncertainty). The concept of deterministic formability itself is not entirely new. Bibas et al. [8] define a learnability measure (regret) as the log sum of the predictive probabilities normalized by the probabilities from . However, its connection with Bayesian posterior variance, comparison with deterministic Laplace approximation through basin decomposition [13], or its influence on full conformal prediction do not follow automatically. (iii) We show that on larger models and various tasks this notion of formability still can be used as a practical tool to quantify epistemic uncertainty. Our work also connects with many other uncertainty quantification approaches. While the posterior distribution of parameters is a canonical way to quantify epistemic uncertainty, on larger models it is computationally near-impossible to compute. Laplace Redux [13] approximates this distribution as a Gaussian around the MAP estimate. However, as we show in Prop. 3.3 it is agnostic to other basins of attraction, and therefore miscalculates the epistemic uncertainty where other highly plausible basins disagree with the current predictive distribution. A remedy to that is MC-dropout [17], and ensemble training [31], which are parameter centric approximations. Formability however is a data-centric approximation. On computation, the comparison depends on whether multiple model inference or test-time training is more favorable. Schweighofer et al. [41] instead searches for adversarial models with high disagreement to the baseline, while attaining high posterior plausibility. In comparison, formability is a local measure, searching for such models through retraining. The comparison results as shown in Figure 17 are very close. Our ultimate goal is also not to come up with a new notion of uncertainty, but to showcase what is the underlying uncertainty quantification in conformal prediction.
Figure 17. Cost against quality, one marker per measure, one column per dataset. [Horizontal] Wall-clock seconds per evaluation point (log), timed with CUDA synchronization. The softmax controls are a forward pass that is already made, and are drawn hollow at the left edge. [Vertical] Misclassification AUROC [Top, higher is better], and selective E-AURC [Bottom, lower is better]. Means over calibration splits, bars ±1 standard error on both axes. QUAM (Schweighofer et al., [41]) is drawn at two budgets – the two most probable classes attacked for 50 steps, and every class for 100 – joined by a line from the cheap budget to the full one.
Supplementary to Theory
We first prove the sequential updating property used in § 3.2.
Lemma B.1 (Sequential Bayesian updating). *Let be the posterior from the prior and evidence , and be the posterior from the prior and evidence . For a fixed , assume its
label to be independent of given , and (recalling Eq. (3)). Then equals the posterior computed from the prior on the single observation ; i.e.,
Proof. For two sets of evidence and , Bayes’ rule with the joint evidence gives
Bayes’ rule on alone yields , and the law of total probability expands . Substituting both,
This means that the posterior after observing can be used as the prior when conditioning on , and it results in the same posterior as conditioning jointly on both sets of evidence.
In our case, and , therefore and . With fixed, we have , and the denominator of Eq. (12) becomes , which is positive by assumption.
Lemma 3.1. The (per-label) variance of the calibration posterior predictive distribution at is proportional to the formability at :
and
As a result, for a finite label set, the total posterior variance (epistemic uncertainty) satisfies
Proof. Using Eq. (11),
Substituting it into Eq. (4) yields
Multiplying both sides by results in Eq. (5).
For the remainder of the proof, we will use the following rule: for any we have
As , its posterior variance is bounded by
Replacing the variance in Eq. (5) with this inequality gives , and hence . Therefore,
For a finite number of labels, the expected squared Euclidean norm is
which proves Eq. 6.
Proof of Thm. 3.2
Before proving Thm. 3.2, we first prove two lemmas: (i) Lemma B.2 shows how conditioning on the pseudo-labeled test point moves the posterior expectation w.r.t. the expectation under the prior. In our case the posterior is the augmented posterior, and the prior is the calibration posterior. (ii) Lemma B.3 records the stability of the conformal quantile under coordinatewise shifts of the scores. The proof relies on two auxiliary lemmas: the first shows how conditioning on the pseudo-labeled test point moves any posterior expectation, and the second records the stability of the conformal quantile under coordinatewise shifts of the scores.
Lemma B.2. For any function with ,
Proof. Using Eq. (11),
By the definition of covariance,
and . Dividing by completes the proof.
Note that Eq. (5) in Lemma 3.1 is the special case , since the covariance of a variable with itself is its variance.
Lemma B.3 (Quantile stability under score shifts). The map defined in Eq. (1) is non-decreasing in every coordinate, and translation equivariant whenever the quantile is finite, i.e., for every . For any shift vector ,
Proof. is the -quantile of the empirical distribution over the multiset , and hence returns the -th smallest element of this multiset, where the rank is fixed by and (with when ; we assume and are such that the quantile is finite). Increasing any coordinate cannot decrease any order statistic, which gives monotonicity, and adding the same to every score adds to every finite order statistic, which gives translation equivariance. For the sandwich, note that coordinatewise . Applying monotonicity and then translation equivariance to both sides yields Eq. (15).
Theorem 3.2. Under Bayesian training, with scores , and , , the following statements hold: (i) the update in each calibration score is exactly and bounded in absolute value by . As a result, with denoting the closed interval , we have (ii - Decomposition):
Proof. Part (i). Under Bayesian training, the calibration scores are posterior predictive probabilities: and . Applying Lemma B.2 with gives
The bound is the Cauchy–Schwarz inequality for the covariance.
Part (ii). We first bound the threshold drift. By definition, with and . Defining as the difference in the scores after the update, we have with , so Lemma B.3 gives
hence
where the last step is the Cauchy–Schwarz bound of part (i).
Next, Eq. (2) is an algebraic identity that holds for any three fits: . Under Bayesian refitting where the formability is defined as Eq. (4), , we write (given ). Therefore, . By Eq. (18), we have
which proves Eq. 9.
Proof of Prop. 3.3
Proposition 3.3. Let be twice differentiable at , and be an open neighborhood of . Define . Let be a probability density that vanishes on , and for satisfy , and set . Then:
(i) Every maximizer of U$ and is a maximizer of . In particular maximizes , and .
(ii) With and ,
Proof. (i) On , , so , which means that is maximized at the same points as in . Outside ,
Hence, the maximizers of and are the same in , and is one of them. Since on the open set (meaning that the distribution is a scaled version of in that region), the two log-densities have the same Hessian at , hence the same (the disappears when differentiating). The gradient depends only on the model and on . Therefore .
(ii) Write . Expectations are linear in the density, so
Subtracting the square of the first from the second,
where the last three terms equal . Subtracting from both sides results in Eq. 10, and the inequality follows from .
Proof of Prop. 3.4
Proposition 3.4. Let be a basin containing , and suppose that for some and , and some outside of the basin containing the MAP estimate, we have
Then, if , the augmented MAP . In either case .
Proof. Adding the candidate to the calibration set adds to the log posterior. Thus the augmented MAP globally maximizes . With , and , we have:
Since maximizes the calibration log posterior, . Subtracting from the preceding inequality therefore yields , or . This bound holds whether or not . If that condition holds, then , whereas every model in assigns probability at most . Hence .
Supplementary Results, and Experiments
Synthetic Experiments
Anatomy of the Split-to-Full CP
This experiment replicates the transition from split to full conformal prediction in a two-dimensional setup where every quantity of Eq. (2) is directly visible. Figure 3 in § 3.1 shows a compact presentation of the result (one correct, and one wrong candidate per test point). Figure 8 reports the complete anatomy for every candidate label at both test points, in addition to the score function at each stage.
Figure 8. Complete anatomy of the split-to-full margin.
Setup. We sample three classes from overlapping Gaussians (means on a triangle with pairwise distance 4, standard deviation 1.5). The model is a multinomial logistic regression over fixed RBF
features, so that every refit is an exact convex MAP problem. The pretrained model is trained on 3 points per class (with a prior centered at zero), and every fine-tuning minimizes the cross-entropy plus the L2 pull with ; i.e. the MAP objective under the prior . The calibration set has points, the score is the predictive probability, and . We fine-tune once on (the base model ), and once per candidate on (the full models ). We examine two test points that differ only in their local calibration density: one is placed inside the class-0 cluster, the other away from every calibration point.
Description for Figure 8. Rows iterate over the candidate labels . There are two blocks, one for the test point in the dense area (top), and the other (bottom) the sparse one. Columns from left to right are (i) the base predictive field together with its level set at the base threshold (dashed): the region inside the contour is where the label would enter the prediction set under the base model. (ii) The same field after fine-tuning on the pseudo-label, , with the level set at (solid). (iii) Deformation . The calibration markers are colored by the shift of their own score, from Thm. 3.2. The figure also shows the quantities , , and on one scale shared across all plots. (iv) Shows every score through the three regimes (split, base, full): thin lines are the calibration scores, the dashed line is the threshold, the shaded area is where the label is included in the set, and the bold line is the candidate itself, with filled markers indicating membership at that stage.
As expected, at the dense region the self-pull is resisted by the surrounding calibration points (and the training) with for the wrong candidates. In other words, the deformation is small, and sharply localized. The threshold also barely moves, and the two level sets remain relatively steady. Conversely, at the sparse test point where there is much less resistance from other points, fine-tuning on any pseudo-label bends the model freely (– for every candidate), no calibration score reacts (, hence ), and as a result, every candidate – the wrong ones included – flips into the full set. The takeaway is that the full CP loses exactly where the formability is large.
Aleatoric against Epistemic Uncertainty
Figure 4 in § 3.1 reads two uncertainty measures on the setup of § C.1.1 while the aleatoric part of the uncertainty is moved on purpose. The generating law is known, so the aleatoric part is known in closed form: with the class means (at pairwise distance ), and the common standard deviation 1.5, the Bayes posterior at is , and we report , with TV the total variation distance to the uniform law over the classes. It is 1 where the label is a coin flip among the three classes, where it is a coin flip between two of them, 0 where it is determined, and it depends neither on the model nor on the sample. The two measures are read from the model fine-tuned on : the normalized softmax entropy , and the formability variance of Eq. 6, with , and from one MAP refit per pseudo-label, exactly as in § C.1.1.
Setup. Model, pretraining (3 points per class), calibration (), prior, and are those of § C.1.1. The grid of RBF centers is the one of that experiment extended by the same spacing (to centers), so that it covers the wider configurations, and the longer paths below. The data are drawn once as standard normal offsets , and placed as , so that the sample moves continuously with . The means are the means of § C.1.1 scaled about their centroid to pairwise distance ( is the configuration of Figure 3), and the pretrained model, and are refitted at every . In Figure 4(b), runs from 8 to 0, and . In (c), and (d), , and for along a unit direction : in (c) towards the center of class 0, which the path reaches at ; in (d) towards the midpoint of classes 1 and 2, which it reaches at . The path of lies, up to 0.01, on the perpendicular bisector of and , so the Bayes posterior stays a two-way coin flip between the two classes along the rest of it. Every curve is the mean over 5 draws of the data, and the band is their range. The aleatoric reference does not depend on the draw.
Reading. In (b), the label at is a three-way coin flip at every (the centroid is, up to 0.02, equidistant from the three means), so the aleatoric reference is at least 0.976 throughout. The softmax entropy reads this: its mean stays between 0.82 and 0.92 (the range over the draws is 0.59–0.98). The labeled points, however, pile up around as the classes converge – the Gaussian kernel density of at (bandwidth 1.1) grows from 0.002 to 0.046 – and the formability variance reads this: it falls in every draw, from 0.276 to 0.061 on average ( from 0.94 to 0.20). The two measures move independently: the entropy is flat, while the formability variance falls by a factor of 4.5. In (c), the aleatoric reference falls from 0.994 to 0.079 at , and to 0.002 at . Both measures fall towards , the entropy from 0.92 to 0.32, and the formability variance from 0.166 to 0.024, as they must: with near one every plausible model agrees, and (Lemma 3.1) leaves no room for disagreement. Both reach their minimum at (0.22, and 0.012), where the labeled points start to thin out (the density is 0.018 there, 0.009 at , 0.003 at , and below from on), and then both rise: the formability variance to 0.37, since a model that no data resist is free to deform towards any label (; every refit lifts its pseudo-label to about 0.7), and the entropy to 0.99, since the RBF logits decay to the bias away from the data, and the softmax returns to the near-uniform bias. The entropy therefore reads the same value, its maximum, at the centroid and at the far end of the path, where the aleatoric reference is 0.994, and 0: the first reading is aleatoric, the second is entirely epistemic, and the entropy cannot tell the two apart. In (d), the aleatoric reference falls from 0.994 to 0.62 at the midpoint , and then settles at 0.50: the label remains a coin flip between classes 1, and 2 along the rest of the path. The entropy reads 0.74 at the midpoint, its minimum 0.69 at , and then climbs back to 0.99. The formability variance reads 0.12 at the midpoint, and rises to 0.37 as the density falls from 0.014 to 0. Between and the aleatoric part is constant, so the rise of the entropy from 0.69 to 0.99 is entirely epistemic; the entropy does not say so, and the formability variance does. Wherever the entropy is high, then, the formability variance separates the two cases: it is small where the entropy is high because the labels are mixed and the labeled points are many (the centroid at small , the neighborhood of ), and large where the entropy is high because nothing constrains the model (the centroid at , the far ends of both paths).
Exact Evaluation on Bayesian Linear Regression
The margin decomposition for full - split CP in § 3.2 is general. However, using the Bayesian learning regime we can show that is a function of epistemic uncertainty. Such a setup is unrealistic for large-scale deep learning, for which we discuss the same concept under deterministic fine-tuning in § 3.3. Meanwhile, the Bayesian setup is exactly realizable in a small Gaussian-linear model, where every quantity of § 3.2 can be evaluated in closed form. The synthetic dataset is drawn from plus noise: 40 pretraining points uniform on , and the calibration set is drawn from the mixture , therefore its density is high around and low elsewhere (). The model is with , , and is a bias plus 12 Gaussian bumps from centered on a grid over . The score is the likelihood , which takes values in . The bound of Lemma 3.1 therefore applies to it as it does to a class probability, and the candidates range over a grid on . The pretrained model is trained with prior , and .
Closed forms. For write and . The calibration posterior is with and , and is the same update with the pair added. This update is the refit that produces , and in Figure 5. The moments that enter Lemma 3.1 and Thm. 3.2 are, with ,
which are used to compute the scores and the candidate mass , the variances and , and the forming kernel of Eq. (7). Each identity of § 3.2 is then compared against the quantity it predicts, computed independently from the refit .
Conclusion. Our sanity check (which is also shown in Figure 5) succeeds as over all candidate–test pairs, the two independent computations are close up to the numerical instability; i.e.
, , throughout, and lies inside the band of Eq. 9 with a smallest slack of (inside the kernel band of Eq. (18), ). The formability is the dominant term wherever the neighborhood is sparse ( against at the sparse test point), which is why the full set inflates there and not at the dense point, where .
The Frozen Image Backbones
Every image experiment in this paper trains a small head on top of a frozen ImageNet ([14])-pretrained backbone, and the feature space from which the formability is defined is the output of this backbone (this especially matters for CIFAR-10 experiments with feature reduction). The ImageNet output layer is removed, the remaining network embeds each image once, and only the head is trained, or refitted. The backbone itself is never updated, which helps us to reduce the cost of per-candidate refit from the entire model to only the head. Table 1 lists the specifications of the ResNet ([20]), ViT ([15]), and ConvNeXt ([33]) backbones. ViT-H/14 uses the SWAG weights of Singh et al. [42].
Table 1: The frozen ImageNet-pretrained backbones over which the image experiments are run. Weights is the torchvision checkpoint. Original counts the parameters of the classifier as loaded, and fixed ones that remain once its ImageNet output layer is removed, which is the form in which we load it. is the dimension of the embedding the head sees, and it is also the dimension the PCA reduction of § C.1.5 starts from. The top-1 accuracy is the reference number reported for the same checkpoint on ImageNet-1k.
| Backbone | Weights | Original | Fixed | dim. | top-1 (%) |
| ResNet-18 | IMAGENET1K_V1 | 11.7M | 11.2M | 512 | 69.8 |
| ResNet-50 | IMAGENET1K_V2 | 25.6M | 23.5M | 2048 | 80.9 |
| ViT-B/16 | IMAGENET1K_V1 | 86.6M | 85.8M | 768 | 81.1 |
| ConvNeXt-B | IMAGENET1K_V1 | 88.6M | 87.6M | 1024 | 84.1 |
| ViT-H/14 | IMAGENET1K_SWAG_E2E_V1 | 633.5M | 632.2M | 1280 | 88.6 |
Table 1.
Full vs. Split CP over the Feature Dimension
Setup. We use a two-layer GCN ([27]) over the 2879-dimensional bag-of-words node features of CoraML, and a two-layer MLP over the 512-dimensional embedding of a frozen ImageNet-pretrained ResNet-18 for CIFAR-10. Full CP trains the model from initialization on the training set, the calibration set, and the pseudo-labeled test point. We reduce the features by PCA to for CoraML, and for CIFAR-10, and sweep , and . The score is the predictive probability of § 2, and . The adaptation floor is the size of the posterior conformal set i.e. split CP on the model trained on the training, and the calibration set together, which needs a single fit per calibration set, and no per-candidate refit.
Results. In Figure 9, and Figure 10 the plots go from the full feature space to the smallest . The top row reports the average set size against , where the hollow markers are the approximate full CP with , and the bottom row reports the empirical coverage of the same sets against the nominal . Full, and split CP are at the nominal level in every panel which shows the sanity of the result. The adaptation floor is not (as expected): it under-covers by a wide margin in the full feature space ( on CoraML at , and on CIFAR-10 at ), and it approaches as grows, or shrinks. This is expected, since scores the calibration points it was trained on, and the amount of this optimism is again a formability – now read at the calibration points.
Figure 9. Full (blue, solid) vs. split (orange, dashed) CP on CoraML (transductive node-classification, GCN) over the calibration set size , with the 2879-d bag-of-words features [Left], and their PCA reduction to , 16, and 8 [Middle Left to Right]. [Top] Average set size against the adaptation floor (dotted). Hollow markers are the approximate full CP with . [Bottom] Empirical coverage of the same sets against the nominal (black, dashed). Bands are standard deviation across calibration splits.
Figure 10. Full (blue, solid) vs. split (orange, dashed) CP on CIFAR-10 (two-layer MLP over a frozen ResNet-18) over the calibration set size , with the 512-d embedding [Left], and its PCA reduction to , 32, and 16 [Middle Left to Right]. [Top] Average set size against the adaptation floor (dotted). Hollow markers are the approximate full CP with . [Bottom] Empirical coverage of the same sets against the nominal (black, dashed). Bands are standard deviation across calibration splits.
Similarly in set size, we observe that by increasing the calibration points, the difference between the two approaches decreases, until full CP crosses the split CP. Notably this effect is more visible in CIFAR-10 than in CoraML. Mainly this is since the training size of CIFAR-10 is larger than that of CoraML. We also observe that reducing the feature dimension closes the performance gap.
An interesting side observation is that the question of whether the full CP is more efficient than the split CP can be answered without the need to compute the full CP. One can simply estimate it with a small sample rate () by computing a leave-one-out refit on the calibration set, and that point alone. This is a relatively cheap operation to check the performance gap, without the need for expensive computation. The method turns out to be a good approximation as shown in both figures (the hollow markers are the approximate full CP with ).
We report the extended experiment on the gap over a grid of , and , on CoraML (2879-), and on CIFAR-10 (the 512- embedding). The results are shown in Figure 11. This is also same as our expectation. Another takeaway from this result is that the training set size also decreases the performance, and as the training size increases, the effect of the calibration set size becomes less significant.
Figure 11. The gap in average set size over the training set size , and the calibration set size (positive: split CP is smaller, negative: full CP is smaller), on CoraML with 2879-d features [Top], and CIFAR-10 with the full 512-d embedding [Bottom]. [Left] The gap over the grid. [Middle] The gap against , one line per . Hollow markers are the CoraML extension arm up to . [Right] The gap against , one line per . In the line panels, the shade encodes the size of the other set (light to dark = small to large), and bands are standard error of the paired difference across calibration splits. .
Adaptive Prediction Sets
In this work we focus on threshold prediction sets as the score function ((Sadinle et al., 2019)): it directly sets the conformity score to the predictive probability of the candidate label; i.e. . That object is very easy to work with and it directly connects to the decomposition of Eq. (2) to the updates of the predictive distribution after refits. However, conformal prediction is agnostic to the choice of score, and the same margin-decomposition can apply on other score functions as well. It is an important question to ask what interpretation does the full-split gap (and in particular the margin gap) has under that different score.
We repeat the experiment shown in Figure 10, this time under the adaptive prediction sets (APS) score [38]. Result is shown in Figure 12. This ablation does not need to train a new model since we have already cached the refits from the previous experiment.
Figure 12. The grid of Figure 11 read under the randomized APS score [38], off the same refits: the gap in average set size over the training set size , and the calibration set size (positive: split CP is smaller), on CoraML with 2879-d features [Top], and CIFAR-10 with the full 512-d embedding [Bottom]. [Left] The gap over the grid. [Middle] The gap against , one line per . Hollow markers are the CoraML extension arm up to . [Right] The gap against , one line per . In the line panels, the shade encodes the size of the other set (light to dark = small to large), and bands are standard error of the paired difference across calibration splits. The color scale is centered at zero as in Figure 11, but it spans a range about two and a half times wider. No cell of either grid is negative. .
APS applies the same conformal prediction as shown in § 2, but instead on the following score function:
and we use the randomized variant, with one drawn per (point, rank) once per split, and reused by every refit. The score therefore remains a deterministic function of the predictive distribution, and the exchangeability argument is untouched. Everything else – the grid, the test points, the models, and – is as in Figure 11.
Result. Split CP is smaller in every cell of both grids, and by a wide margin, and by increasing the training points or calibration points the gap converges to zero. APS can be read as a fractional knapsack problem over the probability distribution: take the smallest number of labels (all with equal weight), until cumulative probability is captured. Then conformalization (computing the quantile) is an answer to: what cumulative in model’s predictive probability can capture the probability in the true distribution. The gap is therefore a measure of how much the model’s predictive distribution is misaligned with the true distribution, and the gap converges to zero as the model sees more labels and becomes better aligned with the true distribution. However deriving the same formability decomposition as in § 3.2 is not straightforward, specially since the score involves the ranking of the labels, and the refit can change this ranking. We leave this as an open question for future work.
Breakdown of Uncertainty Quantification
In Figure 13 we report the detailed comparison against other uncertainty measures. Formability is close to the best methods on graphs and images, while on language softmax entropy performs better. The full–split set-size gap performs poorly across all three modalities, as it quantizes the continuous formability signal into a binary signal.
Figure 13. Formability against families of uncertainty measures, for in-distribution data, on graphs, images and language. For each modality the relative performance is gathered from different datasets and models.
Figure 14 extends the image comparison of Figure 6, and Figure 7 across different backbones and distribution shifts. Formability stays close to the best uncertainty measures for ranking errors on clean images, while for shift detection the result depends on the type of shift and is weaker on Gaussian noise.
Figure 14. Formability against uncertainty measures across frozen image backbones and distribution shifts. [Top] Misclassification AUROC (higher is better) and selective E-AURC (lower is better, log scale) on clean CIFAR-10, CIFAR-100, ImageNet, and ImageNet-V2, with five backbones in each block. [Bottom] Shift-detection AUROC for CIFAR-10-C at three severities, Street View House Numbers (SVHN), CIFAR-100, held-out classes, and ImageNet-V2 against ImageNet validation images. The blue line is . Bars are ±1 standard error over splits. Other colors distinguish Laplace, ensemble and MC-dropout, QUAM, feature-based, and softmax measures.
Figure 15 further extends the image comparison of Figure 6 by grouping CIFAR–10 images according to human-label disagreement (which can be considered a proxy for aleatoric uncertainty). Formability remains close to the best methods consistently within these groups, while ranking the errors generally becomes harder as the disagreement increases.
Figure 15. Error ranking within bins of human-label disagreement on CIFAR-10H. The 1000 CIFAR-10 test images are grouped by the entropy of about 51 human labels per image: unanimous (), low (), medium (), and high (). “All” is the pooled result. Each column is a frozen backbone. [Top] Misclassification AUROC (higher is better). [Bottom] Selective E-AURC (lower is better, log scale). Blue: with standard error over splits. Other lines: last-layer Laplace, ensemble BALD, QUAM, Mahalanobis, and softmax entropy.
Ablation on training set size. In addition we also stress-test the measures against the number of labels the model has seen during training. Figure 16 shows this comparison across various training budgets.
Figure 16. The epistemic measures of § 4 against the number of labels the model has seen, (log), at the end of training (epoch 300), on CoraML with the 2879-d node features [Left], and CIFAR-10 with the 512-d embedding [Right]. The accuracy of the base model at each rung is printed along the top. [Top] Misclassification AUROC (higher is better; dotted: chance). [Bottom] Selective E-AURC (lower is better). Every measure ranks the same base model on the same 200 evaluation points of a split, with held fixed. Blue circles: from the refits. Purple diamonds: the linearized last-layer Laplace (solid), the all-parameter empirical-Fisher form (dashed), and the sampled last-layer Laplace (dotted). Grey squares: ensemble BALD (solid), and MC-dropout BALD (dashed). Green triangles: deep kNN. Orange: softmax entropy. Red crosses: the set-size gap . Lines are means over 8 calibration splits, bands standard error.
The models are the ones of § C.1.5. We vary the number of labels the model has seen, , from 50 to 1600 on CoRaML, and from 50 to 3000 on CIFAR–10 – with the calibration set held at . Note that here our main argument is not to show a powerful conformal prediction, but to show the quality of formability in capturing epistemic uncertainty. Therefore the small calibration size is justified for computational efficiency. Columns are the two datasets, with the accuracy of the base model at each rung along the top. [Top] The misclassification AUROC (higher is better; the dotted rule is chance). [Bottom] The selective E-AURC (lower is better). In all training budgets, formability remains among the best methods.
Cost against quality. The refits are the most expensive reading of the epistemic uncertainty by construction: one fine-tuning per pseudo-label, per test point. We evaluate the cost, and (AUROC, E-AURC) on different measures for five datasets: CoraML, CiteSeer, and PubMed with the two-layer GCN of § 4, and CIFAR-10, and ImageNet-100 (its validation images) with the two-layer MLP over the 512-d embedding of the frozen ResNet-18 (Table 1). The refits, the measures, and the two metrics are as in § 4. Every measure is timed per evaluation point with CUDA synchronization. The fit of the base model is charged to no one, since every measure needs it, and the softmax controls are a forward pass that is already made. The refits are fitted in a batch over the candidate labels, which is how full CPU is run throughout the paper. One at a time, they would cost one to two orders of magnitude more. QUAM is timed at two budgets: every class attacked for 100 steps, and the two most probable classes for 50.
Figure 17 shows the result: one column per dataset. The horizontal axis is the wall-clock seconds per evaluation point (log). The vertical axis is the misclassification AUROC [Top], and the selective E-AURC [Bottom]. Markers are means over the calibration, and bars are ±1 standard error on both axes. The two QUAM budgets are joined by a line, from the cheap one to the full one.
Formability on Language Models
For language model tasks, we use four-option question answering: each model assigns probabilities to the answer letters A–D, and the conformal set contains a subset of these letters. We do not evaluate free-form generations. Even keeping most of the parameters frozen, running the same pipeline as the images, and graphs can be significantly more expensive. Therefore we approximate formability by a local update of a LoRA adapter. In general we follow the full conformal approximation from Abad Martinez et al. [2], but this time over the LoRA adapter parameters, for the classification task.
Setup. We use six model–dataset combinations: Llama-1B, Llama-3B ([19]), Qwen-1.5B, and Qwen-3B (Qwen Team, 2024) on MMLU ([22]) humanities and business, and Llama-1B on MedQA ([26]) and MedMCQA ([35]). The pretrained language model is frozen, and a LoRA adapter is fitted on 2000 labeled training questions. The LoRA adapter is applied with rank , and scaling , dropout 0, on query and value matrices of the attention layers. The same setup is also used for the ensemble model. From separate pools of 6000 calibration questions, and 800 evaluation questions, we vary the calibration set size from 50 to 2000, with 20 calibration draws. The score is the probability assigned to an answer letter. Every uncertainty measure ranks the same evaluation questions, using the misclassification AUROC, and selective E-AURC as mentioned in § 4. Softmax entropy, kNN, Mahalanobis and the ensemble are not computed from . They come from a separate cached model fitted on the training questions plus one fixed set of calibration questions. Formability is however computed from damped influence steps that start at that same anchor adapter. The comparison is not entirely fair, but it is the best we can do given the expensive computational cost of the refits on LLMs.
Approximate refits. Let denote the adapter fitted on the training questions, and updated by a single damped influence step over calibration points, and let denote the adapter fitted on those same questions plus . These are the base, and refit whose difference defines formability. The pseudo-labeled test question is the only additional observation. We approximate the two fits by damped influence steps in the adapter parameters ([28, 2]), rather than optimizing a new adapter for every candidate. In particular, the additional step from the base has the form
where , and is the adapter at which the gradients are cached. Here is the empirical Fisher of the existing training questions (note that a preconditioner containing the calibration questions, but not the test question would break their exchangeability). Writing for the combined size of the training, and calibration questions, and , we use
Each uses the question’s observed label . This differs from the generalized Gauss–Newton matrix (the model Fisher for softmax cross-entropy), which averages over all four labels with weights . This observed-label outer product is not, in general, a reliable approximation to the model Fisher or the Hessian [30]. Cached gradients, and their bilinear products give the changes of the candidate probability, and the calibration scores without running every refit in the main sweep. We vary the independently of the calibration size (every calibration size is run at each ). Fig. 18 shows the results at , with . With , and denoting the predictions of the two approximate fits, we compute , , and . The same approximate refit changes the calibration scores, and hence the full-conformal threshold. This is a local approximation to adapter refitting, and does not update the frozen language-model weights.
Result. Similar to the image and graph experiments, the approximate formability shows a performance that is competitive with the best method and performs better than many other approaches. However, the performance of the method is not as strong as the softmax entropy, and in general the results are not at all comparable with the image baseline. Our intuition is that with all the compromises we made to make the experiments feasible (and the training to fit on a single GPU), it is very much possible that the refit approximation is not as accurate as it should be. Finding a more efficient way to compute the refit, or a better approximation is an interesting direction for future work. The results are shown in Figure 18.
Figure 18. Formability on four-option language-model tasks. Each column ranks errors on the same evaluation questions. [Top] Misclassification AUROC (higher is better). [Bottom] Selective E-AURC (lower is better).
References
- [2]Javier Abad Martinez, Umang Bhatt, Adrian Weller, and Giovanni Cherubin. Approximating full conformal prediction at scale via influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 6631–6639, 2023. doi: 10.1609/aaai.v37i6.25814. URL https://ojs.aaai.org/index.php/AAAI/article/view/25814.DOI
- [3]Anastasios N Angelopoulos, Rina Foygel Barber, and Stephen Bates. Theoretical foundations of conformal prediction. arXiv preprint arXiv:2411.11824, 2024. URL https://arxiv.org/abs/2411.11824.
- [4]Anastasios Nikolas Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I. Jordan. Uncertainty sets for image classifiers using conformal prediction. International Conference on Learning Representations, 2021. URL https://api.semanticscholar.org/CorpusID: 221995507.arxiv.org/abs/2009.14193
- [5]Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. Predictive inference with the jackknife+. The Annals of Statistics, 49(1):486–507, 2021. doi: 10.1214/20-AOS1965. URL https://arxiv.org/abs/1905.02928.
- [6]Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816–845, 2023. doi: 10.1214/23-AOS2276. URL https://arxiv.org/abs/2202.13415.
- [7]Umang Bhatt, Adrian Weller, and Giovanni Cherubin. Fast conformal classification using influence functions. In Proceedings of the Tenth Symposium on Conformal and Probabilistic Prediction and Applications, volume 152 of Proceedings of Machine Learning Research, pp. 303–305, 2021. URL https://proceedings.mlr.press/v152/bhatt21a.html.
- [8]Koby Bibas, Yaniv Fogel, and Meir Feder. Deep pNML: Predictive normalized maximum likelihood for deep neural networks. arXiv preprint arXiv:1904.12286, 2019. URL https://arxiv.org/abs/1904.12286.
- [9]Aleksandar Bojchevski and Stephan Günnemann. Deep Gaussian embedding of graphs: Unsupervised inductive learning via ranking. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=r1ZdkJ-0W.
- [10]Siu Lun Chau, Soroush H. Zargarbashi, Yusuf Sale, and Michele Caprio. Quantifying epistemic predictive uncertainty in conformal prediction. arXiv preprint arXiv:2602.01667, 2026. URL https://arxiv.org/abs/2602.01667. Preprint. Under review.
- [11]Alvaro H. C. Correia, Fabio Valerio Massoli, Christos Louizos, and Arash Behboodi. An information theoretic perspective on conformal prediction. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://papers.nips.cc/paper_files/paper/2024/hash/b6fa3ed9624c184bd73e435123bd576a-Abstract-Conference.html.arxiv.org/abs/2405.02140
- [12]Luben Miguel Cruz Cabezas, Vagner Silva Santos, Thiago Ramos, and Rafael Izbicki. Epistemic uncertainty in conformal scores: A unified approach. In Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, volume 286 of Proceedings of Machine Learning Research, pp. 443–470. PMLR, 2025. URL https://proceedings.mlr.press/v286/cruz-cabezas25a.html.arxiv.org/abs/2502.06995
- [13]Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux – effortless Bayesian deep learning. In Advances in Neural Information Processing Systems, volume 34, 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/hash/a7c9585703d275249f30a088cebba0ad-Abstract.html.arxiv.org/abs/2106.14806
- [14]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009.
- [15]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mustafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
- [16]Runa Eschenhagen, Erik Daxberger, Philipp Hennig, and Agustinus Kristiadi. Mixtures of Laplace approximations for improved post-hoc uncertainty in deep learning. Bayesian Deep Learning Workshop, NeurIPS, 2021. URL https://arxiv.org/abs/2111.03577.
- [17]Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050–1059. PMLR, 2016. URL https://proceedings.mlr.press/v48/gal16.html.arxiv.org/abs/1506.02142
- [18]Yonatan Geifman, Guy Uziel, and Ran El-Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=SJfb5jCqKm.arxiv.org/abs/1805.08206
- [19]Aaron Grattafiori et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783.
- [20]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016.arxiv.org/abs/1512.03385
- [21]Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=HkG4Ti9xl.
- [22]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ.
- [23]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022. URL https://api.semanticscholar.org/CorpusID:235458009. Preprint. Under review.arxiv.org/abs/2106.09685
- [25]Sebastián Jiménez, Mira Jürgens, and Willem Waegeman. Why machine learning models fail to fully capture epistemic uncertainty. arXiv preprint arXiv:2505.23506, 2025. URL https://arxiv.org/abs/2505.23506v1.
- [26]Di Jin, Eileen Pan, Nassim Oufattolle, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
- [27]Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=SJU4ayYgl.arxiv.org/abs/1609.02907
- [28]Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885–1894, 2017. URL https://proceedings.mlr.press/v70/koh17a.html.
- [29]Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
- [30]Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical Fisher approximation for natural gradient descent. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/hash/46a558d97954d0692411c861cf78ef79-Abstract.html.arxiv.org/abs/1905.12558
- [31]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://arxiv.org/abs/1612.01474.
- [32]Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111, 2018. doi: 10.1080/01621459.2017.1307116. URL https://arxiv.org/abs/1604.04173.
- [33]Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976–11986, 2022.
- [34]Radford M. Neal. Bayesian Learning for Neural Networks, volume 118 of Lecture Notes in Statistics. Springer, 1996. URL https://link.springer.com/book/10.1007/978-1-4612-0745-0.DOI
- [35]Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarsubbu. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pp. 248–260. PMLR, 2022.arxiv.org/abs/2203.14371
- [36]Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.04765, 2018. URL https://arxiv.org/abs/1803.04765.
- [38]Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. In Advances in Neural Information Processing Systems, volume 33, 2020. URL https://papers.nips.cc/paper/2020/hash/244edd7e85dc81602b7615cd705545f5-Abstract.html. Preprint. Under review.arxiv.org/abs/2006.02544
- [40]Yusuf Sale, Paul Hofman, Timo Löhr, Lisa Wimmer, Thomas Nagler, and Eyke Hüllermeier. Label-wise aleatoric and epistemic uncertainty quantification. In Negar Kiyavash and Joris M. Mooij (eds.), Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, volume 244 of Proceedings of Machine Learning Research, pp. 3159–3179. PMLR, 15–19 Jul 2024. URL https://proceedings.mlr.press/v244/sale24a.html.arxiv.org/abs/2406.02354
- [41]Kajetan Schweighofer, Lukas Aichberger, Mykyta Ielanskyi, Günter Klambauer, and Sepp Hochreiter. Quantification of uncertainty with adversarial models. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://papers.nips.cc/paper_files/paper/2023/hash/3e0b96206965f5f05b0b4550c0e73ff0–Abstract-Conference.html.arxiv.org/abs/2307.03217
- [42]Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting weekly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 804–814, 2022.arxiv.org/abs/2201.08371
- [43]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005. URL https://alrw.net/.
- [44]Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. In Advances in Neural Information Processing Systems, volume 33, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/322f62469c5e3c7dc3e58f5a4d1ea399–Abstract.html.arxiv.org/abs/2002.08791
- [45]Soroush H. Zargarbashi, Simone Antonelli, and Aleksandar Bojchevski. Conformal prediction sets for graph neural networks. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 12292–12318, 2023. URL https://proceedings.mlr.press/v202/h-zargarbashi23a.html.
- [46]Yuwei Zong and Yanwen Xu. Comparative analysis of conformal prediction: Split, full, and adaptive approaches for statistical and neural network models. Proceedings of the Design Society, 5:981–990, 2025. doi: 10.1017/pds.2025.10112. URL https://doi.org/10.1017/pds.2025.10112.














