betaFeedback

WorksComputer Science

ShiftGuard: Evaluating Confidence and Feature Distance for Risk-Controlled Image Classification

Abstract

A classifier that is usually right still makes confident mistakes, and a deployed system needs some way to decide when to answer and when to abstain. Cheap post-hoc scores such as maximum softmax probability (MSP) and predictive entropy are the standard answer. Distances in feature space are a popular alternative, and fusing several signals into one score is a natural next step. We ask whether adding a feature distance signal to MSP and entropy improves error detection, and whether it increases the share of predictions that can be accepted at a certified error rate when only a small labelled calibration set is available. We train a CIFAR-adapted ResNet-18 from scratch on CIFAR-10 (92.92% test accuracy, one seed) and compare five reliability scores: MSP, entropy, distance to the predicted-class centroid, a logistic fusion of MSP and entropy, and a logistic fusion of all three signals. Acceptance thresholds are chosen on a held out calibration set with Clopper Pearson upper bounds, Bonferroni corrected over a prespecified family of 315 rules at 95% simultaneous confidence. Confidence based scores were best. Error AUROC was 0.9230 for MSP, 0.9233 for entropy and 0.9235 for MSP plus entropy, against 0.8814 for the three signal fusion and 0.8224 for distance alone. At a 5% risk target with 1,000 calibration images, entropy accepted 84.82% of test images with 2.05% observed error, while three signal fusion accepted 80.61% with 2.21%. With 250 calibration images, neither distance nor the three-signal fusion could certify the 5% target at all. The results come from one seed, one dataset and one architecture, with no corruption experiments, so they say nothing about distribution shift detection. Within these limits, the distance signal did not earn its added complexity.

Introduction

Image classifiers are usually judged by accuracy, but accuracy hides a practical problem. A network that is right 93% of the time is wrong 7% of the time, and when it is wrong it is often very sure of itself [1]. A system built around such a classifier needs a second decision on top of the prediction: accept it, or hand the image to a person or a slower model. This is selective classification, and its roots go back to the reject option of Chow [2] and the formal treatment of El-Yaniv and Wiener [3]. Geifman and El-Yaniv [4] brought it to deep networks and showed how to choose a rejection threshold with a statistical guarantee on the error rate of the accepted predictions.

The cheapest way to build a rejection rule is to reuse something the classifier already computes. The maximum softmax probability is the usual baseline for spotting errors [5], and the entropy of the softmax distribution is a close relative. Both look only at the output layer. A different family of methods looks inside the network instead. These ask how far an input’s learned representation sits from the representations of examples the model handled correctly, with Trust Score [6], Mahalanobis distance [7] and nearest-neighbour distance [8] as well-known examples. The two families see different things. It is therefore tempting to combine them, for instance by fitting a small logistic regression on the signals. If the signals carry complementary information, the fused score should rank errors better and should let more predictions through at the same error rate.

Whether that actually happens, and whether it still happens when the labelled data available for calibration are scarce, is the question this paper asks. Scarcity matters because a certificate is only as strong as the data behind it. With a few hundred labelled calibration images, the statistical upper bound on accepted-set risk is wide, and a rule that looks good on the calibration data may not be probably good. A method with worse error ordering can lose certified coverage in this regime. All five scores receive the same per-rule confidence adjustment; the bound does not directly penalise the number of fitted logistic coefficients.

We built a small evaluation framework, which we call ShiftGuard, to answer it. A CIFAR-adapted ResNet-18 is trained from scratch, and a reliability layer on top produces five scores from the frozen classifier. Thresholds are chosen on separate calibration data and certified with finite-sample bounds, and everything is evaluated once on the official test set. The name reflects the original motivation of guarding a classifier against distribution shift. The experiments in this paper use clean, in-distribution data only, so the name should not be read as a claim of demonstrated shift detection. We return to this in Section 7.

Our research question is the following.

Does adding a feature-distance signal to confidence and entropy improve error detection, and does it increase coverage at a statistically controlled error rate, particularly when calibration data are limited?

Our contributions are these.

  1. A controlled comparison of five post-hoc reliability scores on a single frozen ResNet-18, with disjoint data for training, checkpoint selection, class-centroid estimation, fusion fitting, threshold calibration and testing.

  1. A risk-control audit that keeps three quantities apart: the target risk, the calibration upper bound used to certify a rule, and the error rate observed on the test set. It

covers two risk targets (5% and 10%) and three calibration budgets (250, 500 and 1,000 images), with one simultaneous confidence level over all 315 prespecified rules.

  1. An empirical answer that is negative for fusion. Confidence-based scores matched or beat both distance-based variants on every ranking metric and on certified coverage, and the three-signal fusion could not certify the 5% target with 250 calibration images and had substantially lower coverage with 500.

  1. An auditable saved-output package with an independent script for score reconstruction, certificate verification and paired bootstrap analysis.

We make no novelty claim for the statistical machinery. Confidence fusion is common, and the binomial bound with a Bonferroni correction is an established construction. The contribution is the empirical comparison.

The paper is organised as follows. Section 2 covers related work. Section 3 describes the scores, the fusion and the certification procedure. Section 4 gives the experimental setup, Section 5 the results, Section 6 a discussion, Section 7 the limitations and Section 8 the conclusion. The appendix holds complete certification tables, extra figures and exploratory earlier experiments.

Selective classification and failure prediction. The reject option was studied for pattern recognition by Chow [2], who characterised the trade-off between error and rejection rates. El-Yaniv and Wiener [3] gave a theoretical framework for selective classification with a risk-coverage curve. Geifman and El-Yaniv [4] applied it to deep networks, ranked inputs by softmax confidence, and proposed an algorithm that picks a threshold with a guaranteed bound on selective risk using a binomial confidence interval. The area under the risk-coverage curve (AURC) was used as a summary metric in later work on estimating and improving selective risk [9]. SelectiveNet [10] trains the abstention decision into the network. For ranking errors without a rejection threshold, Hendrycks and Gimpel [5] showed that MSP is a strong baseline for detecting misclassified inputs, and later work learns a separate confidence network for failure prediction [11]. Large comparisons of uncertainty estimators on image classifiers keep finding that simple confidence scores are hard to beat once classifiers are trained well [12], [13]. Our study is a small, in-distribution instance of that question with a distance signal added.

Feature-space and distance-based scores. Several methods score inputs by their position in representation space. Trust Score [6] compares the distance to the predicted class’s high-density region with the distance to the nearest other class. The Mahalanobis detector [7] fits class-conditional Gaussians with a shared covariance on penultimate features and scores inputs by distance to the nearest class mean, and nearest-neighbour methods use distances to training examples [8]. Energy scores [14] come from the logits and sit between the two families. These methods were mostly developed and evaluated for out-of-distribution detection.

Our distance score is deliberately simpler. It is the root-mean-square Euclidean distance, in per-dimension standardised coordinates, from an input’s feature vector to the centroid of its predicted class. It ignores correlations between feature dimensions and class multimodality, and it is not Mahalanobis distance or Trust Score. We chose it to isolate whether a basic geometric signal adds anything, and we do not claim it represents the best that distance-based methods can do.

Combining uncertainty signals. Ensembles [15] and Monte Carlo dropout [16] improve uncertainty by averaging multiple predictions, at the price of extra forward passes. Cheaper alternatives fit a small model on top of existing signals, which is the approach we take with logistic regression. The logistic combination optimises likelihood of the error labels, which is a different objective from ranking, and we will see in Section 6 that this difference may matter. Benchmarks of post-hoc selective-classification methods report that outcomes depend on the classifier and the signal [17], which is one reason we report the full comparison and not only the final fused score.

Calibration. Modern networks tend to be overconfident [1]. Expected calibration error (ECE) bins predictions by confidence and averages the gap between confidence and accuracy [18], and the Brier score measures squared probability error [19]. We report these for the classifier and also for the error probabilities that the fused scores produce, because ranking quality and probability calibration are different properties.

Statistical risk control. Distribution-free methods choose a decision rule so that a risk is bounded with high probability over the calibration sample. Risk-controlling prediction sets [20] and Learn then Test [21] treat rule selection as multiple hypothesis testing over a grid of candidate parameters. Conformal prediction is the best-known relative [22]. The version we use is the simplest member of this family. For each candidate threshold on a grid fixed in advance, we compute a one-sided Clopper-Pearson upper bound [23] on the accepted-set error rate and correct for the size of the grid with a Bonferroni adjustment [24]. This is the same idea as the guaranteed-risk selective classification of Geifman and El-Yaniv [4], extended over several methods and budgets.

Evaluation under shift. Predictive uncertainty can degrade sharply under dataset shift [25], and corrupted-image benchmarks exist for measuring robustness [26]. Even without deliberate corruption, a new test set drawn by a nominally identical process can shift accuracy [27]. These results are why we state the IID assumption behind our certificates explicitly, and why we do not extend the claims to shifted data.

Methodology

Problem setup

A trained classifier ff maps an image xx to logits z(x)∈R10z(x) \in\mathbb{R}^{10} and a penultimate feature vector h(x)∈R512h(x) \in\mathbb{R}^{512}. The predicted class is

y^(x)=arg max⁡k zk(x)(1)\hat{y}(x) = \underset{k}{\operatorname{arg\,max}}\ z_k(x) \tag*{(1)}

with softmax probabilities pk(x)=exp⁡(zk(x))∑jexp⁡(zj(x))p_k(x) = \frac{\exp(z_k(x))}{\sum_j \exp(z_j(x))}. Define e(x,y)=1{y^(x)≠y}e(x,y) = 1\{\hat{y}(x) \ne y\}. The classifier is frozen after training; reliability scores do not retrain it.

A reliability score s(x)s(x) is oriented so that higher means more likely to be wrong. A selective classifier accepts the prediction when s(x)≤τs(x) \le\tau for a threshold τ\tau and abstains otherwise. For a set of NN examples, the coverage of a threshold is the fraction accepted, and the selective risk is the error rate among accepted examples:

C^(τ)=N−1∑i=1N1{si≤τ},\widehat{C}(\tau) = N^{-1}\sum_{i=1}^{N}1\{s_i \le\tau\},
R^(τ)=∑i:si≤τei∑i1{si≤τ}.(2)\widehat{R}(\tau) = \frac{\sum_{i:s_i\le\tau}e_i}{\sum_i1\{s_i\le\tau\}}. \tag*{(2)}

When no example is accepted, R(τ)R(\tau) is undefined. We never replace it with zero.

The five reliability scores

MSP. The score is one minus the maximum softmax probability,

sMSP(x)=1−max⁡kpk(x).(3)s_{\mathrm{MSP}}(x) = 1 - \max_k p_k(x). \tag*{(3)}

so lower confidence gives a higher score.

Entropy. The score is the Shannon entropy of the softmax distribution in natural logarithms, with probabilities clipped at 10−1510^{-15} inside the logarithm for numerical stability,

sent(x)=−∑k=110pk(x)log⁡pk(x).(4)s_{\mathrm{ent}}(x) = -\sum_{k=1}^{10}p_k(x)\log p_k(x). \tag*{(4)}

Feature distance. Only correctly classified examples in an independent 1,000-image reference set are used. Per-dimension means μ\mu and standard deviations σ\sigma are estimated from their features, with SD floored at 0.001. Set u(x)=(h(x)−μ)/σu(x) = (h(x)-\mu)/\sigma, with elementwise division. The centroid ckc_k averages standardised features of correct reference examples of class kk. The score is the RMS distance to the predicted-class centroid:

sdist(x)=1512∥u(x)−cy^(x)∥22.(5)s_{\mathrm{dist}}(x) = \sqrt{\frac{1}{512}\lVert u(x)-c_{\hat{y}(x)}\rVert_2^2}. \tag*{(5)}

A class with fewer than two correct reference examples would fall back to the pooled correct-example centroid. This did not occur here, since every class had between 86 and 98 correct reference examples. We keep the reference set disjoint from fusion fitting to avoid optimistic in-sample distances. Independent calibration of a fixed score is essential for the certificate.

MSP + entropy. A logistic regression is fitted on the standardised pair (MSP, entropy), with the error indicator as the label. The fitted score is the predicted error probability.

All three signals. The same logistic regression is fitted on the standardised triple (MSP, entropy, distance).

For both fused scores we use a standard scaler followed by unweighted logistic regression with L2 regularisation strength C = 1, fixed in advance and not tuned. The estimator is fitted on a 1,000-image fusion-fitting set that is disjoint from the reference set, the calibration set and the test set. The fitted weights are stored. In the three-signal model the standardised coefficients were 0.767 (MSP), 1.558 (entropy) and 0.588 (distance), with intercept -3.146. In the two-signal model they were 0.836 and 1.804 with intercept -2.977.

Score convention audit.: Let q=max⁡kpkq = \max_k p_k. The standalone MSP error score is 1−q1-q, but the first fusion input is qq itself. This is verified by the saved input matrix and by reconstructing both fitted probabilities from the exported coefficients to numerical precision. For v=(q,H,d)v=(q,H,d) and fitting-set means aa and scales bb, the fitted probability is

P^(e=1∣v)=sigmoid⁡(β0+∑jβjvj−ajbj).(6)\widehat{P}(e=1\mid v)=\operatorname{sigmoid}\left(\beta_0+\sum_j\beta_j\frac{v_j-a_j}{b_j}\right). \tag*{(6)}

The two-signal model omits dd. Reparameterising the first input as 1−q1-q requires negating its coefficient and replacing its mean by 1−a11-a_1; it does not change predictions. The positive recorded coefficient on qq is conditional on the other signals and is not a standalone confidence-to-error relationship.

Error-detection and selective-prediction metrics

Error AUROC treats an incorrect prediction as the positive class and measures how well the score separates errors from correct predictions [28]. Ties in the score do not affect it beyond the standard rank treatment.

Accepting in increasing score order gives s(1)≤⋯≤s(N)s_{(1)} \le\cdots\le s_{(N)}. The prefix risk is Rk=k−1∑i=1ke(i)R_k = k^{-1}\sum_{i=1}^{k}e_{(i)}. We use the discrete area under the risk–coverage curve:

AURC⁡=1N∑k=1NRk(7)\operatorname{AURC}=\frac{1}{N}\sum_{k=1}^{N}R_k \tag*{(7)}

Lower is better. Groups of exactly tied scores use the expected prefix risk under a random ordering within the group, so that the result does not depend on the order in which tied examples happen to appear. We also report selective risk at fixed coverages of 50%, 70%, 80% and 90%, computed at ceil⁡(coverage×N)\operatorname{ceil}(\text{coverage}\times N) accepted examples.

Calibration of the classifier uses the negative log-likelihood of the true class (probabilities clipped to [10−15,1][10^{-15},1]), the multiclass Brier score, and ECE with 15 equal-width confidence bins. For the fused scores, which output error probabilities, we report the binary Brier score and a binary ECE on the same 15-bin scheme. These describe whether the numbers can be read as probabilities, which is separate from how well they rank.

Risk-controlled threshold selection

Ranking metrics do not choose a threshold. To do that with a guarantee, we use independent calibration data. The procedure has four steps. Step 1, candidate thresholds. For each method, the candidate thresholds are fixed before any calibration label is used. They are the unique values among 21 equally spaced quantiles (0, 0.05, ..., 1) of that method’s scores on the fusion-fitting set. The calibration and test sets play no part in forming the grid.

Step 2, bounds. For a calibration budget nn, the first nn calibration images are used. A threshold τ\tau accepts mm examples of which kk are errors. The one-sided Clopper-Pearson upper confidence bound on the accepted-set risk is the (1−δ′)(1-\delta') quantile of a Beta(k+1,m−k)(k+1,m-k) distribution,

U(k,m,δ′)=Beta⁡−1(1−δ′;k+1,m−k).(8)U(k,m,\delta') = \operatorname{Beta}^{-1}(1-\delta';k+1,m-k). \tag*{(8)}

and U=1U=1 when m=0m=0 or k=mk=m. The per-rule level is δ′=δ/∣F∣\delta'=\delta/|F|, where ∣F∣|F| is the number of rules in the prespecified family and δ=0.05\delta=0.05.

Step 3, family size. The family contains every combination of method, candidate threshold and calibration budget. With 5 methods, 21 candidate thresholds each and 3 budgets, ∣F∣=315|F|=315, giving δ′=0.05/315≈1.59×10−4\delta'=0.05/315\approx1.59\times10^{-4}. The two risk targets add no further tests, because the same upper-bound event is used for both.

Step 4, selection. For a method, budget and target α\alpha, a rule is eligible if it accepts at least 20 calibration examples and U≤αU\le\alpha. Among eligible rules, the one with the largest calibration acceptance count is chosen, with ties broken by the smaller upper bound and then the smaller threshold. If no rule is eligible, the method returns "not certified", accepts no test examples, and its conditional error rate is reported as undefined and never as zero.

The guarantee that follows is this. If the calibration and future clean examples are IID draws from one population, and the model, the scores and the candidate grids are fixed before calibration labels are looked at, then with probability at least 0.95 over the calibration sample, every rule in the family has a population accepted-set error rate no larger than its computed bound UU. A selected rule therefore has population risk at most its bound, which is at most α\alpha. The guarantee concerns a population rate under stated assumptions. It does not apply to individual images, to the particular test sample, or to arbitrary distribution shift. Because the bound holds simultaneously over a fixed finite family, it remains valid even though the selection step looks across the whole grid.

The calibration budgets 250, 500 and 1,000 are nested prefixes of the same 1,000 calibration images. The budgets are therefore evaluations of one sample at different sizes, and results across budgets are not independent replications.

Three quantities that must stay separate

We use three different risk numbers and they should never be read as one.

The target risk α\alpha is the requirement we set, 5% or 10%.

The calibration upper bound UU is a conservative statistical bound computed from calibration data, and it is what makes a rule eligible.

The observed test risk is the actual fraction of accepted test predictions that were wrong. It is a measurement on one finite sample and fluctuates around the true population risk. It can be above or below the target. It is not what certifies a rule.

Because UU is deliberately conservative, observed test risk is typically far below α\alpha, as Section 5.4 shows.

Experimental Setup

Dataset and data roles

We use CIFAR-10 [29], which has 32×3232\times32 colour images in ten classes. The archive was downloaded from a pinned mirror revision and checked against the official archive’s MD5 hash (Appendix B). The official 50,000-image training set is divided into five disjoint subsets, and the official 10,000-image test set is kept apart.

The risk-calibration subset was set aside without using labels before the remaining pool was stratified, so its class counts are not exactly balanced (they range from 83 to 118 per class). The training, selection, reference and fitting subsets are stratified by class. The reference set holds 931 correctly classified examples (69 errors), and the fitting and calibration sets contain 76 and 81 classifier errors, respectively. The test set contains 708 errors. No test label was used to train, to select a checkpoint, to fit a score, to set a candidate grid or to choose a threshold.

Classifier and training

The classifier is a ResNet-18 [30] trained from random initialisation with no pretrained weights. For 32×3232\times32 inputs we replace the first convolution with a 3×33\times3, stride-1 convolution with 64 filters and remove the initial max-pooling layer. The remaining residual stages follow the standard torchvision ResNet-18. The model has 11,173,962 parameters and outputs a 512-dimensional feature vector after global average pooling, followed by a linear layer with ten outputs.

Training uses AdamW [31] with an initial learning rate of 0.001, weight decay 0.0001 and batch size 128, for 40 epochs. The learning rate follows cosine annealing [32] down to a minimum of 0.00001. Training images receive random 32-pixel crops with 4-pixel padding and random horizontal flips. All images are normalised with the standard CIFAR-10 channel means (0.4914, 0.4822, 0.4465) and standard deviations (0.2470, 0.2435, 0.2616). Selection, reference, fitting, calibration and test images use deterministic preprocessing without augmentation. The checkpoint is the epoch with the highest accuracy on the 2,000-image selection set. That was epoch 39, with a selection accuracy of 93.95%. The seed was 42.

Additional diagnostics are provided in Appendix D.

Software, hardware and cost

The run session records Google Colab execution with a Tesla T4 GPU and Python 3.13.16. The recorded environment was PyTorch 2.11.0 (CUDA 13.0), torchvision 0.26.0, NumPy 2.1.3, scikit-learn 1.6.1, SciPy 1.16.3 and Matplotlib 3.10.0 [33, 34, 35, 36, 37]. Training took 2,837 seconds (about 47 minutes, around 71 seconds per epoch), not counting the time to upload the final checkpoint. Peak process memory was 1,928 MiB. and peak CUDA allocation 877 MiB. On the test set, the forward pass took 0.268 ms per image and feature extraction 6.2 seconds in total, and computing a combined score took 0.008 ms per image. The reliability layer therefore adds almost nothing to inference cost. Analysis ran in float32 or float64 as recorded in the code.

Figure 1. Data roles and information flow. Scores come from a frozen classifier; threshold grids use fitting data, certificates use calibration data, and final outcomes use the official test set.

RoleImagesPurpose
Training45,000Classifier weights
Selection2,000Checkpoint epoch
Reference1,000Correct-example centroids
Fusion fitting1,000Estimator + threshold grid
Risk calibration1,000Threshold certificates
Official test10,000Final evaluation

Table 1. DISJOINT DATA ROLES. THE FIRST FIVE ROWS PARTITION THE OFFICIAL TRAINING POOL.

SettingValue
ArchitectureCIFAR-adapted ResNet-18
Training40 epochs; seed 42; batch 128
OptimiserAdamW; weight decay 0.0001
Learning rate0.001; cosine minimum 0.00001
FusionStandardScaler + logistic; C = 1
Candidate family5 methods; 21 thresholds; 3 budgets
Calibration budgets250 / 500 / 1,000
Risk targets5% / 10%
Confidence95% simultaneous over 315 rules
Acceptance floor20 calibration examples

Table 2. PRESPECIFIED MAIN-RUN CONFIGURATION.

The Colab workflow saves checkpoints and intermediate results to persistent storage and resumes interrupted training from the last completed epoch. The draft reports a 20-test suite and a prior CPU smoke run. The supplied results archive contains no test source or execution log, so those development checks are not independently verified here. The saved scores, disjoint split indices, coefficients, bounds and selected-rule outcomes were independently audited for this manuscript.

Protocol integrity

Candidate threshold grids come from fusion-fitting scores only. Calibration data are used only to compute bounds and select a rule from the fixed grid. The reported protocol reserves test data until the classifier, fusion models, grids and certificates are frozen. The results archive supports checking saved artefacts, but cannot independently establish every aspect of execution order. The repository records SHA-256 hashes of the checkpoint and the configuration so that an evaluation can be checked against the exact model and settings (Code and Data Availability).

Results

All results in this section come from one training seed (42) and the official 10,000-image test set. No across-seed intervals are available, and we say so wherever a comparison is close.

MetricValue
Accuracy92.92%
Correct / incorrect9,292 / 708
NLL0.3789
Multiclass Brier0.1202
15-bin ECE0.0488

Table III. Frozen classifier performance on the official test set.

MethodAUROCAURC
MSP0.922990.00860
Entropy0.923320.00857
Distance0.822420.01845
MSP + entropy0.923540.00856
All three0.881450.01357

Table IV. Error detection: higher AUROC and lower AURC are better.

Classifier performance

Additional diagnostics are provided in Appendix D.

Additional diagnostics are provided in Appendix D.

The classifier is accurate enough to make a meaningful reliability study. Its ECE of 0.0488 is modest but not negligible. On the 1,000-image fitting set, the mean maximum softmax probability was 0.981 while accuracy was 92.4%, so the network is overconfident in the usual way. The confusion matrix shows that errors are concentrated in a few animal classes. The largest single confusions are dog predicted as cat (67 images) and cat predicted as dog (66 images), followed by cat predicted as bird (28), bird as deer (26) and cat as deer (25). Per-class recall is lowest for cat (83.7%) and dog (89.6%) and highest for automobile (97.1%) (Appendix D).

Error detection

Three confidence-based scores are nearly identical. MSP, entropy and the two-signal fusion differ in the third decimal place of AUROC and in the fifth of AURC. That is expected, because for a ten-class softmax output MSP and entropy are strongly related, and their rank correlation on the test set rounds to 1.000. The two-signal fusion has the best numbers (0.92354 and 0.00856), but its edge over MSP or entropy alone is tiny, and Section 5.7 shows that its paired interval includes zero.

The distance signal alone is clearly worse, with an AUROC of 0.822 against 0.923 for MSP. It is not uninformative. The mean standardised distance was 0.558 for correctly classified test images and 0.714 for misclassified ones, so errors do sit farther from their predicted centroid on average. But as a ranking of errors it is weaker than confidence.

Adding distance to the two-signal fusion made things worse, not better. The three-signal score has an AUROC of 0.881, which is 0.042 below the two-signal fusion, and an AURC of 0.01357 against 0.00856, an increase of about 59%. It sits between the distance signal and the confidence signals. This is the central result of the paper: in this run, fusing the feature-distance signal with confidence and entropy lowered error-detection quality.

Selective risk at fixed coverage

The count of errors makes the gaps concrete. When the 5,000 most trusted predictions are accepted, entropy lets through 8 errors, while the three-signal fusion lets through 34 and distance alone 48. Put the other way, entropy places 700 of the 708 errors in the least trusted half of the data. At 70% coverage the counts are 43 for entropy and 111 for the three-signal score, and at 80% they are 98 and 171. The gap closes at 90% coverage (281 versus 286 errors). So the extra signal costs the most at the top of the ranking, where a few confident mistakes are what a deployment would most like to avoid, and costs almost nothing once most of the data are accepted. At 90% coverage, rejecting the 1,000 least trusted images removes 425 of the 708 errors under MSP (60%) and 422 under the three-signal score.

Controlled-risk coverage

With the full 1,000-image calibration set and a 5% target, entropy accepted 8,482 of the 10,000 test images (84.82%) and 174 of those were wrong, an observed accepted-error rate of 2.05%. The three-signal fusion accepted fewer, 8,061 (80.61%), with 178 errors (2.21%). Distance alone was the most conservative at 61.96% coverage. MSP and the two-signal fusion matched entropy to within a few images. At the 10% target, entropy accepted 93.97% of images at 4.45% observed error, and the three-signal fusion accepted 93.75% at 4.36%, which is a very small difference.

Two things are worth stating plainly. First, the observed test risk is well below the target in every row, around 2% for a 5% target and around 4.4% for a 10% target. The upper bounds are conservative by construction, since they must hold simultaneously for 315 rules at 95% confidence. Second, the 10% target is not demanding for this classifier. The overall test error rate is 7.08% (708/10,000), so accepting every prediction would already meet it on this sample. The 5% target is the informative one, and it is where the methods separate.

Effect of the calibration budget

More calibration data gave more certified coverage. For entropy at the 5% target, test coverage rose from 70.84% with 250 calibration images to 80.56% with 500 and 84.82% with 1,000. The extra data tighten the upper bound, so rules that accept more examples become eligible. The two methods that use distance were hit hardest by a small budget. With 250 calibration images, neither distance alone nor the three-signal fusion could certify the 5% target, so both returned "not certified" and rejected everything, while MSP, entropy and the two-signal fusion each certified about 71% coverage. With 500 images the three-signal fusion certified only 67.67% against 80.56% for entropy. The gap narrowed with 1,000 images (80.61% against 84.82%) and did not close. So in this experiment the added complexity did not become useful as calibration data grew. It became less costly.

At the 10% target, coverage for the confidence methods was the same at 500 and 1,000 images, because the same thresholds were chosen. This is a property of the finite grid. The best

Calibration of the error probabilities

Risk-coverage curves for MSP, Entropy, Feature distance, MSP + entropy, and All three (ShiftGuard)

Figure 2. Risk-coverage curves on 10,000 clean test images. Lower is better. AURC averages prefix risks with expected random-order treatment of exact ties. One training seed.

Method50%70%80%90%
MSP0.16 (8)0.61 (43)1.23 (98)3.14 (283)
Entropy0.16 (8)0.61 (43)1.23 (98)3.12 (281)
Distance0.96 (48)2.24 (157)3.36 (269)5.12 (461)
MSP + entropy0.16 (8)0.61 (43)1.24 (99)3.13 (282)
All three0.68 (34)1.59 (111)2.14 (171)3.18 (286)

Table V. Observed test risk in percent (error count) at fixed coverage.

Observed test risk at selected coverage levels for five rejection methods and four requested coverage levels

Figure 3. Observed test risk (%) at fixed coverage, with identical test images for every method. One seed; no population guarantee is attached to these fixed-coverage summaries.

eligible grid point stayed the same once enough calibration data were available.

Test coverage of certified rules

Figure 4. Test coverage of certified rules. One simultaneous 315-rule family covers all methods, thresholds and budgets. Not-certified rules reject every example; their conditional risk is undefined. Validity assumes IID clean data.

Calibration of the error probabilities

The fused scores output estimated error probabilities, so we also checked them as probabilities. The two-signal fusion had a binary Brier score of 0.0533 and an ECE of 0.0261, and the three-signal fusion had a Brier score of 0.0535 and an ECE of 0.0192. The three-signal score is slightly better calibrated in the ECE sense and equal in Brier score, while ranking errors clearly worse. Ranking and calibration are different properties. A score can give well-scaled probabilities and still order examples badly at the confident end. These ECE values come from 15-bin estimates with a small number of errors, and sparse bins fluctuate, so we do not read much into the 0.007 gap.

Sampling uncertainty (post hoc)

The pipeline reports single-seed point estimates, so we added one analysis after the fact to judge how large a difference would have to be to stand out from test-sample noise.

Coverage versus calibration-label budget

Figure 5. Certified-rule test coverage versus nested calibration budget at each risk target. Budgets are dependent prefixes, not repeated experiments. Zero coverage indicates not certified.

Using the saved test scores, we drew 2,000 paired bootstrap resamples of the 10,000 test images and recomputed error AUROC for every method on each resample. This captures uncertainty from which test images were drawn. It does not capture variability from training, which would need more seeds. It was run on frozen scores and informed no selection or threshold. We disclose it as post hoc.

The intervals for the confidence-only comparisons contain zero and do not resolve their small differences. The deficit of the three-signal fusion relative to the two-signal fusion is about 0.042 and its interval is far from zero, so on this test set the estimated deficit is large relative to the bootstrap variation. The single-seed caveat still applies. The interval indicates that the deficit is large relative to the estimated test-sample variability. It does not establish that the effect would appear for another training run.

Discussion

Why distance did not help

We did not test why, so what follows are hypotheses that fit the numbers.

The first is redundancy. A well-trained network that has fit its training data nearly perfectly (the final training loss was 0.0068) concentrates its confident predictions on feature vectors close to class centroids, so the information in distance and the information in confidence overlap. The rank correlation between the distance score and either MSP or entropy was about 0.39, which is real but moderate. The part of distance that is not already in confidence may be mostly noise for the purpose of detecting errors.

The second is the fusion objective. A logistic regression fitted to maximise the likelihood of the error labels is not trained to rank, and its output is a weighted sum of standardised features. The MSP signal is very skewed, with mean 0.981 and standard deviation 0.074 on the fitting set, so after standardisation a handful of low-confidence examples have extreme values while the large mass of confident correct examples is packed near one end. Among those, differences in MSP and entropy are tiny in raw units but may still carry rank information about which are likely to be wrong. Adding a distance term of comparable scale lets noise in distance reorder that crowded region. The pattern in Table 5 matches this story: the fused score loses the most at 50% to 80% coverage, the confident end of the ranking, and has almost caught up by 90% coverage. This is a hypothesis about the mechanism. A direct test would fit a rank-based fusion or compare the ordering inside the confident region, which we did not do.

The third is the weakness of the distance signal itself. Our distance is a diagonal, standardised centroid distance. A full-covariance Mahalanobis distance, a Trust Score or a nearest-neighbour distance might be a stronger signal and might behave differently. Our result is about this simple distance, and we do not claim it generalises to the best distance-based methods.

Ranking quality and certified coverage move together here

Confidence-only scores perform strongly both in error ranking and in certified coverage, though small within-group differences need not preserve an exact ordering. Confidence-based scores were at the top on every metric and the distance-based ones were below. Certification adds a second cost for a worse score. A rule that accepts the same number of images with more errors has a higher upper bound and fails the target sooner, which is why distance and the three-signal fusion fell to "not certified" when the budget was small. In a regime where labels are scarce, a noisier score is penalised twice, once in ranking and again in the bound.

ComparisonDifferenceInterval
Entropy minus MSP+0.00033[-0.00002, +0.00067]
MSP+Entropy minus MSP+0.00054[-0.00041, +0.00147]
MSP+Entropy minus Entropy+0.00021[-0.00046, +0.00089]
Combined minus MSP+Entropy-0.04209[-0.04917, -0.03519]
Combined minus MSP-0.04155[-0.04881, -0.03466]
Distance minus MSP-0.10058[-0.11014, -0.09104]

Table VIII. Post hoc paired AUROC differences: 2,000 resamples; 95% percentile intervals.

How to read the certificates

The accepted-set error rates observed on the test set (about 2% for a 5% target) are much lower than the targets. This should not be read as the procedure being miscalibrated. The certificate controls a worst-case bound over a family of 315 simultaneous rules at 95% confidence, and the Bonferroni correction pushes each individual bound well out. The price of simultaneous validity is conservatism. A valid less conservative design, such as a smaller prespecified family or an appropriate sequential testing scheme, could raise certified coverage; this was not evaluated. We kept a single family covering every displayed method, threshold and budget so that the comparison across methods rests on one common guarantee.

The guarantee applies to a population of clean images from the same source as the calibration data. The calibration images come from the CIFAR-10 training pool and the evaluation images from the official test set. These are nominally the same distribution, but a nominally identical new test set can still shift accuracy [27], and nothing in our experiments can verify the IID assumption.

What is a finding and what is a hypothesis

Findings, for this classifier, seed and dataset: the error-detection and AURC numbers in Table 4; the counts in Table 5; the certified coverages in Table 6 and Table 7; the failure of the distance-based methods to certify the 5% target with 250 images; the bootstrap intervals in Table 8.

MethodTargetCal. accepted/errorsUTest accepted/errorsCoverageTest risk
MSP5%845/154.04%8480/17584.80%2.06%
MSP10%935/478.10%9365/40893.65%4.36%
Entropy5%846/154.03%8482/17484.82%2.05%
Entropy10%938/508.47%9397/41893.97%4.45%
Distance5%622/114.53%6196/10261.96%1.65%
Distance10%887/579.90%8933/44589.33%4.98%
MSP + entropy5%846/154.03%8479/17484.79%2.05%
MSP + entropy10%936/498.36%9388/40793.88%4.34%
All three5%799/174.62%8061/17880.61%2.21%
All three10%929/478.16%9375/40993.75%4.36%

Table 6. Certification audit with 1,000 calibration images. U is the calibration upper bound; test risk is descriptive.

MethodTarget2505001,000
MSP5%70.7980.3684.80
Entropy5%70.8480.5684.82
Distance5%n.c.n.c.61.96
MSP + entropy5%70.8480.5584.79
All three5%n.c.67.6780.61
MSP10%88.6193.6593.65
Entropy10%88.5693.9793.97
Distance10%71.8284.9889.33
MSP + entropy10%88.5493.8893.88
All three10%88.3493.7593.75

Table 7. Certified-rule test coverage (%); n.c. denotes not certified and undefined conditional risk.

Hypotheses we did not test: that distance is redundant with confidence for a well-trained network; that the logistic objective and the skew in MSP explain the loss at the confident end; that a stronger distance signal would change the result; that the same ordering would hold across seeds, datasets and architectures.

Limitations

  1. One training seed. The full ResNet-18 experiment has a single seed (42). The paired bootstrap covers test-sample variability and says nothing about training variability. Differences among MSP, entropy and the two-signal fusion are too small to interpret at this level.

  1. One dataset and one architecture. All results come from CIFAR-10 and a CIFAR-adapted ResNet-18 trained from scratch.

from scratch. A pretrained network, a different architecture or a harder dataset might rank the signals differently. Earlier exploratory results described in the draft are not part of the submitted evidence; they are not used to support the main comparison.

  1. No shift experiments. The code includes a corruption module, but the corruption list for this run is empty, so there are no corruption results. We make no claim about distribution-shift detection or robustness, and the name ShiftGuard should not be taken as evidence of either. The certificates apply under the stated IID assumption and do not extend to shifted data.

  1. A simple distance. The distance signal is a diagonal, standardised distance to the predicted-class centroid, computed from penultimate features. It ignores feature correlations and multimodal classes. Stronger distance-based detectors exist and were not tested.

  1. Fixed estimator and grid. The logistic regression uses a prespecified C = 1 and was not tuned, and the candidate grid has 21 quantile thresholds. Other fusion models, other regularisation and finer grids might change the results.

  1. Conservative, nested certification. The simultaneous 315-rule family makes the bounds conservative, and the three budgets are nested prefixes of one calibration set, so they are not independent. Observed test risk is a single-sample estimate.

  1. IID assumption. The calibration pool is the CIFAR-10 training split and the evaluation set is the official test split. The guarantee relies on these behaving like draws from one population, which cannot be verified here.

  1. Small fitting and calibration sets. The fitting set has 76 errors and the calibration set 81, so every fitted coefficient and bound rests on a small number of mistakes. This is the intended low-label regime but it limits precision.

  1. Post hoc analysis. The bootstrap in Section 5.7 was added after the main run. It used frozen scores and informed no choices, but it was not prespecified.

Conclusion And Future Work

We asked whether a feature-distance signal improves on confidence and entropy for detecting classification errors and for accepting predictions at a controlled error rate. On a from-scratch CIFAR-10 ResNet-18 with 92.92% test accuracy, the answer in this run is no. MSP, entropy and their logistic fusion ranked errors almost identically (AUROC 0.9230, 0.9233 and 0.9235), while distance alone (0.8224) and the three-signal fusion (0.8814) were worse. With 1,000 labelled calibration images and a 5% risk target, entropy certified a rule that accepted 84.82% of test images at an observed error of 2.05%, against 80.61% at 2.21% for the three-signal fusion. With 250 calibration images the distance-based methods could not certify the 5% target at all. A small calibration set made the added complexity a liability and a larger one reduced the cost without removing it.

The strongest supported statement is narrow. For this classifier and seed, a simple standardised centroid distance did not earn its place beside confidence and entropy, and a more complex score made certification harder when labels were scarce. We do not claim that distance signals are useless, that the result holds for other models or data, or that these scores detect distribution shift.

Several directions follow. The most useful is more training seeds, so that the comparison has a real across-run variance, and more architectures and datasets. Stronger distance signals, including full-covariance Mahalanobis distance, Trust Score and nearest-neighbour distances, should be tested under the same certification protocol. Rank-based fusion, or fusion that treats the confident region more carefully, would test the explanation in Section 6.1. Finally, the corruption module already in the code can support a shift evaluation with frozen clean thresholds, which would tell us whether certificates degrade under corruption and whether distance helps there, in a separately evaluated follow-up. Any change motivated by the test results here should be evaluated on fresh data and reported as a follow-up analysis, with no tuning on the test set.

CODE AND DATA AVAILABILITY

The author-provided repository is https://github.com/mitravarun123/Shift_Name-. Public availability was not independently confirmed during preparation. The accompanying research archive preserves the submitted result folder, including configuration, split indices, saved predictions, estimator metadata, certificates and training history. It contains no classifier checkpoint or training source. The manuscript package contains all figures, typeset tables, CSV data and the independent score/certificate audit and bootstrap script. Reproduction of training requires the source repository and checkpoint or a fresh run; numerical reproduction from the saved scores is documented in the package README. No project licence is asserted.

Complete Experimental Configuration and Data Integrity

The main experiment is risk_control_v1, architecture resnet18, scope full, seed 42, device CUDA, batch size 128, 40 epochs, initial learning rate 0.001, minimum learning rate 0.00001 and weight decay 0.0001. The run uses two CPU threads, zero data-loader workers, no training/test limit and no corruptions. Feature dimension is 512. Saved figures are 600-DPI PNG exports. Classifier training completed all 40 epochs; the best checkpoint is epoch 39 with selection accuracy 0.9395. Final-epoch selection accuracy is 0.9375 and training loss is 0.0068. Training took approximately 2,837 seconds. Peak RSS is 1,927.91 MiB and peak CUDA allocation 876.59 MiB.

The recorded session specifies Python 3.13.16, PyTorch 2.11.0+cu130, torchvision 0.26.0+cu130, CUDA 13.0, NumPy 2.1.3, scikit-learn 1.6.1, SciPy 1.16.3, Matplotlib 3.10.0 and PyYAML 6.0.3, on a Tesla T4. These are historical run metadata, not the environment used to prepare the manuscript.

The dataset archive is recorded as CIFAR-10 Python format, mirror MIT-OL-AI-D/cifar-10-python, revision a48007227f9e2cd0af96175f4afb5ac4965e261b, with archive MD5 c58f30108f718f92721af3b95e74349a.

The saved index arrays independently verify that training, selection, reference, fitting and calibration are disjoint and partition all 50,000 training indices. Training class counts range from 4,483 to 4,516, selection from 199 to 201, reference and fitting from 99 to 101, and calibration from 83 to 118. Test indices belong to the distinct official test pool.

Complete Risk-Certification Audit

Every row below uses the common confidence family of 315 rules and δ=0.05\delta= 0.05. A failed certificate has zero coverage and undefined conditional risk (shown as —). Columns distinguish calibration accepted/error counts, upper risk UU, test accepted/error counts, test coverage and observed test risk. No test outcome enters certificate selection.

BudgetMethodTargetCal. accepted/errorsUUTest accepted/errorsCoverageTest risk
250MSP5%178/04.80%7079/4670.79%0.65%
250MSP10%226/79.54%8861/24488.61%2.75%
250Entropy5%178/04.80%7084/4670.84%0.65%
250Entropy10%225/68.89%8856/24388.56%2.74%
250Distance5%——0/00.00%—
250Distance10%185/38.03%7182/17971.82%2.49%
250MSP + entropy5%178/04.80%7084/4670.84%0.65%
250MSP + entropy10%225/68.89%8854/24588.54%2.77%
250All three5%——0/00.00%—
250All three10%224/79.62%8834/26388.34%2.98%
500MSP5%401/44.22%8036/10380.36%1.28%
500MSP10%469/229.23%9365/40893.65%4.36%
500Entropy5%401/44.22%8056/10580.56%1.30%
500Entropy10%468/229.25%9397/41893.97%4.45%
500Distance5%——0/00.00%—
500Distance10%416/189.09%8498/35584.98%4.18%
500MSP + entropy5%401/44.22%8055/10580.55%1.30%
500MSP + entropy10%468/229.25%9388/40793.88%4.34%
500All three5%345/44.89%6767/10367.67%1.52%
500All three10%466/229.29%9375/40993.75%4.36%
1000MSP5%845/154.04%8480/17584.80%2.06%
1000MSP10%935/478.10%9365/40893.65%4.36%
1000Entropy5%846/154.03%8482/17484.82%2.05%
1000Entropy10%938/508.47%9397/41893.97%4.45%
1000Distance5%622/114.53%6196/10261.96%1.65%
1000Distance10%887/579.90%8933/44589.33%4.98%
1000MSP + entropy5%846/154.03%8479/17484.79%2.05%
1000MSP + entropy10%936/498.36%9388/40793.88%4.34%
1000All three5%799/174.62%8061/17880.61%2.21%
1000All three10%929/478.16%9375/40993.75%4.36%

Table IX. TABLE IX: All 30 prespecified method/budget/target selections.

Mathematical Details and Validity of Selection

For fixed scoring and threshold functions, define population acceptance Aτ=1{s(X)≤τ}A_{\tau} = \mathbb{1}\{s(X) \leq\tau\}, coverage Cτ=P(Aτ=1)C_{\tau} = \mathbb{P}(A_{\tau} = 1) and conditional risk rτ=P(Y^≠Y∣Aτ=1)r_{\tau} = \mathbb{P}(\hat{Y} \ne Y \mid A_{\tau} = 1) when Cτ>0C_{\tau} > 0. Conditional on the accepted calibration count M=mM = m, the accepted error count is binomial with parameters (m,rτ)(m,r_{\tau}) under the stated IID assumption. Thus a one-sided Clopper–Pearson upper bound has failure probability at most δ′\delta' for each fixed rule. Conditioning on the independent training/reference/fitting data fixes the score and grid. The union bound gives

P(∃f∈F:rf>Uf)≤∑f∈Fδ′=∣F∣δ′=0.05.(9)\mathbb{P}(\exists f \in\mathcal{F}: r_f > U_f) \le\sum_{f \in\mathcal{F}} \delta' = |\mathcal{F}|\delta' = 0.05. \tag*{(9)}

No independence among candidate tests is required: nested budgets and shared examples do not invalidate this union bound. On the simultaneous event, any data-selected rule satisfying Uf≤αU_f \le\alpha has population conditional risk at most α\alpha. Reusing the same bounds for two risk targets introduces no additional failure event. These statements provide no guarantee under arbitrary distribution shift and do not bound a particular finite test-set error count.

With zero accepted errors, the exact bound simplifies to

U(0,m,δ′)=1−(δ′)1/m,m≥1.(10)U(0,m,\delta') = 1 - (\delta')^{1/m}, \qquad m \ge1. \tag*{(10)}

At δ′=0.05/315\delta' = 0.05/315, at least 171 zero-error accepted calibration examples are needed to certify a 5% target. This explains the 250-image setting: the confidence-only rules accept 178 calibration examples without error and have upper risk approximately 4.80%. The minimum-acceptance rule of 20 is therefore weaker than the statistical requirement in this particular zero-error case. With no accepted examples the implemented bound is one and the rule is ineligible; its conditional risk remains undefined.

Exact-tie treatment in AURC

Suppose a score-tied group contains gg examples and bb errors, preceded by tt examples and aa errors. Under uniform random ordering inside the tied group, the expected prefix risk after jj tied examples is

E[Rt+j]=a+jb/gt+j,j=1,…,g.(11)\mathbb{E}[R_{t+j}] = \frac{a + jb/g}{t+j}, \qquad j = 1,\ldots,g. \tag*{(11)}

Summing these expected risks over all groups and dividing by NN makes AURC invariant to the arbitrary ordering of exact ties. Fixed-coverage summaries use ⌈cN⌉\lceil cN\rceil accepted examples. They describe the test ranking and do not identify independently certified deployment thresholds.

Probability metrics

For true class yiy_i, softmax probabilities pikp_{ik}, bin counts nbn_b, average confidence q‾b\overline{q}_b and accuracy a‾b\overline{a}_b,

NLL=−1N∑ilog⁡max⁡(pi,yi,10−15),(12)\mathrm{NLL} = -\frac{1}{N}\sum_i \log\max(p_{i,y_i}, 10^{-15}), \tag*{(12)}
Briermulti=1N∑i∑k=110(pik−1{yi=k})2,(13)\mathrm{Brier}_{\mathrm{multi}} = \frac{1}{N}\sum_i\sum_{k=1}^{10}(p_{ik} - \mathbb{1}\{y_i = k\})^2, \tag*{(13)}
ECE15=∑b:nb>0nbN∣q‾b−a‾b∣.(14)\mathrm{ECE}_{15} = \sum_{b:n_b>0}\frac{n_b}{N}\lvert\overline{q}_b - \overline{a}_b\rvert. \tag*{(14)}

ECE uses 15 equal-width bins on [0,1][0,1], with the rightmost bin including one. Binary error-probability metrics replace class correctness by error labels and use (p^i−ei)2(\widehat{p}_i-e_i)^2 for the binary Brier score. These calibration metrics do not measure ranking quality.

Appendix D: Additional Diagnostics

The following figures are supplied clean-test or training diagnostics. They share the main experiment’s seed 42 and do not provide independent replications. The confusion matrix gives class recalls of 94.0, 97.1, 90.1, 83.7, 94.2, 89.6, 95.2, 94.9, 95.2 and 95.2% in the standard CIFAR-10 class order (airplane through truck). The largest off-diagonal counts are dog-to-cat 67 and cat-to-dog 66.

Error-detection ROC curves for ResNet-18 on the clean test set

Figure 6. Clean-test error-detection ROC curves, with incorrect predictions as positives. One training seed.

Reliability diagrams for classifier correctness and fused error probabilities

Figure 7. Reliability diagrams for classifier correctness and fused error probabilities. Fifteen equal-width bins; marker sizes reflect bin counts. Sparse bins can fluctuate.

Classification confusion matrix for ResNet-18 on CIFAR-10

Figure 8. Official CIFAR-10 test confusion matrix; entries are image counts.

Training loss, checkpoint-selection accuracy and learning rate

Figure 9. Training loss, checkpoint-selection accuracy and learning rate. Epoch 39 is selected by the independent selection subset, not test results.

Estimator Audit and Post Hoc Uncertainty

The saved first fusion input equals maximum softmax probability, not its complement. Fitting means are (0.980919, 0.048832) and scales (0.074233, 0.158986) for the two-signal model. Adding distance appends mean 0.574360 and scale 0.138404. Standardised coefficients are (0.835964, 1.803954) with intercept −2.977201, and (0.767487, 1.557528, 0.587546) with intercept −3.145717 for the three-signal model. Reconstructing both error probabilities from the saved input matrix reproduces the stored scores to floating-point precision. Estimator pickle/joblib files are preserved as evidence but are not executed by the audit script.

The paired bootstrap was recomputed for manuscript preparation from frozen test scores: 2,000 resamples of 10,000 image indices with replacement, NumPy default random generator seed 0, shared indices across methods, and 2.5th/97.5th percentiles of paired AUROC differences. It is explicitly post hoc, conditional on this trained network, and does not represent training-seed variability. The bootstrap intervals are not adjusted for simultaneous inference over the six displayed contrasts. The accompanying CSV contains the recomputed intervals and the script regenerates them.

Evidence, Reproducibility and Scope

The independent audit verifies all 315 candidate upper bounds against the exported counts, every one of the 30 selected method/budget/target rules, their test accepted/error counts, all five AUROCs, disjoint training-pool partitions, and both logistic scoring functions. The archive records checkpoint SHA-256 daa75ae61d0d41844ceafcf906e85391a5befcec326e9f71b64b5508ce24b379 and configuration SHA-256 a486c2cb090d86d412f13c4a0e5ae2089c593554315fc947e713c10da9e4259e. These identifiers are reported metadata; the classifier checkpoint itself is absent, so its hash cannot be independently recomputed from this archive.

The source draft also describes earlier small-CNN runs, a two-epoch pilot, and a 20-test software suite. Their underlying results and test logs are absent from the submitted archive. They are excluded from this paper’s quantitative evidence rather than treated as verified additional experiments. All empirical conclusions concern the supplied 40-epoch ResNet-18 run. Training architecture, augmentation and optimiser details beyond the saved configuration are author-reported protocol descriptions, not an independent audit of unavailable training source.

The raw 315-row candidate table is supplied as data/calibration_candidate_bounds.csv; the full selection audit is data/clean_risk_control.csv. To reproduce numerical checks and bootstrap intervals, install NumPy, pandas and SciPy and run python scripts/verify_and_bootstrap.py results in the research archive. This recomputes evaluation summaries from saved outputs without training a model. Full training reproduction requires the training source, dataset and checkpoint workflow. No training was performed in preparing this manuscript.

References

References

  1. [1]Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning.arxiv.org/abs/1706.04599
  2. [2]Chow, C. K. (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41-46.
  3. [3]El-Yaniv, R. and Wiener, Y. (2010). On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11:1605-1641.DOI
  4. [4]Geifman, Y. and El-Yaniv, R. (2017). Selective classification for deep neural networks. In Advances in Neural Information Processing Systems 30.arxiv.org/abs/1705.08500
  5. [5]Hendrycks, D. and Gimpel, K. (2017). A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations.
  6. [6]Jiang, H., Kim, B., Guan, M. Y., and Gupta, M. (2018). To trust or not to trust a classifier. In Advances in Neural Information Processing Systems 31.arxiv.org/abs/1805.11783
  7. [7]Lee, K., Lee, K., Lee, H., and Shin, J. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems 31.arxiv.org/abs/1807.03888
  8. [8]Sun, Y., Ming, Y., Zhu, X., and Li, Y. (2022). Out-of-distribution detection with deep nearest neighbors. In Proceedings of the 39th International Conference on Machine Learning.arxiv.org/abs/2204.06507
  9. [9]Geifman, Y., Uziel, G., and El-Yaniv, R. (2019). Bias-reduced uncertainty estimation for deep neural classifiers. In International Conference on Learning Representations.arxiv.org/abs/1805.08206
  10. [10]Geifman, Y. and El-Yaniv, R. (2019). SelectiveNet: A deep neural network with an integrated reject option. In Proceedings of the 36th International Conference on Machine Learning.arxiv.org/abs/1901.09192
  11. [11]Clarke, C., Thome, N., Bar-Hen, A., Cord, M., and Pérez, P. (2019). Addressing failure prediction by learning model confidence. In Advances in Neural Information Processing Systems 32.
  12. [12]Galil, I., Dabbah, M., and El-Yaniv, R. (2023). What can we learn from the selective prediction and uncertainty estimation performance of 523 ImageNet classifiers? In International Conference on Learning Representations.
  13. [13]Ashukha, A., Lyzhov, A., Molchanov, D., and Vetrov, D. (2020). Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In International Conference on Learning Representations.arxiv.org/abs/2002.06470
  14. [14]Liu, W., Wang, X., Owens, J. D., and Li, Y. (2020). Energy-based out-of-distribution detection. In Advances in Neural Information Processing Systems 33.arxiv.org/abs/2010.03759
  15. [15]Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30.arxiv.org/abs/1612.01474
  16. [16]Gal, Y. and Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning.arxiv.org/abs/1506.02142
  17. [17]Cattelan, L. F. P. and Silva, D. (2024). How to fix a broken confidence estimator: Evaluating post-hoc methods for selective classification with deep neural networks. In Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, PMLR 244, pp. 547–584. https://proceedings.mlr.press/v244/cattelan24a.html.arxiv.org/abs/2305.15508
  18. [18]Naeini, M. P., Cooper, G. F., and Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence.pubmed.ncbi.nlm.nih.gov/25927013
  19. [19]Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3.doi.org/10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
  20. [20]Bates, S., Angelopoulos, A., Lei, L., Malik, J., and Jordan, M. (2021). Distribution-free, risk-controlling prediction sets. Journal of the ACM, 68(6):1-34.arxiv.org/abs/2101.02703
  21. [21]Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I., and Lei, L. (2022). Learn then Test: Calibrating predictive algorithms to achieve risk control. arXiv:2110.01052 (revised 2022). https://arxiv.org/abs/2110.01052.
  22. [22]Angelopoulos, A. N. and Bates, S. (2023). Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494-591.
  23. [23]Clopper, C. J. and Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404-413.DOI
  24. [24]Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293):52-64.DOI
  25. [25]Ovadia, Y., Fertig, E., Ren, J., et al. (2019). Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32.arxiv.org/abs/1906.02530
  26. [26]Hendrycks, D. and Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations.
  27. [27]Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. (2019). Do CIFAR-10 classifiers generalize to CIFAR-10? In Proceedings of the 36th International Conference on Machine Learning.arxiv.org/abs/1806.00451
  28. [28]Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8):861-874.DOI
  29. [29]Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Technical report, University of Toronto.
  30. [30]He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.arxiv.org/abs/1512.03385
  31. [31]Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations.
  32. [32]Loshchilov, I. and Hutter, F. (2017). SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations.arxiv.org/abs/1608.03983
  33. [33]Paszke, A., Gross, S., Massa, F., et al. (2019). PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32.arxiv.org/abs/1912.01703
  34. [34]Harris, C. R., Millman, K. J., van der Walt, S. J., et al. (2020). Array programming with NumPy. Nature, 585:357-362.
  35. [35]Pedregosa, F., Varoquaux, G., Gramfort, A., et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.arxiv.org/abs/1201.0490
  36. [36]Virtanen, P., Gommers, R., Oliphant, T. E., et al. (2020). SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods, 17:261-272.
  37. [37]Hunter, J. D. (2007). Matplotlib: A 2D graphics environment. Computing in Science and Engineering, 9(3):90-95.DOI

Paper details

vlnc:W_848syeza
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents