betaFeedback

WorksComputer Science

A Canary in the Neural Mine: Catching an Image Classifier's Silent, Confident Failures

Abstract

Deployed vision models auto-accept confident predictions, so high-confidence errors propagate silently. Maximum Softmax Probability (MSP), the standard failure predictor, has a structural blind spot: as a scalar read from the classifier head, it is first-order blind to feature variation orthogonal to its confidence gradient. Through a factorial analysis of feature-space detectors, we identify a conditional-redundancy principle explaining why they fail to complement MSP: the class-separability geometry they measure is already encoded in MSP and collapses under fusion. The signal that survives is class-conditional typicality on the off-manifold residual—and because this residual's distribution varies by backbone, from near-Gaussian to heavy-tailed, it must be estimated non-parametrically. We instantiate these findings as CANARY, a training-free detector that computes a per-class kNN distance on the PCA residual of the penultimate feature, percentile-calibrates it on validation data, and fuses it with MSP through a single validation-tuned weight. CANARY's benefit follows a headroom law: gains grow as baseline MSP quality falls, so a single clean-validation number forecasts the improvement. With every parameter selected on in-distribution data alone, CANARY matches or improves 382 of 385 shift cells (e-AURC; 379 on AUROC) across five backbones, with worst-case regression within seed noise (−0.0002-0.0002 AUROC) and the largest gains on natural distribution shifts (ImageNet-A/R, ObjectNet)—where deployed models are most exposed.

Introduction

Deployed vision models operate under selective classification: they auto-accept predictions when confident and abstain when uncertain. The safety of this pipeline relies entirely on accurate failure prediction, defined as the model’s ability to rank correct predictions above incorrect ones. When ranking fails at high confidence, errors propagate silently into deployment.

Maximum Softmax Probability (MSP) remains the canonical baseline for failure prediction, but it suffers from a structural blind spot. By collapsing the

From a DD-dimensional feature space to a scalar confidence, MSP is first-order insensitive to an entire (D−1)(D-1)-dimensional hyperplane of feature variation—on ViT-B/16, 767 of 768 dimensions (§ 4.1). Feature-space methods such as ViM and the Mahalanobis family exploit exactly this discarded geometry—yet, designed for OOD detection, they consistently underperform MSP when transplanted to failure prediction (§ 2).

To understand exactly what information in the feature space complements MSP, we decompose the design space of feature-based failure prediction into a factorial experiment over three axes: projection basis, distance metric, and reference conditioning. This analysis reveals a conditional-redundancy principle: the class-separability geometry that most feature detectors measure is redundant with MSP under fusion—the classifier head has already extracted those directions. What survives fusion is off-manifold typicality, and it imposes two requirements: the score must be non-parametric, because the residual’s distribution varies by backbone, and class-conditional, because global typicality breaks under natural shifts (§ 4).

We introduce CANARY, a minimal, training-free instantiation of these findings: a per-class kNN distance on the PCA residual of the penultimate feature, percentile-calibrated and fused with MSP through a single validation-tuned weight. Because CANARY isolates precisely the signal MSP lacks, its benefit is predictable: fusion gains grow as baseline MSP quality falls (Figure 1), so a single clean-validation number forecasts the gain before deployment—small where MSP is already a near-ceiling ranker, largest on weak backbones and natural shifts. A same-architecture control isolates the driver: two ViT-B/16 fine-tunes differing only in MSP quality show gains of +0.42 vs. +0.08 pp, ruling out architectural confounds.

Figure 1. The Headroom Law across shift families. CANARY’s fusion AUROC gain (y-axis) is inversely related to the backbone’s baseline MSP AUROC on the clean ImageNet-1k validation set (x-axis). This predictable negative-slope relationship holds consistently across both synthetic ((a) ImageNet-C) and natural shifts ((b) ImageNet-A, (c) ImageNet-R), with natural shifts exhibiting the most substantial gains. Dashed line: a same-architecture control (two ViT-B/16 fine-tunes) isolating MSP quality as the driver of the gain (§ 6).

This predictability yields a concrete operational property—empirically, CANARY is never worse than the MSP it augments. Across 385 shift cells on five backbones, it matches or improves 382 cells (e-AURC; 379 AUROC); the exceptions (3 on e-AURC, 6 on AUROC) lie within seed noise (worst case −0.0002 AUROC). Its gains concentrate on natural, real-world shifts (ImageNet-A, ImageNet-R, ObjectNet) rather than synthetic corruptions.

Contributions:

  • Conditional-Redundancy Principle. A factorial analysis over basis, distance, and conditioning showing that class-separability geometry is redundant with MSP under fusion; the complementary signal is non-parametric, class-conditional, off-manifold typicality (§ 4).

  • Headroom law. Fusion gains decrease predictably with baseline MSP quality, isolated from architecture by a same-architecture control and validated across nine ImageNet backbones (including self-supervised DINOv2), a different label space (CIFAR-100), and a held-out covariate shift (ObjectNet) (§ 6).

  • CANARY. A training-free, post-hoc detector realizing the blueprint; it matches or exceeds MSP on 382 of 385 shift cells with worst-case regression within seed noise, with the largest gains on natural shifts (§ 5–§ 6).

Related Work

Logit- and Activation-Based Detection. MSP [16] remains the canonical baseline for failure prediction. Logit-space corrections, such as MaxLogit [13], Energy [25], Doctor [10], and GEN [26], rely purely on the information already exposed by the linear classifier head, inherently inheriting MSP’s structural blind spot. Because any score read from the logit vector alone is invariant to feature variation orthogonal to the classifier head, it cannot retroactively distinguish confident errors from correct predictions; extensive evaluations confirm none consistently outperform MSP [19]. Activation-shaping heuristics (e.g., ReAct [34], ASH [4], DICE [35]) modify features before logit computation. However, being calibrated primarily on CNN statistics, they suffer from severe architecture sensitivity. When transplanted to structurally different models like Vision Transformers, their performance often degrades sharply.

Feature-Space Detection. To bypass the logit bottleneck, methods like Mahalanobis [23], Deep kNN [36], ViM [39], and Kernel PCA [6] leverage broader feature geometry. However, these methods were designed specifically for out-of-distribution (OOD) detection. Detecting a confidently misclassified in-distribution sample is structurally distinct from flagging an anomalous OOD input. Consequently, when transplanted to failure prediction, they consistently—and often substantially—underperform the MSP baseline. Furthermore, heuristic parameter-free combinations (e.g., ViM adding an energy term, or NNGuide [31] scaling logits) implicitly assume their constituent tests provide independent signals. As we demonstrate, this assumption fails: class-geometry scores are conditionally redundant with MSP under fusion, meaning a feature-based partner like fDBD [24] contributes only through its off-manifold component. Closest to our design is SIRC [40, 41], which combines the softmax score with a class-agnostic feature norm (a feature L1L_1 [18] or ViM residual norm). In contrast, our approach uses a class-conditional off-manifold residual. Prior work has neither isolated the individual contributions of these design choices nor explained their inconsistent outcomes—a gap our factorial analysis addresses (§ 4).

Failure Prediction and Selective Classification. Dedicated failure predictors like TrustScore [20], ConfidNet [2], and OpenMix [44] score misclassifications directly. ConfidNet and OpenMix train auxiliary heads, whereas TrustScore operates post-hoc and ranks predictions by agreement with class-conditional neighbors in feature space. Against this backdrop, none of these predictors isolates the signal that survives fusion with MSP.

Preliminaries

Selective Classification. Let f:X→RCf:\mathcal{X} \to\mathbb{R}^{C} be a neural network trained to classify inputs x∈Xx \in\mathcal{X} into one of CC classes, with predicted class y^=arg⁡max⁡cfc(x)\hat{y} = \arg\max_{c} f_c(x). In the selective classification framework, the model is equipped with a confidence scoring function s(x)∈Rs(x) \in\mathbb{R} and a threshold τ\tau: it outputs the prediction y^\hat{y} if s(x)≥τs(x) \ge\tau, and abstains otherwise. An ideal score ranks all correct predictions above all incorrect ones, so that errors are rejected first as the threshold τ\tau increases.

The MSP Baseline and Feature Space. We decompose ff into a feature extractor and a linear classifier head. Let ϕ(x)∈RD\phi(x) \in\mathbb{R}^{D} denote the penultimate feature vector; the logits are z(x)=Wϕ(x)+bz(x) = W\phi(x) + b, with W∈RC×DW \in\mathbb{R}^{C \times D} and b∈RCb \in\mathbb{R}^{C} the classifier weights and biases. The predicted distribution is p(x)=Softmax⁡(z(x))p(x) = \operatorname{Softmax}(z(x)), and Maximum Softmax Probability (MSP), the canonical failure-prediction baseline, defines confidence as the highest class probability, sMSP(x)=max⁡cpc(x)s_{\mathrm{MSP}}(x) = \max_{c} p_c(x). Because sMSPs_{\mathrm{MSP}} is derived solely from the CC-dimensional logit vector z(x)z(x), it collapses the DD-dimensional feature geometry—a structural bottleneck we formally analyze in § 4.

Evaluation Protocol: Isolating Ranking Quality. The Risk–Coverage curve plots selective risk (the error rate on the accepted subset) against coverage as the threshold τ\tau varies; AURC is its integral. Raw AURC is confounded by base accuracy—a more accurate model makes fewer errors and so scores a lower AURC regardless of ranking quality, and the oracle optimum AURC∗\mathrm{AURC}^{*} is fixed entirely by the base error rate. To isolate ranking quality we follow the fd-shifts protocol [19] and adopt the Excess AURC, e-AURC = AURC–AURC∗\mathrm{AURC}^{*} [9] (reported ×103\times10^{3}), as our primary metric: zero for a perfect predictor, lower is better, and comparable across backbones of differing accuracy. As a secondary, gaming-resistant evidence we report AUGRC [38] (P(accept∩wrong)P(\mathrm{accept} \cap\mathrm{wrong}), which abstaining on easy samples cannot inflate); AUROC serves only as a rank-sanity check.

What Complements MSP? A Factorial Analysis of the Feature Residual

To build a detector that complements MSP without degrading baseline performance, we first isolate the exact information discarded by the classifier head, and then measure which of that discarded signal actually survives fusion. Existing feature-space methods entangle several design choices over a chosen feature, specifically the projection subspace (basis), the distance metric, and the reference population (conditioning). This entanglement makes it impossible to attribute their failures to any single factor. We deconstruct this space through a controlled factorial experiment (Figure 2). The orthogonal blind spot (§ 4.1) and the conditional-redundancy principle (§ 4.2) establish exactly why the feature residual is the correct geometric object. Following this, we determine how to evaluate it along the three fundamental axes of distance, basis, and conditioning.

Three-panel bar charts showing standalone factorial effects, absorbed conditioning effects under fusion, and ImageNet-A fused gains

Figure 2. The residual factorial and its fate under fusion. (a) Standalone main effects on the ViT-B/16 penultimate residual (validation AUROC): conditioning strictly dominates (Δ5.6\Delta5.6 pp), while distance (Δ1.5\Delta1.5 pp) and basis (Δ2.1\Delta2.1 pp) are secondary. Marginals exclude the AE to prevent its Mahalanobis degeneracy from inflating the secondary axes. (b) Fused with MSP, this large standalone advantage collapses to ≈0.2\approx0.2 pp across backbones (conditional redundancy). (c) Per-class conditioning remains essential for robustness: on natural-adversarial ImageNet-A its fused gain stays positive on every backbone while global weakens and reverses (ViT-B/16).

The Orthogonal Blind Spot of Linear Heads

Every logit-based score factors through z(x)=Wϕ(x)+bz(x) = W\phi(x) + b, and MSP further reduces zz to the scalar φ(z)=max⁡cSoftmax⁡(z)c\varphi(z) = \max_c \operatorname{Softmax}(z)_c. By the chain rule, ∇ϕsMSP=WT∇zφ\nabla_{\phi}s_{\mathrm{MSP}} = W^{\mathsf{T}}\nabla_z\varphi spans a single direction in RD\mathbb{R}^{D}. This scalar collapse leaves MSP first-order blind to the (D−1)(D-1)-dimensional hyperplane orthogonal to that gradient—on ViT-B/16, 767 of the model’s 768 feature dimensions. Since the gradient direction varies with the input, this statement is local. It therefore supplies only a necessary condition: a complementary signal must vary transverse to MSP’s gradient. Which transverse component actually survives fusion is an empirical question, answered by the factorial analysis (§ 4.2–§ 4.5). This limitation persists even when the head has full column rank in feature space (here D=768<C=1000D=768 < C=1000, so WW can span all of RD\mathbb{R}^{D}): a full-rank head guarantees that the logit map discards nothing, but MSP’s scalar collapse of the logits still does.

The Conditional-Redundancy Principle

Cross-entropy training is expected to concentrate class-discriminative variation in the directions the head reads; if so, a distance computed in the full DD-dimensional space (Mahalanobis [23], Deep kNN [36]) is dominated by class-separability geometry—precisely what MSP already encodes. Conditioning is by far the dominant standalone axis (Δ5.6\Delta5.6 pp AUROC; Figure 2a), dwarfing distance (Δ1.5\Delta1.5 pp) and basis (Δ2.1\Delta2.1 pp). But standalone performance is the wrong criterion for a detector deployed in fusion with MSP. Once fused, the conditioning advantage collapses to ≈0.2\approx0.2 pp on every backbone (Figure 2b): its variance is shared with MSP, which already encodes the class separability that conditioning exploits. We call this the conditional-redundancy principle: class-geometry signals are redundant with MSP under fusion. A detector escapes redundancy only by projecting away the principal subspace and scoring the residual r(x)r(x).

Distance: The Necessity of a Non-Parametric Estimator

Beyond isolating r(x)r(x), the scoring function must accurately reflect the residual’s underlying distribution. Existing methods like ViM and Mahalanobis summarize the residual with parametric quantities (such as an L2L_2 norm or a shared-covariance Gaussian), implicitly assuming a zero-mean, near-isotropic density. The validity of this assumption is strongly backbone-dependent (Figure 3; Appendix § 8). For instance, the per-class residuals of ViT-B/16 and DINOv2 remain close to Gaussian (median excess kurtosis 0.12 and 0.24), whereas those of Swin-B and the CNNs are heavily leptokurtic (median 1.0–2.1).

Figure 3. Residual non-Gaussianity is backbone-dependent. Median per-class excess kurtosis (xx) vs. bimodal-class fraction (yy), one point per backbone. The kurtosis axis separates the near-Gaussian and heavy-tailed regimes; bimodality does not. Per-class densities in Appendix § 8.

Consequently, no single parametric density transfers across these structural regimes. The standard shared-covariance assumption under-fits near-Gaussian residuals by ignoring class-specific variations. Conversely, modeling a full per-class covariance over-fits the heavy-tailed regimes (Swin-B and the CNNs), performing strictly worse than the shared-covariance baseline. The optimal parametric choice fluctuates with the backbone’s geometry, and incorporating mixture components fails to close the performance gap (Appendix § 9). A non-parametric kNN estimator sidesteps this dilemma entirely. By measuring the distance to the kk nearest training-set neighbors, it reproduces the best parametric density on the near-Gaussian backbones (within ∼0.01 AUROC) and is the single best estimator where tails are heavy. No parametric family is uniformly best, whereas kNN is uniformly near-best.

Basis: PCA and the Coordinate Problem

Under the deployed per-class kNN estimator, the choice of basis is nearly inconsequential. PCA, AE, and VAE [21] all perform within a 0.2 pp margin, with the VAE’s nominal 0.1 pp advantage falling entirely within crossfit seed noise (Figure 4). Because kNN evaluates local geometry directly, it is inherently coordinate-free, making the specific factorization of the residual largely irrelevant. The same holds for a cosine-kernel basis: replacing PCA with the cosine-kernel PCA of [6] moves CANARY’s fused AUROC by at most 0.0007 across four backbones (Appendix § 17).

Figure 4. Basis is inconsequential under kNN.

We therefore adopt PCA. Training-free, deterministic, and parameter-free, it matches the best performance at zero added computational cost. The single exception is the overcomplete TopK-SAE [8], trailing by ∼2 pp. Its dynamic, per-input atom selection projects same-class residuals into varying coordinate systems. This inconsistency dissolves the cohesive per-class clusters that kNN relies on to measure typicality (Appendix § 10).

Conditioning: Class-Conditional vs. Global Typicality

The last axis is the reference population. A global residual score asks whether a feature is atypical of the entire training set, whereas a class-conditional score asks whether it is atypical of its predicted class y^\hat{y}. For synthetic, near-isotropic corruptions a global score suffices, but structured natural shifts such as ImageNet-A and ImageNet-R induce class-dependent off-manifold variation, and under fusion the two scores diverge sharply (Figure 2c). There a global reference, blind to the predicted class, is actively misled exactly where per-class typicality still tracks error. Most of the standalone conditioning gap is redundant with MSP and collapses under fusion (§ 4.2), but this robustness residual is not, and it is systematic. Across backbones the per-class residual variance shows a moderate positive correlation with the per-class failure rate (Spearman 0.44–0.64; Appendix § 11). An error is best read as a violation of the predicted class’s off-manifold structure, exposing the limits of global atypicality under shift.

Summary. Each competing design fails under fusion for its own reason: a full-space distance duplicates MSP, a parametric norm mismodels the residual’s distribution, and a global reference conditions on the wrong population. What survives all three tests—a non-parametric, class-conditional typicality on the orthogonal residual—is the blueprint for the method we detail in § 5.

Method

Our proposed method, CANARY, is the minimal instantiation of the blueprint distilled in § 4, coupling the resulting residual score to MSP through a single validation-tuned weight. Let ϕ(x)∈RD\phi(x) \in\mathbb{R}^{D} denote the penultimate feature of a frozen classifier, and y^(x)\hat{y}(x) its predicted class. CANARY fits strictly on the ID-correct training samples, D✓={(xi,yi)∣y^(xi)=yi}\mathcal{D}^{\checkmark} = \{(x_i, y_i) \mid\hat{y}(x_i) = y_i\}, to capture the typical geometry of correctly classified features.

Low-Rank PCA Residual

We compute PCA on the D✓\mathcal{D}^{\checkmark} features. Let μ∈RD\mu\in\mathbb{R}^{D} be the mean and VK∈RD×KV_K \in\mathbb{R}^{D \times K} the top-KK eigenvectors. The residual is the projection of the centered feature onto the orthogonal complement:

r(x)=(I−VKVK⊤)(ϕ(x)−μ).(1)r(x) = \left(I - V_K V_K^{\top}\right)\left(\phi(x) - \mu\right). \tag*{(1)}

This projection isolates the feature variation left unexplained by the class-separating directions, the redundant component MSP already captures. We fix K=512K = 512. The residual’s KK-dependent variation is precisely that redundant class geometry. Fusion therefore absorbs it, rendering the fused score insensitive to KK on every backbone, even where the residual’s overall gain is large (Appendix § 12).

Per-Class kNN Distance

For each class cc, we group the D✓\mathcal{D}^{\checkmark} residuals into a reference set Rc={r(xi)∣(xi,yi)∈D✓, yi=c}\mathcal{R}_c = \{r(x_i) \mid(x_i, y_i) \in\mathcal{D}^{\checkmark},\ y_i = c\}. At inference, we score r(x)r(x) by its Euclidean distance

d(x)=∥r(x)−r(k)∥2d(x) = \left\|r(x) - r^{(k)}\right\|_2

to its kk-th nearest neighbor r(k)r^{(k)} (k=100k = 100) in the predicted class’s reference set Ry^(x)\mathcal{R}_{\hat{y}(x)}. These reference sets merely store the training residuals, requiring no covariance fitting or auxiliary training.

Percentile Calibration

Because the kNN distance d(x)d(x) is unbounded and backbone-dependent, while MSP operates on an unrelated scale, the two signals must be placed on a common footing prior to fusion. Passing a score through the empirical CDF of its reference distribution performs the probability integral transform, mapping the score to a pp-value that is uniform on [0,1][0, 1] for typical inputs. Let V\mathcal{V} denote the set of correctly classified validation inputs:

sr(x)=1−F^(d(x)),F^(t)=1∣V∣∑xj∈V1[d(xj)≤t],(2)s_r(x) = 1 - \widehat{F}\bigl(d(x)\bigr), \qquad\widehat{F}(t) = \frac{1}{|\mathcal{V}|} \sum_{x_j \in\mathcal{V}} \mathds{1}\bigl[d(x_j) \le t\bigr], \tag*{(2)}

where 1[⋅]\mathbb{1}[\cdot] denotes the indicator function (1 if the condition holds, 0 otherwise). A higher srs_r thus marks a more typical, likely-correct input.

We calibrate MSP identically against its own validation-correct CDF to obtain sms_m. Both signals are now pp-values computed against a fixed validation reference, reflecting precisely what a deployed system can calculate without access to test labels. These standardized pp-values provide well-calibrated inputs for a Fisher-inspired log-linear combination [7].

Weighted-Fisher Fusion

Percentile calibration yields two pp-values per input: sm(x)s_m(x) from MSP and sr(x)s_r(x) from the residual. A Fisher-style combination multiplies the two pp-values (equivalently, sums their logs). We apply a weighted formulation in the logarithmic domain:

S(x;w)=wln⁡sm(x)+(1−w)ln⁡sr(x).(3)S(x; w) = w \ln s_m(x) + (1 - w) \ln s_r(x). \tag*{(3)}

At w=12w = \frac{1}{2}, this recovers the canonical Fisher combination (sm(x)sr(x)s_m(x)s_r(x)), which assumes equally informative signals. However, our signals are strongly asymmetric. By the conditional-redundancy principle, MSP already captures the residual’s class-relevant signal. Fusion absorbs this redundancy, isolating only the residual’s unique off-manifold component. MSP thus supplies the dominant evidence, prioritized through the validation-tuned weight ww.

We tune ww via cross-fitting on the clean validation set. Specifically, we split the set in half, maximizing AUROC on one and evaluating on the other. We repeat this across five random seeds and average the results. Because the fusion relies on a single scalar weight, this low-capacity formulation mitigates over-adaptation to the validation splits. The optimal weight w∗w^* is then frozen for all subsequent shift evaluations. We treat this as a log-linear pooling rule rather than a hypothesis test: since sms_m and srs_r are dependent (§ 4.2), we neither invoke nor rely on the test’s independence-based null distribution; ww is tuned empirically and the score is used only for ranking.

Experiments

Setup

Datasets. We study failure prediction on ImageNet-1k [3, 33], where the objective is to robustly separate a model’s correct predictions from its own misclassifications under distribution shift. ImageNet-1k validation (50k) serves as the clean, in-distribution reference. We evaluate robustness across three distribution-shift families: ImageNet-C [15] (15 corruptions ×\times 5 severities), ImageNet-A [17] (natural adversarials), and ImageNet-R [14] (renditions). ObjectNet [1] (an independently collected, controlled real-world covariate shift) is held out as a further generalization test. To test transfer to a different label space, we additionally evaluate on CIFAR-100 [22] and its corrupted variant CIFAR-100-C [15].

Backbones. To establish architectural generality, we evaluate five main backbones: ViT-B/16 [5] as the primary architecture, along with ResNet-50 [12], Swin-B [27], ConvNeXt-B [28], and EfficientNet-B4 [37]. We further probe representational generality using the self-supervised DINOv2 [30]. To isolate the drivers of CANARY’s gain within a single architecture family, we include three additional ViT configurations: one differently fine-tuned ViT-B/16 that lowers baseline MSP quality while keeping the architecture, size, and pretraining data constant; and two scaled variants (ViT-S/16 and ViT-L/16) that shift MSP headroom through capacity alone, with ViT-L/16 approaching the accuracy ceiling. Finally, the CIFAR-100 experiments employ the standard OpenOOD v1.5 [43] ResNet-18 and a WideResNet-28-10 [42] trained with a standard recipe.

For every backbone, we fix CANARY’s basis rank K=512K=512 (K=D−128K=D-128 for the narrower 384-d ViT-S/16 and 512-d CIFAR ResNet-18) and neighbor count k=100k=100, and cross-fit the single fusion weight ww strictly on that backbone’s own clean validation split. Full details on the pretrained checkpoints and the per-backbone optimum w⋆w^\star are provided in Appendix § 13.

Baselines and metrics. CANARY is compared against 19 post-hoc baselines spanning logit-based, activation-shaping, feature-based, and softmax-combination families. The primary metric is e-AURC, the accuracy-corrected excess area under the risk–coverage curve, which ensures fair comparability across backbones of differing base accuracies; AUGRC and AUROC serve as secondary metrics.

Experimental Results

CANARY ranks first across backbones and metrics. On the main ImageNet-1k backbones, CANARY outperforms all 19 baselines on e-AURC, AUGRC, and AUROC, the sole exception being a within-seed-noise near-tie with SIRC (Residual) on ResNet-50 (Appendix § 15). Table 1 highlights three backbones spanning the MSP-headroom range, with full-panel comparisons deferred to Appendix § 14. The performance margin over MSP follows the headroom law: the gap peaks where the baseline MSP is weakest and narrows toward parity as MSP improves. Consequently, absolute e-AURC reductions vary by over an order of magnitude, from 9.5 (×103\times10^3) on the low-MSP EfficientNet-B4 to a tighter 0.4 on the high-MSP ViT-B/16. The standalone feature-based detectors trail by a wide margin, even the strongest of them (TrustScore or fDBD), by 0.02–0.03 AUROC; the logit scores (MaxLogit, Energy, GEN) trail throughout. The two SIRC variants isolate the signal question within a fixed combination function: SIRC (Residual) (an off-manifold residual norm) beats SIRC (L1) (a global feature norm) on every backbone, independent evidence that the complementary signal is off-manifold rather than global magnitude—the choice CANARY sharpens further with a per-class residual, surpassing both.

MethodViT-B/16 e-AURC ↓ViT-B/16 AUGRC ↓ViT-B/16 AUROC ↑ConvNeXt-B e-AURC ↓ConvNeXt-B AUGRC ↓ConvNeXt-B AUROC ↑EfficientNet-B4 e-AURC ↓EfficientNet-B4 AUGRC ↓EfficientNet-B4 AUROC ↑
Logit-based
MSP [16]58.298.20.853761.496.10.853187.5155.30.8318
MaxLogit [13]84.7108.60.801689.7104.80.8083123.0165.20.7854
Energy [25]97.6114.40.7739112.4114.40.7601182.4189.10.6746
Doctor [10]58.998.70.851461.796.30.852088.8155.90.8286
GEN [26]85.3109.00.799379.0103.40.8149103.6161.40.8027
Activation-shaping
ASH [4]159.4138.50.6568108.6114.50.7593236.0210.40.5777
ReAct [34]99.4115.10.770791.6108.50.7901175.1191.60.6637
DICE [35]104.7117.30.7601136.0124.70.7089252.4217.30.5455
Feature-based
ViM [39]128.8128.50.706994.0111.10.7761123.3171.50.7538
Deep kNN [36]270.5173.60.4939247.1160.50.5251193.8186.60.6837
MDS [23]320.1189.10.423177.6104.90.8070109.7166.90.7756
MDS++ [29]302.9182.10.455980.3106.00.8021104.1164.90.7857
RMDS [32]260.2171.80.503970.5102.10.821895.8161.30.8021
RMDS++ [29]163.7138.80.655974.7104.20.8121104.6165.60.7826
NNGuide [31]89.0110.90.790178.5104.70.8085108.9165.70.7808
fDBD [24]76.8105.80.815067.5100.40.830095.1159.60.8106
TrustScore [20]69.3104.00.826262.799.70.835789.6160.60.8073
Softmax-combination
SIRC (L1) [40]65.4102.10.836262.796.40.851589.7156.00.8285
SIRC (Residual) [40]60.199.70.847159.396.00.853383.1155.10.8317
CANARY (ours)57.898.00.854556.995.10.857678.0153.30.8406

Table 1. Failure prediction across three backbones spanning the MSP-headroom range (low→high). Mean over the 77 distribution-shift cells (ImageNet-C 15×5, ImageNet-A, ImageNet-R). e-AURC and AUGRC are reported ×10310^{3}. Best in bold, second and third _underlined_.

CANARY is the only detector never worse than MSP. By degrading to MSP when the residual provides no new information, CANARY matches or exceeds the baseline in 382 of 385 shift cells on e-AURC (379 on AUROC). Table 2 compares this robustness against all baselines that achieve the top rank in at least one cell. While CANARY is the single best detector in 308 of 385 cells by e-AURC, no competitor matches its reliability. Its nearest rival here, the combination baseline SIRC (Residual), tops 35 cells but is not never-worse (worst-cell −0.039 AUROC, −15.0 e-AURC); the standalone feature detectors fare worse still—fDBD, RMDS, and CANARY’s closest sibling TrustScore collapse far below MSP on backbones whose geometry does not suit them, with worst-case e-AURC regressions of 25.9, 352.8, and 46.1 (×10310^{3}), respectively, against CANARY’s 0.1. By anchoring the residual to MSP, CANARY keeps its worst-case regression bounded to the seed-noise floor, gracefully trading marginal peak performance on a few cells rather than collapsing on any.

Methode-AURC (×103\times10^{3})AUROC
Top-1 Cells≥\geq MSP CellsMax DropTop-1 Cells≥\geq MSP CellsMax Drop
MSP3——3——
SIRC (Residual)35289−15.056215−0.0394
fDBD1653−25.91540−0.0662
TrustScore16130−46.11317−0.1334
Doctor464−17.0459−0.0318
RMDS234−352.8415−0.4378
ASH11−249.701−0.3340
GEN012−45.8111−0.1154
CANARY (ours)308382−0.1289379−0.0002

Table 2. Table 2: Reliability across the five main backbones (385 shift cells). “Top-1 Cells” counts cells where the method is the single best of the 20 detectors; “≥\geq MSP Cells” counts cells where the method matches or exceeds MSP; Max Drop is its largest regression relative to MSP. Only detectors that top at least one cell are listed. CANARY dominates the win-count and is the only detector with a bounded worst case; SIRC (Residual)’s top-1 cells are concentrated on ResNet-50, where it is a near-tie with CANARY within seed noise (Appendix § 15).

The headroom law across architectures. The dependence of the performance margin on MSP quality is a consistent cross-architecture trend rather than a per-model artifact. Figure 1 plots the fusion AUROC gain against each backbone’s clean-validation MSP AUROC. The backbones align along a single negatively sloped trend, and the optimal fusion weight w∗w^{*} tracks this available headroom, decreasing monotonically from 0.95 on high-MSP backbones to 0.70 on the weaker EfficientNet-B4. This trend holds across both synthetic (ImageNet-C) and natural (ImageNet-A/R) shifts, extends to the self-supervised DINOv2, and is mirrored on the held-out ObjectNet (Appendix § 16).

Crucially, this law holds even within a single architecture. By fixing the ViT-B/16 structure and ImageNet-21k pretraining while varying only the fine-tuning strategy (augreg with MSP 0.8783 vs. augreg2 with 0.8651), the variant with the weaker baseline MSP achieves substantially larger gains (+0.42 vs. +0.08 pp AUROC across shifts, and +1.04 vs. ≈0\approx0 on ObjectNet). This isolates MSP quality as the primary driver of the performance margin, free from architectural confounds.

Generality Across Regimes

CIFAR-100: the never-worse property transfers to a different label space. We evaluate CANARY on CIFAR-100 to test whether its never-worse property holds in a genuinely different label space (100 classes, disjoint from ImageNet). To this end, we use two CIFAR-native models (the standard OpenOOD v1.5 ResNet-18 and a WideResNet-28-10 trained with a standard recipe) that are entirely independent of ImageNet on both the data and pretraining axes. CANARY stays at or above MSP on 136 of 150 cells across the two backbones, with the 14 exceptions on WRN-28-10 all falling within a −0.0004 AUROC margin that represents effective ties rather than true losses. The mean gains are small because both are CNN backbones whose off-manifold residual is nearly uninformative (standalone residual AUROC ≈0.55\approx0.55–0.570.57 vs. ≈0.75\approx0.75 on ViT features). Consequently, the fusion weight sits at the MSP anchor (w⋆=0.9w^{\star} = 0.9–0.950.95), consistent with the headroom law. While the boundary-based detector fDBD achieves a higher shift-mean on these cleanly separated features, it lacks the never-worse property across the ImageNet backbones, dropping by up to 0.066 AUROC relative to MSP (Table 2). CANARY trades a marginally lower peak for an empirically bounded worst case across every setting we evaluate.

AxisBackboneMSPCANARY (Ours)Best Baseline (AUROC↑\uparrow)
AUROC↑\uparrowAUROC↑\uparrow≥\geq MSP CellsMax Drop
Label SpaceCIFAR-100 ResNet-180.79150.795975/75+0.0005†^{\dagger}fDBD (0.8007)
CIFAR-100 WRN-28-100.78710.788261/75−0.0004-0.0004fDBD (0.7996)
RepresentationDINOv2 (self-supervised)0.85190.853877/77+0.0010†^{\dagger}SIRC (L1) (0.8550)

Table 3. Generality across label spaces and representations. We evaluate CANARY beyond the standard ImageNet setting on a different label space (CIFAR-100) and a self-supervised backbone (DINOv2). “≥\geq MSP Cells” denotes the number of shift cells where CANARY matches or outperforms MSP. †\dagger indicates that CANARY outperforms MSP on every cell. AUROC values represent the mean across all shift cells.

DINOv2: the never-worse property extends to a self-supervised representation. On DINOv2, a representation learned entirely without class labels, CANARY still improves over MSP, never regresses on any shift cell, and its gain grows monotonically with corruption severity. The fact that the off-manifold residual remains informative on a representation trained without supervision indicates that the mechanism is intrinsic to the feature geometry, rather than an artifact of label-supervised training. Although SIRC (L1) attains a marginally higher shift-mean on DINOv2 (0.8550 vs. 0.8538), it trails CANARY on all five main backbones.

Model scale: the never-worse property holds across the ViT family. We further vary model capacity along the ViT hierarchy, specifically evaluating ViT-S/16 and ViT-L/16, the smaller and larger siblings of the primary ViT-B/16. On the smaller ViT-S, the residual continues to complement MSP, and CANARY maintains strict baseline dominance across every shift cell. On the larger ViT-L, where the baseline MSP approaches the accuracy ceiling, the residual is less consistent in adding novel information: while the mean gain persists, CANARY marginally trails MSP on 11 of 77 cells, though all such cases fall within seed noise (worst case −0.0005-0.0005 AUROC). Across scales, CANARY never falls meaningfully below MSP, while the number of cells at or above MSP decreases as available headroom shrinks (77 →\to 74 →\to 66; Appendix § 18).

Computational Cost

CANARY adds negligible overhead to existing MSP pipelines. Fitting is training-and backpropagation-free: given the extracted features, building the PCA basis and per-class reference set requires ≈\approx1.9 TFLOP, and the resulting reference set occupies just ≈\approx0.6 GB for ImageNet-1k on ViT-B/16. At inference, a 768→\to256 projection and a single L2L_2 distance add merely ≈\approx1.3 MFLOP per query. On an NVIDIA GeForce RTX 4080, this translates to an overhead of ≈\approx50 μ\mus per unbatched query, averaging under 2 μ\mus batched.

Conclusion

This work elucidates the mechanisms governing MSP-complementary failure prediction: the conditional-redundancy principle shows that the class-relevant geometry of a frozen feature space is absorbed by MSP under fusion, so only the orthogonal, off-manifold component contributes novel information. CANARY operationalizes this principle, measuring class-conditional non-parametric typicality on the PCA residual and fusing it with MSP via a single validation-tuned weight. Evaluated under a strictly deployment-realistic protocol without retraining or OOD data exposure, CANARY delivers its largest gains where MSP is weakest, closely following the headroom law, while degrading gracefully toward MSP where the baseline is already strong. Crucially, it remains empirically never-worse, not regressing meaningfully below the baseline MSP across hundreds of distribution shifts, diverse backbone architectures, and even a different label space.

Limitations. We identify two primary limitations. First, while our findings are robustly validated across diverse shift sets and backbones and grounded in a geometric rationale, they remain empirical. We have yet to provide a rigorous mathematical formalization regarding the joint distribution of the fused scores. The headroom law, in particular, is an empirical regularity established over the nine backbones and shift families studied here rather than an analytical derivation, and its exact functional form may differ beyond this regime. Second, CANARY is structurally blind to purely on-manifold confident failures, where a shifted input is genuinely typical of an incorrect class and leaves no off-manifold trace. However, this represents only a subset of hard natural shifts. On ImageNet-A, for instance, CANARY still improves upon MSP because many challenging samples retain an off-manifold component; only the strictly on-manifold remainder lies beyond its reach.

Future work. Subsequent research should aim to formalize the conditions under which the empirical never-worse property holds, and to derive an analytical bound for the headroom law, currently established as an empirical forecast. Methodologically, closing the on-manifold gap requires integrating external signals (e.g., boundary-distance or raw-input heuristics). Fusing these with CANARY’s off-manifold typicality offers a promising path toward comprehensive failure prediction without surrendering its never-worse reliability. Finally, expanding CANARY to dense prediction tasks (e.g., per-pixel typicality), adversarial settings, and high-stakes domains (e.g., medical imaging) remains an important direction.

Acknowledgments This work was supported by the National Research Foundation of Korea (NRF) grant (No. RS-2026-25490405) and by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant (No. RS-2024-00428780), both funded by the Korea government (MSIT).

Non-Gaussianity of the Residual Across All Backbones

Section 4 summarized the non-Gaussianity regimes (Figure 3); this section provides the full per-backbone picture. On the same per-class PCA residual that the detector deploys, we measure, per class, heavy-tailedness (pooled per-dimension excess kurtosis κ\kappa; 0 = Gaussian) and multi-modality (Hartigan’s dip test on the dominant within-class direction; bimodal if p<0.05p < 0.05).

Per-backbone densities. For each backbone, Figure 5 plots the standardized residual density against a fitted Gaussian on a logarithmic axis. To provide a transparent view of the within-backbone spread, we pair the most non-Gaussian class with the median (typical) class. Swin-B and the CNNs (ConvNeXt-B, ResNet-50, and EfficientNet-B4) exhibit heavy tails even at the median. The high-MSP ViT-B/16 remains near-Gaussian throughout. While the extreme class of the self-supervised DINOv2 is explicitly bimodal, its typical class remains near-Gaussian, aligning with its minority 24% bimodal fraction.

Figure 5. Per-backbone residual non-Gaussianity. We pair each backbone’s most non-Gaussian class (left) with its median (typical) class (right). Each panel plots the standardized residual density (solid) against a fitted Gaussian (dashed) on a logarithmic yy-axis, annotated with its regime and excess kurtosis κ\kappa.

Regime summary. Figure 3 in the main text maps the median per-class excess kurtosis against the bimodal-class fraction, separating the backbones into distinct tail regimes. Swin-B and the CNNs occupy the heavy-tailed corner, while ViT-B/16 and DINOv2 are near-Gaussian. Table 4 summarizes the measured statistics, highlighting that DINOv2 carries the highest bimodal fraction despite being predominantly near-Gaussian.

BackboneBimodal % (Dip p<0.05p < 0.05)Median Excess KurtosisDominant Regime
ViT-B/1610.8%0.12Near-Gaussian
Swin-B22.0%2.07Heavy-tailed
ConvNeXt-B24.0%1.09Heavy-tailed
ResNet-505.4%1.04Heavy-tailed
EfficientNet-B48.1%1.10Heavy-tailed
DINOv224.4%0.24Near-Gaussian, most bimodal classes

Table 4. Per-class residual statistics and geometric regimes across backbones.

Density-Model Ablation

This section expands on Section 4.3. The tail heaviness of the per-class residual is backbone-dependent, so no single parametric family transfers across architectures. A non-parametric kNN, by contrast, adapts robustly to each.

The best parametric density flips with the regime. Table 5 reports the standalone residual-only AUROC for different estimators, using the deployed per-class conditioning and k=100k = 100 for the kNN.

EstimatorViT-B/16ConvNeXt-BSwin-BResNet-50EfficientNet-B4DINOv2
Median excess kurtosis0.121.092.071.041.100.24
Mahalanobis (shared Σ\Sigma)0.7280.7660.7540.7120.7550.707
Diagonal GMM (best M≤32M \le32)0.7400.7640.7520.7150.7570.725
Full per-class Σ\Sigma Gaussian0.7480.7460.7400.7030.7490.735
Per-class kNN (k=100k = 100)0.7470.7710.7600.7220.7580.729

Table 5. Standalone AUROC across parametric and non-parametric estimators.

On the near-Gaussian backbones (ViT-B/16, DINOv2), the full per-class covariance Gaussian is the strongest parametric density and closely matches kNN, while the shared-covariance Mahalanobis of prior work trails it by failing to capture class-specific covariances. This ordering inverts on the heavy-tailed backbones (Swin-B and the CNNs), where the full covariance over-fits the tails and drops below the shared-covariance baseline. The kNN estimator is uniformly competitive—matching the best parametric density on the near-Gaussian backbones (within ∼0.01\sim0.01 AUROC) and leading on the heavy-tailed ones.

Multimodality is not the primary lever. Sweeping the number of mixture components in a per-class diagonal GMM does not reach kNN performance on any backbone, and full-covariance GMMs with M>1M > 1 consistently degrade the result. This holds even for DINOv2, the most bimodal backbone (24% of classes flagged by dip test). Because its typical class remains near-Gaussian (median excess kurtosis 0.24; median dip p=0.82p = 0.82), fitting extra modes merely over-fits the unimodal majority. The residual’s departure from Gaussianity is primarily a heavy-tail phenomenon, not a mode-count issue.

kNN robustness. The kNN estimator requires no regime-specific tuning or covariance estimation. It consistently matches or exceeds the shared-covariance Mahalanobis baseline, providing an AUROC gain of 0.003 to 0.022. Against the best parametric density on each backbone, it leads on every heavy-tailed backbone and stays within 0.006 AUROC on the near-Gaussian ones.

Why the TopK SAE basis underperforms

Section 4.4 evaluates the choice of basis under the deployed per-class kNN estimator. This section provides a finer geometric perspective underlying those results.

The SAE residual does not occupy a fixed subspace. Instead, its overcomplete code selects a different set of active neurons per input. Consequently, same-class residuals lack a shared coordinate system, loosening the per-class clusters that kNN relies on. Because the code is trained strictly to reconstruct, its leftover is dominated by hard-to-reconstruct, instance-specific details.

Empirically, reconstruction quality and failure-prediction performance do not align. PCA at a bottleneck of K=256K = 256 leaves 27.8% of the feature variance unexplained (0.2775). The overcomplete TopK SAE (latent dimension 6144, sparsity 32) leaves a smaller 26.2% (0.2623) unexplained. However, under the per-class kNN, the SAE yields an AUROC of 0.7264, falling strictly behind PCA’s 0.7662—a ∼\sim4 pp gap at this reconstruction-matched K=256K = 256; against the deployed K=512K = 512 PCA (AUROC 0.747) the SAE still trails by the ∼\sim2 pp quoted in Section 4.4. A smaller reconstruction residual thus does not buy a better failure-prediction score. This confirms that the cause is fundamentally geometric rather than a matter of residual size. Any fixed basis recovers the off-manifold geometry equally well, making PCA the natural, training-free choice.

Figure 6. The SAE underperformance is geometric. On ViT-B/16, despite reconstructing slightly better than PCA, the overcomplete TopK SAE scores worse under the per-class kNN.

Per-Class Residual Variance Predicts Failure

The advantage of the per-class conditioning discussed in Section 4.5 is systematic. Specifically, a class whose in-distribution (ID) correct residuals are more dispersed is inherently more failure-prone. For each class cc, we measure the per-class residual total variance over its ID-correct training residuals:

tr⁡(Σc)=1∣Cc∣∑i∈Cc∥ri−μc∥2,(4)\operatorname{tr}(\Sigma_c) = \frac{1}{|\mathcal{C}_c|} \sum_{i \in\mathcal{C}_c} \lVert r_i - \mu_c \rVert^2, \tag*{(4)}

where CcC_c is the set of ID-correct training samples for class cc, and μc\mu_c is their residual mean. Alongside this, we calculate the per-class validation failure rate, defined as failc=1−acc(val∣true class c)\mathrm{fail}_c = 1 - \mathrm{acc}(\mathrm{val}\mid\mathrm{true\ class}\ c). We then compute the Spearman rank correlation between the residual variances {tr⁡(Σc)}\{\operatorname{tr}(\Sigma_c)\} and the failure rates {failc}\{\mathrm{fail}_c\} across all 1,000 classes for each backbone.

BackboneSpearman ρ\rho (Variance vs. Failure)
ViT-B/160.55
ConvNeXt-B0.64
Swin-B0.61
ResNet-500.49
EfficientNet-B40.61
DINOv20.44

Table 6. Spearman rank correlation between per-class residual variance and failure rate.

The correlation is consistently positive and moderate on every backbone, ranging from 0.44 to 0.64. Consequently, the per-class residual spread systematically tracks where errors are most likely to occur. A global reference inherently discards this crucial class-specific failure information by pooling all classes into a single distribution.

Robustness to the Basis Rank KK

CANARY is KK-insensitive across backbones. Sweeping KK at matched fractions of the feature dimension DD and evaluating across all shift families, the fused AUROC of CANARY varies by at most 0.0021 over K/D∈[0,0.9]K/D \in[0, 0.9]—even on the 2048-d ResNet-50, where K=512K = 512 leaves the bottom three-quarters of the spectrum as the residual (Figure 7a). This is not a vanishing-gain artifact: on the large-gain Swin-B (§ 6.2), the score moves by under 0.001 across K∈{0,…,512}K \in\{0, \ldots, 512\}.

Figure 7. Robustness to the PCA basis rank KK. (a) On the five main backbones, the CANARY score is robust to the PCA rank KK (varying by ≤0.0021\le0.0021). (b) A global ViM-style residual norm is strongly KK-dependent. The yy-axis is the mean AUROC over the full 77 shift cells.

Why fusion absorbs the KK-dependence. A small KK retains the class-mean directions in the residual—the class-separating axes the classifier head reads, and thus already encoded by MSP (§ 4.2). Conditioned on MSP, these carry no additional information, so fusion discards them regardless of KK. This robustness is specific to CANARY’s per-class position statistic: a global ViM-style norm depends on the subspace dimension and swings by 0.014–0.151 across the same backbones (Figure 7b).

We fix K=512K = 512 at no performance cost. Because the projection is orthogonal, the trailing (D−K)(D-K)-dimensional residual preserves the full-DD distances, so the reference set is stored and searched in that reduced space (256-d for ViT-B).

Narrower backbones (D≤512D \le512) instead set K=D−128K = D - 128, retaining a minimal 128-d residual (ViT-S/16, CIFAR ResNet-18).

Backbone Checkpoints and Hyperparameters

Every backbone is a standard pretrained checkpoint, evaluated frozen. Table 7 details each checkpoint (a timm tag for the ImageNet-1k backbones), its clean top-1 accuracy, the MSP validation AUROC that anchors the headroom law, its penultimate feature dimension DD, the basis rank KK, and the cross-fitted fusion weight w⋆w^\star. The neighbor count (k=100k = 100) and the basis rank are not tuned per backbone but fixed by a single rule, K=min⁡(512,D−128)K = \min(512, D - 128): cap the basis at 512 principal components while always retaining at least a 128-dimensional residual.

BackboneCheckpointTop-1 (%)MSP AUROCDDKKw⋆w^\star
ImageNet-1k
ViT-B/16 (main)vit_base_patch16_224.augreg_in21k_ft_in1k84.50.87837685120.95
ViT-B/16 (within-arch control)vit_base_patch16_224.augreg2_in21k_ft_in1k85.10.86517685120.80
ViT-S/16vit_small_patch16_224.augreg_in21k_ft_in1k81.40.87643842560.95
ViT-L/16vit_large_patch16_224.augreg_in21k_ft_in1k85.80.883710245120.95
ResNet-50resnet50.a1_in1k80.40.875720485120.95
Swin-Bswin_base_patch4_window7_224.ms_in22k_ft_in1k85.30.860710245120.80
ConvNeXt-Bconvnext_base.fb_in22k_ft_in1k85.80.871410245120.80
EfficientNet-B4efficientnet_b4.ra2_in1k82.70.848217925120.70
DINOv2vit_base_patch14_dinov2.lvd142m83.80.87377685120.95
CIFAR-100
ResNet-18Pre-trained on OpenOOD v1.577.30.87525123840.90
WideResNet-28-10Standard training recipe81.70.88526405120.95

Table 7. Backbone checkpoints and CANARY hyperparameters. K=512K = 512 for every backbone with D≥640D \ge640, and K=256/384K = 256/384 on the 384-d ViT-S / 512-d CIFAR ResNet-18 (each leaving a 128-d residual).

Full Experimental Results

Main Backbones

Tables 8–13 report all 20 detectors on the five main backbones. ImageNet-C is summarized by its four corruption groups [15]; ImageNet-A/R are the full sets. e-AURC is reported ×103\times10^{3}. Best detector is in bold and the second/third underlined. Each table’s bottom row (Δ\Delta vs. MSP) is CANARY−MSP (↑\uparrow = CANARY improves over MSP, ↓\downarrow = worse).

MethodNoise e-AURC↓Noise AUROC↑Blur e-AURC↓Blur AUROC↑Weather e-AURC↓Weather AUROC↑Digital e-AURC↓Digital AUROC↑ImageNet-A e-AURC↓ImageNet-A AUROC↑ImageNet-R e-AURC↓ImageNet-R AUROC↑
Logit-based
MSP53.90.857564.40.853150.50.859851.20.8551235.70.6842114.60.8272
MaxLogit79.60.807489.30.809178.20.803879.90.7944265.60.6443117.00.8204
Energy92.70.7804102.60.782390.20.776993.20.7642280.70.6142126.70.8039
Doctor54.70.855065.10.850951.10.857752.10.8524236.50.6821113.90.8283
GEN82.40.801689.30.808378.60.801680.00.7925261.70.6503116.80.8227
Activation-shaping
ASH144.60.6857169.20.6683150.00.6575158.30.6279339.60.5252216.50.6850
ReAct96.80.7733104.00.779992.00.773493.30.7636281.30.6125131.30.7975
DICE99.30.7685110.20.768296.50.7635100.40.7493290.80.5952141.50.7825
Feature-based
ViM127.10.7075140.60.7090119.00.7093116.60.7070288.80.5938211.70.6935
Deep kNN268.50.4921312.80.4770264.80.4877232.80.5103359.70.4809333.00.5341
MDS352.00.3708359.50.4097298.40.4427268.20.4575369.10.4677478.40.3530
MDS++319.10.4331338.00.4415286.30.4667264.50.4745358.50.4932403.70.4587
RMDS288.40.4600289.20.4995236.40.5259220.00.5250351.60.4897445.80.4038
RMDS++168.60.6444164.80.6788148.10.6675157.70.6412354.30.4905304.20.5959
NNGuide85.20.791993.70.799382.00.792983.10.7838277.20.6272122.50.8085
fDBD74.80.814282.90.819570.60.817870.00.8122230.80.699294.10.8559
TrustScore66.70.826476.40.826460.70.832760.30.8294241.40.6776147.90.7758
Softmax-combination
SIRC (L1)54.50.855470.30.837355.20.845767.70.8197236.20.6806115.90.8224
SIRC (Residual)61.10.836665.90.847950.90.857851.50.8524233.40.6880114.70.8243
CANARY (ours)53.80.857764.10.853650.10.860650.50.8565234.50.6865113.50.8289
(Δ\Delta vs. MSP)0.2↑0.0002↑0.3↑0.0005↑0.4↑0.0009↑0.7↑0.0014↑1.3↑0.0023↑1.1↑0.0017↑

Table 8. ViT-B/16.

BackboneMethodNoise e-AURC↓Noise AUROC↑Blur e-AURC↓Blur AUROC↑Weather e-AURC↓Weather AUROC↑Digital e-AURC↓Digital AUROC↑ImageNet-A e-AURC↓ImageNet-A AUROC↑ImageNet-R e-AURC↓ImageNet-R AUROC↑
ViT-S/16MSP67.60.852981.00.842665.00.851262.70.8492209.70.6556126.70.8196
ViT-S/16CANARY67.30.853380.40.843464.20.852461.90.8505208.70.6577122.70.8243
ViT-B/16MSP53.90.857564.40.853150.50.859851.20.8551235.70.6842114.60.8272
ViT-B/16CANARY53.80.857764.10.853650.10.860650.50.8565234.50.6865113.50.8289
ViT-L/16MSP41.70.866351.60.861743.10.861148.00.8463206.60.7236105.10.8325
ViT-L/16CANARY41.90.866151.30.862442.90.861947.20.8486204.50.7262103.60.8347

Table 9. ViT model scale: MSP vs. CANARY. ViT-S/16, ViT-B/16, and ViT-L/16 on ImageNet-C by corruption group (cell-means) and ImageNet-A/R. e-AURC ×103\times10^{3} (↓\downarrow), AUROC (↑\uparrow); better of MSP/CANARY in bold. CANARY is at or above MSP on almost every cell, the gains shrinking to seed-noise ties on ViT-L (−0.0002 AUROC on its Noise group).

MethodNoise e-AURC↓Noise AUROC↑Blur e-AURC↓Blur AUROC↑Weather e-AURC↓Weather AUROC↑Digital e-AURC↓Digital AUROC↑ImageNet-A e-AURC↓ImageNet-A AUROC↑ImageNet-R e-AURC↓ImageNet-R AUROC↑
Logit-based
MSP79.00.851595.30.828173.90.849375.40.8448129.80.5538126.90.8244
MaxLogit83.60.8431101.80.816981.70.836280.60.8339130.30.5404137.30.8094
Energy93.70.8133113.00.782688.10.818387.70.8128131.80.5165143.40.7912
Doctor78.80.851895.40.827773.70.849875.20.8450129.50.5583126.20.8262
GEN83.60.8418100.10.818177.10.841680.10.8342128.10.5677124.60.8290
Activation-shaping
ASH132.80.7611155.70.7304140.50.7371132.90.7403126.40.5628208.90.7177
ReAct490.30.2407466.20.2497487.40.2380468.60.2512147.20.4089501.80.2325
DICE96.00.8079116.20.775090.20.813589.70.8078131.30.5223143.50.7907
Feature-based
ViM225.20.6230220.30.6423195.60.6539204.80.6380134.40.5047194.10.7153
Deep kNN212.40.6517232.20.6230221.90.6208213.90.6261140.10.4637260.60.6247
MDS146.70.7401156.70.7296130.40.7507138.70.7361146.30.4112151.30.7542
MDS++125.50.7718142.60.7437119.90.7621121.30.7580149.40.3769173.20.7191
RMDS127.20.7629143.30.7386113.90.7684116.80.7600148.00.3876160.50.7462
RMDS++117.80.7799135.10.7510110.80.7755108.30.7750149.60.3648182.30.7110
NNGuide93.60.8155113.40.782188.50.817888.10.8126131.70.5161149.50.7877
fDBD89.20.8322100.10.820979.50.835583.30.8283129.30.5518107.70.8374
TrustScore106.80.7978126.20.767894.80.8038101.50.7882142.80.4204150.10.7803
Softmax-combination
SIRC (L1)79.10.851296.10.825474.20.848875.60.8440129.80.5546127.10.8243
SIRC (Residual)79.20.849993.60.831572.50.852074.50.8466129.80.5537123.70.8289
CANARY (ours)78.60.851894.20.829672.70.850874.60.8458129.40.5562121.40.8306
(∆ vs. MSP)0.4↑0.0003↑1.1↑0.0015↑1.2↑0.0015↑0.7↑0.0011↑0.4↑0.0025↑5.5↑0.0062↑

Table 10. ResNet-50.

MethodNoise e-AURC↓Noise AUROC↑Blur e-AURC↓Blur AUROC↑Weather e-AURC↓Weather AUROC↑Digital e-AURC↓Digital AUROC↑ImageNet-A e-AURC↓ImageNet-A AUROC↑ImageNet-R e-AURC↓ImageNet-R AUROC↑
Logit-based
MSP65.90.843880.30.839170.10.819766.90.8286250.20.6685128.30.8197
MaxLogit109.20.7802121.50.7880106.00.7585109.90.7563292.50.6152179.20.7744
Energy163.10.6710171.90.6934159.20.6400157.70.6481332.40.5388223.30.7059
Doctor66.70.841381.00.837073.60.809968.70.8226252.30.6639128.20.8198
GEN86.60.800295.20.808598.40.755389.10.7740274.50.6393124.60.8194
Activation-shaping
ASH162.50.6784176.50.6902158.10.6454158.60.6555324.80.5592265.60.6609
ReAct116.10.7335129.00.7393123.40.6951110.20.7217317.50.5578150.90.7666
DICE206.40.5811216.60.6059195.10.5634191.00.5754362.70.4861281.60.6187
Feature-based
ViM106.40.7577111.60.7745105.60.736996.40.7535284.90.6217127.40.7930
Deep kNN141.00.7244135.60.7581133.30.7067133.30.7122281.60.6333119.00.8246
MDS88.50.785097.00.795088.20.769979.30.7853270.50.6406111.90.8137
MDS++87.50.790698.30.795292.50.761180.00.7861326.30.5434127.30.7797
RMDS78.00.808286.90.814673.10.800069.20.8101258.60.6598101.10.8306
RMDS++81.40.800993.60.800284.40.776773.80.7982334.40.5290134.60.7637
NNGuide87.00.791195.00.800890.20.766982.60.7802320.50.5717128.40.7852
fDBD78.10.807986.00.817078.90.792671.90.8032245.00.677197.70.8534
TrustScore65.00.828277.00.826058.00.830557.30.8315246.40.6770122.10.7992
Softmax-combination
SIRC (L1)67.60.841581.80.837471.60.817069.00.8246251.10.6667130.90.8174
SIRC (Residual)62.70.843475.10.839868.60.817359.80.8371248.00.6668110.00.8297
CANARY (ours)59.30.850771.80.846064.80.826258.10.8391244.00.6798102.40.8388
(∆ vs. MSP)6.7↑0.0068↑8.5↑0.0069↑5.2↑0.0065↑8.8↑0.0105↑6.2↑0.0113↑25.8↑0.0191↑

Table 11. Swin-B.

MethodNoise e-AURC↓Noise AUROC↑Blur e-AURC↓Blur AUROC↑Weather e-AURC↓Weather AUROC↑Digital e-AURC↓Digital AUROC↑ImageNet-A e-AURC↓ImageNet-A AUROC↑ImageNet-R e-AURC↓ImageNet-R AUROC↑
Logit-based
MSP55.50.857272.10.849050.70.863055.20.8527226.60.6982109.30.8373
MaxLogit85.30.809999.30.813277.90.817084.30.8003263.40.6616129.80.8209
Energy110.00.7578122.60.770098.70.7698107.40.7469287.30.6207143.60.8005
Doctor55.80.856172.50.847750.90.862255.50.8515227.30.6964108.80.8381
GEN75.30.816187.70.818667.70.824174.60.8072251.90.6699106.60.8375
Activation-shaping
ASH104.90.7595116.30.772895.50.7666105.60.7436279.00.6344157.40.7795
ReAct88.50.7900102.00.793878.70.801086.10.7820273.50.6296117.30.8198
DICE134.40.7030146.60.7241121.50.7175129.30.6945325.70.5671181.40.7525
Feature-based
ViM93.20.7717104.60.781881.60.784486.80.7708266.40.6386114.20.8098
Deep kNN146.60.7196145.90.7516134.40.7250145.70.7024263.90.6667115.90.8420
MDS75.60.804688.20.809366.90.814870.00.8043246.60.664494.50.8376
MDS++77.00.802490.90.804569.10.809772.00.8005278.00.6181106.70.8136
RMDS68.00.820380.80.822160.20.829562.70.8215240.50.674291.90.8406
RMDS++70.70.813885.50.812263.70.819965.70.8134282.40.6116110.50.8041
NNGuide74.90.809488.00.811466.50.818472.60.8032274.20.6376108.50.8200
fDBD65.10.828777.20.830858.00.837261.40.8265211.40.716078.70.8677
TrustScore58.30.837375.50.827252.00.845855.00.8385200.30.7344102.60.8294
Softmax-combination
SIRC (L1)56.80.855573.40.847751.90.861456.50.8509227.00.6977111.10.8361
SIRC (Residual)54.40.856070.00.847748.60.864152.50.8544225.70.6946100.20.8431
CANARY (ours)52.20.860367.20.852946.60.867550.60.8579215.00.711791.80.8519
(∆ vs. MSP)3.3↑0.0031↑4.8↑0.0039↑4.0↑0.0044↑4.5↑0.0051↑11.6↑0.0135↑17.5↑0.0146↑

Table 12. ConvNeXt-B.

MethodNoise e-AURC↓Noise AUROC↑Blur e-AURC↓Blur AUROC↑Weather e-AURC↓Weather AUROC↑Digital e-AURC↓Digital AUROC↑ImageNet-A e-AURC↓ImageNet-A AUROC↑ImageNet-R e-AURC↓ImageNet-R AUROC↑
Logit-based
MSP79.80.847397.50.826178.20.838882.80.8303230.60.6240140.40.8133
MaxLogit109.90.8097127.10.7931120.00.7813122.40.7739241.20.6005192.90.7655
Energy164.50.7126181.90.6989183.70.6577187.00.6466259.10.5418265.00.6513
Doctor80.70.845198.80.823079.10.836285.00.8251230.80.6232140.10.8139
GEN96.70.8184112.10.802893.10.8086103.10.7928234.40.6205124.80.8273
Activation-shaping
ASH223.70.6118238.50.6016243.80.5471225.50.5682292.10.4613365.70.5045
ReAct177.40.6703183.20.6789159.10.6667176.50.6437249.20.5698195.20.6938
DICE235.20.5854245.30.5874261.90.5103256.60.5135271.00.5037362.70.4957
Feature-based
ViM129.80.7467138.50.7410101.40.7789118.80.7530247.30.5837125.50.8111
Deep kNN133.30.7683138.30.7657125.30.7640136.70.7470237.60.6051115.50.8292
MDS114.20.7720124.70.763290.50.7965102.80.7778252.20.5731118.30.8184
MDS++104.20.7931118.30.778286.90.802196.20.7880281.50.4723137.90.7664
RMDS97.70.8041110.30.790379.10.819087.40.8069257.00.5602117.30.8177
RMDS++105.60.7888120.60.771286.30.801594.40.7895294.30.4443115.60.7422
NNGuide104.90.7946123.50.768192.30.7974104.00.7795273.40.5187139.80.7794
fDBD93.40.8194104.90.808183.20.817891.40.8059230.50.6371104.00.8510
TrustScore90.40.8105106.00.792573.60.822779.60.8137238.80.6148123.40.8091
Softmax-combination
SIRC (L1)81.20.845699.20.823980.50.835586.10.8244230.50.6235144.70.8090
SIRC (Residual)79.70.839895.40.818470.20.846477.10.8346230.70.6205118.90.8344
CANARY (ours)73.80.851889.40.832365.70.851273.30.8398225.10.6396108.90.8413
(∆ vs. MSP)6.0↑0.0045↑8.1↑0.0062↑12.5↑0.0124↑9.5↑0.0095↑5.5↑0.0156↑31.5↑0.0280↑

Table 13. EfficientNet-B4.

ObjectNet

Table 14 reports every method on the held-out ObjectNet set (16,825 images). CANARY matches or exceeds MSP on ObjectNet across all five backbones, and no baseline shares this never-worse property.

MethodViT-B/16ResNet-50Swin-BConvNeXt-BEfficientNet-B4
e-AURC↓AUROC↑e-AURC↓AUROC↑e-AURC↓AUROC↑e-AURC↓AUROC↑e-AURC↓AUROC↑
Logit-based
MSP103.20.8217120.10.8104134.10.7941106.50.8181129.30.8023
MaxLogit120.30.7970131.30.7947197.80.7343141.90.7859176.40.7540
Energy134.10.7717139.60.7723256.80.6434164.70.7503240.60.6474
Doctor102.80.8220119.50.8118134.70.7927106.00.8189130.00.8011
GEN116.80.8017122.10.8078144.60.7791113.00.8081129.20.8015
Activation-shaping
ASH224.90.6500256.10.6415295.80.5975165.70.7406323.50.5454
ReAct133.80.7728505.60.2601177.00.7117138.70.7687306.30.5428
DICE144.50.7561141.20.7687295.70.5752183.30.7190300.30.5505
Feature-based
ViM261.50.6087307.00.5709133.90.7767135.80.7593154.70.7611
Deep kNN371.40.4998466.00.3491251.20.6743300.80.5760219.40.7089
MDS122.50.8000162.80.7533114.10.802099.00.8171132.40.7853
MDS++130.60.7871159.20.7445115.80.801999.00.8179123.20.7939
RMDS108.00.8193133.50.7821101.10.822089.80.8306114.50.8080
RMDS++108.30.8180125.00.7943105.40.814290.80.8285117.80.7977
NNGuide137.10.7691136.50.7748133.80.7774106.60.8074132.10.7857
fDBD108.00.8166117.70.8131104.50.818194.70.8250113.50.8186
TrustScore126.70.7791141.70.7674112.60.7956109.30.7912133.30.7711
Softmax-combination
SIRC (L1)105.00.8166120.50.8095136.80.7915108.40.8166131.90.7996
SIRC (Residual)105.10.8166119.00.8123113.80.8128100.80.8213114.10.8176
CANARY (ours)103.20.8217118.40.8121113.30.808695.80.8258110.10.8197
(Δ vs. MSP)0.1↑0.00001.7↑0.0018↑20.8↑0.0146↑10.7↑0.0077↑19.2↑0.0174↑

Table 14. ObjectNet. Failure prediction on ObjectNet across the five main backbones. e-AURC ×10310^3; best per column in bold, 2nd/3rd underlined. The bottom row (Δ vs. MSP) is CANARY−MSP (↑ = CANARY improves over MSP, ↓ = worse).

DINOv2 (self-supervised)

Table 15 reports all 20 detectors on DINOv2 on ImageNet-C by corruption group and ImageNet-A/R. e-AURC is reported ×103\times10^3. Per column the best detector is in bold and the second/third _underlined_. The bottom row (Δ\Delta vs. MSP) is CANARY−-MSP (↑\uparrow = CANARY improves over MSP, ↓\downarrow = worse).

DINOv2 (self-supervised). Failure prediction on the self-supervised DINOv2 backbone: ImageNet-C by corruption group + ImageNet-A/R. Best per column bold, 2nd/3rd underlined.

MethodNoise e-AURC↓\downarrowNoise AUROC↑\uparrowBlur e-AURC↓\downarrowBlur AUROC↑\uparrowWeather e-AURC↓\downarrowWeather AUROC↑\uparrowDigital e-AURC↓\downarrowDigital AUROC↑\uparrowImageNet-A e-AURC↓\downarrowImageNet-A AUROC↑\uparrowImageNet-R e-AURC↓\downarrowImageNet-R AUROC↑\uparrow
Logit-based
MSP70.90.842670.40.848145.30.863247.90.8588204.90.7290104.40.8258
MaxLogit93.00.799099.20.799975.50.798377.20.7899237.00.6932114.70.8132
Energy100.40.7841108.90.781283.90.778885.30.7698249.50.6726123.10.7994
Doctor70.30.843770.50.847645.40.862548.00.8581205.30.7278104.00.8262
GEN92.30.797698.30.799374.70.797576.50.7892239.20.6900112.60.8148
Activation-shaping
ASH144.00.7002160.40.6913128.00.6831128.40.6731302.40.5954211.10.6793
ReAct101.80.7815111.20.776885.30.775286.80.7661251.30.6675125.20.7957
DICE112.00.7610122.40.756194.20.754596.20.7435274.30.6378156.90.7482
Feature-based
ViM143.80.6896163.80.6754124.90.6850123.80.6796313.80.5706197.70.6658
Deep kNN152.00.7090165.50.7080148.50.6783149.30.6646278.40.6291137.20.7828
MDS104.90.7646110.40.772584.40.777184.90.7703280.30.6126127.00.7711
MDS++93.80.7924100.20.794980.90.788481.20.7809256.80.6471104.20.8130
RMDS99.00.777898.60.794075.60.796076.10.7898276.90.6175119.20.7834
RMDS++94.90.788694.30.803574.10.800674.50.7942264.60.6384108.50.8052
NNGuide88.60.803096.20.802574.10.799576.30.7899247.10.6775110.00.8125
fDBD93.60.796397.70.800377.70.793078.10.7864211.40.722999.60.8399
TrustScore68.90.844863.20.859547.20.860749.20.8559199.80.7330105.70.8143
Softmax-combination
SIRC (L1)64.40.853068.70.848744.50.864146.60.8603200.40.734593.70.8443
SIRC (Residual)67.70.847570.70.845144.70.864146.90.8603205.60.7251102.90.8261
CANARY (ours)69.30.844768.80.850044.40.865046.90.8607203.10.7311101.20.8289
(Δ\Delta vs. MSP)1.6↑\uparrow0.0021↑\uparrow1.6↑\uparrow0.0019↑\uparrow0.9↑\uparrow0.0018↑\uparrow1.0↑\uparrow0.0018↑\uparrow1.9↑\uparrow0.0021↑\uparrow3.2↑\uparrow0.0031↑\uparrow

Table 15. DINOv2 (self-supervised). Failure prediction on the self-supervised DINOv2 backbone: ImageNet-C by corruption group + ImageNet-A/R. Best per column bold, 2nd/3rd underlined.

CIFAR-100

Tables 16–17 report all 20 detectors on CIFAR-100-C (a 100-class label space disjoint from ImageNet) for the two CIFAR-native backbones, by corruption group and overall Average. e-AURC is reported ×103\times10^3. Per column the best detector is in bold and the second/third _underlined_. Each table’s bottom row (Δ\Delta vs. MSP) is CANARY−-MSP (↑\uparrow = CANARY improves over MSP, ↓\downarrow = worse).

MethodNoiseBlurWeatherDigitalAverage
e-AURC↓AUROC↑e-AURC↓AUROC↑e-AURC↓AUROC↑e-AURC↓AUROC↑e-AURC↓AUROC↑
Logit-based
MSP161.30.7095123.40.785778.70.831790.10.8184110.20.7915
MaxLogit180.50.6878136.50.763794.50.8034106.10.7922126.00.7667
Energy184.90.6769141.40.752199.20.7918111.20.7803130.80.7552
Doctor161.60.7085123.60.784979.00.830590.30.8175110.50.7905
GEN183.30.6790139.90.754597.70.7943109.60.7828129.20.7576
Activation-shaping
ASH262.30.5693261.00.5994228.80.6178242.10.6165247.60.6028
ReAct185.80.6692150.00.7336110.50.7686125.00.7569140.00.7363
DICE184.10.6786140.70.753198.60.7928110.70.7810130.20.7562
Feature-based
ViM333.20.4439411.50.3527410.90.3141418.40.3324397.50.3552
Deep kNN340.10.3989427.40.3042438.10.2489443.80.2700417.20.2993
MDS219.30.6019206.20.6248182.80.6236186.60.6380197.30.6234
MDS++165.90.6736137.40.7476102.80.7777111.50.7738127.00.7478
RMDS170.60.6556128.20.765790.30.801697.70.7983118.40.7619
RMDS++163.60.6711122.20.777585.40.812194.50.8053113.30.7729
NNGuide173.70.6935129.70.772291.40.8055104.60.7919121.60.7706
fDBD139.60.7406118.00.793377.80.832588.40.8213103.70.8007
TrustScore148.70.7150122.70.776282.00.820094.30.8044109.50.7832
Softmax-combination
SIRC (L1)163.00.7045125.10.781779.50.829891.20.8157111.50.7881
SIRC (Residual)155.40.7158123.70.783879.00.830490.70.8163109.30.7913
CANARY (ours)150.30.7202120.60.789177.00.834087.90.8214106.20.7959
(∆ vs. MSP)11.0↑0.0107↑2.8↑0.0034↑1.7↑0.0023↑2.2↑0.0030↑4.0↑0.0045↑

Table 16. CIFAR-100 — ResNet-18.

MethodNoiseBlurWeatherDigitalAverage
e-AURC↓AUROC↑e-AURC↓AUROC↑e-AURC↓AUROC↑e-AURC↓AUROC↑e-AURC↓AUROC↑
Logit-based
MSP168.60.6746120.60.780868.20.841383.70.8237106.40.7871
MaxLogit181.50.6577126.10.769576.60.823093.00.8072115.10.7714
Energy184.60.6498128.60.762279.40.814896.30.7985118.10.7635
Doctor168.90.6736120.60.780468.40.840384.00.8228106.60.7863
GEN183.40.6514128.00.763578.60.816495.40.8001117.20.7649
Activation-shaping
ASH197.20.6407134.80.759080.10.8182100.20.7987123.50.7617
ReAct235.90.5708208.40.6136180.60.6170194.60.6177202.80.6071
DICE184.50.6499128.50.762479.30.814996.20.7986118.00.7635
Feature-based
ViM252.80.5912317.40.4693293.40.4524331.80.4362301.90.4803
Deep kNN337.10.3902426.10.2727435.20.2015450.30.2212417.20.2635
MDS139.40.7044115.50.784567.10.838080.60.823698.10.7932
MDS++144.90.6976114.00.786865.40.842979.30.827598.00.7948
RMDS158.70.6685116.60.781267.30.837881.70.8221102.60.7847
RMDS++148.50.6899114.90.785665.40.843079.30.827498.90.7929
NNGuide173.50.6696126.00.777076.10.824691.90.8064113.10.7750
fDBD145.20.7094116.90.789866.00.846279.50.830299.00.7996
TrustScore148.70.6870118.00.779166.50.819180.40.8233100.40.7885
Softmax-combination
SIRC (L1)170.20.6700121.70.777968.80.839684.80.8211107.40.7843
SIRC (Residual)154.20.7031119.60.781467.20.843182.90.8251102.80.7938
CANARY (ours)164.80.6781119.70.781567.60.841883.40.8239105.20.7882
(∆ vs. MSP)3.8↑0.0035↑0.8↑0.0007↑0.6↑0.0006↑0.3↑0.0002↑1.2↑0.0011↑

Table 17. CIFAR-100 — WideResNet-28-10.

SIRC per-backbone behavior

SIRC (Residual)’s strength is concentrated rather than broad. Almost all of its top-1 cells—31 of 35 (e-AURC) and 51 of 56 (AUROC)—fall on ResNet-50 alone, where it and CANARY tie within seed noise on both metrics. On the other four backbones it tops at most four cells.

BackboneMSP e-AURC ↓MSP AUROC ↑MSP Top-1 CellsCANARY e-AURC ↓CANARY AUROC ↑CANARY Top-1 CellsSIRC (Residual) e-AURC ↓SIRC (Residual) AUROC ↑SIRC (Residual) Top-1 Cells
ViT-B/1658.20.85373 / 357.80.854567 / 6860.10.84714 / 2
ResNet-5082.30.83890 / 081.30.840133 / 1581.20.840731 / 51
Swin-B74.20.82980 / 066.60.837761 / 5969.70.83160 / 2
ConvNeXt-B61.40.85310 / 056.90.857674 / 7359.30.85330 / 1
EfficientNet-B487.50.83180 / 078.00.840673 / 7483.10.83170 / 0

Table 18. SIRC per-backbone behavior. “Top-1 Cells” = per-backbone single-best count of the 20 detectors (e-AURC / AUROC). Showing both metrics exposes the seed-noise-level gaps—on ResNet-50, SIRC (Residual) and CANARY tie on both, and even disagree on which metric each tops.

That concentration is the point: a global residual norm helps only where an architecture’s failure modes happen to align with it, whereas CANARY’s per-class residual isolates a signal that transfers—top-1 cells on every backbone and never regressing past the seed-noise floor.

The Headroom Law on ObjectNet

Figure 1 is shown in the body on the ImageNet-C/-A/-R panels. Figure 8 confirms it reproduces on ObjectNet on the same forecastable validation MSP xx-axis.

Figure 8. The headroom law on ObjectNet. CANARY’s fusion AUROC gain over MSP on ObjectNet, plotted against each backbone’s ImageNet-1k validation MSP AUROC. The same negative slope as the Figure 1 panels reproduces here.

Comparison with Kernel PCA

Kernel PCA (KPCA) [6] detects out-of-distribution inputs by measuring the reconstruction error of features projected onto a non-linear principal subspace. To compare it with CANARY, we evaluate KPCA in three distinct roles: as a standalone failure detector, fused with MSP, and as an alternative basis for CANARY’s residual. These evaluations span four diverse backbones: ViT-B/16, ResNet-50, Swin-B, and the self-supervised DINOv2. Following the original method, we implement its two explicit-feature-map kernels—cosine (CoP; ϕ(z)=z/∥z∥\phi(z) = z/\lVert z\rVert) and cosine-Gaussian (CoRP; using random Fourier features with M=4096M=4096). We fit PCA on the correctly classified in-distribution validation features, retaining components that explain 90% of the variance, and score each input by its reconstruction error. All reported results represent the mean AUROC across 77 shift cells, under the same out-of-sample protocol used in our main experiments.

KPCA as a failure detector. On its own, KPCA reconstruction error provides a weak failure signal (0.68–0.75 AUROC), falling well below the MSP baseline (Table 19). When fused with MSP using CANARY’s percentile calibration and weighted-Fisher formulation, KPCA regresses below MSP in 6 of 8 cases (by up to 0.027 AUROC) and only marginally exceeds it in two (CoP on Swin-B and DINOv2, by at most 0.0035). Furthermore, once fused, the non-linear CoRP kernel consistently trails CoP across all backbones. In contrast, CANARY improves upon MSP across all four backbones, with DINOv2 being the sole instance where CoP+MSP attains a slightly higher mean than CANARY (0.8553 vs. 0.8538). Since both methods share an identical fusion rule, this discrepancy isolates the quality of the underlying signal: a class-agnostic, whole-feature reconstruction error is largely redundant with MSP (§ 4.2). Consequently, its contribution under fusion is small and inconsistent in sign across backbones.

BackboneMSPCoPCoRPCoP + MSPCoRP + MSPCANARY (ours)
ViT-B/160.85370.69450.71580.84920.83050.8545
ResNet-500.83890.67940.68330.83850.81240.8401
Swin-B0.82980.72650.75450.83330.82750.8377
DINOv20.85190.70860.73970.85530.84860.8538

Table 19. KPCA reconstruction error as a failure detector. Evaluated standalone and fused with MSP via CANARY’s percentile calibration and weighted-Fisher formulation. Mean AUROC over 77 shift cells; best per row in bold.

KPCA’s native fusion rule. Following [11], KPCA [6] multiplicatively combines the reconstruction error with a logit score via (1−e~)⋅slogit(1-\tilde{e}) \cdot s_{\mathrm{logit}}, where e~\tilde{e} is the in-distribution percentile of the error. When applied to MSP, this rule substantially degrades performance, dropping AUROC by 5.6–10.6 pp on every backbone (Table 20); its Energy-based variant similarly falls 6.8–14.8 pp below MSP. We attribute this to the rule’s design: built to separate in-distribution from OOD inputs, the multiplicative factor rescales MSP by a class-agnostic error, thereby perturbing the within-distribution confidence ranking on which failure prediction relies.

Table 20: KPCA’s multiplicative fusion. The rule of [6, 11] applied to MSP, compared against CANARY. Mean AUROC across 77 shift cells; best per column in bold. e~\tilde{e} denotes the in-distribution percentile of the reconstruction error.

MethodViT-B/16ResNet-50Swin-BDINOv2
MSP0.85370.83890.82980.8519
(1−e~)⋅MSP(1-\tilde{e})\cdot\mathrm{MSP}, CoP0.75210.75800.76610.7734
(1−e~)⋅MSP(1-\tilde{e})\cdot\mathrm{MSP}, CoRP0.74800.74050.77360.7758
CANARY (ours)0.85450.84010.83770.8538

Table 20.

KPCA as CANARY’s basis. To evaluate KPCA as a representation basis, we replace CANARY’s linear PCA basis with the cosine-kernel basis (CoP’s explicit feature map, i.e., PCA on ℓ2\ell_2-normalized features). Holding the per-class kNN, calibration, and fusion constant, this substitution changes CANARY’s AUROC by at most 0.0007 (Table 21). The kernel’s non-linearity thus yields no tangible benefit, consistent with kNN’s reliance on local neighborhood geometry rather than on a particular choice of coordinates. This result extends the basis invariance of § 4.4 (PCA, AE, and VAE) to a cosine-kernel basis.

Table 21: CANARY with a cosine-kernel (CoP) basis. Mean AUROC over 77 shift cells.

BackboneLinear-PCA basisCoP basisΔ\Delta
ViT-B/160.85450.8543−0.0002-0.0002
ResNet-500.84010.8399−0.0002-0.0002
Swin-B0.83770.8370−0.0007-0.0007
DINOv20.85380.8541+0.0003+0.0003

Table 21.

Model scale: ViT-S/16 and ViT-L/16

ViT-S/16 and ViT-L/16 probe the model-scale axis of § 6.3. As the smaller and larger siblings of the primary ViT-B/16, they differ only in capacity. They sit outside the main tables, which span distinct architectures, and are therefore reported here. Capacity monotonically raises the clean-validation MSP quality, and CANARY’s behavior tracks the resulting headroom.

BackboneVal MSP AUROC↑\uparrowShift AUROC↑\uparrow MSPShift AUROC↑\uparrow CANARYCANARY vs. MSP Δ\Delta (pp)≥\geq MSP CellsMax Drop
ViT-S/160.87640.84580.8469+0.1177/77+0.0002†^{\dagger}
ViT-B/16 (main)0.87830.85370.8545+0.0874/77−0.0002
ViT-L/160.88370.85630.8573+0.1066/77−0.0005

Table 22. Performance across the ViT model scales. Shift AUROCs are the mean across all shift cells; Val MSP is the clean-validation AUROC. † indicates that CANARY outperforms MSP on every shift cell.

On the smaller ViT-S, the residual carries a complementary failure signal, allowing CANARY to improve over MSP on all 77 shift cells, thereby maintaining strict baseline dominance at a lower capacity. On ViT-L, capacity pushes MSP to the accuracy ceiling; the per-class kNN residual is already correlated with MSP on validation (Pearson +0.52), and its standalone AUROC (0.7523, comparable to ViT-B’s 0.7465) confirms that the pipeline is sound. Rather, the residual is less consistent in providing novel information beyond what MSP already captures. Consequently, while ViT-L’s mean improvement (+0.10 pp) is comparable to that of ViT-S, the gains are no longer uniform; CANARY marginally trails MSP on 11 of 77 cells, but the worst-case drop (−0.0005 AUROC) falls within seed noise, rendering it an effective tie.

References

  1. [1]Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., Katz, B.: ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In: Adv. Neural Inform. Process. Syst. (2019)
  2. [2]Corbière, C., Thome, N., Bar-Hen, A., Cord, M., Pérez, P.: Addressing failure prediction by learning model confidence. In: Adv. Neural Inform. Process. Syst. (2019)arxiv.org/abs/1910.04851
  3. [3]Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A large-scale hierarchical image database. pp. 248–255 (2009)
  4. [4]Djurisic, A., Bozanic, N., Ashok, A., Liu, R.: Extremely simple activation shaping for out-of-distribution detection. In: Int. Conf. Learn. Represent. (2023)arxiv.org/abs/2209.09858
  5. [5]Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: Int. Conf. Learn. Represent. (2021)
  6. [6]Fang, K., Tao, Q., Lv, K., He, M., Huang, X., Yang, J.: Kernel PCA for out-of-distribution detection. In: Adv. Neural Inform. Process. Syst. pp. 134317–134344 (2024)
  7. [7]Fisher, R.A.: Statistical Methods for Research Workers. Oliver and Boyd, Edinburgh (1925)
  8. [8]Gao, L., Dupré la Tour, T., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., Wu, J.,: Scaling and evaluating sparse autoencoders. In: Int. Conf. Learn. Represent. (2025)arxiv.org/abs/2406.04093
  9. [9]Geifman, Y., Uziel, G., El-Yaniv, R.: Bias-reduced uncertainty estimation for deep neural classifiers. In: Int. Conf. Learn. Represent. (2019)arxiv.org/abs/1805.08206
  10. [10]Granese, F., Romanelli, M., Gorla, D., Palamidessi, C., Piantanida, P.: DOCTOR: A simple method for detecting misclassification errors. In: Adv. Neural Inform. Process. Syst. pp. 5669–5681 (2021)
  11. [11]Guan, X., Liu, Z., Zheng, W.S., Zhou, Y., Wang, R.: Revisit PCA-based technique for out-of-distribution detection. In: Int. Conf. Comput. Vis. pp. 19431–19439 (2023)DOI
  12. [12]He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. pp. 770–778 (2016)arxiv.org/abs/1512.03385
  13. [13]Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., Song, D.: Scaling out-of-distribution detection for real-world settings. In: Int. Conf. Mach. Learn. pp. 8759–8773 (2022)
  14. [14]Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., Gilmer, J.: The many faces of robustness: A critical analysis of out-of-distribution generalization. In: Int. Conf. Comput. Vis. pp. 8340–8349 (2021)
  15. [15]Hendrycks, D., Dietterich, T.: Benchmarking neural network robustness to common corruptions and perturbations. In: Int. Conf. Learn. Represent. (2019)
  16. [16]Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: Conf. Learn. Represent. (2017)
  17. [17]Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., Song, D.: Natural adversarial examples. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 15262–15271 (2021)
  18. [18]Huang, R., Geng, A., Li, Y.: On the importance of gradients for detecting distributional shifts in the wild. In: Adv. Neural Inform. Process. Syst. pp. 677–689 (2021)arxiv.org/abs/2110.00218
  19. [19]Jaeger, P.F., Lüth, C.T., Klein, L., Bungert, T.J.: A call to reflect on evaluation practices for failure detection in image classification. In: Int. Conf. Learn. Represent. (2023)arxiv.org/abs/2211.15259
  20. [20]Jiang, H., Kim, B., Guan, M., Gupta, M.: To trust or not to trust a classifier. In: Adv. Neural Inform. Process. Syst. (2018)arxiv.org/abs/1805.11783
  21. [21]Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: Int. Conf. Learn. Represent. (2014)arxiv.org/abs/1312.6114
  22. [22]Krizhevsky, A.: Learning multiple layers of features from tiny images. Tech. rep., University of Toronto (2009)
  23. [23]Lee, K., Lee, K., Lee, H., Shin, J.: A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In: Adv. Neural Inform. Process. Syst. (2018)arxiv.org/abs/1807.03888
  24. [24]Liu, L., Qin, Y.: Fast decision boundary based out-of-distribution detector. In: Int. Conf. Mach. Learn. pp. 31728–31746 (2024)arxiv.org/abs/2312.11536
  25. [25]Liu, W., Wang, X., Owens, J., Li, Y.: Energy-based out-of-distribution detection. In: Adv. Neural Inform. Process. Syst. pp. 21464–21475 (2020)arxiv.org/abs/2010.03759
  26. [26]Liu, X., Lochman, Y., Zach, C.: GEN: Pushing the limits of softmax-based out-of-distribution detection. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 23946–23955 (2023)DOI
  27. [27]Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin Transformer: Hierarchical vision transformer using shifted windows. In: Int. Conf. Comput. Vis. pp. 10012–10022 (2021)arxiv.org/abs/2103.14030
  28. [28]Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A ConvNet for the 2020s. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 11976–11986 (2022)
  29. [29]Müller, M., Hein, M.: Mahalanobis++: Improving OOD detection via feature normalization. In: Int. Conf. Mach. Learn. pp. 45151–45184 (2025)
  30. [30]Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jégou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: DINOv2: Learning robust visual features without supervision. Trans. Mach. Learn Res. (2024)arxiv.org/abs/2304.07193
  31. [31]Park, J., Jung, Y.G., Teoh, A.B.J.: Nearest neighbor guidance for out-of-distribution detection. In: Int. Conf. Comput. Vis. pp. 1686–1695 (2023)arxiv.org/abs/2309.14888
  32. [32]Ren, J., Fort, S., Liu, J., Roy, A.G., Pad hy, S., Lakshminarayanan, B.: A simple fix to Mahalanobis distance for improving near-OOD detection. arXiv preprint arXiv:2106.09022 (2021)arxiv.org/abs/2106.09022
  33. [33]Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
  34. [34]Sun, Y., Guo, C., Li, Y.: ReAct: Out-of-distribution detection with rectified activations. In: Adv. Neural Inform. Process. Syst. pp. 144–157 (2021)arxiv.org/abs/2111.12797
  35. [35]Sun, Y., Li, Y.: DICE: Leveraging sparsification for out-of-distribution detection. In: Eur. Conf. Comput. Vis. pp. 691–708 (2022)arxiv.org/abs/2111.09805
  36. [36]Sun, Y., Ming, Y., Zhu, X., Li, Y.: Out-of-distribution detection with deep nearest neighbors. In: Int. Conf. Mach. Learn. pp. 20827–20840 (2022)arxiv.org/abs/2204.06507
  37. [37]Tan, M., Le, Q.: EfficientNet: Rethinking model scaling for convolutional neural networks. In: Int. Conf. Mach. Learn. pp. 6105–6114 (2019)arxiv.org/abs/1905.11946
  38. [38]Traub, J., Bungert, T.J., Lüth, C.T., Baumgartner, M., Maier-Hein, K.H., Maier-Hein, L., Jäger, P.F.: Overcoming common flaws in the evaluation of selective classification systems. In: Adv. Neural Inform. Process. Syst. pp. 2323–2347 (2024)arxiv.org/abs/2407.01032
  39. [39]Wang, H., Li, Z., Feng, L., Zhang, W.: ViM: Out-of-distribution with virtual-logit matching. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 4921–4930 (2022)arxiv.org/abs/2203.10807
  40. [40]Xia, G., Bouganis, C.S.: Augmenting softmax information for selective classification with out-of-distribution data. In: Asian Conf. Comput. Vis. pp. 664–680 (2023)arxiv.org/abs/2207.07506
  41. [41]Xia, G., Bouganis, C.S.: Augmenting the softmax with additional confidence scores for improved selective classification with out-of-distribution data. Int. J. Comput. Vis. 132(9), 3714–3752 (2024)DOI
  42. [42]Zagoruyko, S., Komodakis, N.: Wide residual networks. In: Brit. Mach. Vis. Conf. pp. 87.1–87.12 (2016)arxiv.org/abs/1605.07146
  43. [43]Zhang, J., Yang, J., Wang, P., Wang, H., Lin, Y., Zhang, H., Sun, Y., Du, X., Li, Y., Liu, Z., Chen, Y., Li, H.: OpenOOD v1.5: Enhanced benchmark for out-of-distribution detection 2(3), 1–32 (2024)arxiv.org/abs/2306.09301
  44. [44]Zhu, F., Cheng, Z., Zhang, X.Y., Liu, C.L.: OpenMix: Exploring outlier samples for misclassification detection. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 12074–12083 (2023)arxiv.org/abs/2303.17093

Paper details

vlnc:W_ah28azfa
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents