betaFeedback

WorksComputer Science

When Does an AUROC Comparison Survive a Change in Population Mix?

Abstract

Does an AUROC comparison survive a change in the proportions of tasks, populations, or image corruptions? A pooled-AUROC ranking of failure detectors or uncertainty scores describes one particular mix of groups. We study two fixed scores for a common binary target under mixtures of fixed group-conditional laws. Using the cross-group AUC decomposition, we prove that every rational symmetric quadratic comparison geometry is realizable by finite scalar-score distributions. Universal dominance is thus coNP-hard, and even exhaustive checks of every mixture on fewer than KK groups can miss a KK-group reversal. A grid bound gives a dimension-uniform approximation certificate with sharp 1/m1/m order. Common score marginals across groups make the comparison numerator linear, with an error envelope for imperfect alignment. These results yield exact finite-bank audits, explicit reversal mixtures and distances, and simultaneous population statements. For failure detection with a frozen ResNet-50 on ImageNet-C, 15 of 18 confidence-score comparisons win in every corruption family on a discovery split, and all 15 are finite-bank certified over all family mixtures on a held-out split; over all 15 corruption types, of 14 discovery winners the corrected grid certifies 10, leaves 3 unresolved, and finds one confirmation reversal of at least 0.05 AUROC. An audit of all 11,019 released Stable LM 2 12B outputs and 35 uncertainty methods finds 66 confirmation-set comparisons that win within all four tasks yet reverse under mixing. Discovery-fitted marginal rank alignment removes all 66; an alignment-error envelope certifies 83 comparisons. Label checks also change several selected reversals. The findings distinguish robustness of frozen scores from robustness after calibration; they do not establish human-error detection, robustness to unseen corruptions, or generalization beyond the evaluated banks.

This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

Introduction

An AUROC leaderboard answers a question about a particular population. Deployment can change the proportions of tasks, sites, user groups, or image corruptions while leaving each group’s joint distribution of labels and scores unchanged. A practical question is then: does the observed ordering of two scores persist throughout the allowed changes in that mixture?

The question arises directly in failure detection and uncertainty estimation for vision models. A confidence score for an image classifier, such as the maximum softmax probability, is judged by how well it ranks misclassified images above correctly classified ones. On a corruption benchmark such as ImageNet-C [7], the pooled AUROC of such a score is computed over a fixed mix of corruption types and severities, and a deployed camera does not see that mix. The same holds for uncertainty scores of language models evaluated on a fixed mix of tasks.

This question differs from reporting a high overall AUROC or a favorable AUROC in every group. AUROC compares a positive example to an independently drawn negative example, so cross-group comparisons contribute to the pooled metric. The xAUC decomposition already makes this explicit [13]. Recent work further studies AUC non-collapsibility, cluster decomposition, and failure-mixture effects [3, 19, 26]. Our starting point is this established decomposition. Our question is a universal comparison: what evidence suffices to certify an ordering over a continuum of deployment mixtures, and what can simpler checks miss?

We give two answers that materially constrain such audits. First, any rational symmetric quadratic form, after a positive rescaling, is the AUROC difference of two realizable scalar scores under group-mixture shift, even when every group has positive-label probability 1/21/2. Thus the apparent difficulty of a quadratic comparison is not an artifact of allowing matrices that no scores could produce: universal dominance is coNP-hard. Second, for every m<Km < K, two scores can be ordered correctly by every mixture supported on at most mm groups, yet reverse on the uniform mixture of all KK groups. In this construction, one score has perfect AUROC inside every group. Neither within-group checks nor exhaustive low-order mixture checks establish universal dominance.

We then build MIXAUDIT, an audit for a specified mixture set. It returns a valid lower bound, an explicit reversing mixture when found, and an unresolved status when the statistical or computational evidence is insufficient. The audit uses exact rational optimization for small group counts and simultaneous paired U-statistic intervals for population statements. The statistical intervals remain valid when the mixture is selected using the same evaluation data. We also quantify how far a reference mixture can move before a specified reversal becomes possible. A corrected finite grid gives a dimension-independent approximation bound for the AUROC gap with optimal 1/m1/m order. Aligning each method’s score marginals across groups yields a tractable comparison, with an explicit bound for departures from exact alignment.

The theory does not assert that a reversal is common or that an arbitrary mixture describes a realistic deployment. We evaluate the audit in two domains. In vision, we audit four confidence scores of a frozen ResNet-50 for its top-1 errors on ImageNet-C, with corruption families (K=4K = 4) or all 15 corruption types (K=15K = 15) as groups. In language, we audit a released Stable LM 2 12B score bank containing 11,019 question-answering outputs and 35 uncertainty methods, with all 595 pairs included. Exact results for these finite banks are distinguished from population confidence statements. Figure 1 summarizes the problem, the audit, and the main results.

Overview of comparison under group-mixture shift and MIXAUDIT

Figure 1. Overview. (a) Pooled AUROC ranks two scores AA and BB for one target (e.g., two failure detectors for one classifier) on one mixture of groups such as tasks or corruption types. We ask whether the ordering holds for every mixture in a declared set, not for one target population or a generic drift ball. (b) In the realizable construction of Theorem 3 (c=17/24c = 17/24), AA wins in every group and on every two-group mixture, yet BB wins on the shaded disc. (c) MIXAUDIT returns a certificate, an explicit reversing mixture with its distance, or “unresolved”. (d) Finite-bank results on held-out data: marginal rank alignment removes all 66 language-model reversals (Table 3); vision results are in Table 2.

Scope and related work. The quadratic AUC decomposition, concentration tools, and global-optimization primitives are established tools. We contribute the scalar-score realizability reduction, the hierarchy of insufficient mixture checks, a sharp ratio-specific grid bound, and a comparison identity with an error bound for marginal alignment. Their integration yields a reproducible comparison audit. We do not propose a new uncertainty estimator or failure detector, or claim a first certificate for shifted AUROC. Weber et al. [28] already certify AUC under Hellinger drift; Li et al. [14] study transport to a specified target population; distributionally robust AUC training addresses a different optimization problem [5, 17]. Our structure is a fixed finite collection of group distributions with uncertain mixture weights, which permits sharp finite-bank conclusions and exposes the limits of subgroup-only validation. Groupwise quantile maps and transport-based post-processing already serve fairness and ranking objectives [15, 22]. Covariate-adjusted ROC analysis has long used negative-class placement values [10]. We study the universal comparison induced by fixed score representations, including the consequences of applying those established transformations. Our aligned identity and approximation envelope describe when within-group comparisons suffice; they do not prescribe a fairness objective or replace a deployment-specific utility analysis.

Comparison Under Group-Mixture Shift

Let G∈{1,…,K}G \in\{1,\ldots,K\} denote a prespecified group and let Z=(Y,SA,SB)Z = (Y,S_A,S_B), where Y∈{0,1}Y \in\{0,1\} is one common binary target and larger scores indicate Y=1Y = 1. The two scores can be classifiers for the same outcome or uncertainty estimators for the same base model’s errors, such as failure detectors for one image classifier. Comparing error detectors attached to different base models generally changes YY and is outside this formulation.

The conditional laws Pg=L(Z∣G=g)P_g = \mathcal{L}(Z \mid G = g) are fixed. A deployment mixture q∈ΔK={q≥0:1Tq=1}q \in\Delta_K = \{q \ge0 : \mathbf{1}^{\mathsf{T}}q = 1\} defines Pq=∑gqgPgP_q = \sum_g q_g P_g. Write πg=Pg(Y=1)\pi_g = P_g(Y = 1) and p(q)=πTqp(q) = \pi^{\mathsf{T}}q. We assume 0<πg<10 < \pi_g < 1 for the universal simplex statements; more generally restrict QQ to mixtures with 0<p(q)<10 < p(q) < 1. No conclusion covers within-group distribution shift unless it is included separately in the audit.

Let ϕ(s,t)=1{s>t}+121{s=t}\phi(s,t) = \mathbb{1}\{s > t\} + \frac{1}{2}\mathbb{1}\{s = t\}, with independent draws for the positive and negative observations. The cross-group comparison difference is

dgh=E[ϕ(SAg,SAh)−ϕ(SBg,SBh)∣Yg=1,Yh=0].(1)d_{gh} = \mathbb{E}\left[\phi(S_A^g,S_A^h) - \phi(S_B^g,S_B^h) \mid Y_g = 1, Y_h = 0\right]. \tag*{(1)}

Define symmetric matrices

B=sym⁡(diag⁡(π) d diag⁡(1−π)),(2)B = \operatorname{sym}\left(\operatorname{diag}(\pi)\,d\,\operatorname{diag}(1-\pi)\right), \tag*{(2)}
C=12(π1T+1πT)−ππT.(3)C = \frac{1}{2}\left(\pi\mathbf{1}^{\mathsf{T}} + \mathbf{1}\pi^{\mathsf{T}}\right) - \pi\pi^{\mathsf{T}}. \tag*{(3)}

Here sym⁡(M)=(M+MT)/2\operatorname{sym}(M) = (M + M^{\mathsf{T}})/2. The familiar cross-group decomposition gives the following useful representation.

Proposition 1 (Comparison identity). For any admissible mixture,

Δ(q):=AUC⁡A(q)−AUC⁡B(q)=qTBqqTCq,(4)\Delta(q) := \operatorname{AUC}_A(q) - \operatorname{AUC}_B(q) = \frac{q^{\mathsf{T}}Bq}{q^{\mathsf{T}}Cq}, \tag*{(4)}

qTCq=p(q)(1−p(q))>0q^{\mathsf{T}}Cq = p(q)(1-p(q)) > 0.

Consequently Δ(q)≥τ\Delta(q) \ge\tau for all q∈Qq \in Q if and only if min⁡q∈QqT(B−τC)q≥0\min_{q\in Q} q^{\mathsf{T}}(B-\tau C)q \ge0.

A proof is included in Appendix § 9 for completeness. On the full simplex, the condition for τ=0\tau= 0 is precisely copositivity of BB: xTBx≥0x^{\mathsf{T}}Bx \ge0 for all x≥0x \ge0. For two groups it reduces to

B11≥0,B22≥0,B12+B11B22≥0.(5)B_{11} \ge0,\quad B_{22} \ge0,\quad B_{12} + \sqrt{B_{11}B_{22}} \ge0. \tag*{(5)}

The off-diagonal condition explains why two positive within-group gaps do not suffice.

Why the common mixture matters. The positive and negative group weights are coupled: rg+(q)=qgπg/p(q)r_g^+(q) = q_g\pi_g/p(q) and rg−(q)=qg(1−πg)/(1−p(q))r_g^-(q) = q_g(1-\pi_g)/(1-p(q)). If these two class-conditional weights could instead vary independently over their simplices, the worst gap would simply be min⁡g,hdgh\min_{g,h} d_{gh}. Universal dominance in that larger shift class is equivalent to entrywise nonnegativity of dd. The coupled shift studied here admits more comparisons as robust, but has the harder geometry characterized below.

The estimand must be declared. A macro-average of within-task AUROCs is a different metric: it intentionally omits cross-task comparisons. It is appropriate when rankings are used only within each task. Pooled AUROC and this audit apply when a common score ranks examples across tasks or populations. A separate monotone transformation in each group can change that pooled ordering; a common strictly increasing transformation of one method cannot. We freeze score orientation, groups, labels, and any calibration before evaluation.

Allowed mixtures. We support a rational polytope Q⊆ΔKQ \subseteq\Delta_K, including group-share bounds and TV⁡(q,q0)=12∥q−q0∥1≤r\operatorname{TV}(q,q_0) = \frac{1}{2}\lVert q-q_0\rVert_1 \le r. The full simplex is a stress test. A deployment claim must use an externally justified QQ; choosing a convenient region after observing the results changes the claim.

What Subgroup Checks Cannot Establish

The matrix representation alone leaves a possible escape: perhaps matrices generated by real scalar scores form an easy subclass. They do not.

Theorem 2 (Realizable quadratic comparisons). For every rational symmetric K×KK \times K matrix DD with ∣Dgh∣≤1\lvert D_{gh}\rvert\le1, there exist finite-support joint distributions of two scalar scores and one binary label, with πg=1/2\pi_g = 1/2 for every group, such that

Δ(q)=εqTDq (q∈ΔK),ε=14(K+1).(6)\Delta(q) = \varepsilon q^{\mathsf{T}}Dq \ (q \in\Delta_K), \qquad\varepsilon= \frac{1}{4(K+1)}. \tag*{(6)}

The distributions have a description of polynomial size in the input. It is therefore coNP-hard to decide whether one score weakly dominates another for every mixture, even under equal group prevalences and finite scalar-score supports.

Construction. In negative group hh, both scores equal hh. Baseline positive scores are uniform on {1/2,3/2,…,K+1/2}\{1/2,3/2,\ldots,K+1/2\}. In positive group gg, perturb the other score’s tail probability at threshold hh by εDgh\varepsilon D_{gh}. Telescoping differences define nonnegative probabilities: each interior mass changes by at most 2ε2\varepsilon, less than its baseline mass. The cross-AUC differences are exactly the prescribed perturbations. Appendix § 9 verifies realizability, bit complexity, and the reduction from matrix copositivity [21]. This is an exact worst-case complexity statement; it does not claim that every practical small-KK audit is hard or establish hardness at a fixed, dimension-independent approximation margin.

Theorem 3 (An arbitrary hierarchy of missed reversals). For any integers 1≤m<K1 \le m < K, there exist two scores, with equal prevalence 1/21/2 in every group, such that

AUCA⁡(eg)=1 for every g, and\operatorname{AUCA}(e_g)=1\ \text{for every }g,\ \text{and}
Δ(q)>0whenever ∣supp⁡(q)∣≤m,\Delta(q)>0\quad\text{whenever }|\operatorname{supp}(q)|\le m,
Δ(1/K)<0.(7)\Delta(1/K)<0. \tag*{(7)}

The construction can have

AUCA⁡(q)=12+12∥q∥2,AUCB⁡(q)=c,(8)\operatorname{AUCA}(q)=\frac{1}{2}+\frac{1}{2}\lVert q\rVert^2,\qquad\operatorname{AUCB}(q)=c, \tag*{(8)}

for any 12+12K<c<12+12m\frac{1}{2}+\frac{1}{2K}<c<\frac{1}{2}+\frac{1}{2m}.

Here SA=4G+YS_A=4G+Y is perfect within groups but orders different groups almost entirely by group identity. For SBS_B, all negatives have score zero and each positive has score 11 with probability cc, otherwise −1-1, independently of its group. The gap is controlled by the concentration of qq. For c=12+14(K−1+m−1)c=\frac{1}{2}+\frac{1}{4}(K^{-1}+m^{-1}), both the smallest positive gap on all mm-group mixtures and the magnitude of the negative uniform gap equal 14(m−1−K−1)\frac{1}{4}(m^{-1}-K^{-1}). This supplies an explicit quantitative example, including the case m=K−1m=K-1; Figure 1(b) shows K=3K=3, m=2m=2.

These examples do not show that subgroup reporting is useless. They show precisely which additional claim it cannot support. A continuum audit must account for the interactions between groups; isolated group wins, all two-group wins, and a favorable reference mixture are insufficient in general.

A Certified Comparison Audit

Exact Finite-Bank Geometry

For a released evaluation bank, define PgP_g to be the empirical distribution of its rows. Pairwise win counts, including half credit for ties, give rational BB and CC exactly. This is a fully specified finite-distribution estimand; optimizing it introduces no sampling assertion. MIXAUDIT minimizes q⊤Dqq^\top Dq on Q\mathcal{Q}, with D=B−τCD=B-\tau C for a desired AUROC margin τ\tau.

For a cell P⊆Q\mathcal{P}\subseteq\mathcal{Q} with vertices v1,…,vJv_1,\ldots,v_J, write q=∑iαiviq=\sum_i\alpha_i v_i. Then

min⁡i,jvi⊤Dvj≤min⁡q∈Pq⊤Dq≤min⁡ivi⊤Dvi.(9)\min_{i,j}v_i^\top Dv_j\le\min_{q\in\mathcal{P}}q^\top Dq\le\min_i v_i^\top Dv_i. \tag*{(9)}

Subdivision tightens these bounds. Maintaining cells that cover the feasible set, every completed iteration returns a global lower bound and a feasible upper-bound witness; an uncorrected grid or local optimizer cannot supply the former. Rational arithmetic avoids interpreting numerical tolerances as proof. For small KK, enumerating stationary points on all faces gives an exact fallback, including zero-margin cases. These are classical quadratic-optimization and copositivity techniques [1, 2], adapted here to the comparison estimand. Their cost can grow exponentially; we apply them only with K≤4K\le4.

Theorem 4 (A corrected grid certificate). Let m≥2m\ge2, Gm={a/m:a∈N0K,∑gag=m}\mathcal{G}_m=\{a/m:a\in\mathbb{N}_0^K,\sum_g a_g=m\}, γm=min⁡q∈GmΔ(q)\gamma_m=\min_{q\in\mathcal{G}_m}\Delta(q), γ=min⁡q∈ΔKΔ(q)\gamma=\min_{q\in\Delta_K}\Delta(q), and b=max⁡gΔ(eg)b=\max_g\Delta(e_g). For arbitrary group prevalences in (0,1)(0,1),

mγm−bm−1≤γ≤γm.(10)\frac{m\gamma_m-b}{m-1}\le\gamma\le\gamma_m. \tag*{(10)}

The 1/m1/m correction has optimal order: realizable scores with K≥2mK\ge2m can satisfy γm−γ≥1/(4m)\gamma_m-\gamma\ge1/(4m).

This specializes the classical random-walk argument for simplex optimization [23] to the normalized AUROC ratio. It is valid without an inverse-minimum-prevalence factor. Thus a grid with its correction can certify dominance, while an uncorrected positive grid minimum cannot. There are (K+m−1m)\binom{K+m-1}{m} grid points: fixed-accuracy approximation and exact weak dominance have different complexity. The support hierarchy is consistent with this bound; its missed reversals shrink at order 1/m1/m.

Reversal distance. For a reference mixture q0q_0 and a meaningful gap κ>0\kappa>0, define

rκ=inf⁡{TV⁡(q,q0):q∈Q, Δ(q)≤−κ},(11)r_\kappa=\inf\{\operatorname{TV}(q,q_0):q\in\mathcal{Q},\ \Delta(q)\le-\kappa\}, \tag*{(11)}

with rκ=+∞r_\kappa=+\infty if the set is empty. Bisection on the radius and certified minimization of q⊤(B+κC)qq^\top(B+\kappa C)q yields a bracket. A feasible reversal gives an upper bound; a strictly positive global lower bound on the smaller ball gives a lower bound. We use κ=0.01\kappa=0.01 for a practically interpretable sensitivity analysis. Positive-margin reversal avoids conflating a numerically tiny tie with a material ranking change.

Population Statements After Mixture Selection

For population inference, suppose the audit receives ng≥2n_g\ge2 independent iid observations from each fixed group law, independently of model fitting and protocol choices. Define

h(z,z′)=y(1−y′){ϕ(sA,sA′)−ϕ(sB,sB′)},h(z,z')=y(1-y')\{\phi(s_A,s'_A)-\phi(s_B,s'_B)\},
k(z,z′)=12{h(z,z′)+h(z′,z)}.(12)k(z,z')=\frac{1}{2}\{h(z,z')+h(z',z)\}. \tag*{(12)}

The kernel is symmetric, has range [−1/2,1/2][-1/2,1/2], and satisfies Bgh=E[k(Zg,Zh′)]B_{gh}=\mathbb{E}[k(Z_g,Z'_h)]. Estimate off-diagonal entries by all cross-group pairs and diagonal entries by unordered pairs of distinct observations. The diagonal is an unbiased U-statistic, which differs from the empirical-bank product estimator by a factor ng/(ng−1)n_g/(n_g-1).

Theorem 5 (A simultaneous comparison envelope). Let JJ score comparisons be fixed before evaluation, L=JK(K+1)/2L=JK(K+1)/2, mgh=min⁡(ng,nh)m_{gh}=\min(n_g,n_h) for g≠hg\ne h, and mgg=⌊ng/2⌋m_{gg}=\lfloor n_g/2\rfloor. For any δ∈(0,1)\delta\in(0,1), set

egh=log⁡(2L/δ)2mgh.(13)e_{gh}=\sqrt{\frac{\log(2L/\delta)}{2m_{gh}}}. \tag*{(13)}

With probability at least 1−δ1-\delta, simultaneously for all JJ comparisons and every q∈ΔKq \in\Delta_K,

q⊤(B^−E)q≤q⊤Bq≤q⊤(B^+E)q.(14)q^\top(\widehat{B}-E)q \le q^\top Bq \le q^\top(\widehat{B}+E)q. \tag*{(14)}

Thus the same statement holds for any data-selected mixture and mixture set. It does not justify data-selected score training, group definitions, or labels outside the prespecified family.

The proof uses permutation blocking for the U-statistics and a union bound; it does not treat the quadratically many pairs as independent observations. Clipping entry intervals to [−1/2,1/2][-1/2,1/2] preserves coverage. The implementation also intersects these bands with empirical-Bernstein intervals from independent, fixed matched pairs [20], dividing the error budget between the two constructions. This can exploit correlation between the competing scores. Complete formulas and the extension to independent clusters are in Appendix § 9.§ 5.

If B‾≤B≤B‾\underline{B} \le B \le\overline{B} entrywise, a nonnegative certified minimum of q⊤Bqq^\top Bq proves dominance throughout QQ on the coverage event. A feasible qq with q⊤B‾q<0q^\top\overline{B}q < 0 proves a reversal at that mixture. Otherwise the result is unresolved. This three-way rule separates an actual ordering from insufficient evidence. Prevalence estimation is unnecessary for the sign; quantitative AUROC-gap intervals add simultaneous prevalence bounds and retain the positive denominator.

The audit also reports within-group, macro, and reference-mixture AUROCs as descriptive baselines; they are not substituted for a continuum certificate.

A Tractable Boundary: Aligning Score Distributions

Calibration changes the frozen score and therefore changes the comparison problem. A useful boundary occurs when each method has the same marginal score distribution in every group. The two methods need not share a marginal. Let

ag=πg(1−πg)Δ(eg)a_g = \pi_g(1-\pi_g)\Delta(e_g).

Theorem 6 (Comparison after marginal alignment). If L(Ss∣G=g)\mathcal{L}(S_s \mid G=g) does not depend on gg for each s∈{A,B}s \in\{A,B\}, then, including scores with ties,

Δ(q)=∑gqgagp(q)(1−p(q)).(15)\Delta(q) = \frac{\sum_g q_g a_g}{p(q)(1-p(q))}. \tag*{(15)}

Universal dominance on the full simplex is equivalent to nonnegative within-group gaps. More generally let δs,gh=∥Fs,gmid−Fs,hmid∥∞\delta_{s,gh} = \lVert F^{\mathrm{mid}}_{s,g} - F^{\mathrm{mid}}_{s,h} \rVert_\infty and wgh=min⁡{πg+πh,2−πg−πh}w_{gh} = \min\{\pi_g+\pi_h, 2-\pi_g-\pi_h\}. Then

∣p(q)(1−p(q))Δ(q)−q⊤a∣≤∑g<hqgqhwgh(δA,gh+δB,gh).(16)\left|p(q)(1-p(q))\Delta(q)-q^\top a\right| \le\sum_{g<h}q_gq_hw_{gh}\left(\delta_{A,gh}+\delta_{B,gh}\right). \tag*{(16)}

Here Fmid(t)=P(S<t)+12P(S=t)F^{\mathrm{mid}}(t)=\mathbb{P}(S<t)+\frac{1}{2}\mathbb{P}(S=t). The exact identity uses the AUC placement-value representation. Groupwise randomized probability-integral transforms align marginals and preserve within-group AUROCs in expectation, using unlabeled calibration data in the population limit. Quantile alignment and demographic-parity transport are established methods [11]; negative-class placement calibration is also classical [10]. We contribute the comparison identity and its error bound, not a new calibration algorithm.

Alignment changes cross-group rankings, can discard useful group risk information, and estimated transforms need not align new data exactly. The completed calibration control below measures this boundary rather than assuming that raw-score reversals survive preprocessing.

Experiments

We report two completed real-data studies, a released language-model score bank (§ 5.1 and § 5.2) and a vision failure-detection study (§ 5.3), and a synthetic coverage study (§ 5.4).

Released Language-Model Score Bank

We reanalyze the released LM-Polygraph Stable LM 2 12B base-model results [27]: MMLU (5,700 rows), CoQA (2,000), GSM8K (1,319), and TriviaQA (2,000). All 35 released scalar uncertainty methods are finite on all 11,019 rows, so there is no selective deletion. The original benchmark did not claim the pooled-mixture robustness studied here; this is a new analysis of its saved outputs. No model inference is rerun.

Before computing new comparison outcomes, we froze all 595 unordered method pairs, the primary released Accuracy label, score orientation, and a deterministic cluster split. CoQA questions sharing a story remain together; identical input strings in the other tasks remain together. Discovery and confirmation contain 5,605 and 5,414 rows. This is a retrospectively specified analysis protocol, not prospective preregistration. Each pair is oriented by the discovery winner at q0=(1/4,1/4,1/4,1/4)q_0=(1/4,1/4,1/4,1/4), with that orientation fixed on confirmation. The comparison domains are the full simplex and the restricted region qg≥0.05q_g \ge0.05. Witnesses and five displayed pairs are selected using discovery only; complete pair tables include every outcome.

Exact finite-bank results. On discovery, exact continuum optimization classifies 419 pairs as mixture-dependent and 176 as universally ordered. Of the 238 pairs whose discovery winner also wins in every individual task, 62 nevertheless reverse under a mixture. At the unchanged discovery witnesses, 390 of the 419 reversals and 55 of those 62 persist on confirmation (Figure 2). The restricted domain still has 350 mixture-dependent discovery comparisons. These counts describe this finite bank and comparison family, not a prevalence estimate for all uncertainty methods or a population significance test.

Figure 2. All 595 primary-endpoint pairs. Blue points denote pairs where the discovery reference winner wins within each task; gray points have mixed within-task ordering. Left: equal-task-reference advantage versus gap at the selected discovery stress mixture. Right: the same stress mixtures evaluated on disjoint confirmation clusters. The stress grid supplies feasible gaps; continuous rational optimization independently establishes the reversal counts. These are finite-bank comparisons, not population confidence plots.

Pair A−BA-BReferenceMinimum withinStress: releasedStress: task checkStress: AlignScore
Token entropy–Mahalanobis+.15085+.08707−.12342+.21626+.30665
Sequence probability–NLI degree+.07991+.01424−.07932−.07592−.04745
ROUGE-L–normalized entropy+.01628+.03048−.10170+.09561+.14390
CCP–NLI eccentricity+.03952+.01298−.06089−.05539−.04968
NLI eccentricity–Jaccard eigenvalue−.01490−.00809−.11478+.00920+.07352

Table 1. All five discovery-selected examples, evaluated on confirmation. The two methods and the stress mixture remain fixed across label checks. Positive values favor the first method. “Task check” uses the prespecified task-dependent scoring rules in Appendix § 11; it is not human adjudication. Spectral and density names are abbreviated only in this table.

The corrected denominator-40 grid certifies 168 of the 176 discovery-wide orderings and finds all 419 negative witnesses; eight cases are unresolved until exact optimization. On confirmation it certifies 156 of 173 orderings, with 17 unresolved. Exhaustive two-group continuum checks find every raw-bank reversal here. Thus the hierarchy in Theorem 3 is a worst-case theoretical limitation, not an additional four-way failure observed in these data.

Labels and stability limit the interpretation. We reproduce all released Accuracy values from saved answers. In CoQA this is literal stripped-string equality, which differs materially from a semantic notion of correctness. Using the same primary pairs and mixtures, only 13 of the 62 discovery within-task-win reversals remain negative on confirmation under all three label definitions. Table 1 shows both stable reversals and sign changes, and Figure 3 traces the first three pairs along two-task mixtures. For the second example, a one-AUROC-point reversal first becomes possible at discovery TV distance [.12988,.13086][.12988,.13086]; the separately optimized confirmation radius is [.16602,.16699][.16602,.16699]. The smallest-shift discovery witness has a positive confirmation gap +.00725+.00725. A large-stress reversal can replicate even when the exact estimated radius location does not.

Figure 3. The first three illustrations under the frozen discovery-only selection rule. Only the two displayed task proportions vary; the other two task masses are zero. Each discovery-selected method A wins in all four individual tasks, yet the intermediate mixtures can favor BB. Lines evaluate the exact finite-bank AUROC function on a dense display grid, using the released Accuracy endpoint. Confirmation uses disjoint input/story clusters. These are not population-confidence curves. The first and third examples change sign under the alternative correctness checks in Table 1; that label sensitivity is retained.

Calibration Is a Decisive Control

After the primary analysis, we froze a separate control that fits group-specific marginal or negative-class rank transforms on discovery and applies them to confirmation. This control is explicitly post hoc. Strict monotone interpolation preserves all 140 within-task AUROCs exactly in the saved bank; empirical step-CDF controls have only small additional tie effects. Table 3 compares the same confirmation rows and fixed pair orientations. Marginal alignment removes all 66 reversals among the same 239 strict within-task winners. Overall mixture dependence remains because many methods have different within-task advantages. Calibration is not a new method here, and these results do not support a claim that the strongest raw scale pathology persists after alignment.

ScoresMixed pairsWithin-win reversalsEnvelope certificates
Raw422660
Marginal rank347083
Negative-class rank34810

Table 3. Confirmation-bank control, all 595 pairs. “Within-win reversals” uses the same 239 strict all-task winners. “Envelope” counts universal dominance certificates from the marginal-alignment error bound.

The median method’s largest pairwise group KS distance falls from .7856 to .0844 after strict marginal alignment. The finite-bank envelope in Theorem 6 certifies 83 comparisons using within-group gaps and marginal discrepancies alone. All 47,600 checked matrix entries across raw and four calibrated controls obey its bounds exactly. These are deterministic checks on empirical laws, not sampling-coverage observations. The implementation uses ordinary KS distances, which upper-bound the mid-CDF distances required by the theorem. The negative-class row is evaluated with this marginal envelope; its zero count does not

rule out certificates tailored to negative-class alignment.

Vision: Failure Detection Across Corruption Mixtures

The vision study (O1) asks whether the audit transfers to a second domain and to more groups. It was specified in advance as an optional extension (Appendix § 13.§ 2); inference was run once. We use frozen torchvision ResNet-50 IMAGENET1K_V2 weights on 1,000 ImageNet validation images selected by a fixed hash, under the 15 standard ImageNet-C corruption types at severities 1, 3, 5 [7]: 45,000 corrupted-image evaluations from verified official archives. The target is Y=1Y = 1 when top-1 classification is wrong, so each score is a failure detector for the same classifier. We compare negative maximum softmax probability [8], softmax entropy, negative top-two logit margin, and negative log-sum-exp of logits at temperature one (the energy score [16]), giving six pairs at each severity, 18 comparisons in all. The classifier is not refit. Image IDs are split by a hash into 342 calibration, 316 discovery, and 342 confirmation images, and all corrupted versions of one image stay together. Top-1 error differs widely across groups (over all 1,000 images), from 24.7% (brightness, severity 1) to 93.0% (impulse noise, severity 5), which is the setting in which the mix of corruptions can matter.

At each severity, the corruption types are the groups. We report the prespecified noise, blur, weather, and digital family aggregation (K=4K = 4), audited with exact stationary-face and branch-and-bound methods, and the full K=15K = 15 analysis, audited only with the corrected grids m=2,3,4m = 2, 3, 4 (Theorem 4). Each comparison is oriented once by raw discovery AUROC and kept fixed in both panels: raw scores and the marginal-rank control of § 4.3. A reversal is material at 0.01 AUROC.

Table 2 gives the outcome. For the four-family setting, 15 of 18 comparisons are all-family winners on discovery. All 15 are certified as continuum winners on confirmation, the three non-winners carry explicit confirmation reversal witnesses, and no frozen discovery witness reverses. Under image-cluster resampling (2,000 replicates; pointwise, descriptive, not a population certificate) every raw-panel winner keeps its 95% interval above −0.01 at the frozen witness; in the marginal-rank panel 14 of 15 do. For all 15 corruptions, 14 of 18 comparisons win every group on discovery; of these 14, the corrected confirmation grids certify 10, leave 3 unresolved, and give an explicit confirmation reversal for one (severity 5, negative top-two margin against negative log-sum-exp; confirmation gap at most −0.050 at the witness mixture). Of the four non-winners, three carry reversal witnesses and one is unresolved. No frozen discovery witness reverses materially. Raw and marginal-rank controls give identical counts.

GroupsPanelWinsFrozen rev. (material)Cert.Rev. witnessUnres.
K=4K = 4Raw15/180 (0)1530
K=4K = 4Marginal rank15/180 (0)1530
K=15K = 15Raw14/18– (0)1044
K=15K = 15Marginal rank14/18– (0)1044

Table 2. Vision study (O1): ResNet-50 failure detection on ImageNet-C, 18 comparisons (six score pairs at three severities). Wins: all-group winners on discovery. Cert.: confirmation continuum certificates. Rev. witness: explicit confirmation reversal witnesses. Unres.: unresolved. Frozen rev.: among discovery winners, reversals of the frozen discovery witness on confirmation, material ones in parentheses; for K=15K = 15 only the material frozen-grid count is computed (–).

Unlike on the raw language bank, within-family wins are thus certified over the continuum of family mixtures on the confirmation images (a finite-bank statement), while with 15 groups winning every corruption type does not by itself carry over to every mixture. This is one frozen model on one benchmark and a transfer check of the audit, not support for the theorems.

Sampling Coverage and Power

We simulate the K=3K = 3, c=17/24c = 17/24 hierarchy with truly independent samples, 500 repetitions per sample size, a fixed comparison, and total error probability .05. Both the all-pairs Hoeffding bands and their paired-variance intersection cover every matrix entry in all 500 repetitions at each of ng=500,2000,8000,32000n_g = 500, 2000, 8000, 32000. A count of 500/500 has a 95% Wilson interval [.9924,1][.9924, 1]; this is compatible with conservatism rather than evidence of exact 95% calibration. At the known uniform reversing mixture, the intersection certifies the negative sign in 0, 0, 32, 500 repetitions, respectively; Hoeffding alone does so in 0, 0, 0, 236. Appendix § 12 gives all outcomes and the power figure. We do not apply these iid-row intervals to correlated CoQA questions or describe the released bank as a population-certified leaderboard.

Further validation. A prespecified replication with an independent language model and human-audited correctness labels (E1) is specified in Appendix § 13.§ 1; it has not been run, and no claim here depends on it. A smaller auxiliary replication with Qwen3-1.7B (E2) was run and is reported in Appendix § 13.§ 3; it does not replace E1.

Discussion and Conclusion

A pooled AUROC ordering has a population-mixture domain. Within-group wins and low-order stress tests cannot establish its universal validity in general, even for realizable scalar scores. Corrected grids and exact small group-count optimization can supply that missing certificate, and marginal alignment identifies a tractable boundary that removes the cross-group scale effect that otherwise makes the comparison difficult. On the language-model bank, raw reversals among within-task winners occur but are sensitive to labels and calibration. On ImageNet-C, the audit certifies the corruption-family winners among four confidence scores of one frozen ResNet-50, while over 15 corruption types it finds one winner that reverses and leaves three unresolved.

Limitations. The empirical evidence covers two finite banks: a released language-model bank whose labels are task-dependent string matches rather than validated correctness, and one frozen classifier on one corruption benchmark. It does not speak to unseen corruptions or other models, and the independent replication (E1) has not been run. Every observed raw reversal was already detectable on a two-group edge, so the higher-order hierarchy is a worst-case result. Distribution-free population certificates need large samples, and mixture certification does not address within-group shift, fairness, or downstream decision utility.

LLM Usage Statement

The authors developed the research ideas and proofs. AI tools provided coding, debugging, writing, and editing support, including polishing the exposition and presentation of the proofs. The authors reviewed and verified the resulting material and retain responsibility for the research questions, methods, proofs, experiments, interpretation, and final content.

Appendices

These appendices contain the complete proofs, the algorithms and software description, the full protocol and counterevidence for the language-model score-bank audit, the synthetic and comparator validation, the prespecified validation protocols (the essential study E1, which has not been run; the vision study O1; and the auxiliary language replication E2), and a brief statement of scope and limitations. Unprefixed theorem, equation, table, figure and section numbers refer to the main paper; numbers prefixed with “S” and lettered sections refer to these appendices.

Complete Proofs

All scores are real-valued and measurable; their joint laws with the binary target may be arbitrary. Independent draws, including the two draws used for AUROC, are explicitly distinguished from correlated score coordinates on the same example. Ties always receive half credit.

Comparison Identity and the Two-Group Criterion

Proof of Proposition 1. By Bayes’ rule the positive-class group mass is qgπg/p(q)q_g\pi_g/p(q) and the negative-class group mass is qh(1−πh)/(1−p(q))q_h(1-\pi_h)/(1-p(q)). Independence of the two AUROC draws gives

Δ(q)=∑g,hqgqhπg(1−πh)dghp(q)(1−p(q)).\Delta(q)=\frac{\sum_{g,h}q_gq_h\pi_g(1-\pi_h)d_{gh}}{p(q)(1-p(q))}.

The quadratic form is unchanged by symmetrization. Since 1Tq=1\mathbf{1}^{\mathsf{T}}q=1, qTCq=(πTq)(1Tq)−(πTq)2=p(q)(1−p(q))q^{\mathsf{T}}Cq=(\pi^{\mathsf{T}}q)(\mathbf{1}^{\mathsf{T}}q)-(\pi^{\mathsf{T}}q)^2=p(q)(1-p(q)). The denominator is positive on the stated domain, so subtracting τ\tau and multiplying by it proves the margin equivalence. For the full simplex, every nonzero nonnegative vector xx is a positive multiple of q=x/(1Tx)q=x/(\mathbf{1}^{\mathsf{T}}x); homogeneity proves the copositivity equivalence. □

For independently variable class-conditional group weights r+,r−r^{+},r^{-}, the gap is ∑g,hrg+rh−dgh\sum_{g,h}r_g^{+}r_h^{-}d_{gh}, a convex combination of entries of dd. Its minimum is the smallest entry, attained at corresponding point masses. The common-qq coupling imposes a different feasible set of weights.

Proof of the two-group criterion, Eq. (5). Let B=(abbc)B=\begin{pmatrix}a&b\\b&c\end{pmatrix}. Testing coordinate vectors gives a,c≥0a,c\geq0. If a,c>0a,c>0 but b<−acb<-\sqrt{ac}, the vector (c,a)(\sqrt{c},\sqrt{a}) gives a negative value. If a=0a=0, c>0c>0, b<0b<0, use (t,1)(t,1) for t>c/(−2b)t>c/(-2b); the symmetric case is identical. If a=c=0a=c=0, b<0b<0, use (1,1)(1,1). This proves necessity including zero-diagonal cases. Conversely, for nonnegative u,vu,v,

au2+2buv+cv2=(a u−c v)2+2(b+ac)uv≥0.au^2+2buv+cv^2=(\sqrt{a}\,u-\sqrt{c}\,v)^2+2(b+\sqrt{ac})uv\geq0.

For an exact edge minimum on q=(t,1−t)q=(t,1-t), evaluate the two endpoints and, when a−2b+c>0a-2b+c>0, the stationary point t=(c−b)/(a−2b+c)t=(c-b)/(a-2b+c) if it is in [0,1][0,1]. Nonpositive curvature requires only endpoints. With unequal prevalences this minimizes the numerator, not necessarily the AUROC ratio. □

Realizability and Computational Hardness

Proof of Theorem 2. Set u=1/(K+1)u=1/(K+1) and ε=1/[4(K+1)]\varepsilon=1/[4(K+1)]. In negative group hh, both scores equal the integer hh. In positive group gg, score BB is uniform on j+1/2j+1/2, j=0,…,Kj=0,\ldots,K, and score AA has the masses

wg0=u−εDg1,w_{g0}=u-\varepsilon D_{g1},
wgj=u+ε(Dgj−Dg,j+1),1≤j<K,w_{gj}=u+\varepsilon(D_{gj}-D_{g,j+1}),\qquad1\leq j<K,
wgK=u+εDgK.w_{gK}=u+\varepsilon D_{gK}.

The two endpoint masses are at least 3u/43u/4, and every interior mass is at least u/2u/2, because ∣Dgh∣≤1|D_{gh}|\leq1. Their perturbations telescope to zero, so the masses sum to one. There are no positive-negative score ties. A positive score exceeds the negative score hh exactly for indices j≥hj\geq h, and therefore

P(SAg>h∣Y=1)−P(SBg>h∣Y=1)=∑j=hK(wgj−u)=εDgh.\mathbb{P}(S_A^g>h\mid Y=1)-\mathbb{P}(S_B^g>h\mid Y=1)=\sum_{j=h}^{K}(w_{gj}-u)=\varepsilon D_{gh}.

The desired comparisons are thus genuine scalar-score comparisons.

For a joint realization, choose the two positive score coordinates independently given (G,Y)(G,Y), and define a feature X=(G,U,V)X=(G,U,V) with scorers sA(X)=Us_A(X)=U and sB(X)=Vs_B(X)=V. On negative examples set U=V=GU=V=G. The scorers do not take the label as an argument. Give every group label probability 1/21/2. There are KK negative support points and at most K(K+1)2K(K+1)^2 positive support points; their rational probabilities have encoding length polynomial in the encoding of DD. The comparison matrix is B=εD/4B=\varepsilon D/4, and the denominator is 1/41/4, proving Δ(q)=εq⊤Dq\Delta(q)=\varepsilon q^{\top}Dq.

For hardness, take an arbitrary rational symmetric copositivity instance MM, let a=max⁡{1,max⁡g,h∣Mgh∣}a=\max\{1,\max_{g,h}|M_{gh}|\}, and apply the construction to D=M/aD=M/a. Positive scaling preserves copositivity, and the scaling and construction have polynomial encoding size and can be computed in polynomial time. Universal dominance holds if and only if MM is copositive. Since copositivity testing is coNP-hard [21], the claimed restricted dominance problem is coNP-hard. This reduction establishes exact decision hardness, not a fixed-margin approximation lower bound. □\square

Support Hierarchy and an Analytic Radius

Proof of Theorem 3. Let πg=1/2\pi_g=1/2. Set SA=4g+YS_A=4g+Y in group gg. A positive and negative from the same group are ranked correctly, and for different groups the positive is ranked higher exactly when its group index is larger. Both class-conditional group mixtures equal qq, so

AUC⁡A(q)=∑gqg2+∑g>hqgqh=12+12∑gqg2.\operatorname{AUC}_A(q)=\sum_g q_g^2+\sum_{g>h}q_gq_h=\frac{1}{2}+\frac{1}{2}\sum_g q_g^2.

Let SB=0S_B=0 on negatives and let its positive value be 11 with probability cc, otherwise −1-1, independently of the group. Then AUC⁡B(q)=c\operatorname{AUC}_B(q)=c. These assignments are realized jointly by the score-coordinate construction used in the preceding proof. If qq has at most mm nonzero coordinates, Cauchy–Schwarz gives ∑gqg2≥1/m\sum_g q_g^2\ge1/m. Choose 1/2+1/(2K)<c<1/2+1/(2m)1/2+1/(2K)<c<1/2+1/(2m). Every such sparse mixture has positive gap, whereas the uniform KK-group mixture has negative gap. At a pure group, AUC⁡A=1\operatorname{AUC}_A=1. Choosing the midpoint value of cc gives the quantitative gaps stated in the main text. □\square

Figure S1. Analytic construction in Theorem 3, K=3K=3, m=2m=2, c=17/24c=17/24. Every boundary mixture favors A by at least 1/241/24; the center favors B by 1/241/24. Thus exhaustive checks along all three edges miss an interior reversal. Color shows the AUROC difference.

The construction also gives an exact radius oracle for implementation checks. Take q0=e1q_0=e_1 and a margin κ>0\kappa>0 such that t=2(c−κ)−1t=2(c-\kappa)-1 lies in [1/K,1][1/K,1]. At TV distance rr from e1e_1, one has q1=1−rq_1=1-r, and the minimum possible squared norm is (1−r)2+r2/(K−1)(1-r)^2+r^2/(K-1), attained by equal remaining coordinates. Thus

rκ=K−1K(1−Kt−1K−1).(17)r_\kappa=\frac{K-1}{K}\left(1-\sqrt{\frac{Kt-1}{K-1}}\right). \tag*{(17)}

If t<1/Kt < 1/K, no such reversal exists; if t≥1t \ge1, the reference already meets the target. The equality case t=1/Kt = 1/K is attained only at the uniform mixture. Equation (S1) is a test oracle for positive-margin reversal, not an empirical result.

A Sharp Corrected Grid Bound

Proof of Theorem 4. The denominator is continuous and positive on the compact simplex, so a minimizer q∗q_\ast of the AUROC gap exists. Let I1,…,ImI_1,\ldots,I_m be iid group indices with probabilities q∗q_\ast, and let q^=m−1∑i=1meIi∈Gm\widehat{q}=m^{-1}\sum_{i=1}^{m}e_{I_i}\in\mathcal{G}_m. For every symmetric matrix MM,

mE[q^TMq^]=(m−1)q∗TMq∗+∑gq∗,gMgg.m\mathbb{E}[\widehat{q}^{\mathsf{T}}M\widehat{q}]=(m-1)q_\ast^{\mathsf{T}}Mq_\ast+\sum_g q_{\ast,g}M_{gg}.

Write d=p(q∗)(1−p(q∗))d=p(q_\ast)(1-p(q_\ast)) and T=∑gq∗,gπg(1−πg)T=\sum_g q_{\ast,g}\pi_g(1-\pi_g). Since q^TBq^≥γmq^TCq^\widehat{q}^{\mathsf{T}}B\widehat{q}\ge\gamma_m\widehat{q}^{\mathsf{T}}C\widehat{q}, taking expectations yields

(m−1)γd+∑gq∗,gBgg≥γm{(m−1)d+T}.(m-1)\gamma d+\sum_g q_{\ast,g}B_{gg}\ge\gamma_m\{(m-1)d+T\}.

Moreover Bgg=πg(1−πg)Δ(eg)≤bπg(1−πg)B_{gg}=\pi_g(1-\pi_g)\Delta(e_g)\le b\pi_g(1-\pi_g), so (m−1)(γ−γm)d≥(γm−b)T(m-1)(\gamma-\gamma_m)d\ge(\gamma_m-b)T. The grid contains all vertices, hence γm≤b\gamma_m\le b. Concavity of x(1−x)x(1-x) gives T≤dT\le d; because the coefficient γm−b\gamma_m-b is nonpositive, the preceding inequality implies (m−1)(γ−γm)≥γm−b(m-1)(\gamma-\gamma_m)\ge\gamma_m-b. This is the lower bound. The upper bound follows from Gm⊆ΔK\mathcal{G}_m\subseteq\Delta_K.

For sharp order, use the hierarchy construction with K≥mK\ge m. A denominator-mm grid point has at most mm nonzero coordinates, and its smallest squared norm is 1/m1/m, attained by putting mass 1/m1/m on mm distinct groups. Consequently

γ=12+12K−c,γm=12+12m−c,b=1−c.\gamma=\frac{1}{2}+\frac{1}{2K}-c,\qquad\gamma_m=\frac{1}{2}+\frac{1}{2m}-c,\qquad b=1-c.

Their difference is 12(m−1−K−1)≥1/(4m)\frac{1}{2}(m^{-1}-K^{-1})\ge1/(4m) when K≥2mK\ge2m. The bound in Eq. (10) equals 1/2−c1/2-c, only 1/(2K)1/(2K) below the true minimum. For fixed mm, its correction is asymptotically attained as KK grows. The probabilistic technique is the classical simplex-grid argument; the ratio, prevalence cancellation, and attainable comparison construction are the specialization needed here. □

One immediate corollary clarifies the support hierarchy. If every mixture with at most m≥2m\ge2 groups has nonnegative gap, then the grid also has γm≥0\gamma_m\ge0, while b≤1b\le1, and hence every mixture has Δ(q)≥−1/(m−1)\Delta(q)\ge-1/(m-1). The theorem does not say that checking only a few selected sparse mixtures suffices: its premise quantifies over the complete sparse family. For a desired additive error ϵ\epsilon, the worst-case bound b−γm≤2b-\gamma_m\le2 allows m≥1+2/ϵm\ge1+2/\epsilon, with a grid size polynomial in KK for fixed ϵ\epsilon but potentially impractical at small tolerances.

Simultaneous Sampling Statements

For distinct groups set

B^gh=1ngnh∑i=1ng∑j=1nhk(Zgi,Zhj),\widehat{B}_{gh}=\frac{1}{n_gn_h}\sum_{i=1}^{n_g}\sum_{j=1}^{n_h}k(Z_{gi},Z_{hj}),

and on the diagonal set B^gg=(ng2)−1∑i<jk(Zgi,Zgj)\widehat{B}_{gg}=\binom{n_g}{2}^{-1}\sum_{i<j}k(Z_{gi},Z_{gj}). At most one of the two label indicators in Eq. (12) is nonzero, so k∈[−1/2,1/2]k\in[-1/2,1/2], an interval of length one. Both estimators are unbiased.

Proof of Theorem 5. First take g≠hg\ne h and suppose ng≤nhn_g\le n_h. Independently permute each sample and pair the ngn_g observations of the smaller sample with the first ngn_g of the larger. The average of the paired kernels has the same mean BghB_{gh} and is an average of mgh=ngm_{gh}=n_g independent variables of range length one. Averaging this statistic over all matchings recovers B^gh\widehat{B}_{gh}: every cross pair has the same inclusion probability. Hoeffding’s exponential-moment bound and Jensen’s inequality therefore give

Eexp⁡{λ(B^gh−Bgh)}≤exp⁡{λ2/(8mgh)}.\mathbb{E}\exp\{\lambda(\widehat{B}_{gh}-B_{gh})\}\le\exp\{\lambda^2/(8m_{gh})\}.

On the diagonal, randomly permute the observations, form ⌊ng/2⌋\lfloor n_g/2\rfloor disjoint pairs, and discard the remaining observation if ngn_g is odd. The paired terms are independent. Every unordered pair has equal chance of appearing, so averaging over permutations gives the complete U-statistic. The same exponential-moment inequality holds with mggm_{gg}. Chernoff optimization gives two-sided probability at most 2exp⁡(−2mght2)2\exp(-2m_{gh}t^2). A union bound over LL unique entries across the fixed comparison family proves the simultaneous entrywise event. Finally, qgqh≥0q_gq_h\ge0 for every mixture, so multiplying the entry bounds by these weights and summing proves Eq. (14) for the entire simplex on that single event. Data-dependent selection of qq or Q\mathcal{Q} cannot leave this event. It cannot enlarge the prespecified score or group family for which the event was proved. □

Paired variance-sensitive intersection. We use a known empirical-Bernstein inequality [20 Theorem 4]: for m≥2m \ge2 iid variables in an interval of length one, sample mean W‾\overline{W} and unbiased sample variance sW2s_W^2, a two-sided interval of error probability at most α\alpha has radius

rEB=2sW2log⁡(4/α)m+7log⁡(4/α)3(m−1).r_{\mathrm{EB}}=\sqrt{\frac{2s_W^2\log(4/\alpha)}{m}}+\frac{7\log(4/\alpha)}{3(m-1)}.

It follows by applying the cited one-sided inequality to WW and to −W-W, each with failure probability α/2\alpha/2. Use one matching per matrix entry, fixed independently of observed labels and scores, and the kernel values on those disjoint pairs. They are iid under the assumptions of Theorem 5. Allocate total error δ/2\delta/2 to all-pairs Hoeffding intervals, replacing their logarithm by log⁡(4L/δ)\log(4L/\delta); allocate the remaining δ/2\delta/2 to the LL matched-pair intervals, using log⁡(8L/δ)\log(8L/\delta) above. Intersect the two intervals with [−1/2,1/2][-1/2,1/2]. The union bound proves simultaneous coverage at least 1−δ1-\delta. The intersection need not be centered at the complete U-statistic. If numerical or off-event computations produce an empty intersection, the implementation returns [−1/2,1/2][-1/2,1/2] and records a diagnostic, a conservative fail-open rule. The inequalities are standard; their paired construction here keeps the correlation between the two competing methods.

Exact arithmetic for statistical endpoints. Kernel values are multiples of 1/41/4. The matching means and variances are computed from integer sums as rational numbers. The code encloses logarithms and square roots using 60-digit decimal arithmetic, correctly rounded transcendental operations followed by an outward step, and upward rounding for positive arithmetic. Intersections are then rational and can be passed directly to the exact optimizer. Floating outputs are rounded outward for display. Statistical validity still depends on the sampling assumptions; arithmetic certification cannot supply missing independence.

Quantitative gaps with uncertain prevalence. An independent error allocation δπ\delta_\pi gives simultaneous bounds

π‾g=max⁡{0,π^g−log⁡(2K/δπ)2ng},π‾g=min⁡{1,π^g+log⁡(2K/δπ)2ng}.\underline{\pi}_g=\max\left\{0,\widehat{\pi}_g-\sqrt{\frac{\log(2K/\delta_\pi)}{2n_g}}\right\},\qquad \overline{\pi}_g=\min\left\{1,\widehat{\pi}_g+\sqrt{\frac{\log(2K/\delta_\pi)}{2n_g}}\right\}.

For every qq, let pL=q⊤π‾p_L=q^\top\underline{\pi}, pU=q⊤π‾p_U=q^\top\overline{\pi}, dL=min⁡{pL(1−pL),pU(1−pU)}d_L=\min\{p_L(1-p_L),p_U(1-p_U)\}, and dU=max⁡p∈[pL,pU]p(1−p)d_U=\max_{p\in[p_L,p_U]}p(1-p). The latter equals 1/41/4 if the interval contains 1/21/2, and otherwise its larger endpoint value. If dL>0d_L>0, take the smallest and largest of the four ratios N/dN/d with N∈{q⊤B‾q,q⊤B‾q}N\in\{q^\top\underline{B}q,q^\top\overline{B}q\} and d∈{dL,dU}d\in\{d_L,d_U\}, then intersect with [−1,1][-1,1]. All these bounds hold simultaneously for all mixtures with failure probability at most δ+δπ\delta+\delta_\pi. If dL=0d_L=0, return [−1,1][-1,1]; do not divide by an estimated prevalence. The sign certificates need no prevalence interval. The delivered prevalence and ratio convenience functions use floating-point arithmetic and are not presented as rational certificates. The exact statistical certification path uses the rational sign-envelope endpoints; the completed quantitative finite-bank calculations use exact prevalences.

Independent clusters. Suppose iid independent units WiW_i each contain group records ZigZ_{ig}, which may be arbitrarily dependent within unit. Use the symmetric kernel

Kgh(Wi,Wj)=12{k(Zig,Zjh)+k(Zjg,Zih)},i≠j.K_{gh}(W_i,W_j)=\frac{1}{2}\{k(Z_{ig},Z_{jh})+k(Z_{jg},Z_{ih})\},\qquad i\ne j.

Its expectation is BghB_{gh} and its range has length one. A complete degree-two U-statistic over independent units therefore satisfies the same proof with m=⌊n/2⌋m=\lfloor n/2\rfloor for every entry. Within-unit cross pairs must be excluded. If units contain varying numbers of questions, one must also specify how units and questions are sampled and weighted; merely counting all questions as independent does not instantiate this theorem. The real score-bank analysis consequently makes finite-bank statements, keeps story and duplicate-input clusters together in its split, and does not report an iid population guarantee for its rows.

Marginal Alignment and Its Approximation Error

For any score with law FF, define Fmid(t)=P(S<t)+12P(S=t)F^{\mathrm{mid}}(t)=\mathbb{P}(S<t)+\frac{1}{2}\mathbb{P}(S=t). For iid S,S′S,S', the identity ϕ(S,S′)+ϕ(S′,S)=1\phi(S,S')+\phi(S',S)=1 gives E[Fmid(S)]=1/2\mathbb{E}[F^{\mathrm{mid}}(S)]=1/2 without continuity. For independent, not necessarily identically distributed Sg,ShS_g,S_h it gives

EgFhmid(Sg)+EhFgmid(Sh)=1.(18)\mathbb{E}_gF_h^{\mathrm{mid}}(S_g)+\mathbb{E}_hF_g^{\mathrm{mid}}(S_h)=1. \tag*{(18)}

Lemma S1 (Placement representation). For a binary label with prevalence p∈(0,1)p\in(0,1) and marginal score law FF,

p(1−p)AUC⁡(S;Y)=E[YFmid(S)]−p22.p(1-p)\operatorname{AUC}(S;Y)=\mathbb{E}[YF^{\mathrm{mid}}(S)]-\frac{p^2}{2}.

Proof. Let (Y′,S′)(Y', S') be an independent copy. The first expectation is E[Yϕ(S,S′)]\mathbb{E}[Y\phi(S,S')]. Splitting on Y′Y' yields the desired positive-negative term plus E[YY′ϕ(S,S′)]\mathbb{E}[YY'\phi(S,S')]. Conditional on Y=Y′=1Y=Y'=1, the two positive scores are iid, so their tie-adjusted ordering probability is 1/21/2. The second term is therefore p2/2p^2/2.

Proof of Theorem 6. Let Fs,qmid=∑gqgFs,gmidF^{\mathrm{mid}}_{s,q}=\sum_g q_gF^{\mathrm{mid}}_{s,g}. Applying Lemma S1 to the two methods and subtracting gives

N(q):=p(q)(1−p(q))Δ(q)=∑gqgEg[Y{FA,qmid(SA)−FB,qmid(SB)}].N(q):=p(q)(1-p(q))\Delta(q)=\sum_g q_g\mathbb{E}_g\left[Y\left\{F^{\mathrm{mid}}_{A,q}(S_A)-F^{\mathrm{mid}}_{B,q}(S_B)\right\}\right].

Applying the same lemma within group gg gives ag=Eg[Y{FA,gmid(SA)−FB,gmid(SB)}]a_g=\mathbb{E}_g\left[Y\left\{F^{\mathrm{mid}}_{A,g}(S_A)-F^{\mathrm{mid}}_{B,g}(S_B)\right\}\right]. When each method’s marginal is common across groups, subtracting these identities immediately gives N(q)=qTaN(q)=q^\mathsf{T}a. Pure-group necessity and the nonnegative linear combination establish the dominance equivalence.

For the error bound, for each method and unordered group pair define

Rs,gh=Eg[Y{Fs,hmid(Ss)−Fs,gmid(Ss)}]+Eh[Y{Fs,gmid(Ss)−Fs,hmid(Ss)}].R_{s,gh}=\mathbb{E}_g\left[Y\left\{F^{\mathrm{mid}}_{s,h}(S_s)-F^{\mathrm{mid}}_{s,g}(S_s)\right\}\right]+\mathbb{E}_h\left[Y\left\{F^{\mathrm{mid}}_{s,g}(S_s)-F^{\mathrm{mid}}_{s,h}(S_s)\right\}\right].

Expanding the mixture CDF and pairing the directed terms yields N(q)−qTa=∑g<hqgqh(RA,gh−RB,gh)N(q)-q^\mathsf{T}a=\sum_{g<h}q_gq_h(R_{A,gh}-R_{B,gh}). The absolute value of Rs,ghR_{s,gh} is at most (πg+πh)δs,gh(\pi_g+\pi_h)\delta_{s,gh}. By Eq. (18), replacing YY by the constant one in its definition gives zero. Replacing YY by −(1−Y)-(1-Y) therefore gives exactly the same Rs,ghR_{s,gh}, and supplies the second bound (2−πg−πh)δs,gh(2-\pi_g-\pi_h)\delta_{s,gh}. Take the smaller bound and apply the triangle inequality over methods and group pairs. This proves Eq. (16), including arbitrary ties.

An entrywise envelope. Define Hgh=(ag+ah)/2H_{gh}=(a_g+a_h)/2, Eggalign=0E^{\mathrm{align}}_{gg}=0, and, for g≠hg\ne h,

Eghalign=12wgh(δA,gh+δB,gh).E^{\mathrm{align}}_{gh}=\tfrac{1}{2}w_{gh}(\delta_{A,gh}+\delta_{B,gh}).

The coefficient identity in the proof gives Bgh−Hgh=12(RA,gh−RB,gh)B_{gh}-H_{gh}=\tfrac{1}{2}(R_{A,gh}-R_{B,gh}), hence H−Ealign≤B≤H+EalignH-E^{\mathrm{align}}\le B\le H+E^{\mathrm{align}} entrywise. The same optimizers can use this envelope. It requires valid within-group and marginal-law information; plug-in empirical marginals alone are not a population envelope.

Let δs=max⁡g<hδs,gh\delta_s=\max_{g<h}\delta_{s,gh}. Summing the positive-class bound instead of taking pairwise minima gives ∣N(q)−qTa∣≤(δA+δB)p(q)|N(q)-q^\mathsf{T}a|\le(\delta_A+\delta_B)p(q); the negative-class version gives the same expression with 1−p(q)1-p(q). Dividing by the denominator yields

∣Δ(q)−qTap(q)(1−p(q))∣≤δA+δBmax⁡{p(q),1−p(q)}≤2(δA+δB).(19)\left|\Delta(q)-\frac{q^\mathsf{T}a}{p(q)(1-p(q))}\right|\le\frac{\delta_A+\delta_B}{\max\{p(q),1-p(q)\}}\le2(\delta_A+\delta_B). \tag*{(19)}

This avoids an artificial explosion at rare class prevalences. The sharper pairwise bound vanishes at pure groups and can be substantially smaller.

Oracle rank alignment and ties. For method ss in group gg, let VV be a fresh independent uniform variable, independent of the label, and transform

Us=Fs,g(Ss)+V{Fs,g(Ss)−Fs,g(Ss−)}.U_s=F_{s,g}(S_s)+V\{F_{s,g}(S_s)-F_{s,g}(S_s^-)\}.

The distributional probability-integral transform makes UsU_s uniform. For two unequal score values their transformed intervals are ordered. When the original scores tie, the independent uniform positions rank each way with probability 1/21/2. Thus the original tie-adjusted within-group AUROC is preserved in expectation over the added randomization. This is an oracle construction, not a claim that an estimated empirical step transform has the same property on new data. Finite step transforms can create additional ties; the completed control reports those changes and separately tests a strict monotone interpolation.

An unlabeled finite-calibration implication. For each fixed method and group, suppose an independent calibration sample of size ngcaln_g^{\mathrm{cal}} estimates its marginal CDF. The classical DKW–Massart bound [18] and a union bound over MM methods and KK groups give, with probability at least 1−α1-\alpha,

∥F^s,g−Fs,g∥∞≤ϵg=log⁡(2MK/α)2ngcalfor all s,g.\left\|\widehat{F}_{s,g}-F_{s,g}\right\|_\infty\le\epsilon_g=\sqrt{\frac{\log(2MK/\alpha)}{2n_g^{\mathrm{cal}}}}\qquad\text{for all }s,g.

The same inequality holds for left limits. Coupling the estimated randomized transform with the oracle transform using the same VV gives ∣U^s−Us∣≤ϵg|\widehat{U}_s-U_s|\leq\epsilon_g. Since the oracle is uniform, the CDF of U^s\widehat{U}_s is within ϵg\epsilon_g of the uniform CDF on [0,1][0,1]. Its midpoint CDF has the same bound, being the average of the right and left limits. Therefore for the transformed scores, δA,gh+δB,gh≤2(ϵg+ϵh)\delta_{A,gh}+\delta_{B,gh}\leq2(\epsilon_g+\epsilon_h). Substitution into the alignment envelope gives a valid calibration-error bound. The within-group AUROCs in that envelope are those of the transformed scores; finite empirical calibration need not preserve the original AUROCs. Obtaining their population confidence bounds still requires labeled evaluation data independent of the fitted transforms.

Negative-class calibration as a related boundary. If, for each method, its negative-class score law is invariant across groups, then dghd_{gh} is independent of hh and equals Δ(eg)\Delta(e_g). Thus Δ(q)=∑g[qgπg/p(q)]Δ(eg)\Delta(q)=\sum_g[q_g\pi_g/p(q)]\Delta(e_g). This is the familiar negative-placement/covariate-adjusted ROC setting [10]; cf. the within/cross-group AUC decomposition of Matos et al. [19]. The marginal-alignment identity in Eq. (15) has different weights and does not require negative-class labels to define the population transform. Neither boundary makes preservation of useful cross-group risk information automatic.

Algorithms, Optimization Proofs, and Software

Exact Input Representation

For method ss and groups g,hg,h, let

Wghs=∑i:Ygi=1∑j:Yhj=0(21{Sgis>Shjs}+1{Sgis=Shjs}).W_{gh}^{s}=\sum_{i:Y_{gi}=1}\sum_{j:Y_{hj}=0}\left(2\mathbf{1}\{S_{gi}^{s}>S_{hj}^{s}\}+\mathbf{1}\{S_{gi}^{s}=S_{hj}^{s}\}\right).

This is an integer and can be computed by sorting negative scores and using binary searches for the two tie boundaries. With ngn_g total rows, the exact finite-bank matrix entries are

Bghemp=(WghA−WghB)+(WhgA−WhgB)4ngnh.(20)B_{gh}^{\mathrm{emp}}=\frac{(W_{gh}^{A}-W_{gh}^{B})+(W_{hg}^{A}-W_{hg}^{B})}{4n_gn_h}. \tag*{(20)}

When g=hg=h, the two terms in the numerator are identical. For unbiased population estimation only, replace the diagonal denominator by 4ng(ng−1)4n_g(n_g-1). Empirical prevalences are the exact fractions ng+/ngn_g^+/n_g. These formulas prevent floating score values from introducing numerical ambiguity into the comparison probabilities: score ordering is frozen by the released values, while probabilities and optimization use exact integers and fractions. Ties are determined from the stored values without arbitrary rounding or a tolerance.

Global Quadratic Brackets

Let DD be symmetric and rational and let the nonempty rational mixture polytope be Q={q≥0:1⊤q=1,Hq≤h}Q=\{q\geq0: \mathbf{1}^{\top}q=1,Hq\leq h\}. Every cell is the intersection of QQ with a rational coordinate box. Let its vertices be V={v1,…,vJ}V=\{v_1,\ldots,v_J\}. Define L(P)=min⁡i,jvi⊤DvjL(\mathcal{P})=\min_{i,j}v_i^{\top}Dv_j and U(P)=min⁡ivi⊤DviU(\mathcal{P})=\min_i v_i^{\top}Dv_i.

Proposition S2 (Cell bounds and convergence). For every nonempty cell,

L(P)≤min⁡q∈Pq⊤Dq≤U(P),U(P)−L(P)≤12∥D∥2diam⁡(P)2.L(\mathcal{P})\leq\min_{q\in\mathcal{P}}q^{\top}Dq\leq U(\mathcal{P}),\qquad U(\mathcal{P})-L(\mathcal{P})\leq\tfrac{1}{2}\lVert D\rVert_2\operatorname{diam}(\mathcal{P})^2.

Maintaining a covering partition, pruning cells only when their lower bound is at least a feasible incumbent, and bisecting the longest coordinate interval yields valid global brackets at every completed iteration. It reaches any positive requested absolute bracket width after finitely many refinements if not interrupted.

Proof. Every feasible qq is a convex combination ∑iαivi\sum_i\alpha_i v_i, hence q⊤Dq=∑i,jαiαjvi⊤Dvjq^{\top}Dq=\sum_{i,j}\alpha_i\alpha_jv_i^{\top}Dv_j is a convex combination of the bilinear vertex values. This proves the lower bound, and feasible vertices prove the upper bound. For any vertices u,vu,v,

u⊤Dv=12{u⊤Du+v⊤Dv−(u−v)⊤D(u−v)}≥U(P)−12∥D∥2∥u−v∥2.u^{\top}Dv=\tfrac{1}{2}\{u^{\top}Du+v^{\top}Dv-(u-v)^{\top}D(u-v)\}\geq U(\mathcal{P})-\tfrac{1}{2}\lVert D\rVert_2\lVert u-v\rVert^2.

Taking the minimum gives the gap bound. The minimum active-cell lower bound is a global lower bound because the active cells together with correctly pruned cells cover QQ and every pruned objective is at least the feasible incumbent. The incumbent itself is an upper bound. Longest-coordinate bisection shrinks cell diameters uniformly with depth: after sufficiently many

levels every coordinate width is below a prescribed positive number. The gap bound then falls below the requested tolerance. There are finitely many nodes before that depth. Best-bound refinement cannot leave a larger-gap relevant cell forever while processing only deeper cells, since cells of sufficiently small gap either close the global bracket or are pruned by the current incumbent. Thus finite-tolerance termination follows. Exact zero-margin decisions are not implied by this argument.

The code eliminates qKq_K and enumerates all sets of K−1K-1 linearly independent active inequalities to obtain exact vertices, including lower-dimensional feasible sets. It solves each rational system and checks every original inequality exactly. Cached inverse bases reduce repeated work during bisection. This classical partition method is described to make its certificates auditable, not claimed as a new global-optimization method.

Reference audit algorithm.

  1. Freeze score methods, target labels, groups, reference mixture and allowed mixture domain. Compute all integer rank counts and rational matrices.

  2. For a desired advantage τ\tau, form D=B−τCD=B-\tau C. For a population sign statement substitute the appropriate simultaneous lower or upper matrix; do not substitute an empirical prevalence into a population margin.

  3. Initialize the feasible incumbent at a polytope vertex and insert the initial cell with its bilinear lower bound into a priority queue.

  4. Refine the cell with the smallest lower bound by exact midpoint bisection. Discard empty children; update the feasible incumbent and prune children whose lower bound cannot improve it.

  5. Return nonnegative dominance only if the global lower bound is nonnegative. Return a reversal only with a feasible negative witness. At a budget stop, return the valid interval and unresolved if needed.

  6. For small KK, the stationary-face procedure below can settle exact zero cases. For a quantitative gap or positive-margin radius, use the certified quadratic oracle inside the stated bisection transformations.

Exact Stationary-Face Enumeration

After eliminating the simplex equality, write f(z)=zTRz+2rTz+cf(z)=z^{\mathsf{T}}Rz+2r^{\mathsf{T}}z+c on the compact polytope P={z:Az≤b}P=\{z:Az\le b\}. For every independent subset of active equalities Ez=eEz=e, choose a basis NN of ker⁡E\ker E and solve

Ez=e,NT(Rz+r)=0.(21)Ez=e,\qquad N^{\mathsf{T}}(Rz+r)=0. \tag*{(21)}

Intersect this affine solution set with PP, and retain one rational feasible point if the intersection is nonempty. Include all vertices. Minimize ff over the retained points.

Proposition S3 (Exact fallback). This finite enumeration obtains the exact global minimum, including singular Hessians and lower-dimensional polytopes.

Proof. Let z∗z_\ast be a global minimizer, and let FF be its minimal containing face. Then z∗z_\ast is in the relative interior of FF. An independent basis of its active equalities is among the enumerated sets. Both signs of a small tangential displacement are feasible, so first-order optimality gives Eq. (21). If two solutions z,z′z,z' share this stationary affine set, put d=z′−zd=z'-z. Then Ed=0Ed=0, NTRd=0N^{\mathsf{T}}Rd=0, and NT(Rz+r)=0N^{\mathsf{T}}(Rz+r)=0. Since dd lies in the span of NN, dTRd=dT(Rz+r)=0d^{\mathsf{T}}Rd=d^{\mathsf{T}}(Rz+r)=0, and therefore f(z′)=f(z)f(z')=f(z). Any feasible point of this stationary set has the global-minimum value. The compact intersection of rational affine equations and PP, when nonempty, has a rational vertex, obtainable by adding enough independent active inequalities. The code finds such a point. Every retained point is feasible, so no retained value can be below the global minimum, while the minimizing face supplies equality. No assertion that indefinite quadratic minima occur only at vertices is used.

Normalized Gaps, Radii, and Restricted Grids

For any trial τ\tau, positivity of qTCqq^{\mathsf{T}}Cq gives

min⁡q∈QΔ(q)≥τ⟺min⁡q∈QqT(B−τC)q≥0.\min_{q\in Q}\Delta(q)\ge\tau \quad\Longleftrightarrow\quad \min_{q\in Q}q^{\mathsf{T}}(B-\tau C)q\ge0.

The normalized-gap routine maintains a dyadic bracket in [−1,1][-1,1]. A nonnegative quadratic minimum raises the lower bound; a negative feasible witness lowers the upper bound. If an interrupted oracle cannot decide the sign, the outer routine retains its current bracket. It never infers the worst normalized gap by minimizing the numerator alone.

For equal-mass probability vectors,

TV⁡(q,q0)=max⁡T⊆{1,…,K}∑g∈T(qg−q0,g).\operatorname{TV}(q,q_0)=\max_{T\subseteq\{1,\ldots,K\}}\sum_{g\in T}(q_g-q_{0,g}).

Indeed the maximizing set is the positive-coordinate-difference set, and its sum is half the ℓ1\ell_1 norm. Thus a TV ball is a rational polytope with 2K−22^K-2 nontrivial subset inequalities. At a trial radius rr, minimize q⊤(B+κC)qq^\top(B+\kappa C)q over its intersection with Q\mathcal{Q}, for κ>0\kappa>0. A strictly positive lower bound excludes a reversal throughout that ball; a nonpositive feasible value exhibits a reversal of at least κ\kappa. The feasible witness’s exact TV distance supplies the upper bracket endpoint. The lower endpoint is valid by nesting of TV balls. If no reversal exists over all Q\mathcal{Q}, the radius is infinite. An isolated exact tie cannot satisfy this positive-margin target. The implementation rejects a zero-margin radius request rather than silently interpreting a tie as a strict reversal.

The corrected-grid theorem extends to a nonempty rational polytope whose vertices are v1,…,vJv_1,\ldots,v_J. Treat Pvj=∑gvj,gPgP_{v_j}=\sum_g v_{j,g}P_g as the component distributions, with prevalences π⊤vj\pi^\top v_j. A mixture over these components has the same AUROC as the corresponding point of Q\mathcal{Q}. Applying Theorem 4 to the equally weighted averages of mm vertices gives its bound with b=max⁡jΔ(vj)b=\max_j\Delta(v_j). An arbitrary coordinate grid intersected with a polytope is not this grid and need not satisfy the same correction. For qg≥0.05q_g\ge0.05 with four groups, the transformation q=0.051+0.8tq=0.051+0.8t, t∈Δ4t\in\Delta_4, is exactly a vertex-mixture grid.

Our sharp-order statement is dimension-uniform over varying KK, not a claim that every fixed instance converges at rate only 1/m1/m. Faster instance-dependent quadratic grid bounds are known [6]. The hierarchy used for our lower bound has K≥2mK\ge2m and is fully compatible with those results.

Comparable Baselines

The audit compares the same two frozen scores and the same group law under each baseline. Within-group signs use only the diagonal. Entrywise cross-AUC nonnegativity certifies the larger class of independently shiftable positive and negative mixtures. Exhaustive two-group tests optimize all simplex edges. The uncorrected denominator-40 grid supplies feasible witnesses; the corrected grid supplies a valid lower bound. Exact quadratic certificates settle the small-KK sign problem. The alignment envelope uses only within-group gaps and pairwise marginal-CDF discrepancies.

For a distributional-drift comparator, we implement the paired-loss Hellinger certificate of Weber et al. [28], adapted to the same positive-negative pair law. At q0q_0, put D=ϕ(SA+,SA−)−ϕ(SB+,SB−)∈[−1,1]D=\phi(S_A^+,S_A^-)-\phi(S_B^+,S_B^-)\in[-1,1] and L=(D+1)/2∈[0,1]L=(D+1)/2\in[0,1], with mean μ\mu and variance ν\nu. Define the class-conditional mixture weights rg+(q)r^+_g(q) as in Sec. 2 of the main paper. The labeled group-pair Hellinger affinity is

a(q)=(∑grg+(q0)rg+(q))(∑hrh−(q0)rh−(q)).a(q)=\left(\sum_g\sqrt{r_g^+(q_0)r_g^+(q)}\right)\left(\sum_h\sqrt{r_h^-(q_0)r_h^-(q)}\right).

It is exact with group identities retained and a lower bound on affinity after dropping them. The Gramian bound gives the valid paired gap lower bound

Δ(q)≥2[max⁡{0,a(q)μ−1−a(q)2ν/μ}]2−1,(22)\Delta(q)\ge2\left[\max\left\{0,a(q)\sqrt{\mu}-\sqrt{1-a(q)^2}\sqrt{\nu/\mu}\right\}\right]^2-1, \tag*{(22)}

with the conservative value −1-1 when μ=0\mu=0. The positive part must be taken before squaring; otherwise the bound can be invalid at large shifts. The implementation computes the exact five-point empirical distribution of DD in blocks, retaining ties and method pairing. For a whole TV region it also exposes the analytic affinity lower bound ar=(1−min⁡{1,r/p(q0)})(1−min⁡{1,r/(1−p(q0))})a_r=(1-\min\{1,r/p(q_0)\})(1-\min\{1,r/(1-p(q_0))\}). This region conversion is explicitly conservative; we do not claim it is an optimal implementation of Hellinger-ball certification. Pointwise comparisons use the exact group-pair affinity. Empirical moments here describe the finite bank; population use requires simultaneous moment and prevalence uncertainty.

Reason for Eq. (22). Let ν=P+Q\nu=P+Q, ψP=dP/dν\psi_P=\sqrt{\mathrm{d}P/\mathrm{d}\nu} and ψQ=dQ/dν\psi_Q=\sqrt{\mathrm{d}Q/\mathrm{d}\nu}. These are unit vectors in L2(ν)L^2(\nu) with inner product aa. If a<1a<1, write ψQ=aψP+1−a2 w\psi_Q=a\psi_P+\sqrt{1-a^2}\,w, where ww is a unit vector orthogonal to ψP\psi_P. Then

∫LψPψQ dν=aμ+1−a2∫(L−μ)ψPw dν≥aμ−(1−a2)ν.\int L\psi_P\psi_Q\,\mathrm{d}\nu=a\mu+\sqrt{1-a^2}\int(L-\mu)\psi_Pw\,\mathrm{d}\nu\ge a\mu-\sqrt{(1-a^2)\nu}.

Cauchy–Schwarz also bounds this integral above by EPLEQL\sqrt{\mathbb{E}_P L\mathbb{E}_Q L}. For μ>0\mu>0, taking the positive part, dividing by μ\sqrt{\mu}, and squaring proves the bound. If a=1a=1, the two laws coincide and the formula is equality; if μ=0\mu=0, the stated lower bound is trivial. This proof includes mutually singular components. It is the standard Hellinger/Gramian argument, used as a comparison baseline rather than claimed as a contribution.

Reproducibility

The implementation comprises the exact optimizer, the U-statistic confidence routines, the alignment and Hellinger comparators, a generic audit interface for score tables, the prespecified protocols, integer rank counts, complete pair tables, and the figure code; it will be released with the paper. The public score-bank revision and checksums of all inputs are recorded. No training or generation was rerun for the completed language-model study. The tests cover tied ranks, unequal prevalences, agreement of weighted AUROCs with an independent library, interior and singular minima, non-dyadic zero minima, constraint feasibility, and the analytic hierarchy radius.

Completed Score-Bank Audit: Protocol, Evidence, and Counterevidence

Source, sampling units, and frozen analysis

We analyze author-released LM-Polygraph score banks, associated with the benchmark of Vashurin et al. [27]. The source is the public LM-Polygraph/managers dataset at Hugging Face, at a pinned revision recorded with the code to be released: the four Stable LM 2 12B result files for MMLU, CoQA, GSM8K, and TriviaQA. The source paper identifies the white-box model as Stable LM 2 12B base, without instruction tuning. We use the released generations and scores; we did not run new model inference. The banks do not record the original inference code or checkpoint revision, and the release contains no second base-model bank.

Each downloaded file is pinned by its SHA-256 checksum, size, and source revision, and is read with a restricted data-only loader. The public repository metadata lists an MIT license. The source benchmark and constituent datasets retain their respective attribution requirements.

All 35 common uncertainty methods are retained (Table S2). All 11,019 rows have finite values for all these scores and both released endpoint arrays; the common-complete-case policy excludes zero rows. Scores retain their published orientation, with larger values indicating greater uncertainty. No orientation is selected using AUROC. Every released row, including repeated inputs, remains in the empirical target population.

IDEstimatorIDEstimator
1MaximumSequenceProbability19DegMat_NLI_score_entail
2Perplexity20DegMat_NLI_score_contra
3MeanTokenEntropy21DegMat_Jaccard_score
4MeanPointwiseMutualInformation22Eccentricity_NLI_score_entail
5MeanConditionalPointwiseMutualInformation23Eccentricity_NLI_score_contra
6CCP24Eccentricity_Jaccard_score
7PTrue25SemanticEntropy
8PTrueSampling26SAR
9MonteCarloSequenceEntropy27TokenSAR
10MonteCarloNormalizedSequenceEntropy28SentenceSAR
11LexicalSimilarity_rouge129RenyiNeg
12LexicalSimilarity_rouge230FisherRao
13LexicalSimilarity_rougeL31MahalanobisDistanceSeq_decoder
14LexicalSimilarity_BLEU32RelativeMahalanobisDistanceSeq_decoder
15NumSemSets33RDESeq_decoder
16EigValLaplacian_NLI_score_entail34PPLMDSeq_decoder
17EigValLaplacian_NLI_score_contra35PPLRMDSeq_decoder
18EigValLaplacian_Jaccard_score

Table S2.

We assign exact duplicate inputs to one cluster. For CoQA, we instead extract the complete shared story between the unique Story: marker and its first following Question: marker; all questions from a story remain together. Parsing is checked for every row. After normalizing line endings and stripping outer whitespace, SHA-256 hashes of a fixed protocol identifier, task name, and cluster text assign cluster rows to discovery or confirmation according to whether the first digest byte is below 128. This produces the counts in Table S1. It does not establish independence of benchmark questions beyond these observed clusters.

TaskRowsDiscoveryConfirmationDisc. clustersConf. clustersDisc. errorsConf. errors
MMLU5700284028602830285612871305
CoQA20001059941260226651608
GSM8K1319671648671648300290
TriviaQA20001035965978912448415
Total11019560554144739464226862618

Table S1. Audit sample sizes. Discovery/confirmation assignment is at the input/story-cluster level, so row counts need not be equal. Error counts refer to the released Accuracy endpoint.

The primary protocol was fixed after inspecting schemas, finite counts, duplicate inputs, and example prompt formats, but before computing new score comparisons. It is an outcome-blind retrospective protocol, not prospective preregistration. The reference mixture is g0=(1/4,1/4,1/4,1/4)g_0 = (1/4,1/4,1/4,1/4), in task order MMLU, CoQA, GSM8K, TriviaQA. For each of the 595 unordered pairs, AA is the discovery reference winner. An exhaustive grid of integer compositions of 40 into four parts selects a discovery stress mixture; the same mixture is then evaluated on confirmation. The secondary domain is qg≥0.05q_g \ge0.05, using q=0.051+0.8tq = 0.05 1 + 0.8t for grid point tt. All pairs and null outcomes are reported. Exact rational optimization independently audits both continuous domains, using integer twice-win counts and the empirical-product matrix. Pairs share methods and observations; their counts are descriptive, not 595 independent replications or a random sample from a population of UQ methods.

Table S2. Complete method inventory, with stable identifiers used below. These are the exact released estimator names; all 35 are analyzed without selection.

Endpoint semantics and label sensitivity

The primary binary endpoint is Y=1−AccuracyY = 1 - \mathrm{Accuracy} from the release. It is task-dependent textual agreement, not uniformly human semantic correctness. We independently reproduce every primary label from the stored generations and targets (Table S3). These checks establish the released labels’ observable semantics; they do not recover the unknown historical execution configuration.

TaskReproducing rule for released AccuracyParser changes
MMLUCase-sensitive equality after stripping output and target letter.7/5700
CoQACase-sensitive equality after stripping output and the single stored answer; no punctuation, article, or case normalization.676/2000
GSM8KGreedily remove target prefix through #### and output prefix through The answer is; strip, lowercase, remove ASCII punctuation, and compare.7/1319
TriviaQAStrip, lowercase, remove ASCII punctuation, and compare with any provided target alias.28/2000

Table S3.

Table S3. Exact reproduction of released Accuracy and changes under the frozen task-parser sensitivity. Each primary reconstruction has zero disagreements.

The first secondary endpoint uses Y=1{AlignScore<0.5}Y = 1\{\mathrm{AlignScore} < 0.5\}. The second uses task-specific parsers fixed before comparisons: a leading MMLU answer letter; exact Decimal comparison of GSM8K’s final extracted number, preferring text following the final #### marker; official TriviaQA normalization and alias matching; and normalized single-reference CoQA token-F1 at least 0.5. Missing parses count as incorrect. CoQA’s latter rule is a sensitivity, not its official multireference evaluation. AlignScore and the parser labels are also proxies. The substantial CoQA change is therefore scientifically material, rather than an implementation nuisance.

Table S4. Exact continuum classification. Each entry counts mixture-dependent pairs out of 595; the remaining pairs have a universal ordering. “Interior” means every task has at least 0.05 mass. Each endpoint uses its own discovery-reference orientation, without changing score direction.

EndpointDiscoveryFull simplex ConfirmationFull bankDiscoveryInterior ConfirmationFull bank
Released Accuracy419422419350358355
AlignScore threshold415406404294293288
Task-parser sensitivity388396387287297292

Table S4.

Table S4 reports the 10,710 exact certificates across endpoints, splits, and domains. For the primary discovery bank, 176 pairs exhibit universal dominance and 419 exhibit mixture-dependent ranking. The grid detects every primary discovery sign reversal also established by continuous optimization. Grid gaps remain feasible witness values, not exact minima of the normalized gap.

Of the 419 primary discovery reversals, 390 remain negative at the unchanged witness on confirmation. Among 238 pairs whose discovery winner also wins within every task, 62 reverse, and 55 retain a negative gap at that witness on confirmation. Only 40 of those 62 also retain all four within-task wins on confirmation. These are different properties and should not be conflated.

The frozen illustration rule chooses the five largest discovery reversals among pairs with reference advantage at least 0.005 and a strictly positive gap in every discovery task. No confirmation outcome enters this rule. Table S5 retains all five, including unfavorable label sensitivities. For example, the largest raw reversal disappears under both secondary endpoints, whereas the second and fourth remain negative.

Table S5. Confirmation gaps for all five frozen illustrations, at the same primary discovery witness. IDs refer to Table S2; task order is MMLU/CoQA/GSM8K/TriviaQA. Intervals are descriptive pointwise 95% cluster-bootstrap intervals for the released-Accuracy gap, not simultaneous finite-sample guarantees.

The same sensitivity is computed for every primary pair, with no reorientation or witness reselection. Among the 419 primary discovery reversals, confirmation remains negative for 390 under Accuracy, 206 under AlignScore, 251 under the parsers, and 188 under all three. Among the 62 primary all-task-winner reversals, the respective counts are 55, 20, 13, and 13. Requiring confirmation within-task dominance as well gives 40, 0, and 9 for the individual endpoints. These fixed-witness results are more conservative than reoptimizing separately for each endpoint.

Posthoc calibration and strong stress baselines

After the primary outcomes, but before computing calibration outcomes, we fixed four controls. Each method is transformed within each task using discovery data only: the marginal empirical mid-CDF, the correct-example mid-CDF, and strictly monotone interpolation/extrapolation versions of both. The mid-CDF uses [#(s<x)+#(s=x)/2]/n[\#(s < x) + \#(s = x)/2]/n. The strict versions interpolate between unique calibration values, extrapolate with the end slopes, and use an affine fallback for a constant calibration sample. Monotone floating-point tie correction preserves equal original scores while separating distinct ones. No label-dependent performance tuning is performed. The strict versions preserve all 140 within-task AUROCs exactly; mid-CDF ties change them by at most 0.000236 (marginal) and 0.000535 (correct-example).

Table S6. Posthoc calibration on the same confirmation bank. The within-task-winning subset has 239 pairs for each row. The strict transforms preserve every within-task ranking, so the reduction from 66 reversals to zero cannot be explained by changing within-task discrimination.

ScoresMixture-dependentInteriorAll-task winnersWinners that reverse
Raw422/595358/59523966
Marginal mid-CDF347/595268/5952390
Marginal strict rank347/595268/5952390
Correct-example mid-CDF348/595264/5952391
Correct-example strict rank348/595264/5952391

Table S6.

This is strong limiting evidence: groupwise score alignment removes every all-task-winner reversal under marginal calibration (Table S6). Overall mixture dependence remains because many pairs have different within-task performance orderings. Normalization is not a new technique; LM-Polygraph itself studies quantile normalization [27]. Our audit concerns fixed score representations and explicit deployment mixtures, not an assertion that the source benchmark published an erroneous pooled-AUROC leaderboard. Exhaustive exact two-group stress testing detects every raw primary reversal on discovery, confirmation, and full banks. There are zero observed pairs that pass all two-group continua but fail the full simplex. Thus the proved support hierarchy is a theoretical and synthetic result, not an observed advantage over exhaustive pairwise stress testing in this bank.

Certified radii, descriptive uncertainty, and validation

Table S7 reports exact-optimization intervals for the minimum TV shift producing a 0.01 AUROC reversal. These are optimization brackets, not statistical intervals. The near-minimal discovery witnesses do not always transfer: pairs 1 versus 20 and 6 versus 23 have positive confirmation gaps at those witnesses. The larger original stress witnesses in Table S5 do transfer. For pair 22 versus 18, the confirmation reference already favors BB by more than 0.01, giving radius zero.

PairDiscovery TV bracketConfirmation TV bracketConfirmation gap at discovery witness
3 vs 31[.290039, .291016][.285156, .286133]−.012242
1 vs 20[.129883, .130859][.166016, .166992]+.007247
13 vs 10[.115234, .116211][.103516, .104492]−.013046
6 vs 23[.091797, .092773][.141602, .142578]+.006090
22 vs 18[.083008, .083984][0, 0]−.035620

Table S7. One-point reversal radii for the five fixed examples. Last column evaluates the discovery radius’s feasible witness unchanged on confirmation.

For descriptive uncertainty we resample input/story clusters independently within each confirmation task, retain all rows in each sampled cluster, and renormalize row mass within task. We use 2,000 replicates and seed 18172026. The pointwise percentile intervals in Table S5 neither control all 595 comparisons simultaneously nor establish that benchmark clusters are iid deployment draws. Exact finite-bank statements do not require that interpretation. Independent weighted ROC-AUC calculations using scikit-learn reproduce the matrix calculations, including discrete ties and zero-mass groups. Rank-preservation checks cover every strict calibration variant, method, and task.

All pair tables, integer counts, exact certificates, protocols, parsers, and figure code will be released with the code. Figures are computed from observed scores, not simulated model outputs. A second-model, correctness-validated independent study remains essential before generalizing these results to semantic reliability or deployment populations. The present evidence establishes a reproducible finite-bank audit and its calibration and endpoint boundaries.

All 595 primary-endpoint pairs and the first three fixed-pair mixture curves are shown in Figure 2 and Figure 3 of the main paper.

Completed Synthetic and Comparator Validation

Independent-Sampling Coverage and Power

We use the exact hierarchy distribution in Theorem 3, with K=3K=3, πg=1/2\pi_g=1/2, c=17/24c=17/24, and scores SA=4G+YS_A=4G+Y, SB=0S_B=0 on negatives and SB∈{−1,1}S_B\in\{-1,1\} on positives, with positive probability cc for +1+1. Every repetition independently generates nn observations per group. The true uniform-mixture gap is −1/24-1/24. We compare the all-pairs Hoeffding bands at total error probability δ=.05\delta=.05 with their intersection with fixed-matching empirical-Bernstein bands, using the split error allocation in Appendix § 9.§ 5. There is one prespecified score comparison, so J=1J=1 and L=6L=6.

The generator uses NumPy’s default PCG64 generator with seed 1812026, and 500 independent repetitions at each of n=500,2000,8000,32000n=500,2000,8000,32000. The configuration was fixed before the simulation was run. Coverage means that all six true upper-triangular matrix entries lie in the reported intervals. Power means that the upper matrix has a strictly negative quadratic form at the prespecified uniform mixture; it is not power optimized over mixtures. Both procedures cover all six entries in all 500 repetitions at each sample size. The Wilson 95% interval for each coverage proportion is [.9924, 1]. This limited simulation checks a specific distribution and implementation; the coverage theorem, rather than the simulation, supplies a distribution-free statement.

At the smallest sample size, the intersection can be wider than the standalone Hoeffding interval because its Hoeffding component receives only half the error budget. At larger sizes the variance information tightens the intervals. The substantial sample requirement is retained as a negative finding: exact computation of the finite-bank mixture problem does not remove the cost of distribution-free population inference.

nn per groupMethodJoint coverageReversal certificates95% interval for powerHalf-width
500Hoeffding5000[0, .0076].104696
500Intersection5000[0, .0076].106432
2000Hoeffding5000[0, .0076].052348
2000Intersection5000[0, .0076].037200
8000Hoeffding5000[0, .0076].026174
8000Intersection50032[.0457, .0890].014477
32000Hoeffding500236[.4286, .5158].013087
32000Intersection500500[.9924, 1].006200

Table S8. Completed independent-sampling experiment. Counts are out of 500. The half-width column is the median over repetitions of the largest matrix entry interval half-width. Different interval centers are retained when intersecting the two constructions.

Figure S2. Completed synthetic coverage and fixed-witness power. Points are empirical proportions over 500 repetitions; intervals are Wilson 95% intervals. The target coverage is .95. All model and data in this figure are the explicit synthetic construction; no point represents real-model generalization.

Corrected Grid Against Exact Continuum Optimization

We evaluate Theorem 4 using all denominator-40 grid points on the full four-group simplex for every primary pair on discovery and confirmation. Each grid value and lower certificate is computed exactly from integer rank counts and rational prevalences. A nonnegative lower bound certifies dominance; a negative feasible grid value certifies a reversal; other cases remain unresolved. The exact continuum classification uses the separate stationary-face optimizer. No grid minimum is called an exact continuous normalized-gap minimum.

BankGrid dominanceGrid reversalUnresolvedExact dominanceMedian widthMaximum width
Discovery1684198176.006837.018203
Confirmation15642217173.006948.017595

Table S9. Corrected-grid classifications versus exact classifications, all 595 released-Accuracy pairs in each bank. “Width” is the interval between the corrected lower bound and the feasible grid gap.

All 1,190 comparisons agree whenever the corrected grid reaches a decision; there are no invalid decisions. Every negative continuum case has a grid witness in this bank. This agreement is empirical, whereas the correction’s validity is proved for every admissible instance. The exact grid arithmetic agrees with the separately computed floating grid values to within 2.99×10−162.99 \times10^{-16}. All 1,190 grid runs together took 76.64 s of CPU time.

Paired Hellinger and Total-Variation Comparators

For each of the five discovery-selected illustrations, we evaluate the paired-loss Hellinger bound in Eq. (22) using exact finite-bank moments at the confirmation reference mixture and the exact group-pair affinity to the unchanged discovery stress mixture. We also evaluate the paired total-variation bound ∣Δ(q)−Δ(q0)∣≤2TV⁡(Pq0+⊗Pq0−,Pq+⊗Pq−)|\Delta(q)-\Delta(q_0)| \le2\operatorname{TV}(P_{q_0}^{+} \otimes P_{q_0}^{-}, P_q^{+} \otimes P_q^{-}), with group identities retained. The factor two is the range length of the paired gap loss. These are certificates for the same paired comparison rather than separate, unnecessarily uncoupled certificates for the two scores.

For the five full-simplex witnesses, both generic drift bounds include zero and do not certify a comparison sign; the exact structured audit finds the negative values in Table S5. The five interior-domain witnesses likewise remain unresolved by those bounds. The complete numerical affinity, moments, lower and upper endpoints, and observed gap for each of the ten cases will be released with the code. These outcomes illustrate the cost of relaxing known mixture structure to generic drift. They do not claim a uniform superiority theorem, an optimal Hellinger-ball implementation, or statistical population validity for empirical moments.

A versus BqAccuracy gap [interval]AlignScore gapParser gap
3 vs 31(.525, .475, 0, 0)−.1234 [−.1627, −.0861]+.3067+.2163
1 vs 20(0, 0, .5, .5)−.0793 [−.1110, −.0473]−.0475−.0759
13 vs 10(0, .525, .475, 0)−.1017 [−.1285, −.0742]+.1439+.0956
6 vs 23(0, 0, .5, .5)−.0609 [−.0872, −.0357]−.0497−.0554
22 vs 18(0, .475, .525, 0)−.1148 [−.1418, −.0876]+.0735+.0092

Table S5.

PairDiscovery witness domainConfirmation gapHellinger intervalTV interval
1Full simplex−.1234[−1.000, 1.000][−1.000, 1.000]
2Full simplex−.0793[−1.000, 1.000][−1.000, 1.000]
3Full simplex−.1017[−.991, .994][−1.000, 1.000]
4Full simplex−.0609[−.997, 1.000][−1.000, 1.000]
5Full simplex−.1148[−.999, .998][−1.000, 1.000]
1qg≥.05q_g \ge.05−.0584[−.850, .964][−.971, 1.000]
2qg≥.05q_g \ge.05−.0475[−.770, .850][−1.000, 1.000]
3qg≥.05q_g \ge.05−.0737[−.722, .741][−1.000, 1.000]
4qg≥.05q_g \ge.05−.0406[−.743, .786][−1.000, 1.000]
5qg≥.05q_g \ge.05−.0904[−.776, .760][−1.000, 1.000]

Table S10. Completed pointwise drift comparisons. Pair numbers follow the five rows of Table S5. The intervals use the exact finite-bank paired-loss moments and group-pair drift, with outward-rounded endpoints. All ten intervals include zero.

Finite-Bank Alignment Envelope Checks

For every pair, we compute ordinary two-sample KS distances exactly from the empirical score distributions and use them conservatively in Theorem 6. Ordinary KS upper-bounds mid-CDF distance: the right-limit CDF difference and its left limit both lie in the same supremum interval, hence so does their average. All matrix endpoints are rational. Across 595 pairs, 16 ordered entries per matrix, and five representations (raw, marginal mid-CDF, marginal strict rank, negative-class mid-CDF, negative-class strict rank), all 47,600 matrix entries satisfy the deterministic envelope. The numbers of universal AA-dominance certificates are respectively 0, 82, 83, 0, 0. The last two zeros concern the marginal-discrepancy envelope, whose assumption is not targeted by negative-class calibration. They are not an assessment of the best possible negative-class-alignment certificate.

The raw-to-strict-marginal drop in the median method’s maximum pairwise group KS distance is .78561685 to .0843629. A small discrepancy can make the envelope informative without exact equality of group marginals. All these are deterministic empirical-law comparisons, separate from the unlabeled calibration-sample guarantee in Appendix § 9.§ 6.

Essential Independent Validation and Optional Extension

E1 has not been run; its protocol is stated below and is not evidence for any claim in the paper. Its purpose is to determine whether the practical conclusions extend beyond the released base-model bank and its literal-match labels. The vision study O1 (Appendix § 13.§ 2) and an auxiliary smaller-model language replication E2 (Appendix § 13.§ 3), which does not replace E1, were run.

E1 (Essential, Not Run): Independent Model, Validated Outcomes, and Decision Consequences

Questions. The prespecified questions are: (i) do population-mixture comparisons remain useful for an independently generated model under task-native correctness; (ii) how much of any observed instability survives marginal alignment and exhaustive two-group stress tests; (iii) do certificates and reversal radii predict held-out comparison behavior; and (iv) does the chosen comparison change a global error-review decision at a fixed review budget? No particular answer is assumed. A second model is replication of the audit question, not a paired comparison of two models’ distinct error labels.

Pinned model and data. Use Qwen3-8B with its shipped tokenizer and chat template at a pinned checkpoint revision [24], thinking mode disabled, bfloat16 weights, and no fine-tuning or quantization. The following Hugging Face dataset snapshots are used at pinned revisions.

TaskHugging Face dataset/configuration
MMLUcais/mmlu, per-subject configurations
GSM8Kopenai/gsm8k, main
CoQAstanfordnlp/coqa, default
TriviaQAmandarjoshi/trivia_qa, rc.nocontext

Table 1 (PDF p. 25).

These tasks are MMLU [9], GSM8K [4], CoQA [25], and TriviaQA [12]. The exact revisions and software environment are recorded with the code to be released.

Deterministic selection and independence of decisions. Assign stable row identifiers using the task name and the dataset’s provided ID, or the SHA-256 of its complete input and target JSON when no ID exists. Define a global cluster identifier as the SHA-256 of the complete input text, using the shared story for CoQA. Normalize CRLF and CR to LF, strip leading and trailing whitespace, and encode as UTF-8; do not otherwise normalize or add a task prefix. Thus identical inputs across tasks and all questions from one CoQA story remain in the same cluster. For subset selection sort clusters lexicographically by the digest of p181-selection-v1 + NUL + task name + NUL + global cluster ID, encoded as UTF-8. Break any hash tie lexicographically by cluster ID. For evaluation use the first 100 MMLU test questions per subject (5,700 total if every subject has sufficient rows), all 1,319 GSM8K test problems, the first 2,000 distinct TriviaQA validation questions, and every question in the first 250 CoQA validation stories. If a source has fewer eligible units than requested, retain all and report the shortfall; do not replace it based on scores or labels. Keep original question multiplicity inside selected CoQA stories. Reserve independent calibration data: up to 10 MMLU validation questions per subject, the first 1,000 GSM8K training questions, 1,000 TriviaQA training questions, and the first 100 CoQA training stories under the same hash rule. Exclude any exact input overlap between calibration and evaluation, report it, and do not refill it.

Use an independently domain-separated split digest: SHA-256 of the UTF-8 encoding of p181-replication-v1 + NUL + global cluster ID. Values below 128 in its first byte give discovery, otherwise confirmation. Do not include a task name in this split digest or reuse the subset-selection digest. Calibration data fit transforms only; discovery selects pair orientation and witness mixtures; confirmation evaluates the unchanged selections. Preserve every scored row in the finite-bank estimand. Training-data contamination of public benchmarks by the pretrained model is not excluded by this protocol and must be listed as a limitation; it does not create a claim of new training generalization.

Prompts and generation. Use a single user message and no task-specific system prompt. The task messages are exactly the following templates, with bracketed fields replaced by dataset text:

MMLU: [question]
A. [choice A]
B. [choice B]
C. [choice C]
D. [choice D]
Answer with only A, B, C, or D.

GSM8K: [question]
Solve the problem.  End with "The answer is [number]".

TriviaQA: [question]
Answer with a short phrase only.

CoQA: Story: [story]
[history]Question: [current question]
Answer with a short phrase only.

Every displayed line break inside a template is one LF character. CoQA [history] is empty at the first turn; otherwise concatenate, in dataset order, Question: [previous question] + LF + Answer: [previous reference answer] + LF for each preceding turn, using the dataset’s primary reference answer. No additional blank lines or spaces are inserted. The MMLU prefix is the original question followed by the four original choices in order; the symbolic [number] in the GSM8K instruction is literal text, not replaced before generation. The greedy answer determines the binary target for every uncertainty method. Use deterministic greedy generation, no sampling, minimum new tokens 2, maximum new tokens 16 (MMLU), 512 (GSM8K), and 64 (CoQA and TriviaQA); stop on EOS. Save token IDs, token log probabilities, entropy ingredients, decoded answer, stop reason, and truncation flag. Truncated or unparsable answers remain in the study and are scored by the declared task rule. For sampling-based uncertainty methods, draw 10 additional answers at temperature 1.0, top-pp = 1.0, and top-kk = 50, seed 181, with the same token limits. Sampling answers do not replace the greedy target. Hash and retain all outputs before computing new comparison tables.

Score methods and fairness of comparison. The essential panel is MaximumSequenceProbability, Perplexity, MeanTokenEntropy, CCP, PTrue, SemanticEntropy, and SAR: all 21 unordered pairs. Use the LM-Polygraph implementations at a pinned source revision, with the same default estimator definitions. The two auxiliary checkpoints are microsoft/deberta-large-mnli (CCP and SemanticEntropy) and cross-encoder/stsb-roberta-large (SAR), at pinned revisions. PTrue uses the same Qwen checkpoint and the source’s default True-token prompt, without an external judge. Use sequence CCP with 10 token alternatives; SemanticEntropy with class_probability_estimation=sum, entropy_estimation=mean, and use_unique_responses=False; and SAR with t=0.001t = 0.001. Do not substitute an estimator if it fails. The raw-score panel, marginal-rank control, and negative-class rank control use identical greedy outputs and labels. Marginal transforms use calibration scores without labels; negative-class transforms use only calibration examples declared correct by the task rule. Both controls use the strict piecewise-linear mid-rank interpolation with positive-slope end extrapolation defined in Appendix § 11, fitted only on that independent calibration sample. The implementation uses exact rational interpolation to preserve close-score ordering. If integer scores are needed for rank counting, encode the order of all exact transformed values with one common map per method across groups; this preserves every comparison and does not change the fitted pointwise function. No confirmation-dependent tie repair or refitting is allowed. No score orientation is chosen from confirmation AUROC. If a method fails on any row, report the failure counts and cause, retain the planned panel as incomplete, and show both a common-complete-case sensitivity and the unchanged remaining methods; do not silently delete a failing method from the registered family.

This panel includes token-level, verbalized, semantic-sampling, and relevance-weighted scores. The contribution is the comparison audit, so winning against these uncertainty estimators is not a success criterion. The audit baselines are pooled AUROC at q0q_0, within-task and macro AUROC, entrywise cross-AUC dominance, all two-group continua, uncorrected and corrected grids with m∈{10,20,40}m \in\{10, 20, 40\}, exact continuum optimization, the marginal-alignment envelope, and the paired Hellinger/TV comparator. All use the same allowed mixtures and score pairs. If a task-by-split bank contains only one class, mark its within-task AUROC and the full-simplex audit under the paper’s positivity assumption as undefined. Retain every row; do not resample, reassign splits, or relabel examples to create both classes. The implementation rejects such inputs. Separately reported restricted-domain analyses must state their changed estimand and admissible mixture region. If a calibration task has no correct examples, mark its negative-class rank control unavailable rather than borrowing labels from discovery or confirmation.

Primary correctness and human audit. Use first valid leading answer-letter extraction for MMLU, exact final numeric-answer comparison for GSM8K, official normalized alias exact match for TriviaQA, and official multi-reference CoQA token F1 thresholded at 0.5. The continuous CoQA F1 and thresholds 0.25, 0.75, 1.0 are prespecified sensitivities. Parser failure is incorrect. For numerical answers, remove thousands separators and compare exact Decimal values; use the final numeric token after the final The answer is marker, or the final numeric token if the marker is absent. No judge model defines the primary label.

The pinned Hugging Face CoQA export omits additional reference answers and original conversation IDs. For validation scoring only, join it to the official CoQA development set using exact story text, ordered question texts, and ordered primary-answer texts. A story-only join is insufficient. The completed source-consistency check uniquely matches all 500 validation conversations and 7,983 turns; each turn has four reference entries. The official evaluation script computes turn F1 by averaging the maximum over references after leaving each reference out in turn. Additional references enter neither prompts nor subset selection. Training calibration uses its available single reference and the evaluator’s single-reference branch. The join report will be released with the code. This check validates reference availability, not model answers.

Freeze all eligible confirmation row identifiers and operational labels before annotation. Within each task-by-label stratum sort the stable row identifiers lexicographically. Using one PCG64 generator with seed 18172027, iterate tasks in the order MMLU, CoQA, GSM8K, TriviaQA, and then labels 0, 1; in a stratum of size NN, draw min⁡(100,N)\min(100,N) indices uniformly without replacement, giving at most 200 answers per task. Save the sampled identifiers before annotation; do not replace difficult cases. Randomize the pooled presentation order with a separate generator (seed 18172028) and omit stratum and operational-label fields. Two independent annotators see the question, required context, references, and answer, but no uncertainty scores, mixture, method names, or operational label. They label task correctness as correct, incorrect, or indeterminate; a third adjudicator resolves disagreements. Retain indeterminate cases and report optimistic and pessimistic sensitivity bounds rather than deleting them. Report agreement, adjudication counts, and label error rates weighted back to stratum sizes. For a finite-bank interval, invert the hypergeometric sampling tails within each of the eight task-by-label strata, allocating .05/16 to each one-sided tail, then combine the resulting stratum bounds with known stratum weights. Do not treat balanced annotation sampling as a representative unweighted population.

If the audit finds more than 5% weighted mislabeling in a task, or shows a systematic answer-format failure, adjudicate that task’s entire confirmation bank before making a semantic-error claim. Freeze the repaired label rule without inspecting uncertainty-comparison outcomes and report the original and adjudicated results side by side. If full adjudication is not completed, restrict the conclusion to the operational task metric. No favorable comparison under an inaccurate label is counted as validation of human-error detection.

Mixture regions, witnesses, and primary endpoints. Use q0=(1/4,1/4,1/4,1/4)q_0 = (1/4, 1/4, 1/4, 1/4). Report the full simplex, qg≥0.05q_g \ge0.05, and their intersections with TV radii r∈{0.05,0.10,0.20,0.30}r \in\{0.05, 0.10, 0.20, 0.30\} about q0q_0. The full simplex is a stress domain, not an asserted deployment law. The bounded-share TV regions are declared deployment scenarios; no claim that a particular organization experiences them is permitted without data. For all 21 pairs report exact finite-bank dominance, explicit reversals, worst-gap brackets of width at most 10−410^{-4}, and reversal-radius brackets of width at most 10−310^{-3} for k∈{0.01,0.02}k \in\{0.01, 0.02\}. Select pair orientation from the raw discovery AUROC at q0q_0 and retain that orientation in all three score panels; break exact ties by the listed method order. For each panel and declared region, select a discovery near-minimizer of the normalized AUROC gap with a certified optimality bracket of width at most 10−410^{-4}. On confirmation evaluate these exact same mixtures before any separate reoptimization. Include unchanged and reoptimized results as different columns. A one-point negative gap is the prespecified materiality threshold, not a significance threshold.

Review-budget consequence. At budgets b∈{0.05,0.10,0.20}b \in\{0.05, 0.10, 0.20\}, route the largest-score fraction bb to review under each mixture, using random fractional allocation at the threshold to attain exactly the budget. On a finite bank, row ii in group gg has mass qg/ngq_g/n_g. Compute error recall among reviewed mass, error rate among accepted mass, and change from the equal-task mixture. This is a transparent simulation of a fixed review budget, not a deployed human study. Evaluate raw and calibrated scores at q0q_0 and at the unchanged discovery AUROC near-minimizer selected for each panel and declared region. A TV radius specifies a region, not an evaluation mixture. These are AUROC-selected mixtures; no review-budget objective is optimized over a region or used to select a confirmation mixture. AUROC reversal does not logically imply reversal of any one budget metric; agreement and disagreement must both be reported. Do not tune the review budget to produce agreement.

Uncertainty and resources. Report pointwise paired cluster bootstrap intervals (2,000 resamples, seed 181) for discovery-fixed confirmation witnesses and review-budget contrasts, resampling stories or duplicate-input clusters within tasks. These are approximate conditional intervals. Do not bootstrap score pairs as independent observations. For formal population statements, separately define an iid cluster sampling target, take one prespecified hash-selected question per independent cluster, and apply the simultaneous envelope at total δ=.05\delta= .05 across all 63 comparisons (21 pairs in each of the raw, marginal-rank, and negative-class-rank panels). The simultaneous mixture envelope already covers every declared region and adaptive witness within that family. CoQA label-threshold sensitivities are descriptive unless additional multiplicity allocation is explicitly made before evaluation. This changes the target from row-weighted to cluster-representative performance and must be reported as such. If independence is not defensible, report no population certificate. Do not lower the family size or enlarge the error budget after seeing inconclusive intervals. Report the number of exact solves and all unresolved optimizer stops. Run generation once; repeated greedy seeds are not independent model replications.

How outcomes change the claims. If the independent study shows material comparison changes inside the bounded TV scenarios and unchanged-witness transfer under validated labels, the paper may claim practical mixture sensitivity beyond the released bank. If marginal alignment again removes all hidden within-task-win reversals, retain the tractable-boundary interpretation; this supports the mechanism and does not establish a universally useful raw-score failure. If every reversal is found by two-group checks, keep the higher-support hierarchy as theory only. If raw reversals disappear under correctness validation, make no semantic-error claim. If population envelopes remain mostly unresolved, state the sample-efficiency limitation and keep population guarantees as conditional results. If AUROC changes do not affect the fixed review budgets, do not claim downstream utility. A null second-model result limits empirical scope; it does not falsify the theorems, and no post-hoc tasks are added to obtain a favorable replication.

O1: A Second Domain and More Groups

Use frozen torchvision ResNet-50 IMAGENET1K_V2 weights on the first 1,000 ImageNet validation image IDs under the domain-separated p181-selection-v1 subset-selection hash, and ImageNet-C severities 1, 3, 5 [7]. The 15 standard corruption types are Gaussian noise, shot noise, impulse noise, defocus blur, glass blur, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic transform, pixelate, and JPEG compression. Define Y=1Y = 1 when top-1 classification is wrong. Compare negative maximum softmax probability, softmax entropy, negative top-two logit margin, and negative log-sum-exp of logits at fixed temperature one, giving six pairs. Apply the shipped weight preprocessing and save logits; do not refit the classifier. Partition selected image IDs using SHA-256 of the UTF-8 encoding of p181-vision-split-v1 + NUL + image ID. Interpret the first eight digest bytes as an unsigned big-endian integer and take its remainder modulo three for calibration, discovery, and confirmation, respectively. These partitions have approximately equal sizes; report their actual sizes. All corrupted versions of one image remain together, and the split digest is independent of subset selection.

At each fixed severity treat the corruption types as groups; separately report the prespecified noise, blur, weather, and digital family aggregation (K=4K = 4) and the full K=15K = 15 analysis. Apply exact stationary-face and branch-and-bound audits to the four-family setting. For K=15K = 15, report only the corrected grids m=2,3,4m = 2, 3, 4: lower and upper brackets, explicit witnesses, grid-point counts, CPU times, and unresolved cases. The exact polytope construction is intended for small KK and is not applied at K=15K = 15. The same image-cluster kernel in Appendix § 9.§ 5 governs any population interval. Report raw and marginal-rank controls, all pair counts, certificate widths, finite bank reversals, processing times, and coverage only under an explicit independent-image sampling model. This evaluates domain transfer and scaling. Do not infer robustness to unseen corruption distributions.

O1 result. These results are also reported in § 5.3 and Table 2 of the main paper; the complete record follows. Inference over all 45,000 corrupted-image evaluations (verified official archives) was run once. For K=15K = 15, the general vertex oracle would enumerate (3014)\binom{30}{14} bases of the unconstrained simplex, which is impractical, so the aggregation uses the closed form for the unconstrained simplex (its vertices are the unit vectors), which agrees exactly with the oracle on small cases. This substitution was made after inference; no protocol setting, threshold, or split changed. The split produced 342 calibration, 316 discovery, and 342 confirmation images. Each of the 18 comparisons (six score pairs at three severities) is oriented once by raw discovery AUROC and kept fixed in both panels; a reversal is material at 0.01 AUROC.

GroupsPanelWinsFrozen rev. (material)Cert.Rev. witnessUnres.
K=4K = 4Raw15/180 (0)1530
K=4K = 4Marginal rank15/180 (0)1530
K=15K = 15Raw14/18− (0)1044
K=15K = 15Marginal rank14/18− (0)1044

Table 1 (PDF p. 28).

Wins counts all-group winners on discovery; Cert. counts confirmation continuum certificates; Rev. witness counts explicit confirmation reversal witnesses; Unres. counts unresolved comparisons. Frozen rev. counts, among discovery winners, reversals of the frozen discovery witness on confirmation, with material ones in parentheses; for K=15K = 15 only the material frozen-grid count is computed (−).

For the four-family setting (K=4K = 4), 15 of 18 comparisons are all-family winners on discovery. All 15 are certified as continuum winners on confirmation, the three non-winners carry explicit confirmation reversal witnesses, and no frozen discovery witness reverses. Under image-cluster resampling (2,000 replicates; pointwise, descriptive, not a population certificate) every raw-panel winner keeps its 95% interval above −0.01 at the frozen witness; in the marginal-rank panel 14 of 15 do. For all 15 corruptions (K=15K = 15), 14 of 18 comparisons win every group on discovery; the 14 winners give 10 confirmation certificates, 3 unresolved comparisons and one explicit confirmation reversal (severity 5, negative top-two margin against negative log-sum-exp; confirmation gap at most −0.050-0.050 for m=2,3,4m = 2, 3, 4); the four non-winners give three reversal witnesses and one unresolved comparison. No frozen discovery witness reverses materially. Raw and marginal-rank controls give identical counts. Each audit used under 10 s of CPU. This is one frozen model on one corruption benchmark; it says nothing about unseen corruption distributions.

E2: Auxiliary Smaller-Model Language Replication (Not E1)

E2 runs the E1 code path (prompts, generation budgets, the seven score methods, all three panels, mixture regions, frozen raw-discovery orientation, and the 0.01 materiality threshold) with three deviations. The model is the smaller Qwen3-1.7B in float16 (pinned revision) instead of Qwen3-8B in bfloat16. Selection is capped at 250 MMLU, GSM8K and TriviaQA questions and 60 CoQA stories for evaluation (100, 100, 100 and 30 for calibration), giving 1,715 evaluation rows instead of E1’s 12,948. Discovery and confirmation use a separate split salt, so E2’s splits do not coincide with E1’s. E2 was added after the E1 protocol was frozen, uses operational correctness labels only (no human audit), and cannot satisfy or replace any E1 endpoint. All 2,526 rows completed: 811 for calibration and 1,715 for evaluation (863 discovery, 852 confirmation), with 21 method pairs per panel. All computations used one pinned configuration; model generations and NLI scores for all 2,526 rows and SAR scores for 1,999 rows were computed before an interruption, and the remaining 527 SAR rows and every analysis step after resuming. No row was generated twice.

TaskEval. rowsError rateParser fail.Trunc.
MMLU25575.7%169238
CoQA96044.2%04
GSM8K25017.6%06
TriviaQA25081.2%00

Table 4.

PanelWinsFrozen rev. (material)Univ. cert.Median radiusBoot. CI > 0 / < 0
Raw4/211 (1)40.2191 / 1
Marginal rank4/211 (1)40.0270 / 0
Neg.-class rank4/211 (0)40.0001 / 0

Table 5.

Error rate is the share of evaluation rows labelled incorrect. Wins counts pairs that win within every task on discovery. Frozen rev. counts, among those winners, confirmation reversals at the frozen full-simplex discovery witness (material ones, at most −0.01-0.01, in parentheses). Univ. cert. counts the pairs (of 21) for which the method ranked first on raw discovery at q0q_0 is certified on the confirmation bank to be at least as good at every mixture. This is a finite-bank certificate; no population certificate is computed. In every panel these are the three P(True)P(\mathrm{True}) comparisons and semantic entropy over CCP, which does not win within every task. In both aligned panels three further pairs are certified in the reverse direction. Median radius is the median discovery upper bound on r0.01r_{0.01} (Eq. (11)) over the pairs where it is finite (16, 18 and 16 of 21; the rest have r0.01=+∞r_{0.01}=+\infty). A value of 0 means the discovery gap at q0q_0 is already at or below −0.01-0.01 under the raw-discovery orientation. In each aligned panel 9 of 21 pairs are at 0, so the medians are not comparable across panels. Boot. CI counts winners whose pointwise 95% interval at the frozen witness lies entirely above or entirely below zero. The intervals come from resampling input or story clusters within each task on the confirmation split (2,000 replicates); they are descriptive, not simultaneous.

Four comparisons win every task on discovery: maximum sequence probability over CCP, and perplexity, mean token entropy, and SAR each over P(True)P(\mathrm{True}). The three P(True)P(\mathrm{True}) comparisons are finite-bank certified on the confirmation split at every mixture in all three panels. Three caveats apply. No population certificate is computed. Under marginal alignment all three are mixture dependent on discovery (minima −0.032-0.032, −0.039-0.039, −0.038-0.038). At the frozen witness, the pointwise bootstrap interval excludes zero for 1, 0 and 1 of the three in the raw, marginal-rank and negative-class-rank panels.

Maximum sequence probability over CCP is mixture dependent on confirmation. Its frozen discovery witness puts all weight on GSM8K and TriviaQA (about 0.52 and 0.48) and none on MMLU or CoQA. At that witness the confirmation gap is −0.037-0.037 for raw scores, −0.026-0.026 after marginal strict-rank alignment, and −0.002-0.002 after negative-class strict-rank alignment. The reversal is therefore material in the first two panels and not in the third. Under cluster resampling the raw interval is [−0.058,−0.014][-0.058, -0.014], entirely below −0.01-0.01. The marginal-rank interval [−0.056,0.005][-0.056, 0.005] and the negative-class-rank interval [−0.027,0.022][-0.027, 0.022] include zero. So, on a second model, a within-task winner again loses at a mixture chosen on discovery and held fixed. On the completed Stable LM 2 12B bank, marginal alignment removed all 66 raw within-task-winner reversals (of 239 winners). Here it does not remove the single raw reversal as a point estimate (−0.026-0.026), although that interval includes zero.

Three limits bound this result. First, the frozen MMLU budget of 16 new tokens is too short for this model, which usually does not begin with an option letter, the only form the parser accepts. 238 of 255 MMLU evaluation rows hit the budget, and 169 could not be parsed (169 of the 193 MMLU errors), so MMLU labels here mostly measure answer-format compliance. The headline reversal’s witness gives MMLU no weight. Second, TriviaQA uses exact alias matching and labels 81.2% of rows incorrect. Third, labels are operational, with no human audit. E2 shows that the reversal phenomenon appears on a second model, smaller than the E1 model and from the same family, on a capped and differently split selection. It is not evidence about Qwen3-8B, and E1 remains essential.

Implementation status. The audit, grid correction, marginal and metric envelopes, calibration controls, statistical intervals, and generic score-table interface are implemented and tested. The E1 model generation and human annotations were not completed; no output from them is used. O1 and E2 were run and are reported above. The replication configuration will be released with the code; no result is filled by an expectation or simulated model output.

Scope and Limitations

The realizability reduction and the support hierarchy are mathematical existence results; the grid, optimization, statistical, and alignment bounds hold under their stated assumptions; and the empirical results are exact properties of finite released score banks. We do not claim a new uncertainty estimator, AUC decomposition, copositivity algorithm, or a first certificate for shifted AUC. The full simplex is a stress domain rather than a realistic deployment law, exact methods become expensive for many groups, distribution-free population intervals can be conservative at benchmark sizes, and marginal rank alignment can remove genuine differences in base risk and is not a fairness criterion. The independent-model study E1 (Appendix § 13.§ 1) has not been run.

References

  1. [1]Stefan Bundfuss and Mirjam Dür. Algorithmic copositivity detection by simplicial partition. Linear Algebra and its Applications, 428(7):1511–1523, 2008. 4DOI
  2. [2]Stefan Bundfuss and Mirjam Dür. An adaptive linear approximation algorithm for copositive programs. SIAM Journal on Optimization, 20(1):30–53, 2009. 4DOI
  3. [3]Daniel Cieślak and Jacek Łubiński. Better policies, worse verifier audits. arXiv preprint arXiv:2610.29029, 2026. 1arxiv.org/abs/2610.29029
  4. [4]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 25
  5. [5]Siran Dai, Qianqian Xu, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. DRAUC: An instance-wise distributionally robust AUC optimization framework. In Advances in Neural Information Processing Systems, pages 44658–44670, 2023. 2arxiv.org/abs/2311.03055
  6. [6]Etienne de Klerk, Monique Laurent, and Zhao Sun. An error analysis for polynomial optimization over the simplex based on the multivariate hypergeometric distribution. SIAM Journal on Optimization, 25(3):1498–1514, 2015. 18arxiv.org/abs/1407.2108
  7. [7]Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019. 1, 8, 28
  8. [8]Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017. 8
  9. [9]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. 25
  10. [10]Holly Janes and Margaret S. Pepe. Adjusting for covariate effects on classification accuracy using the covariate-adjusted receiver operating characteristic curve. Biometrika, 96(2):371–382, 2009. 2, 5, 16pubmed.ncbi.nlm.nih.gov/22822245
  11. [11]Ray Jiang, Aldo Pacchiano, Tom Stapleton, Heinrich Jiang, and Silvia Chiappa. Wasserstein fair classification. In Proceedings of the 35th Uncertainty in Artificial Intelligence Conference, pages 862–872, 2020. 5arxiv.org/abs/1907.12059
  12. [12]Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, 2017. 25arxiv.org/abs/1705.03551
  13. [13]Nathan Kallus and Angela Zhou. The fairness of risk scores beyond classification: Bipartite ranking and the XAUC metric. In Advances in Neural Information Processing Systems, 2019. 1arxiv.org/abs/1902.05826
  14. [14]Bing Li, Constantine Gatsonis, Issa J. Dahabreh, and Jon A. Steingrimsson. Estimating the area under the ROC curve when transporting a prediction model to a target population. Biometrics, 79(3):2382–2393, 2023. 2DOI
  15. [15]Pengxi Liu, Yi Shen, Matthew M. Engelhard, Benjamin A. Goldstein, Michael J. Pencina, Nicoleta J. Economou-Zavlanos, and Michael M. Zavlanos. FairPOT: Balancing AUC performance and fairness with proportional optimal transport. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 1611–1622, 2025. 2arxiv.org/abs/2508.03940
  16. [16]Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. In Advances in Neural Information Processing Systems, pages 21464–21475, 2020. 8arxiv.org/abs/2010.03759
  17. [17]Wenbo Ma and Miguel A. Lejeune. A distributionally robust area under curve maximization model. Operations Research Letters, 48(4):460–466, 2020. 2DOI
  18. [18]Pascal Massart. The tight constant in the Dvoretzky–Kiefer–Wolfowitz inequality. The Annals of Probability, 18(3):1269–1283, 1990. 15DOI
  19. [19]João Matos, Ben Van Calster, Richard D. Riley, Paula Dhiman, and Gary S. Collins. Collapsibility of performance metrics in clinical predictive AI. arXiv preprint arXiv:2608.30568, 2026. 1, 16arxiv.org/abs/2608.30568
  20. [20]Andreas Maurer and Maximiliano Pontil. Empirical Bernstein bounds and sample variance penalization. In Proceedings of the 22nd Conference on Learning Theory, 2009. 5, 14arxiv.org/abs/0907.3740
  21. [21]Katta G. Murty and Santosh N. Kabadi. Some NP-complete problems in quadratic and nonlinear programming. Mathematical Programming, 39(2):117–129, 1987. 3, 12
  22. [22]Preetam Nandy, Cyrus DiCiccio, Divya Venugopalan, Heloise Logan, Kinjal Basu, and Noureddine El Karoui. Achieving fairness via post-processing in web-scale recommender systems. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 715–725, 2022. 2arxiv.org/abs/2006.11350
  23. [23]Yurii Nesterov. Random walk in a simplex and quadratic optimization over convex polytopes. CORE Discussion Paper 2003/71, CORE, Université catholique de Louvain, 2003. 4
  24. [24]Qwen Team. Qwen3-8B model card and checkpoint. https://huggingface.co/Qwen/Qwen3-8B, 2025. 25
  25. [25]Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019. 25arxiv.org/abs/1808.07042
  26. [26]Agus Sudjianto and Alice J. Liu. Decomposing global AUC into cluster-level contributions for localized model diagnostics. arXiv preprint arXiv:2508.07495, 2025. 1arxiv.org/abs/2508.07495
  27. [27]Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. Benchmarking uncertainty quantification methods for large language models with LM-Polygraph. Transactions of the Association for Computational Linguistics, 13:220–248, 2025. 5, 19, 21
  28. [28]Maurice G Weber, Linyi Li, Boxin Wang, Zhikuan Zhao, Bo Li, and Ce Zhang. Certifying out-of-domain generalization for blackbox functions. In Proceedings of the 39th International Conference on Machine Learning, pages 23527–23548. PMLR, 2022. 2, 18arxiv.org/abs/2202.01679

Paper details

vlnc:W_rvk5f59n
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

AI tools

View statement
Looking for related papers…
Contents