Calibration-Constrained Diagnostics for Evaluating LLM Self-Evaluation

Introduction

As large language models (LLMs) are increasingly embedded in autonomous reasoning pipelines, a practical question arises: when a model reports confidence, is that confidence informative for confidence-gated downstream decisions? Existing uncertainty quantification (UQ) tools (ECE [23], Brier score, ensemble and semantic-entropy scorers [19, 13, 84], and confidence elicitation [81, 29])

Preprint. report aggregate, distribution-level summaries. Aggregate summaries admit cancellation: a model placing half its mass near chance confidence and half at near-certainty can display low ECE yet be unreliable in both regimes, and can be preferred over a model with better category-level calibrated behavior.

Rather than propose another scalar uncertainty score, we evaluate whether self-reported confidence remains meaningful after simultaneously imposing discrimination, per-category calibration, and positive-margin constraints. We define two category-level, VC/fat-shattering motivated diagnostic proxies on an accuracy-by-calibration surface: PVC counts categories with self-evaluation accuracy SEA $\ge\gamma$, and C-PVC additionally requires per-category calibration error $\le\tau$. We aggregate across $(\gamma,\tau)$ via Volume Under Surface (VUS) and a positive-margin (PM) restriction PM-VUS over $\gamma> \tau+ 1/2$, the region in which a standard uniform-convergence interpretation is available under explicit capacity-envelope assumptions. These are empirical evaluation proxies, not literal VC-dimension estimates.

Contributions. (i) A reusable black-box evaluation protocol: three-stage pairwise self-evaluation requiring only generation, binary self-selection, and a confidence report, applicable to closed/API-only models and agentic systems. (ii) Calibration-constrained diagnostic metrics (PVC, C-PVC, VUS, PM-VUS) that separate discrimination, per-category calibration, and positive-margin behavior, complementary to ECE/Brier and ensemble UQ scorers. (iii) A controlled instantiation on Math-360 (a 360-problem diagnostic testbed in a supporting role, not a leaderboard) with cross-domain validation on TruthfulQA [39] and CommonsenseQA [67], plus a MATH-500 [27] robustness check in the appendix (Appendix § 8). (iv) Empirical finding: across 13 models (7B–32B plus Claude 3.5 Haiku) and four benchmarks, JiuZhang3.0-7B is best by ECE (0.199) and Brier (0.049) yet has PM-C-PVC-VUS only 0.019; Qwen2.5-32B-Instruct has worse ECE (0.260) but $\approx11\times$ higher PM-C-PVC-VUS (0.209): a deployment-relevant ranking reversal hidden by aggregate metrics. Across 17,420 predictions, the mean correct–incorrect confidence gap is 0.005; 21/52 model-dataset pairs show a negative gap. (v) Robustness: rankings are stable under 7 alternative categorizations (Spearman $\ge0.75$), 10–30% synthetic judge-label perturbation, and unanimous-only re-analysis.

Scope. We do not claim to estimate VC dimension, that LLMs universally cannot self-evaluate, or that existing UQ tools should be replaced. We claim that aggregate uncertainty summaries can hide a deployment-relevant failure mode under black-box access and per-category calibration constraints.

Relation to the workshop version. This paper builds on our earlier workshop paper [79], published as a non-archival paper at the NeurIPS 2025 LAW workshop and posted as version 1 of this arXiv record, which introduced the PVC and C-PVC measures, the pairwise self-evaluation protocol, and the Math-360 testbed, and reported results for eleven 7–8B models. The present version extends that work in four ways: (i) it reframes PVC and C-PVC as VC/fat-shattering motivated empirical evaluation proxies rather than capacity estimates, and adds the positive-margin restriction PM-VUS with the corresponding uniform-convergence interpretation; (ii) it enlarges the evaluation to 13 models, including Qwen2.5-32B-Instruct and Claude 3.5 Haiku, and centers the analysis on the ranking reversal between aggregate calibration metrics and calibration-constrained diagnostics; (iii) it adds robustness analyses under alternative categorizations, judge-label perturbation, and unanimous-only reference labels; and (iv) it releases the full evaluation artifact (protocol, prompts, reference labels, model outputs, and reproduction scripts).

Background

Classical VC dimension. For a binary hypothesis class $\mathcal{H} \subseteq\{h:\mathcal{X}\to\{0,1\}\}$, a set $\{x_1,\ldots,x_n\}$ is shattered by $\mathcal{H}$ if every labeling $(y_1,\ldots,y_n)\in\{0,1\}^n$ is realized by some $h\in\mathcal{H}$; VC($\mathcal{H}$) is the size of the largest such set [72, 12] (Schapire and Ben-David 2014). If VC($\mathcal{H}$) $=d$, standard uniform convergence gives, for any $\delta\in(0,1)$ and sample of size $n$,

with probability $\ge1-\delta$, where $R$ and $\widehat{R}$ are the true and empirical risks. Classical VC theory is restricted to binary outputs and is inadequate for LLMs with probabilistic predictions.

Fat-shattering dimension. For bounded real-valued classes $\mathcal{F} \subseteq[0,1]^{\mathcal{X}}$ and scale $\alpha> 0$, a set is $\alpha$-shattered if there exist thresholds $s_i$ such that for every sign pattern $\sigma\in\{\pm1\}^{n}$ some $f \in\mathcal{F}$ satisfies $\sigma_i(f(x_i)-s_i) \geq\alpha$ for all $i$; $\operatorname{fat}_{\alpha}(\mathcal{F})$ is the largest such $n$ [7, 1, 8]. For binary classes $\operatorname{fat}_{\alpha}(\mathcal{F}) = \operatorname{VC}(\mathcal{F})$ on $\alpha\in(0,1)$ and zero for $\alpha\geq1$. Fat-shattering is the scale-sensitive extension of VC that we use to motivate the positive-margin region.

Methods

Convention: idealized vs. empirical. Throughout, $\operatorname{PVC}_{\gamma}(\mathcal{F})$ and $\operatorname{C\text{-}PVC}_{\gamma}^{\tau}(\mathcal{F})$ denote idealized population-level capacity notions defined via shattering. Their empirical counterparts (no hat for readability) are category-level black-box diagnostics computed from SEA and calibration error on observed categories. The acronym “PVC” is retained to reflect the motivating idealized capacity notion; all empirical PVC/C-PVC values reported in this paper should be read as category-count diagnostics over a fixed evaluation design, not as literal VC-dimension estimates.

Idealized PVC and C-PVC

We extend the classical VC framework to probabilistic predictors $f : \mathcal{X} \to\Delta(\{0,1\})$, where $\Delta(\{0,1\})$ is the set of distributions over the binary output space. Rather than requiring exact realizability, we require each labeling to be realized in probability.

Definition 1 (Idealized Probabilistic VC Capacity). Fix $\gamma\in(0,1]$. $\operatorname{PVC}_{\gamma}(\mathcal{F})$ is the largest $d$ such that some $\{x_1,\ldots,x_d\} \subseteq\mathcal{X}$ admits, for every labeling $(y_1^*,\ldots,y_d^*) \in\{0,1\}^{d}$, a predictor $f \in\mathcal{F}$ with $\mathbb{P}(f(x_i)=y_i^*) \geq\gamma$ for all $i$.

PVC quantifies discriminative capacity but does not require confidence to match correctness. For high-stakes deployment, calibration is essential:

Definition 2 (Idealized Calibration-aware PVC Capacity). Fix $\gamma\in(0,1]$ and $\tau\in[0,1)$. $\operatorname{C\text{-}PVC}_{\gamma}^{\tau}(\mathcal{F})$ is the largest $d$ such that some $\{x_1,\ldots,x_d\}$ admits, for every labeling, a pair $(f,\hat{p}) \in\mathcal{F}$ with

Unlike population-level ECE [23], C-PVC imposes calibration on fixed inputs, preventing cancellation between over- and under-confidence errors.

LLMs as function classes. A pretrained LLM is not a single function but a family of predictors induced by parameters, prompts, and stochastic decoding. Three perspectives motivate this view: (i) a PAC-Bayesian / parameter-level randomization view [42, 21]; (ii) a prompt-based input-conditioning view in which a prompt $p \in\Pi$ indexes a hypothesis $x \mapsto f_W([p; x])$ [36, 35]; and (iii) a stochastic-decoding view that maps the model to a Gibbs-type probabilistic predictor [20]. Combining these, we fix pretrained parameters $W$ and vary prompts $\pi\in\Pi$ and decoding $\omega\in\Omega$: $\mathcal{F}(W) = \{f(\cdot; W, \pi, \omega) : \pi\in\Pi, \omega\in\Omega\}$. This is the operational hypothesis class to which our idealized capacity notions are applied (Appendix D).

Measuring PVC and C-PVC through Self-Evaluation

Direct shattering verification is intractable ($2^d$ labelings) and statistically unstable at the question level. We therefore operate at the category level: semantic categories (e.g., Algebra, Geometry) act as distributional evaluation units, with performance estimated by aggregating over questions sampled i.i.d. from a category-specific distribution $\mathcal{D}_C$. Categories are coherent reasoning domains, and satisfying the diagnostic constraints at the category level provides evidence of systematic discriminative behavior across problem types.

Evaluation protocol. Three stages (Figure 1): (1) the target model generates two distinct solutions for each problem; (2) the target model selects the preferred solution and reports a confidence on $[0, 100]$ (randomized presentation order, no reference-label access); (3) a three-judge LLM ensemble (Claude 3.7 Sonnet, Amazon Nova Premier, DeepSeek-R1) assigns a reference preference by majority vote. Pairwise comparison is the foundational structure of preference optimization [58, 52] and the de facto standard for LLM-based evaluation [Zheng et al., 2023]. The judge ensemble is used as a stable comparative reference, not as oracle ground truth; the diagnostic signal is the difference between models under a fixed reference.

A category is counted toward the empirical PVC proxy if $\operatorname{SEA}(f,C)\ge\gamma$, and toward the empirical C-PVC proxy if additionally $\operatorname{CalibError}(f,C)\le\tau$. These counts are operational diagnostics under the chosen prompt/decoding protocol, not literal VC-dimension estimates and not certified lower bounds on the true hypothesis-class dimension. Full positioning and the theory-to-practice mapping (inputs $\mapsto$ categories, shattering $\mapsto$ SEA constraint, calibration $\mapsto$ CalibError constraint) are in Appendix § 12.

Aggregation: VUS and Positive-Margin (PM) Restriction

To avoid threshold cherry-picking, we aggregate across $(\gamma,\tau)$ via

Theorem 1 applies only when $\gamma>\tau+1/2$, yielding a positive fat-shattering margin $\alpha=\gamma-\tau-1/2>0$. We thus report the positive-margin (PM) variant

The $1/(\gamma-1/2)$ factor normalizes the $\tau$-range at each $\gamma$; the leading $2$ makes PM-VUS comparable to full-region VUS. Rankings agree with an unweighted PM aggregation (Appendix § 19); “PM” denotes the positive-margin region only, not an empirical generalization guarantee.

Theoretical Results

Two results motivate the PM restriction: monotonicity of the idealized dimensions (Proposition 1) and a positive-margin uniform-convergence motivation (Theorem 1). Theorem 1 does not certify the empirical score; it identifies the region in which calibration-constrained category-level observations are compatible with standard margin-based uniform-convergence reasoning, assuming an external capacity envelope on the induced selection and residual classes. Neither result asserts that empirical category-level counts recover a true VC dimension. Full proofs and a positive-margin interpretation of the PVC proxy are in Appendix K and J.

Proposition 1 (Monotonicity of idealized dimensions). For $\mathcal{F}$ containing deterministic predictors: (i) for $\gamma_1 > \gamma_2$ in $(0,1]$, $\operatorname{VC}(\mathcal{F}) = \operatorname{PVC}_1(\mathcal{F}) \le\operatorname{PVC}_{\gamma_1}(\mathcal{F}) \le\operatorname{PVC}_{\gamma_2}(\mathcal{F})$; (ii) for fixed $\gamma$ and $\tau_1 > \tau_2$ in $[0,1)$, $\operatorname{C\text{-}PVC}_{\tau_2}^{\gamma}(\mathcal{F}) \le\operatorname{C\text{-}PVC}_{\tau_1}^{\gamma}(\mathcal{F}) \le\operatorname{PVC}_{\gamma}(\mathcal{F})$.

Classical VC equals PVC$_1$; relaxing $\gamma$ monotonically increases capacity; calibration constraints can only reduce it. The PVC–C-PVC gap quantifies the empirical cost of calibration.

Positive-margin uniform convergence. For each evaluation example $z = (C, q, J(q))$, let $s_f(z) = \mathbf{1}\{\operatorname{Select}_f(q) = J(q)\}$ and $r_{f,\hat{p}}(z) = \hat{p}_f(q) - s_f(z)$, with induced classes $\mathcal{S} = \{s_f\}$ and $\mathcal{R} = \{r_{f,\hat{p}}\}$.

Theorem 1 (Why the positive-margin region is statistically interpretable under capacity envelopes). Fix $(\gamma,\tau)$ with $\alpha= \gamma- \tau- \frac{1}{2} > 0$, and assume the induced classes admit the capacity envelope $\operatorname{VC}(\mathcal{S}) \le d_s$ and $\operatorname{fat}_{\alpha}(\mathcal{R}) \le d_r$. Let $D_{\gamma,\tau} = d_s + d_r$. For any $\epsilon,\delta\in(0,1)$, if $m \ge\widetilde{O}((D_{\gamma,\tau} + \log(1/\delta))/\epsilon^2)$, then with probability $\ge1-\delta$,

The proof (Appendix K) combines VC uniform convergence on $\mathcal{S}$ [4] with fat-shattering uniform convergence on $\mathcal{R}$ [7, 17]. Crucially, $d_s,d_r$ are assumed complexity envelopes on the induced classes, not the empirical C-PVC count; the theorem identifies the region (PM: $\gamma> \tau+ 1/2$) in which such an envelope can yield uniform convergence. The proxy reports what is observed; the theorem says where observations are theoretically interpretable.

Experiments and Results

Released artifact. The evaluation package is a complete evaluation suite, not a new dataset: Math-360 testbed (360 problems, 8 domains; full release), cross-domain splits for TruthfulQA / CSQA with verbatim prompts, judge-reference labels, model outputs for all evaluated models, the metric code for PVC / C-PVC / VUS / PM-VUS / SEA / CalibError, and scripts that reproduce every main-paper table and figure. The full artifact card is in Appendix A (Table 3). Three reproducibility levels are supported: L1 main-paper tables from released outputs (no API; CPU, <10 min); L2 judge labels from released candidates (judge-API); L3 end-to-end self-evaluation rerun (target-model access).

Experimental Setup

Testbeds. Math-360 is a controlled diagnostic testbed of 360 original math problems across 8 domains (balanced difficulty and subcategory), not a standalone general-purpose benchmark. Cross-domain validation uses TruthfulQA [39], CommonsenseQA [67], and MATH-500 [27], each grouped into 10 semantic categories with 240 questions (Appendices Y.3, Y.4, R).

Models. Thirteen models: eleven 7–8B open models (SFT, SFT+RL, RL-only, distillation, DPO), Qwen2.5-32B-Instruct [82], and Claude 3.5 Haiku [5] as a proprietary mid-scale reference (decoding config in Appendix Y.2).

Reference labels. Majority vote of three LLM judges (Claude 3.7 Sonnet, Amazon Nova Premier, DeepSeek-R1), invalid responses (< 0.4%) excluded, 3-0 agreement 53–75% (Appendix T). The object of evaluation is reference-agreement self-evaluation, not absolute correctness: all compared models face the same randomized majority-vote reference, and the diagnostic signal is the difference between models under that fixed reference (risk mitigations: Appendix F). We report full-region VUS and PM-VUS ($\gamma> \tau+ 1/2$); near-zero PM-C-PVC-VUS reads like a stress-test failure, not a metric defect: evidence that the model rarely enters a regime where high self-selection accuracy and tight calibration coexist. An exploratory SC-VUS proxy (not used in any main claim) is deferred to Appendix N.

Diagnostic Decomposition Across Models

Table 1 is a decomposition, not a leaderboard. Full-region columns summarize category-level behavior across all $(\gamma, \tau)$; the PM-region column asks the stricter question of whether behavior survives under simultaneous accuracy and calibration constraints with a positive margin.

Model	Full Region: PVC-VUS ↑	Full Region: C-PVC-VUS ↑	Full Region: Gap ↓	PM Region ($\gamma> \tau+ 0.5$): PM-C-PVC-VUS ↑	Calibration: CalibError ↓	Accuracy: SEA ↑
Qwen2.5-7B (Pretrain) [82]	5.26	3.76	1.50	0.06	0.313	0.556
Qwen2.5-7B-Instruct (SFT+RL) [82]	5.61	4.08	1.53	0.17	0.291	0.599
Qwen2.5-Math-7B-Instruct [83]	4.70	3.20	1.50	0.002	0.335	0.506
Llama-3.1-8B-Instruct (SFT+DPO) [45]	4.55	2.92	1.63	0.00	0.385	0.495
OpenThinker2-7B (SFT) [51]	5.37	3.50	1.87	0.002	0.357	0.580
DeepSeek-R1-Distill-Qwen-7B [24]	5.10	3.49	1.61	0.07	0.339	0.549
Bespoke-Stratos-7B (SFT) [11]	5.33	3.54	1.79	0.00	0.347	0.570
JiuZhang3.0-7B (SFT) [Zhou et al., 2024]	4.79	3.84	0.95	0.009	0.209	0.516
Ministral-8B-Instruct-2410 [46]	4.88	3.00	1.88	0.00	0.402	0.522
Open-Reasoner-Zero-7B (RL) [26]	5.29	3.51	1.78	0.02	0.352	0.566
s1.1-7B (SFT) [48]	5.83	4.27	1.56	0.12	0.285	0.622

Three observations emerge.

(1) The expressivity–calibration separation is pervasive. Models with higher PVC-VUS (s1.1-7B 5.83, Qwen2.5-7B-Instruct 5.61) exhibit larger PVC–C-PVC gaps (1.56, 1.53); JiuZhang3.0-7B has the smallest gap (0.95) at lower PVC-VUS (4.79), yet still has near-zero PM-C-PVC-VUS (0.009). No 7–8B model simultaneously achieves high PVC-VUS and a small gap.

(2) Training paradigms reshape rather than close the tension. Instruction tuning, RLHF, distillation, and DPO improve raw capability, yet none yields nontrivial PM-C-PVC-VUS at 7–8B (max 0.17, Qwen2.5-7B-Instruct). Higher PVC-VUS correlates with larger gaps: calibration cost grows precisely for the most discriminative models.

(3) Scale helps discrimination but not the PM score (Table 2). With Qwen2.5-32B-Instruct and Claude 3.5 Haiku, PVC-VUS improves modestly [29] but PM-C-PVC-VUS remains limited (Claude 3.5 Haiku 0.036–0.123; Qwen2.5-32B-Instruct 0.148–0.209 across the two pre-specified aggregation subsets). Ordering is preserved; the scarcity of PM-region behavior is not a 7–8B artifact.

Model	Scale	ECE $\downarrow$	Brier $\downarrow$	PVC-VUS $\uparrow$	C-PVC-VUS $\uparrow$	Gap $\downarrow$	PM-C-PVC-VUS $\uparrow$
JiuZhang3.0-7B	7B	0.199	0.049	4.79	3.84	0.95	0.019
Qwen2.5-7B-Instruct	7B	0.291	0.081	5.19	3.70	1.49	0.126
Llama-3.1-8B-Instruct	8B	0.385	0.140	4.40	2.86	1.55	0.000
s1.1-7B	7B	0.293	0.094	5.44	3.94	1.50	0.142
Qwen2.5-32B-Instruct	32B	0.260	0.073	5.57	4.20	1.37	0.209
Claude 3.5 Haiku	Proprietary	0.272	0.079	5.79	4.24	1.55	0.123

Aggregate UQ Can Reverse the Deployment Ranking

What other UQ metrics miss. AUROC/AUPRC test rank separability but are invariant to monotone transforms of confidence and ignore calibration. ECE/Brier test average confidence–accuracy alignment but admit cancellation across categories: high-confidence errors can be offset by low-confidence correct predictions. Selective-risk curves test a chosen operating point. None asks whether each semantic category simultaneously satisfies accuracy and calibration constraints in the positive-margin region. Our diagnostics expose exactly that, and are complementary to existing UQ tools [19, 13, 84], not a replacement.

Table 2 combines the scale extension and the ranking-reversal comparison. Across the scale tiers, raw discrimination (PVC-VUS) improves modestly with scale; PM-C-PVC-VUS remains limited and the ordering is preserved. The aggregate-vs-PM split is the key signal: JiuZhang3.0-7B is the best model by ECE (0.199) and Brier (0.049) yet has PM-C-PVC-VUS only 0.019, while Qwen2.5-32B-Instruct has worse ECE (0.260) but $\approx11\times$ higher PM-C-PVC-VUS.

Mechanism: bimodal confidence cancellation. Inspecting JiuZhang3.0-7B’s confidence distribution explains the mechanism. Roughly 53.9% of its predictions cluster near chance-level confidence ($\hat{p} \approx0.50$, SEA = 0.494), and 44.3% concentrate at near-certainty ($\hat{p} \approx0.985$, SEA = 0.583: a 41.7% error rate at near-full confidence). Averaging across these two regimes pulls aggregate calibration error down, while the high-confidence mass carries catastrophic miscalibration. Aggregate summaries observe only the average and report “well calibrated”; the per-category C-PVC diagnostic shows that no category simultaneously satisfies high accuracy and tight calibration in the PM region. This is not a cosmetic reformulation of existing summaries but a distinct deployment-relevant failure mode: for a confidence-gated pipeline, ECE/Brier recommend JiuZhang3.0-7B while Qwen2.5-32B-Instruct has $\approx11\times$ higher PM-C-PVC-VUS.

Confidence-Correctness Decoupling

A stylistic-cue account would predict that reported confidence meaningfully separates correct from incorrect self-evaluations. The data do not support this. Across all 13 models and 4 datasets (17,420 predictions), the mean confidence gap between correct and incorrect decisions is only 0.005. Across the 52 model–dataset pairs, 21 (40%) show a negative gap: models are on average more confident when they are wrong. The pattern is consistent per domain: Math-360 0.004, MATH-500 0.007, CSQA 0.005, TruthfulQA 0.000. Restricting to unanimous-judge pairs (cleaner reference), the gap remains only 0.008 with a 38.0% high-confidence error rate at confidence $\ge0.9$. Discrimination is partially real, but confidence is not actionable as a deployment gate.

Robustness

Category robustness. Rankings are stable under 7 alternative categorizations (random partitions (3 seeds), difficulty-based splits, and subcategory variants) across all 4 datasets, with C-PVC-VUS Spearman rank correlation in [0.75, 0.92]. This rules out the concern that the ordering is an artifact of the particular semantic partitioning.

Judge-label perturbation. Synthetically flipping 10–30% of judge labels (mean $\pm$ one s.d. over 5 seeds per level) degrades PVC-VUS gradually, as expected from standard concentration. The near-zero PM-C-PVC-VUS regime for models already at that level is stable: JiuZhang3.0-7B stays at 0.019–0.023 under 30% corruption. A label-noise explanation would either depress high-PM models toward zero or inflate low-PM models; neither happens. The central finding is therefore not a judge-noise artifact (Appendix U.2).

Unanimous-only re-analysis. Restricting to samples with 3-0 judge agreement raises mean PM-C-PVC-VUS from 0.073 to 0.320 while preserving the ranking structure (JiuZhang remains near zero at 0.047; s1.1-7B rises to 0.996). The cleaner-label condition does not erase the main distinction (Appendix § 28.§ 1).

Visual Signatures and Cross-Domain Analysis

Figure 2 summarizes the threshold–calibration behavior across models; Figure 3 shows the full 3D C-PVC surfaces. Domain-specific signatures: OpenThinker2-7B peaks on Math-360, Qwen2.5-7B-Instruct on TruthfulQA, s1.1-7B on CSQA; JiuZhang3.0-7B has the smallest PVC–C-PVC gap across all three.

Related Work

LLM reasoning and self-reflection. Large language models have achieved strong performance on complex reasoning tasks [31, 76, 15], evaluated on benchmarks including MathVista [37], Bongard-Logo [50], RAVEN [Zhang et al., 2019], and ARC [16]. Techniques such as DPO [58] and GRPO [61, 24, 82, 85] further enhance capabilities. Self-reflection research spans educational psychology [14, 22, 28, 34, 49, 54, 59, 71, 75] to LLM methodologies [63, 10]. Contemporary approaches (Reflexion [64], Self-Refine [40], CRITIC [55], and the iterative DPO/SFT R3 pipeline [73]) implement iterative refinement (typically with an external larger-model teacher that a calibrated self-evaluator could in principle replace), while Metacognitive Prompting [70] and metacognitive language models [74] attempt formalization. However, most frameworks prioritize final accuracy over assessing reasoning quality [47].

Calibration and uncertainty quantification. Calibration has been studied extensively [23, 53, 29], with techniques including temperature scaling [57], conformal prediction [3], deep ensembles [32], Bayesian approximations [20, 77], and balanced entropy measures [78]. Recent LLM-specific UQ benchmarks and tools include LM-Polygraph [19], UQLM [13], and the broader benchmarking of LLM uncertainty [84]. Confidence elicitation from LLMs has also been studied empirically [81]. Our framework is complementary to these tools rather than a replacement: aggregate UQ scores (ECE, Brier, semantic-entropy, ensemble scorers) report a distribution-level summary and can admit cancellation effects; our per-category, calibration-constrained diagnostics instead ask whether calibration-constrained self-evaluation behavior survives uniformly within the PM region (§ 5.3).

LLM-as-a-judge: bias, position, and scale effects. Model-based judging has well-documented limitations: LLM judges can favor their own generations [Panickssery et al., 2024], exhibit position and length biases [62], and only the largest judges reliably align with humans [69]. We partially mitigate these effects via a three-judge majority vote with randomized solution order and report agreement statistics (Appendix § 27) and label-perturbation robustness (§ 5.5); we do not claim judge labels are oracles.

Alignment and optimization. RLHF [52], DPO [58], Contrastive Preference Learning [25], self-rewarding mechanisms [86], and GRPO [61] have enhanced LLM capabilities [66]. Our CV-PCV metric evaluates whether such training improves introspective reliability or merely suppresses calibration errors.

Evaluation design and protocol choice. A growing body of work documents that evaluation design can change conclusions: holistic multi-scenario benchmarks make method comparisons more robust [38]; large collaborative benchmarks show that single-metric leaderboards understate capability variation [65]; and behavioral audits show that task-frequency and prompt phrasing systematically bias apparent performance [43]. Our diagnostics are in the same spirit, applied to the specific setting of self-reported confidence: we show that the aggregation rule of a calibration metric (full-surface vs. positive-margin, per-category vs. pooled) can reverse deployment-level conclusions under fixed reference labels and fixed models.

Theoretical foundations. The VC dimension [72, 12, 60] and fat-shattering dimension [1, 7, 8] provide foundations for model capacity and generalization. Extensions to probabilistic predictors [30] and neural networks [44, 4, 68, 9, 17] have not been applied to LLM self-evaluation. Our PVC/C-PCV diagnostics are VC/fat-shattering motivated but are reported as empirical evaluation proxies rather than literal capacity estimates, providing a theory-motivated way to study when calibration-constrained self-evaluation survives in the PM region.

Conclusion

We proposed a black-box, calibration-constrained evaluation framework for LLM self-evaluation. The framework introduces two theory-motivated, operational diagnostics (PVC and C-PVC) and a positive-margin restriction PM-VUS, which is intentionally stringent: it asks whether calibrated self-evaluation survives in the positive-margin region where Theorem 1 provides a uniform-convergence motivation under an assumed capacity envelope.

Across 13 models from 7B to 32B plus a proprietary mid-scale system and four benchmarks, we identify a deployment-relevant ranking reversal: the ECE/Brier-best model (JiuZhang3.0-7B) has near-zero PM-C-PVC-VUS while Qwen2.5-32B-Instruct (worse ECE) has $\approx11\times$ higher PM-C-PVC-VUS. The mechanism is bimodal confidence clustering that cancels in averages but fails per-category calibration. Complementary analyses (mean correct–incorrect gap 0.005 across 17,420 predictions, categorization robustness at Spearman $\ge0.75$, stability under 10–30% judge-label perturbation) rule out categorization, judge noise, and model scale as artifacts. Optimizing directly for PM-C-PVC-VUS and extending to open-ended or multi-turn self-evaluation are natural next steps; claim boundaries and limitations are in Appendices § 10–§ 14.

Artifact / Evaluation Card

This appendix summarizes the released evaluation artifacts in a Dataset/Evaluation Card format, intended to make intended use, limitations, provenance, and reproducibility explicit for downstream users.

Field	Description
Artifact name	LLM Self-Evaluation Diagnostics Package (v1.0): protocol, metrics, testbed, and scripts.
Components	(i) three-stage self-evaluation protocol (two candidate-generation prompts, self-selection prompt, judge prompt, under prompts/); (ii) Math-360 (data/math360/; 360 original problems, 8 domains, 5 subcategories, difficulty-stratified); (iii) cross-domain category splits for TruthfulQA, CommonsenseQA, MATH-500 (data/cross_domain/); (iv) model outputs (model_outputs/*_self_eval.csv); (v) three-judge reference labels (reference_labels/; schema documented); (vi) selfeval package (models, judges, evaluation, experiment, utils) with YAML configs; (vii) scripts for PVC/C-PVC computation, cross-dataset aggregation, end-to-end evaluation, main-table reproduction, and reference-label rebuild; (viii) documentation (README, reviewer README, release-strategy, dataset/evaluation cards, Croissant metadata).
Intended use	Diagnosing reliability of LLM self-reported confidence under per-category accuracy and calibration constraints; comparing models under a fixed reference signal; ablating evaluation assumptions.
Not intended use	General-purpose math leaderboard; absolute human-level capability ranking; single-scalar replacement for ECE/Brier/AUROC; any claim that a model’s self-evaluation is reliable outside the declared protocol.
Math-360 source	Original problems authored for this work, stratified across 8 domains (Arithmetic, Algebra, Calculus, Geometry, Number Theory, Combinatorics, Statistics, Linear Algebra), 5 subcategories each, difficulty-stratified (Easy / Medium / Hard). Released in full.
Reused datasets	TruthfulQA [39], CommonsenseQA [67], MATH-500 [27]. Question text is not redistributed; we release only IDs, category mappings, and derived outputs. Sampling in Appendix § 32.§ 4.
Reference-labeling	Three-judge LLM ensemble (Claude 3.7 Sonnet, Amazon Nova Premier, DeepSeek-R1); pairwise binary preference; majority vote; invalid responses (< 0.4%) excluded. See §3.2 and Table 4.
Reference is not	A human-validated ground truth; an oracle of mathematical correctness; a guarantee of absolute correctness for any individual pair.
Known limitations	Judge bias / self-preference (mitigated via three-judge majority with randomized order); category dependence (stable under 7 alternative categorizations, Spearman ≥ 0.75); small controlled math set of 360 items (validated by cross-domain replication); proprietary judge versions may drift (pinned at experiment time).
Version / frozen split	Math-360 v1.0; TruthfulQA/CSQA/MATH-500 splits v1.0 (Appendix § 32.§ 4). Future revisions will be versioned.
License	Math-360 original data, reference labels, and derived outputs: CC-BY-4.0. selfeval code and reproduction scripts: Apache-2.0. Reused datasets retain upstream licenses.
Hosting	Code and data are hosted together at the public GitHub repository https://github.com/calibration-constrained-diagnostics/selfeval-diagnostics/. Croissant metadata accompanies the data release.
Reproducibility	Three levels: L1 metric-only reproduction of every main-paper table from released model_outputs/ (< 10 min, CPU); L2 judge-label recomputation with judge-API access; L3 end-to-end self-evaluation reruns with target-model access. Prompts, configs, and $(\gamma,\tau)$ grid are pinned in the release.

Use of Large Language Models in Manuscript Preparation

The authors used LLM-based assistants during the preparation of this manuscript. This use was limited to editorial support: grammar and wording suggestions, improving the readability of author-written text, and checking LaTeX formatting. Text produced with such assistance was treated as a draft only; the authors reviewed, revised, and verified every passage before including it.

The research questions, the evaluation protocol, the PVC/C-PVC definitions, the theoretical statements and proofs, the experimental design, and the interpretation of the results are the work of the authors. LLM assistants were not used to produce or modify reported results. LLMs are not authors of this paper. The authors take full responsibility for the entire content of the paper, including any errors that may remain.

This appendix covers LLM use for manuscript preparation only. The separate, methodologically declared use of LLMs as (i) the subject of evaluation and (ii) a reference-labeling ensemble is described in § 3.2, § 5, Appendix § 27, and Appendix § 32.

Limitations and Broader Impacts

Limitations. Several limitations apply. (i) The main table targets 7–8B open models; the scale extension adds Qwen2.5-32B-Instruct and Claude 3.5 Haiku but not frontier-scale systems. (ii) Empirical PVC/C-PVC are operational diagnostic proxies under black-box access, not literal VC-dimension estimators; exact shattering is unverifiable and the true hypothesis class is unobservable. (iii) Reference labels come from a three-judge LLM ensemble rather than human annotation; majority voting with label-perturbation robustness (Appendix § 28.§ 2) and unanimous-only re-analysis (Appendix § 28.§ 1) support comparative use, but some label noise persists and proprietary judge versions may drift. (iv) The two-solution pairwise protocol is a specific reduction of self-evaluation; open-ended generation and multi-turn self-critique are not directly covered. (v) Tasks are reasoning-centric (Math-360, MATH-500, TruthfulQA, CSQA); behavior on subjective or creative tasks is future work. (vi) The framework is diagnostic and does not propose a training remedy.

Broader impacts. This paper is primarily diagnostic: it releases tools and benchmark procedures for measuring existing model behavior rather than introducing new model capabilities. The main intended positive impact is helping practitioners identify models prone to overconfident yet incorrect self-assessments before deployment, reducing the risk of silent failures in confidence-gated autonomous pipelines. The main residual risk is misuse of the diagnostic as a false sense of security: favorable PVC/C-PVC scores in evaluated categories do not guarantee reliability in untested settings. We therefore emphasize that PM-C-PVC-VUS should be read alongside, not in place of, other uncertainty metrics and domain-specific safety evaluations. A detailed claim-boundary table (what the artifact enables and does not enable) is in Appendix § 14.

LLMs as Function Classes: Three Views

The category-level diagnostic framework treats an LLM as a family of predictors rather than a single fixed function. Three well-founded perspectives motivate this view.

Bayesian view (parameter-level randomization). Treating the parameter vector $W$ as a random variable $W \sim P$ induces $\{\mathcal{F}_W : W \in\operatorname{supp}(P)\}$. This is standard in PAC-Bayesian theory [42, 21], underlies the NN–GP correspondence [33, 41], and has yielded non-vacuous generalization bounds [18].

Prompt-based view (input-conditioning). With fixed parameters $W$, varying a prompt $p \in\Pi$ induces $\mathcal{F}_{\mathrm{prompt}}(W) = \{x \mapsto f_W([p;x]) : p \in\Pi\}$, consistent with prompts functioning as task-specific adapters [36, 35] (Xiao et al. 2022).

Stochastic decoding view (output-level randomization). Temperature/nucleus sampling induces a distribution over outputs, mapping the model to a probabilistic predictor aligned with Gibbs predictors [42] and MC-dropout Bayesian approximation [20].

Combining these, we fix $W$ and vary prompts $(\Pi)$ and decoding $(\Omega)$: $\mathcal{F}(W) = \{f(\cdot;W,\pi,\omega) : \pi\in\Pi,\omega\in\Omega\}$. This is the operational hypothesis class to which our idealized capacity notions are applied.

Operational Positioning of the PVC/C-PVC Proxies

The empirical PVC/C-PVC quantities are tractable, theory-motivated diagnostics under black-box access, not literal VC-dimension estimators. Direct verification of all $2^d$ labelings is intractable, and the true hypothesis class $\mathcal{F}$ is unobservable under black-box access. What the category-level count provides is not a certified lower bound on the true hypothesis-class dimension, but an operational diagnostic under the chosen prompt/decoding protocol. This positioning preserves the structural separation between discrimination, per-category calibration, and positive-margin region behavior, the three quantities that aggregate uncertainty summaries conflate.

Theory-to-practice mapping. (i) inputs $x_i \mapsto$ semantic categories $C$; (ii) shattering condition $\mathbb{P}(f(x_i) = y_i^*) \ge\gamma\mapsto$ empirical SEA$(f,C) \ge\gamma$; (iii) calibration condition $\lvert\hat{p}(x_i) - \mathbb{E}[1\{f(x_i) = y_i^*\}]\rvert\le\tau\mapsto$ category-level Calibrator$(f,C) \le\tau$. Under this mapping, empirical PVC/C-PVC counts categories satisfying the corresponding operational constraints. These counts can be interpreted as lower-bound evidence for the idealized capacity notion *only under explicit richness assumptions on the prompt/decoding-induced family; we report them as operational evaluation proxies, not distribution-free estimates of the true hypothesis-class dimension. Categories are distributional evaluation units, not literal VC inputs.

Reference-Label Risks and Mitigations

Table 4 summarizes the reference-label risks we anticipate, the mitigations built into the protocol, and the limitations that remain after those mitigations.

Risk	Mitigation	Remaining limitation
Judge bias / self-preference	3-judge ensemble, randomized presentation order	Not a human oracle
Label noise	10–30% synthetic label perturbation (5 seeds)	Synthetic corruption only
Ambiguous pairs (both correct / both wrong)	Unanimous-only (3-0) re-analysis	Reduces effective sample
Style-preference artifacts	Same reference applied across all compared models	May still reward judge-preferred style

Claim Boundary: What This Artifact Enables and Does Not Enable

Table 5 states the claims that the released diagnostic package is designed to support, alongside the claims that it is explicitly not intended to support. This accompanies the per-field declarations in the artifact card (Appendix § 8).

Claims this artifact enables	Claims this artifact does not enable
Compare whether model-reported confidence tracks judge-referenced self-selection correctness across models.	Certify absolute correctness of model reasoning.
Detect category-level calibration/discrimination failures hidden by aggregate UQ summaries.	Provide human-preference alignment without additional human validation.
Analyze whether calibration-constrained self-evaluation behavior survives in the positive-margin (PM) region.	Literally estimate VC dimension or derive distribution-free sample-complexity bounds.
Reproduce and extend PVC/C-PVC/PM-VUS surfaces across new models and datasets under the declared protocol.	Make universal claims that LLMs cannot self-evaluate outside the declared protocol.
Diagnose deployment-relevant failure modes for confidence-gated downstream decisions.	Certify deployment safety of any specific model.

Notation

Table 6 collects the notation used throughout the paper.

Symbol	Description
$\mathcal{X}$	Input space (set of problem categories)
$\mathcal{F}$	Class of probabilistic predictors $f:\mathcal{X}\to\Delta(\{0,1\})$
$\Delta(\{0,1\})$	Set of probability distributions over binary output space $\{0,1\}$
$\gamma\in(0,1]$	Accuracy threshold parameter
$\tau\in[0,1)$	Calibration error tolerance parameter
$\operatorname{VC}(\mathcal{F})$	Classical Vapnik-Chervonenkis dimension of function class $\mathcal{F}$
$\operatorname{PVC}_{\gamma}(\mathcal{F})$	Probabilistic VC dimension at accuracy threshold $\gamma$
\operatorname{C\mathchar`-PVC}_{\gamma}^{\tau}(\mathcal{F})	Calibration-aware PVC dimension with parameters $\gamma$ and $\tau$
$\hat{p}:\mathcal{X}\to[0,1]$	Confidence scoring function associated with predictor $f$
$\operatorname{fat}_{\alpha}(\mathcal{F})$	Fat-shattering dimension at scale $\alpha$
$\alpha=\gamma-\frac{1}{2}$	Margin parameter relating PVC to fat-shattering dimension
$\mathbb{P}(f(x)=y)$	Probability that predictor $f$ outputs label $y$ on input $x$
$\mathbb{E}[\cdot]$	Expectation operator
$\mathbf{1}\{\cdot\}$	Indicator function
$\operatorname{SEA}(f,C)$	Self-Evaluation Accuracy of model $f$ on category $C$
$\operatorname{Judge}(q)$	Reference preference label (three-judge majority vote) for question $q$
$\operatorname{Select}_{f}(q)$	Model $f$’s selected solution index for question $q$
$\operatorname{CalibError}(f,C)$	Calibration error of model $f$ on category $C$
\operatorname{M\mathchar`-VUS}(\mathcal{F})	Volume Under Surface for metric M\in\{\operatorname{PVC},\operatorname{C\mathchar`-PVC},\operatorname{SC}\}
$\operatorname{SC}(\gamma,\tau,\mathcal{F})$	Sample complexity bound with parameters $\gamma,\tau$
$\epsilon$	Generalization error tolerance
$\delta$	Confidence parameter in generalization bounds
$K$	Universal constant in sample complexity bounds
$\Pi$	Set of prompts in prompt-based function class
$W$	Parameter vector of a language model
$\mathcal{P}$	Probability distribution over parameter space
$\mathcal{D}$	Distribution over input space $\mathcal{X}$
$\mathcal{D}_{C}$	Distribution of questions within category $C$

Proof of Proposition 1

Proof. We prove the two statements in order.

Statement 1. For any $\gamma_{1},\gamma_{2}\in(0,1]$ with $\gamma_{1}>\gamma_{2}$:

By definition, $\operatorname{PVC}_{\gamma}(\mathcal{F})$ is the largest integer $d$ such that for every labeling $(y_{1}^{*},\ldots,y_{d}^{*})$, there exists $f\in\mathcal{F}$ satisfying $\mathbb{P}(f(x_{i})=y_{i}^{*})\geq\gamma$ for all $i\in[d]$.

When $\gamma=1$, this requires $\mathbb{P}(f(x_{i})=y_{i}^{*})=1$. Under the assumption that $\mathcal{F}$ contains deterministic predictors, this coincides with classical VC shattering, yielding $\operatorname{PVC}_{1}(\mathcal{F})=\operatorname{VC}(\mathcal{F})$.

For monotonicity, suppose $\gamma_{1}>\gamma_{2}$. Any $\gamma_{1}$-shattered set is also $\gamma_{2}$-shattered since the weaker threshold is easier to satisfy. Thus $\operatorname{PVC}_{\gamma_{1}}(\mathcal{F})\leq\operatorname{PVC}_{\gamma_{2}}(\mathcal{F})$, completing the first chain.

Statement 2. For fixed $\gamma\in(0,1]$ and $\tau_{1}>\tau_{2}$ in $[0,1)$:

First, \operatorname{C\mathchar`-PVC}_{\gamma}^{\tau_{1}}(\mathcal{F})\leq\operatorname{PVC}_{\gamma}(\mathcal{F}) follows directly from definitions. C-PVC requires both accuracy ($\mathbb{P}(f(x_{i})=y_{i}^{*})\geq\gamma$) and calibration ($|\hat{p}(x_{i})-\mathbb{E}[\mathbf{1}\{f(x_{i})=y_{i}^{*}\}]|\leq\tau$). The additional calibration constraint can only reduce or maintain the dimension.

Second, \operatorname{C\mathchar`-PVC}_{\gamma}^{\tau_{2}}(\mathcal{F})\leq\operatorname{C\mathchar`-PVC}_{\gamma}^{\tau_{1}}(\mathcal{F}) holds because $\tau_{2}<\tau_{1}$ imposes a stricter calibration requirement. Any predictor satisfying calibration tolerance $\tau_{2}$ automatically satisfies $\tau_{1}$, so any $\tau_{2}$-shattered set is also $\tau_{1}$-shattered.

Combining these inequalities completes the proof.

Positive-Margin Interpretation of the PVC Proxy

The empirical PVC proxy reported in the main tables is not itself a sample-complexity parameter. It counts observed categories satisfying the $\gamma$-constraint and is a diagnostic indicator, not a distribution-free capacity estimate. The next proposition clarifies the role of PVC: it gives a lower-bound relationship between PVC shattering and fat-shattering of an associated real-valued class at a positive margin, which is a margin interpretation of the $\gamma$-constraint. This is not a generalization bound: lower bounds on class complexity do not control uniform convergence.

Reading of the proposition. This inequality says that if the PVC constraint is satisfied by a set of size $d$, then $\mathcal{G}$ fat-shatters a set of size at least $d$ at margin $\alpha$. It is a structural lower bound on the fat-shattering dimension of $\mathcal{G}$, not an upper bound. Consequently, it does not imply a sample-complexity guarantee in terms of $\operatorname{PVC}_{\gamma}(\mathcal{F})$.

What would be needed for a formal bound. A formal uniform-convergence bound for $\mathcal{G}$ requires an upper bound $\operatorname{fat}_{\alpha}(\mathcal{G})\le D$ for some capacity parameter $D$. When such a $D$ is available, the standard fat-shattering uniform-convergence inequality gives, for a universal constant $K$, $\sup_{g\in\mathcal{G}}\left|m^{-1}\sum_i g(x_i)-\mathbb{E}[g(x)]\right|\le\epsilon$ whenever $m\ge\widetilde{O}((D+\log(1/\delta))/\epsilon^2)$. The PVC proxy does not estimate such a $D$; it is a diagnostic indicator.

Relation to the empirical proxy and SC-VUS. The empirical PVC count in the main tables is an operational indicator of how many observed categories satisfy the $\gamma$-constraint, under the chosen prompt/decoding protocol. It is useful as a comparative diagnostic across models and thresholds, but it is not a distribution-free capacity estimate and not an absolute sample-size guarantee. This is why the main text reports SC-VUS as a relative verification-cost index rather than an absolute sample-size bound.

Proof. Suppose $\{x_1,\ldots,x_d\}$ is $\gamma$-shattered in the PVC sense. For any labeling $(y_1^*,\ldots,y_d^*)\in\{0,1\}^d$, there exists $f=(f_0,f_1)\in\mathcal{F}$ such that $f_{y_i^*}(x_i)\ge\gamma$ for all $i$. Considering $f_1$: if $y_i^*=1$, then $f_1(x_i)\ge\gamma=1/2+\alpha$; if $y_i^*=0$, then $f_0(x_i)\ge\gamma$ so $f_1(x_i)\le1-\gamma=1/2-\alpha$. Hence $\{x_1,\ldots,x_d\}$ is $\alpha$-shattered by $\mathcal{G}$ at thresholds $s_i=1/2$, giving $\operatorname{PVC}_{\gamma}(\mathcal{F})\le\operatorname{fat}_{\alpha}(\mathcal{G})$.

Proposition 2 follows immediately from Lemma 1. We emphasize once more that this is a lower bound on $\operatorname{fat}_{\alpha}(\mathcal{G})$ in terms of $\operatorname{PVC}_{\gamma}(\mathcal{F})$, so it supports the positive-margin interpretation of the $\gamma$-constraint but does not give a sample-complexity bound in $\operatorname{PVC}_{\gamma}$. A formal uniform-convergence statement for $\mathcal{G}$ requires a separate upper bound $\operatorname{fat}_{\alpha}(\mathcal{G})\le D$, as noted above.

Proof of Theorem 1

We prove Theorem 1 by applying standard uniform-convergence results separately to the induced correctness class $\mathcal{S}$ and the induced calibration-residual class $\mathcal{R}$, and combining them via a union bound. The role of C-PVC shattering is clarified at the end of the section.

Standard regularity assumptions.

1. Boundedness: For any $(f,\hat{p})\in\mathcal{F}$, the confidence function $\hat{p}_f(q)\in[0,1]$ and $s_f(z)\in\{0,1\}$; hence $r_{f,\hat{p}}(z)\in[-1,1]$.

2. Measurability: All functions are measurable with respect to the product measure on $\mathcal{Z}=(\mathcal{C},q,J(q))$.

Proof of Theorem 1. Let $S=\{z_i\}_{i=1}^{m}$ be drawn i.i.d. from $\mathcal{D}$, with $z=(C,q,J(q))$. For each $(f,\hat{p})\in\mathcal{F}$, recall

and the induced classes $S = \{s_f : (f, \hat{p}) \in\mathcal{F}\}$, $R = \{r_{f,\hat{p}} : (f, \hat{p}) \in\mathcal{F}\}$. By hypothesis, $\operatorname{VC}(S) \le d_s$ and $\operatorname{fat}_{\alpha}(R) \le d_r$ with $\alpha= \gamma- \tau- \frac{1}{2} > 0$. Write $D_{\gamma,\tau} = d_s + d_r$.

Step 1: Uniform convergence for $S$. Since each $s \in S$ is $\{0,1\}$-valued and $\operatorname{VC}(S) \le d_s$, the standard VC uniform-convergence inequality Anthony and Bartlett, [4] gives a universal constant $K_1 > 0$ such that for any $\epsilon, \delta\in(0,1)$, whenever

it holds with probability at least $1-\delta/2$ that

Step 2: Uniform convergence for $R$. Each $r \in R$ is bounded in $[-1,1]$ and $\operatorname{fat}_{\alpha}(R) \le d_r$ with $\alpha> 0$. The standard fat-shattering uniform-convergence bound for bounded real-valued classes Bartlett and Long, [7]; Colomboni et al. [17] gives a universal constant $K_2 > 0$ such that, whenever

it holds with probability at least $1-\delta/2$ that

Step 3: Union bound. Taking a union bound over the two events and choosing the larger of the two sample-size thresholds gives both inequalities simultaneously with probability at least $1-\delta$. The combined requirement is $m \ge\widetilde{O}((D_{\gamma,\tau}+\log(1/\delta))/\epsilon^2)$, matching the compact form in Theorem 1 after absorbing the logarithmic factors in $1/\epsilon$ and $1/\alpha$ into the $\widetilde{O}(\cdot)$ notation.

Role of the PM condition. The condition $\gamma> \tau+ \frac{1}{2}$ is exactly $\alpha> 0$, which makes Step 2 applicable; outside this region the margin is non-positive and the fat-shattering bound does not apply. The theorem thus motivates the PM restriction: PM-VUS is reported precisely on the region where Step 2 is well-defined.

Motivation lemma: positive-margin interpretation of C-PVC shattering.

The following lemma provides a structural link between the idealized C-PVC shattering condition and the PM condition $\gamma> \tau+ 1/2$. It is not a step in the proof of Theorem 1: it produces a lower bound on a particular auxiliary class, and lower bounds do not control uniform convergence. We include it only to make the positive-margin intuition explicit.

Proof. Let $S = \{x_1,\ldots,x_d\}$ be a set that is $(\gamma,\tau)$-shattered under C-PVC. For any sign pattern $\sigma\in\{-1,+1\}^d$, define $y_i^* = \frac{1}{2}(\sigma_i+1) \in\{0,1\}$. By C-PVC shattering, there exists $(f,\hat{p}) \in\mathcal{F}$ such that for all $i \in[d]$: $\mathbb{P}(f(x_i)=y_i^*) \ge\gamma$ and $|\hat{p}(x_i)-\mathbb{P}(f(x_i)=y_i^*)| \le\tau$.

Case 1 ($\sigma_i=+1$, i.e. $y_i^*=1$). Then $\hat{p}(x_i)\ge\gamma-\tau$, so $g(x_i)=\hat{p}(x_i)-\frac{1}{2}\ge\alpha$.

Case 2 ($\sigma_i=-1$, i.e. $y_i^*=0$). Then $\mathbb{P}(f(x_i)=1)\le1-\gamma$ and $\hat{p}(x_i)\le1-\gamma+\tau$, so $g(x_i)=-\left(\hat{p}(x_i)-\frac{1}{2}\right)\ge\alpha$.

In both cases $\sigma_i g(x_i)\ge\alpha$ for the specific $g\in G_{\mathrm{signed}}$ chosen for sign pattern $\sigma$. Since this holds for every $\sigma\in\{-1,+1\}^d$, $S$ is $\alpha$-fat-shattered at thresholds $s_i=0$.

Remark 1 (Scope of Lemma 2). This lemma does not provide a complexity upper bound. It only shows that the PM condition $\gamma>\tau+1/2$ corresponds to a positive signed margin under the idealized C-PVC constraints. Treating the true labeling $y^*$ as part of the class index also inflates $G_{\mathrm{signed}}$ beyond predictor-only variability, which is another reason it cannot be used directly as a generalization parameter. The role of Theorem 1 is therefore to supply a complexity envelope on actual induced classes ($S$ and $R$), and the PM condition is what aligns that envelope with a positive fat-shattering margin.

Proof of Theorem 1. Let $S = \{z_i\}_{i=1}^{m}$ be drawn i.i.d. from $\mathcal{D}$. For each $(f,\hat{p}) \in\mathcal{F}$, recall $s_f(z) = \mathbf{1}\{\operatorname{Select}_f(q) = J(q)\}$ and $r_{f,\hat{p}}(z) = \hat{p}_f(q) - s_f(z)$, with induced classes $\mathcal{S}$ and $\mathcal{R}$ as in the statement. By hypothesis, $\operatorname{VC}(\mathcal{S}) \le d_s$ and $\operatorname{fat}_{\alpha}(\mathcal{R}) \le d_r$ with $\alpha> 0$.

Step 1: Uniform convergence for $\mathcal{S}$. Each $s_f$ is $\{0,1\}$-valued and $\operatorname{VC}(\mathcal{S}) \le d_s$. The standard VC uniform-convergence inequality [4] gives a universal constant $K_1 > 0$ such that, whenever $m \ge K_1\epsilon^{-2}(d_s + \log(2/\delta))$,

with probability at least $1-\delta/2$.

Step 2: Uniform convergence for $\mathcal{R}$. Each $r_{f,\hat{p}}$ is bounded in $[-1,1]$ and $\operatorname{fat}_{\alpha}(\mathcal{R}) \le d_r$ with $\alpha> 0$. The standard fat-shattering uniform-convergence bound for bounded real-valued classes [7, 17] gives a universal constant $K_2 > 0$ such that, whenever $m \ge K_2\epsilon^{-2}(d_r\log^2(1/\epsilon) + d_r\log(1/\alpha) + \log(2/\delta))$,

with probability at least $1-\delta/2$.

Step 3: Union bound. Choose $K = \max(K_1,K_2)$. Taking a union bound over the two events yields both inequalities simultaneously with probability at least $1-\delta$. Absorbing the logarithmic factors into the $\widetilde{O}(\cdot)$ notation gives the compact form stated in Theorem 1.

Role of the PM condition. The condition $\gamma> \tau+ \frac{1}{2}$ is exactly $\alpha> 0$, which is what makes Step 2 applicable; outside this region the margin is non-positive and the fat-shattering bound does not apply. The theorem thus motivates the PM restriction: PM-VUS is reported precisely on the region where Step 2 is well-defined.

Derivation of PM-VUS Normalization

We explain the normalization factor in the Positive-Margin VUS (PM-VUS) definition and verify its consistency properties.

Definition Recap. The PM-VUS is defined as:

where $\mathcal{G}^* = \{(\gamma,\tau): \gamma> \tau+ \frac{1}{2}, \gamma\in(\frac{1}{2},1], \tau\in[0,\frac{1}{2})\}$ is the provable generalization region.

Purpose of the Factor $\frac{1}{\gamma-1/2}$. For each fixed $\gamma$, the valid range of $\tau$ is $[0,\gamma-\frac{1}{2})$, which has length $\gamma-\frac{1}{2}$. The factor $\frac{1}{\gamma-1/2}$ normalizes the $\tau$-integration at each $\gamma$, ensuring that the contribution from each $\gamma$ slice is independent of the varying $\tau$-range length, and that for $\tau$-independent metrics like PVC, the double integral reduces consistently to a single integral over $\gamma$.

Consistency Verification for PVC. Since PVC depends only on $\gamma$ (not $\tau$), we verify:

Thus, for PVC, the double integral correctly reduces to twice the integral over $\gamma\in(\frac{1}{2},1]$.

Purpose of the Leading Factor 2. After applying the $\frac{1}{\gamma-1/2}$ normalization, the effective $\gamma$-integration domain is $(\frac{1}{2},1]$ with length $\frac{1}{2}$. The factor 2 scales this to match the full $\gamma$-range of length 1, making PM-VUS directly comparable with full-region VUS.

Interpretation. This normalization ensures fair comparison: a model with uniform performance across the PM region will have PM-VUS comparable to its full-region VUS. The gap between them reflects performance degradation in the theoretically guaranteed regime.

Unweighted PM aggregation. To check whether the $1/(\gamma-1/2)$ weighting drives the main ranking, we also compute the unweighted PM integral,

and rank models by $\mathrm{PM\text{-}C\text{-}PVC\text{-}VUS}^{\mathrm{unw}}$. For the eleven 7–8B models in Table 1, the unweighted ordering is identical on the top-3 / bottom-3 and differs only in the middle tier by at most one position. In particular, the ranking reversal in § 5.3 (JiuZhang3.0-7B near-zero; Qwen2.5-32B-Instruct $\approx11\times$ higher) is preserved. The weighting therefore affects absolute numerical values, not the deployment-relevant conclusion.

Calibration Metrics

We define the Category-based Expected Calibration Error (ECE) as the weighted average of category-level calibration errors:

where $C$ is the set of categories, $|C|$ is the sample size per category, and $N$ is the total number of samples. The Actual Error (AE) measures the global failure rate:

Why category-based ECE? Standard ECE groups predictions by confidence bins. Our category-based variant groups by semantic domains, aligning with our theoretical premise that shattering occurs at the category level. This directly links the global ECE metric to the local constraints in C-PVC.

ECE vs. C-PVC. While both assess calibration, they operate differently. C-PVC is a hard constraint: a category contributes only if it strictly satisfies tolerance $\tau$. ECE is a soft average of deviations across all categories. Consequently, low ECE is necessary but not sufficient for high C-PVC.

To illustrate: consider two equally weighted categories where one is perfectly calibrated (error = 0) and another has error 0.14 with tolerance $\tau= 0.1$. The weighted ECE is 0.07, typically considered excellent. However, under C-PVC, the second category violates the tolerance and fails to contribute, effectively halving the dimension despite low ECE. This demonstrates that ECE can mask domain-specific failures through averaging, whereas C-PVC demands uniform reliability across all shattered categories.

Interpretation of SC-VUS

SC-VUS is a relative verification-cost index, not an absolute sample-size bound. Given an assumed capacity envelope $D_{\gamma,\tau}$ on the induced classes (as in Theorem 1), the uniform-convergence inequality takes the form

which is a function of $(\gamma,\tau)$ through $D_{\gamma,\tau}$. Because $D_{\gamma,\tau}$ is not directly observable from black-box evaluation, SC-VUS instead aggregates the empirical quantities reported at each $(\gamma,\tau)$ into a summary index that is comparable across models under a fixed aggregation rule.

Practical use. Models with higher SC-VUS require proportionally more validation under the same aggregation rule than models with lower SC-VUS; no absolute sample-size claim is implied. This index should be read alongside the PVC–C-PVC gap (which quantifies the empirical cost of imposing calibration constraints) and PM-C-PVC-VUS (which reports calibrated capacity in the PM region where positive-margin uniform convergence applies).

Cross-domain behavior. Our cross-domain analysis suggests that models with smaller PVC–C-PVC gaps (e.g. JiuZhang3.0-7B) tend to have more stable calibration-constrained behavior across domains, although this stability does not translate into large PM-C-PVC-VUS. We therefore interpret SC-VUS as a comparative reliability-cost summary rather than a guarantee-generating quantity.

Category-level Measurement for PVC and C-PVC

We compute PVC and C-PVC at the level of problem categories (e.g., Algebra, Geometry) rather than individual questions. This section justifies this design choice.

Statistical tractability. Estimating VC-style capacity at the question level is computationally intractable due to exponential growth of binary labelings ($2^n$). By defining the input space $\mathcal{X}$ as the set of semantic categories, we reduce the hypothesis space to a manageable finite set while preserving semantic diversity. Each category acts as a “macro-input,” enabling efficient shattering computation. Alignment with model organization. LLMs acquire reasoning skills at the domain level before mastering specific problem templates. Measuring shattering over category partitions aligns with this inductive bias, requiring models to succeed across related problems rather than overfitting to single inputs.

Stability in calibration estimation. Individual prediction confidences are noisy. We estimate correctness probability for category $x_i$ via expectation over the category-specific distribution $\mathcal{D}_{x_i}$:

This averaging mitigates single-question variance, providing stable signals for the C-PVC constraint.

Theoretical consistency. This formulation remains consistent with Definition 1. Mapping input $x_i$ to category $C$, the condition $\mathbb{P}(f(x_i)=y_i) \ge\gamma$ translates to:

Rather than demanding uniform correctness on every question (overly restrictive and noisy), this evaluates whether the model reliably solves domain $x_i$ with probability at least $\gamma$, a valid and robust estimate of expressive capacity.

On the Finiteness of the PVC/C-PVC Proxies

This appendix remarks on why the empirical PVC/C-PVC proxies are always finite in our setting. This is a finiteness statement, not a capacity-envelope upper bound on the induced classes required by Theorem 1.

Finiteness via category-level aggregation. Measuring PVC/C-PVC at the category level restricts the effective domain to a finite partition $\{D_1,\ldots,D_M\}$, which immediately yields

This is a bound on the count of categories satisfying the corresponding constraints, not on any idealized VC/fat-shattering dimension of the underlying predictor class.

Relation to classical finite-capacity results. Anthony and Bartlett [ [4] (Theorems 14.18–14.19) establish finite fat-shattering capacity for bounded-weight, Lipschitz feedforward networks. For modern Transformers with residual pathways, softmax attention, and layer normalization, direct capacity analysis is nontrivial and is not attempted here. Our framework does not depend on such architecture-specific bounds: the empirical proxies are finite by construction through the category-level partition, and the role of Theorem 1 is to supply a uniform-convergence motivation under an assumed capacity envelope on the induced classes, not to derive that envelope from architectural assumptions.

Per-Dataset Breakdown of the Main Diagnostic Table

Table 7 reports the per-dataset (Math-360 / TruthfulQA / CSQA) breakdown underlying the cross-dataset aggregate in Table 1, together with the exploratory PM-SC-VUS verification-cost proxy (baseline 283.5). The PM-SC-VUS column is an exploratory verification-cost proxy and is not used in any main claim (interpretation in Appendix § 21).

Model	PVC-VUS ↑	C-PVC-VUS ↑	Gap ↓	PM-PVC-VUS ↑	PM-C-PVC-VUS ↑	PM-SC-VUS ↑	CalibError ↓	SEA ↑
Qwen2.5-7B (Pretrain) [82]	5.26 (4.78/5.66/5.73)	3.76 (3.27/3.93/4.09)	1.50 (1.51/1.73/1.64)	1.25 (1.26/1.15/1.33)	0.06 (0.00/0.01/0.18)	289.77 (283.5/284.5/301.3)	0.313 (0.320/0.318/0.303)	0.556 (0.597/0.565/0.571)
Qwen2.5-7B-Instruct (SFT+RL) [82]	5.61 (4.65/6.13/6.04)	4.08 (3.13/4.73/4.40)	1.53 (1.52/1.40/1.64)	1.78 (1.24/2.07/2.03)	0.17 (0.00/0.47/0.03)	300.30 (283.5/330.6/286.7)	0.291 (0.337/0.247/0.290)	0.599 (0.581/0.613/0.604)
Qwen2.5-Math-7B-Instruct (SFT+RL) [83]	4.70 (4.32/4.63/5.16)	3.20 (3.10/2.95/3.56)	1.50 (1.22/1.68/1.60)	0.65 (0.59/0.54/0.82)	0.002 (0.00/0.00/0.01)	283.76 (283.5/283.5/284.2)	0.335 (0.289/0.388/0.329)	0.506 (0.539/0.463/0.517)
Llama-3.1-8B-Instruct (SFT+DPO) [45]	4.55 (4.84/4.13/4.68)	2.92 (3.51/2.44/2.80)	1.63 (1.33/1.69/1.88)	0.60 (1.47/0.34/0.00)	0.00 (0.00/0.00/0.00)	283.53 (283.5/283.5/283.5)	0.385 (0.284/0.462/0.408)	0.495 (0.606/0.413/0.467)
OpenThinker2-7B (SFT) [51]	5.37 (5.19/5.46/5.46)	3.50 (3.68/3.36/3.47)	1.87 (1.51/2.10/1.99)	1.34 (2.04/0.90/1.09)	0.002 (0.005/0.00/0.00)	283.70 (284.0/283.5/283.5)	0.357 (0.298/0.395/0.378)	0.580 (0.649/0.544/0.546)
DeepSeek-R1-Distill-Qwen-7B (Distill) [24]	5.10 (4.77/5.01/5.51)	3.49 (3.24/3.40/3.83)	1.61 (1.53/1.61/1.68)	1.19 (1.26/1.00/1.31)	0.07 (0.00/0.07/0.13)	290.49 (283.5/291.0/296.9)	0.339 (0.329/0.358/0.330)	0.549 (0.596/0.500/0.550)
Bespoke-Stratos-7B (SFT) [11]	5.33 (4.44/5.67/5.89)	3.54 (2.73/3.95/3.94)	1.79 (1.71/1.72/1.95)	1.17 (0.68/1.39/1.44)	0.00 (0.00/0.00/0.00)	283.53 (283.5/283.5/283.5)	0.347 (0.388/0.315/0.338)	0.570 (0.556/0.567/0.588)
JiuZhang3.0-7B (SFT) [Zhou et al., 2024]	4.79 (4.49/4.55/5.34)	3.84 (3.74/3.22/4.57)	0.95 (0.75/1.33/0.77)	0.49 (0.72/0.08/0.68)	0.009 (0.02/0.00/0.008)	284.47 (285.6/283.5/284.3)	0.209 (0.169/0.309/0.149)	0.516 (0.561/0.454/0.533)
Ministral-8B-Instruct-2410 (SFT+RL) [46]	4.88 (4.21/4.61/5.81)	3.00 (2.50/2.65/3.85)	1.88 (1.71/1.96/1.96)	0.73 (0.53/0.38/1.28)	0.00 (0.00/0.00/0.00)	283.53 (283.5/283.5/283.5)	0.402 (0.414/0.453/0.339)	0.522 (0.528/0.458/0.579)
Open-Reasoner-Zero-7B (RL) [26]	5.29 (4.52/5.47/5.88)	3.51 (2.78/3.71/4.02)	1.78 (1.74/1.76/1.86)	1.19 (0.83/1.13/1.60)	0.02 (0.00/0.06/0.00)	285.60 (283.5/289.7/283.5)	0.352 (0.301/0.347/0.327)	0.566 (0.564/0.540/0.588)
s1.1-7B (SFT) [48]	5.83 (4.76/6.22/6.52)	4.27 (3.22/4.71/4.88)	1.56 (1.54/1.51/1.64)	2.08 (1.25/2.17/2.81)	0.12 (0.00/0.22/0.13)	295.46 (283.5/305.8/297.9)	0.285 (0.311/0.256/0.267)	0.622 (0.593/0.621/0.652)

Evaluation on MATH-500 Benchmark

To assess robustness, we replicate our evaluation on MATH-500 [27]. Table 8 presents the results.

Model	PVC-VUS ↑	C-PVC-VUS ↑	Gap ↓	PM-PVC-VUS ↑	PM-C-PVC-VUS ↑	PM-SC-VUS ↑	CalibError ↓	SEA ↑
Qwen2.5-7B (Pretrain) [82]	3.93	2.56	1.37	0.71	0.00	283.5	0.356	0.561
Qwen2.5-7B-Instruct (SFT+RL) [82]	3.94	2.54	1.40	0.69	0.00	283.5	0.358	0.562
Qwen2.5-Math-7B-Instruct (SFT+RL) [83]	3.68	2.57	1.11	0.20	0.00	283.5	0.304	0.524
Llama-3.1-8B-Instruct (SFT+DPO) [45]	3.97	2.67	1.30	0.75	0.00	283.5	0.331	0.565
OpenThinker2-7B (SFT) [51]	4.36	3.09	1.27	1.43	0.00	283.5	0.294	0.622
DeepSeek-R1-Distill-Qwen-7B (Distill) [24]	3.95	2.52	1.43	0.67	0.00	283.5	0.369	0.563
Bespoke-Stratos-7B (SFT) [11]	4.19	2.88	1.31	1.24	0.11	294.1	0.330	0.597
JiuZhang3.0-7B (SFT) [Zhou et al., 2024]	3.74	3.11	0.63	0.35	0.00	283.5	0.169	0.534
Ministral-8B-Instruct-2410 (SFT+RL) [46]	3.97	2.50	1.47	0.81	0.00	283.5	0.377	0.566
Open-Reasoner-Zero-7B (RL) [26]	4.39	2.99	1.40	1.50	0.00	283.5	0.322	0.626
s1.1-7B (SFT) [48]	4.26	2.93	1.33	1.25	0.00	283.5	0.317	0.607

The findings largely mirror our main evaluation, reinforcing the robustness of our framework. JiuZhang3.0-7B maintains superior calibration, achieving the lowest Gap (0.63) and CalibError (0.169). Open-Reasoner-Zero-7B and OpenThinker2-7B lead in expressiveness (PVC-VUS: 4.39 and 4.36 respectively), consistent with their strong performance on Math-360.

Critically, PM-C-PVC-VUS equals zero for 10 of 11 models; only Bespoke-Stratos-7B achieves nonzero calibrated capacity (0.11) in the provable generalization region. This is notably worse than Math-360 results, where multiple models achieved nonzero PM-C-PVC-VUS, suggesting that reliable self-evaluation becomes more challenging on this benchmark.

Some ranking deviations point to benchmark-specific effects. Llama-3.1-8B-Instruct shows degraded calibration on MATH-500 compared to Math-360, despite maintaining moderate PVC-VUS. These cross-dataset results validate our framework’s stability while highlighting that calibration behavior can vary across problem distributions.

Extended Cross-Domain Analysis

Extended domain-specific signatures (main-text Figure 3): OpenThinker2-7B achieves the highest Math-360 performance (PVC-VUS = 5.19, SEA = 64.9%); Qwen2.5-7B-Instruct peaks on TruthfulQA (PVC-VUS = 6.13); s1.1-7B leads on CSQA (PVC-VUS = 6.52) with consistently high performance across domains. Calibration is more cross-domain consistent than raw discrimination: JiuZhang3.0-7B maintains the smallest PVC–C-PVC gap across Math-360, TruthfulQA, and CSQA (0.75, 1.33, 0.77), suggesting calibration behavior generalizes more robustly across domain boundaries than discrimination. Main-text Figure 2 visualizes the threshold and cross-calibration signatures.

Judge Correlation Study

To assess reference-label consistency, we analyzed correlations between model self-assessments and verdicts from three reference LLMs (Claude 3.7 Sonnet, Amazon Nova Premier, DeepSeek-R1) on Math-360. Figure 6 shows Pearson correlations and agreement rates; Table 9 summarizes overall statistics.

LLM Judge	Correlation	Agreement
Claude 3.7 Sonnet	−0.09	0.53
Amazon Nova Premier	0.18	0.61
DeepSeek-R1	0.01	0.55

Judge characteristics. The three judges exhibit distinct evaluative patterns. Amazon Nova Premier shows the highest alignment with target models (average correlation: 0.18, agreement: 0.61), suggesting evaluation criteria well-matched to 7–8B model reasoning patterns. Claude 3.7 Sonnet exhibits negligible or negative correlations (average: −0.09, agreement: 0.53), indicating stricter or more conservative evaluation standards. DeepSeek-R1 occupies middle ground (average correlation: 0.01, agreement: 0.55), providing balanced assessment without strong bias toward particular model families.

Justification for ensemble. The divergence in evaluative strictness substantiates our multi-judge protocol. Reliance on a single judge risks systematic bias: Claude’s strictness may underestimate capabilities while Nova’s alignment may overestimate them. Notably, the overall accuracy of models often exceeds agreement with any individual judge, indicating that single judges have non-overlapping error modes. The majority-consensus rule integrates these complementary perspectives, reducing sensitivity to individual judge preferences while we do not claim the resulting reference labels are oracle annotations.

Reference-Label Robustness: Unanimous-Only and Label-Perturbation Analyses

This appendix provides the detailed reference-label robustness analyses summarized in § 5.5. Both analyses are designed to test whether the near-zero PM-C-PVC-VUS regime is a judge-noise artifact.

Unanimous-only re-analysis

We restrict the evaluation to pairs on which all three LLM judges agree (3-0 unanimous vote). This removes contested cases at the cost of a smaller effective sample size, and serves as a cleaner-label stress test. Table 10 reports PM-C-PVC-VUS under the full-label and unanimous-only settings.

Model	PM-C-PVC-VUS (full)	PM-C-PVC-VUS (unanimous)
s1.1-7B	0.142	0.996
Qwen2.5-32B-Instruct	0.209	0.597
Claude 3.5 Haiku	0.123	0.588
JiuZhang3.0-7B	0.019	0.047
Llama-3.1-8B-Instruct	0.000	0.038

Restricting to unanimous cases raises mean PM-C-PVC-VUS from 0.073 (full) to 0.320 (unanimous) across the five models shown, with a wider spread (0.047 to 0.996) that preserves the original ranking structure. In particular, JiuZhang3.0-7B stays near zero (0.019 → 0.047), indicating that its calibration-constrained deficit is not driven by judge disagreement.

Synthetic judge-label perturbation

We further test whether the near-zero PM-C-PVC-VUS regime is a label-noise artifact by introducing synthetic perturbation: each reference label is independently flipped with probability $p \in\{0.10, 0.20, 0.30\}$. We repeat this 5 times per perturbation level with independent seeds and report mean ± one standard deviation. Table 11 reports PVC-VUS and PM-C-PVC-VUS for two representative models: s1.1-7B (best-in-class for raw discrimination at the 7B tier) and JiuZhang3.0-7B (near-zero PM-C-PVC-VUS baseline), under 0%, 10%, and 30% flip probability.

Model	Flip	PVC-VUS	PM-C-PVC-VUS
s1.1-7B	0%	20.90	0.142
s1.1-7B	10%	20.09 ± 0.30	0.103 ± 0.053
s1.1-7B	30%	18.42 ± 0.58	0.059 ± 0.064
JiuZhang3.0-7B	0%	17.40	0.019
JiuZhang3.0-7B	10%	17.22 ± 0.18	0.019 ± 0.014
JiuZhang3.0-7B	30%	17.03 ± 0.29	0.023 ± 0.026

Two observations are worth highlighting. First, raw discrimination (PVC-VUS) degrades smoothly with label noise, as expected from standard concentration: s1.1-7B drops from 20.90 to 18.42 at 30% flip probability. Second, PM-C-PVC-VUS for models already near zero under clean labels stays near zero under noise: JiuZhang3.0-7B’s PM-C-PVC-VUS moves only from 0.019 to 0.019–0.023 under 10–30% corruption. A label-noise explanation would predict that flipping labels should either substantially depress high-PM models toward zero (which occurs gradually) or inflate low-PM models away from zero (which does not occur). The observed behavior is consistent with the finding that calibration-constrained capacity in the positive-margin region is genuinely scarce, not an artifact of noisy reference labels.

Decoupling of Reasoning and Self-Evaluation

A fundamental question in the study of metacognition is whether self-evaluation capabilities are decoupled from primary reasoning skills, or merely a byproduct of generation performance. To investigate this, we analyzed the correlation between Solution Accuracy (SA), the proportion of correctly solved problems, and Self-Evaluation Accuracy (SEA), the proportion of correct pairwise judgments, at the category level across all 11 models and four benchmarks.

Weak Correlation Across Domains

As illustrated in Figure 7, we observe consistently weak correlations between generation correctness and evaluation capability across all datasets. This dissociation is particularly pronounced in mathematical reasoning: the Pearson correlation is negligible on Math-360 ($r = 0.079$) and remains weak on MATH-500 ($r = 0.191$). Even on TruthfulQA ($r = 0.148$) and CSQA ($r = 0.186$), correlations remain modest. This statistical independence suggests that a model’s proficiency in solving problems does not guarantee its capacity to discriminate between superior and inferior solutions.

Inverse Performance Patterns

Table 12 provides quantitative evidence of this decoupling through specific inverse patterns: cases where models with lower reasoning performance achieve higher evaluation accuracy.

Model	Math-360		TruthfulQA		CSQA		MATH-500
	SA	SEA	SA	SEA	SA	SEA	SA	SEA
Qwen2.5-7B	83.1	59.7	29.2	56.5	70.0	57.1	63.0	56.1
Qwen2.5-7B-Instruct	89.6	58.1	40.2	61.3	78.7	60.4	70.6	56.2
Qwen2.5-Math-7B-Instruct	90.8	53.9	9.4	46.3	49.4	51.7	79.2	52.4
Llama-3.1-8B-Instruct	63.9	60.6	30.4	41.3	68.5	46.7	41.4	56.5
OpenThinker2-7B	94.7	64.9	39.0	54.4	75.8	54.6	86.7	62.2
DeepSeek-R1-Distill	91.8	59.6	17.5	50.0	57.5	55.0	83.9	56.3
Bespoke-Stratos-7B	90.7	55.6	45.2	56.7	77.3	58.8	76.0	59.7
JiuZhang3.0-7B	61.7	56.1	13.8	45.4	37.1	53.3	36.0	53.4
Ministral-8B-Instruct	75.7	52.8	33.3	45.8	68.8	57.9	52.2	56.6
Open-Reasoner-Zero-7B	91.5	56.4	39.4	54.6	79.0	58.8	76.6	62.6
s1.1-7B	90.7	59.3	44.2	62.1	75.6	65.2	75.4	60.7

Case 1: MATH-500. Open-Reasoner-Zero-7B achieves a Solution Accuracy of 76.6%, substantially lower than OpenThinker2-7B’s 86.7% (a gap of 10.1 percentage points). Yet, Open-Reasoner-Zero-7B exhibits higher Self-Evaluation Accuracy (62.6% vs. 62.2%). If self-evaluation were simply a function of reasoning ability, we would expect the stronger reasoner to also be the better evaluator.

Case 2: TruthfulQA. s1.1-7B records a lower SA (44.2%) than Bespoke-Stratos-7B (45.2%), yet demonstrates vastly superior SEA (62.1% vs. 56.7%, a gap of 5.4 percentage points). This inversion is particularly striking given the similar reasoning performance.

Case 3: Math-360. Llama-3.1-8B-Instruct achieves only 63.9% SA compared to Qwen2.5-7B-Instruct’s 89.6%, yet attains higher SEA (60.6% vs. 58.1%). Despite a 25.7 percentage point deficit in problem-solving, Llama demonstrates superior self-assessment.

Implications for Self-Evaluation Assessment

These findings carry important implications for our framework:

Self-evaluation as an orthogonal skill. The weak correlations and inverse patterns collectively suggest that self-evaluation operates as a distinct metacognitive capability, structurally independent of raw reasoning generation. A model may excel at solving problems while being poor at recognizing correct solutions, or vice versa.

Necessity of specialized metrics. Standard accuracy benchmarks capture only generative proficiency (SA), leaving evaluative reliability (SEA) unmeasured. Our PVC and C-PVC dimensions provide orthogonal insights into model trustworthiness that cannot be inferred from task performance alone.

Practical deployment considerations. For autonomous systems that must self-verify outputs, these results caution against assuming that high task accuracy implies reliable self-assessment. A model selected purely for reasoning performance may exhibit poor introspective calibration, leading to overconfident errors in deployment.

In summary, the empirical decoupling of reasoning and self-evaluation validates the necessity of our specialized framework: metrics like C-PVC capture a fundamentally different aspect of model capability than traditional accuracy measures.

On the Insufficiency of Rank-Based Metrics for Self-Evaluation Assessment

The Isotonic Invariance of AUROC

A natural question in the evaluation of probabilistic reasoning systems is why established discriminative metrics, specifically the Area Under the Receiver Operating Characteristic curve (AUROC), are insufficient for quantifying the reliability of self-evaluation. The fundamental limitation of AUROC in this context stems from its property of isotonic invariance, which renders it blind to the absolute meaning of confidence scores.

Formally, let $\mathcal{D}$ be a distribution over inputs $\mathcal{X}$ and binary labels $\mathcal{Y} \in\{0,1\}$. For a probabilistic predictor $f:\mathcal{X} \to[0,1]$, the AUROC can be interpreted as the probability that a randomly selected positive instance $x^{+}$ is assigned a higher score than a randomly selected negative instance $x^{-}$:

This definition depends exclusively on the ordinal ranking of predictions, independent of their absolute magnitudes. Consequently, for any strictly monotonic transformation $g:[0,1]\to[0,1]$, the AUROC remains invariant:

While this invariance is advantageous for tasks requiring only relative discrimination, it is fundamentally inadequate for autonomous systems where the absolute value of $f(x)$, the reported confidence, must serve as a reliable proxy for the true probability of correctness.

The Calibration Blindness Problem

The isotonic invariance of AUROC leads to a critical failure mode we term Calibration Blindness: a model may exhibit perfect discriminative power while being dangerously miscalibrated.

Illustrative Example. Consider two predictors performing self-evaluation on a category of problems, both achieving 65% accuracy but differing in reported confidence. Predictor $f_A$ (well-calibrated) reports 65% confidence, yielding calibration error $|0.65 - 0.65| = 0$. Predictor $f_B$ (overconfident) reports 99% confidence, yielding calibration error $|0.99 - 0.65| = 0.34$. Any rank-based metric, including AUROC, would treat $f_A$ and $f_B$ as equivalent since their ordinal rankings are identical. However, deploying $f_B$ in high-stakes environments poses serious risks: the system would proceed based on hallucinated certainty, bypassing uncertainty-based safety mechanisms.

How C-PVC Addresses Calibration Blindness

The Calibration-aware Probabilistic VC (C-PVC) dimension provides a principled remedy by introducing a metric explicitly sensitive to the absolute values of confidence scores. For a category to contribute to the C-PVC dimension, a predictor must satisfy both accuracy threshold $\gamma$ and calibration tolerance $\tau$:

Applying this constraint with $\tau= 0.1$:

$f_A : |0.65 - 0.65| = 0.00 \le0.1$ ✓ $f_B : |0.99 - 0.65| = 0.34 > 0.1$ ✗

Thus, C-PVC correctly penalizes $f_B$ while accepting $f_A$, distinguishing between models that merely rank correctly versus those with trustworthy self-knowledge.

The Critical Role of the Positive-Margin (PM) Region

While C-PVC addresses calibration blindness, a further question arises: under what conditions do these measurements admit a uniform-convergence interpretation? This is where the PM region becomes essential.

Theorem 1 establishes that generalization bounds for C-PVC hold only when $\gamma> \tau+ \frac{1}{2}$. This constraint ensures a positive fat-shattering margin ($\alpha= \gamma- \tau- \frac{1}{2} > 0$), which is necessary for uniform convergence guarantees. The PM-VUS metrics restrict evaluation to this theoretically grounded regime:

Why PM-C-PVC-VUS Matters. Consider the contrast between full-region and PM-region metrics from our experiments (Table 1):

Model	C-PVC-VUS (Full)	PM-C-PVC-VUS
Qwen2.5-7B-Instruct	4.08	0.17
s1.1-7B	4.27	0.12
JiuZhang3.0-7B	3.84	0.009

The dramatic drop from C-PVC-VUS to PM-C-PVC-VUS points to a critical insight: most of the measured calibrated capacity exists outside the PM region, where the uniform-convergence motivation of Theorem 1 does not apply. A model may appear well-calibrated under lenient thresholds without its calibration surviving in the positive-margin region.

Connecting the Three Levels. Our framework provides a hierarchical assessment of self-evaluation reliability:

1. AUROC / PVC: Measures pure discriminative capacity: can the model rank correct answers higher? This is necessary but insufficient for trustworthy deployment.

2. C-PVC: Adds calibration constraints: does the model’s confidence align with its accuracy? This addresses calibration blindness but without the positive-margin condition.

3. PM-C-PVC: Restricts to the positive-margin region: the stringent operating region where observed calibration admits a uniform-convergence interpretation under capacity-envelope assumptions. This is the most demanding level of our diagnostic hierarchy.

The Sobering Conclusion. Our experiments show that even the best-performing models achieve PM-C-PVC-VUS $\approx0.17$ (Qwen2.5-7B-Instruct), compared to PM-PVC-VUS $\approx1.78$. This means that within the regime where generalization is provable:

• Models retain moderate discriminative capacity (PM-PVC-VUS $> 0$)

• Models exhibit near-zero calibrated capacity (PM-C-PVC-VUS $\approx0$)

In other words, even when we can guarantee that a model’s discrimination ability will generalize, we cannot guarantee that its calibration will. This fundamental asymmetry, invisible to AUROC and only partially captured by full-region C-PVC, is precisely what PM-C-PVC-VUS measures.

Summary: A Hierarchy of Self-Evaluation Metrics

Table 13 summarizes the progression from rank-based metrics to our full framework. The PM-C-PVC-VUS metric represents the most stringent test of self-evaluation reliability: it requires discrimination, calibration, and provable generalization. The near-zero values observed across all models (best: 0.17 for Qwen2.5-7B-Instruct) underscore our central finding: reliable, generalizable self-evaluation remains largely unattained in current LLMs.

Metric	Discrimination	Calibration	Generalization
AUROC	✓	✗	✗
PVC-VUS	✓	✗	Partial
C-PVC-VUS	✓	✓	✗
PM-PVC-VUS	✓	✗	✓
PM-C-PVC-VUS	✓	✓	✓

Comparative Advantages of the PVC Framework

While standard metrics such as Expected Calibration Error (ECE) and Area Under the ROC Curve (AUROC) provide valuable insights into model performance, they exhibit fundamental limitations when assessing the structural reliability of self-evaluation. The Probabilistic VC (PVC) framework addresses these shortcomings through two key properties: calibration awareness and categorical decomposability.

Beyond Ranking and Averages

As detailed in Appendix § 30, rank-based metrics like AUROC suffer from isotonic invariance, rendering them blind to calibration errors. Similarly, ECE provides only a global average where opposing errors across domains can cancel out, obscuring domain-specific failures. For instance, a model that is overconfident in mathematics but underconfident in factual reasoning may achieve low ECE while being unreliable in both domains individually.

In contrast, C-PVC imposes per-category calibration constraints: each domain must independently satisfy the tolerance $\tau$, preventing error cancellation and ensuring uniform reliability across all evaluated categories.

Categorical Decomposability

A practical advantage of the PVC framework is its natural alignment with categorical evaluation. When the input space admits a disjoint partition into semantic domains $\mathcal{X}_{\mathrm{total}} = \bigcup_i \mathcal{X}_i$ (e.g., $\mathcal{X}_{\mathrm{Math}}, \mathcal{X}_{\mathrm{Logic}}, \mathcal{X}_{\mathrm{Fact}}$), our empirical PVC measurement, defined as the count of shattered categories, is inherently additive:

This formulation offers two practical benefits. First, independent evaluation: each category’s contribution is determined solely by its own accuracy and calibration, unaffected by other categories. Adding or removing a category does not change the shattered status of existing categories. Second, fine-grained diagnostics: the per-category breakdown directly shows which domains contribute to (or detract from) overall capacity, enabling targeted analysis of model strengths and weaknesses.

We note that this additivity holds for our empirical category-count measurement at fixed $(\gamma, \tau)$. When aggregating via VUS, comparisons should be made across evaluations with the same category set to ensure consistency.

Summary of Comparative Advantages

Table 14 summarizes these distinctions. Unlike AUROC (discrimination only) or ECE (global calibration with cancellation), C-PVC combines discrimination, per-category calibration constraints, decomposability, and, via the PM restriction, a positive-margin uniform-convergence motivation, into a single diagnostic framework.

Property	AUROC	ECE	C-PVC
Discrimination-aware	✓	✗	✓
Calibration-aware	✗	✓	✓
Per-category constraints	✗	✗	✓
Decomposable	✗	Partial	✓
Generalization bounds	✗	✗	✓

Detailed Experimental Methodology

This section elaborates on our empirical approach to measuring PVC dimensions in language models, providing implementation details for the framework introduced in § 3.2.

Measurement Protocol

Our empirical evaluation follows a three-stage protocol designed to measure a model’s capacity for reasoning self-assessment while avoiding information leakage. The process is illustrated in Figure 1.

Stage 1: Solution Generation. For each problem instance, the model generates two independent solutions using distinct decoding trajectories: one following standard step-by-step reasoning patterns, and another exploring a non-standard, potentially more intuitive or compact reasoning path. These variants ensure diversity in reasoning quality while remaining entirely model-generated.

Stage 2: Self-Evaluation. The model is presented with both candidate solutions in randomized order and must select the answer it believes to be more correct while assigning a confidence score to its choice. Crucially, the model is not given access to the reference label, ensuring that this choice reflects genuine internal evaluation rather than supervised feedback.

Stage 3: Reference Labeling. To assign the reference preference, a separate three-judge LLM ensemble evaluates the same solution pair with access to the correct final answer. The ensemble assigns the reference label via majority vote. This design keeps the reference signal independent of the model being evaluated and reduces sensitivity to individual judge biases.

Hyperparameters

Table 15 summarizes the decoding and evaluation configuration applied identically to all eleven models: Qwen2.5-7B, Qwen2.5-7B-Instruct [82], Qwen2.5-Math-7B-Instruct [83], DeepSeek-R1-Distill-Qwen-7B [24], Llama-3.1-8B-Instruct [45], OpenThinker2-7B [51], Bespoke-Stratos-7B [11], JiuZhang3.0-7B [Zhou et al., 2024], Ministral-8B-Instruct2410 [46], Open-Reasoner-Zero-7B [26], and s1.1-7B [48]. The same settings were used for all judge ensemble queries.

Stage	Parameter	Value
Solution Generation	Temperature	0.7
Solution Generation	Top-P (nucleus sampling)	0.9
Solution Generation	Max tokens	4096
Solution Generation	Solutions per problem	2
Self-Evaluation	Presentation order	Randomized
Self-Evaluation	Confidence scale	0–100%
Self-Evaluation	Reference-label access	None
External Judging	Judge 1	Claude 3.7 Sonnet [6]
External Judging	Judge 2	Amazon Nova Premier [2]
External Judging	Judge 3	DeepSeek-R1 [24]
External Judging	Aggregation	Majority vote

Math-360 Benchmark Construction

To instantiate the diagnostic protocol on a contamination-aware mathematical setting, we developed Math-360, a controlled diagnostic testbed of 360 original problems. Math-360 is used to support the evaluation protocol under balanced domain, difficulty, and subcategory structure, and is not positioned as a standalone general-purpose math leaderboard; its role in the paper is supporting, with cross-domain validation provided by TruthfulQA, CSQA, and MATH-500. Unlike prior math benchmarks derived from widely available competition problems or textbooks (which may overlap with LLM pretraining corpora), Math-360 emphasizes originality and balanced domain coverage.

Taxonomic Coverage. Table 16 presents the taxonomy of mathematical reasoning categories. The dataset spans eight core domains (Arithmetic, Algebra, Calculus, Geometry, Number Theory, Combinatorics, Statistics, and Linear Algebra), each subdivided into five foundational subtopics. This taxonomy reflects standard mathematics curricula and facilitates interpretable analysis across well-defined conceptual boundaries.

Category	Subcategories
Arithmetic	Basic Operations, Fractions, Percentages, Numerical Approximation, Order of Operations
Algebra	Equations, Inequalities, Polynomials, Functions, Systems of Equations
Calculus	Differentiation, Integration, Limits, Series, Applications
Geometry	Plane Geometry, Coordinate Geometry, Transformations, Mensuration, Trigonometry
Number Theory	Divisibility, Modular Arithmetic, Primes, Diophantine Equations, Number Sequences
Combinatorics	Counting Principles, Permutations, Combinations, Probability, Recursion
Statistics	Descriptive Statistics, Distributions, Hypothesis Testing, Regression, Bayesian Inference
Linear Algebra	Matrices, Determinants, Vector Spaces, Eigenvalues, Linear Transformations

Problem stratification and complexity. Each subcategory contains five problems rigorously stratified across difficulty levels (Easy, Medium, Hard), yielding a total of 360 problems. Crucially, while the Easy tier validates fundamental competencies, the Hard tier incorporates complex, multi-step reasoning tasks. This broad difficulty spectrum is designed to stress-test the reasoning limits of 7–8B models. We note that the examples in Table 17 are selected for brevity and illustrative clarity; the actual dataset includes problems requiring extensive intermediate derivations to prevent ceiling effects.

Category	Subcategory	Example Problem
Arithmetic	Basic Operations	Calculate 238 + 149.
	Fractions	What is $\frac{2}{5}+\frac{1}{5}$?
	Percentages	What is 25% of 80?
	Numerical Approximation	Round 47.68 to the nearest whole number.
	Order of Operations	Calculate $3+4\times2$.
Algebra	Equations	Solve for $x$: $x+5=12$.
	Inequalities	Solve: $x+3>7$.
	Polynomials	Simplify: $(3x^2+2x)+(4x^2-5x+1)$.
	Functions	If $f(x)=2x+3$, find $f(4)$.
	Systems of Equations	Solve the system: $x+y=5$, $x-y=1$.
Calculus	Differentiation	Find the derivative of $f(x)=3x^2+2x-5$.
	Integration	Find $\int(3x^2+2)\,dx$.
	Limits	Evaluate $\lim_{x\to3}(x^2-4)$.
	Series	Find the sum of the first 10 terms of the arithmetic sequence with $a_1=3$ and $d=4$.
	Applications	Find the maximum value of $f(x)=-x^2+6x-5$ on the interval $[0,5]$.
Geometry	Plane Geometry	Find the area of a rectangle with length 8 cm and width 5 cm.
	Coordinate Geometry	Find the distance between the points $(3,4)$ and $(6,8)$.
	Transformations	Reflect the point $(3,5)$ across the $x$-axis.
	Mensuration	Find the circumference of a circle with radius 5 cm.
	Trigonometry	Find $\sin(30^\circ)$.
Number Theory	Divisibility	Determine whether 156 is divisible by 4.
	Modular Arithmetic	Calculate 17 (mod 5).
	Primes	List all prime numbers less than 20.
	Diophantine Equations	Find all integer solutions to $x+y=10$.
	Number Sequences	Find the next number in the sequence: 3, 7, 11, 15, $\ldots$
Combinatorics	Counting Principles	How many different 3-digit numbers can be formed using the digits 1, 2, 3, 4, 5 without repetition?
	Permutations	How many permutations can be formed using all the letters of the word ‘MATH’?
	Combinations	How many ways are there to select 3 books from a shelf containing 7 different books?
	Probability	A fair die is rolled. What is the probability of getting a number greater than 4?
	Recursion	Find the 6th term in the Fibonacci sequence, where $F_1=1$, $F_2=1$, and $F_n=F_{n-1}+F_{n-2}$ for $n\ge3$.
Statistics	Descriptive Statistics	Find the mean of the numbers 4, 7, 9, 3, and 2.
	Distributions	If $X$ follows a normal distribution with mean 10 and standard deviation 2, what is $P(X>12)$?
	Hypothesis Testing	In a hypothesis test, what is the meaning of a Type I error?
	Regression	In a linear regression model, what does the slope coefficient represent?
	Bayesian Inference	State Bayes’ theorem.
Linear Algebra	Matrices	Calculate the sum of matrices $A=\begin{bmatrix}3&5\\1&2\end{bmatrix}$ and $B=\begin{bmatrix}4&0\\6&3\end{bmatrix}$.
	Determinants	Calculate the determinant of the matrix $A=\begin{bmatrix}3&4\\1&2\end{bmatrix}$.
	Vector Spaces	Determine whether the set of vectors $\{(1,0,0),(0,1,0),(0,0,1)\}$ forms a basis for $\mathbb{R}^3$.
	Eigenvalues	Find the eigenvalues of the matrix $A=\begin{bmatrix}2&0\\0&3\end{bmatrix}$.
	Linear Transformations	Let $T:\mathbb{R}^2\to\mathbb{R}^2$ be defined by $T(x,y)=(2x,3y)$. Find $T(1,2)$.

Representative examples. Table 17 provides one representative problem per subcategory, highlighting the range and depth of reasoning skills required. Problems are designed to elicit multi-step thinking, abstraction, and formal manipulation, rather than direct recall or pattern matching. For instance, problems in combinatorics test combinatorial enumeration and recurrence, while those in number theory evaluate modular reasoning and structural properties of integers.

Taken together, the benchmark offers a principled and contamination-resistant testbed for evaluating mathematical reasoning in language models. It supports both overall capability assessment and fine-grained diagnosis of strengths and weaknesses across mathematical domains.

Construction of Non-Mathematical Benchmarks

To investigate the generalizability of our framework beyond mathematical reasoning, we incorporated two established benchmarks: TruthfulQA [39] and CommonsenseQA [67]. However, utilizing these datasets in their raw form presents challenges for our category-level analysis due to inherent class imbalances: the original datasets contain highly uneven distributions across semantic topics, which could bias PVC/C-PVC estimates toward dominant categories.

To ensure statistical stability, we implemented a curation protocol aligned with Math-360’s balanced structure. First, we identified the top-10 major categories per benchmark based on semantic labels to ensure broad domain coverage. Second, we determined the minimum sample count among these categories as a uniform balancing constraint. Third, we performed stratified sampling of exactly 24 questions from each category, yielding a standardized subset of 240 questions per benchmark (10 categories × 24 questions). This uniform distribution ensures that PVC and C-PVC estimates reflect true capability across diverse reasoning patterns rather than being skewed by data volume in specific sub-domains.

Experimental Prompts

Our measurement procedure operationalizes the theoretical concept of PVC dimension through a three-stage prompting process.

First Solution Generation

The target model generates an expert-level solution using the following prompt:

First Solution Prompt

System: You are an expert mathematics educator with deep knowledge across all mathematical domains. Your task is to solve a math problem with exceptional clarity and accuracy.

User: Please solve the following mathematical problem step by step, ensuring complete accuracy: PROBLEM: {problem}

Provide a detailed solution that demonstrates expert-level understanding. Include: 1. Clear explanation of your approach 2. Each step fully worked out 3. Proper mathematical notation 4. Verification of your answer

Make sure your final answer is clearly indicated and correct.

Second Solution Generation

The model generates an alternative solution using a prompt that requests creative approaches:

Second Solution Prompt

System: You are a brilliant mathematics educator known for your creative problem-solving approaches. Your task is to solve a math problem using your new methods.

User: Please solve the following mathematical problem:

PROBLEM: {problem}

Provide an extremely detailed solution that demonstrates understanding.

Include: 1. A clear explanation of your approach using an alternative and creative solution path 2. Each step fully worked out with mathematical reasoning 3. Proper mathematical notation and formalism 4. If applicable, connect this problem to broader mathematical concepts or real-world applications

Make your solution as thorough and educational as possible, showing multiple ways to think about and verify the solution. Focus on approaches that demonstrate mathematical insight and elegance.

This design choice, requesting alternative approaches rather than explicitly introducing errors, allows for natural quality variation while avoiding artificially degraded solutions that would be too obviously flawed.

Self-Evaluation Prompt

The model evaluates both solutions (presented in randomized order), identifies which it believes is superior, and provides a confidence score (0–100%):

Self-Evaluation Prompt

System: You are an expert mathematics evaluator tasked with comparing two solutions to the same problem.

User: Below are two different solutions to the same mathematical problem. Your task is to carefully evaluate both solutions and determine which one is superior in terms of correctness, clarity, and mathematical insight.

PROBLEM: {problem}

SOLUTION A: {solution_a}

SOLUTION B: {solution_b}

Please compare these solutions carefully and respond with: 1. Which solution is better (A or B) 2. Your confidence in this judgment (0-100%) 3. A brief explanation of why one solution is superior to the other

Format your response as: Better solution: [A or B] Confidence: [0-100%] Explanation: [Your analysis]

The reported confidence score directly operationalizes the $\hat{p}(x)$ term in our C-PVC definition (Definition 2), enabling calibration error computation at the category level.

Prompts for Non-Mathematical Benchmarks

For TruthfulQA and CommonsenseQA, we adapted the above prompts by replacing mathematical terminology with domain-appropriate language (e.g., "factual accuracy" for TruthfulQA, "commonsense reasoning" for CommonsenseQA) while preserving the three-stage structure. The self-evaluation prompt structure remains identical, ensuring consistent confidence elicitation across all domains.