betaFeedback

WorksComputer Science

Safe to Omit, Worth Omitting? Keeping the Answer and Cutting Latency When an LLM Agent Skips Another's Message

Abstract

In multi-agent systems, a receiver agent often waits for a helper agent's message (text or its latent KV cache), which can improve performance but adds latency. We study when a receiver agent can skip the message safely (the final answer rarely changes) and beneficially (the time saved exceeds the cost of deciding). Our mechanism reads the receiver agent's confidence without the message and uses a finite-sample test to certify a threshold at which, with high probability, at most 5% of skipped answers change; if none passes, the system keeps communicating. We test it on OpenBookQA, ARC-Challenge, and MMLU-Pro with three Qwen model pairs, on cross-family agents, and on SQuAD and GSM8K. Four insights emerge. First, oracle headroom is weak evidence that messages help: control inputs without answer-specific content, matched to change answers as often as real messages, reproduce a median 76% of the oracle's gain. Second, whether a message can be skipped depends on how often and how predictably it alters the answer, not on its accuracy; every certified KV-cache path is less accurate than the receiver agent alone. Third, where text messages help, skipping keeps most of their gain: on unseen questions in the four such settings, the policies skip 59 to 80% and keep an estimated 76 to 95% of the gain. In two prospective tests with all rules fixed in advance, our mechanism skips a three-agent system's verifier on 39.9% of sealed questions, keeping 12.55 of its 12.80-point gain, and a 32B helper on 68.4% of held-out questions, keeping its whole gain. Fourth, safety does not determine efficiency: at batch size one, skipping text saves hundreds of milliseconds per query and skipping KV cache far less. Certification extends to SQuAD but not to GSM8K. In eleven tests on unseen questions, every certified policy we could test kept its change rate's 95% upper bound below 5% under our answer parsers.

Introduction

Collaborative large language model systems increasingly rely on agents consulting one another to solve complex tasks ([11, 32]). A common paradigm has a helper agent generate a message and pass it to a receiver agent. The message is either natural-language text (Text) or latent intermediate states, such as the key–value (KV) cache in which the helper stores what it has read (Cache-to-Cache, C2C; [15]). While this communication can enhance performance, it introduces a bottleneck: the receiver agent must wait for the helper agent’s output, adding latency and computational overhead Zhang et al. [52]. For practical deployment, it is crucial to determine when such communication is necessary and when the receiver agent can safely skip the message to reduce inference time.

Identifying which queries genuinely require communication is challenging. Work on routing and ensembling motivates keeping several paths with oracle headroom Jiang et al. [25]; Hu et al. [22]: the accuracy of an oracle that picks a correct path for each question, minus the accuracy of the best single path. High headroom means that different paths answer different questions correctly. For agent communication, it would suggest that helper messages carry content the receiver agent lacks, but it is weak evidence of this. Control inputs without answer-specific content, matched in how often they change answers, yield comparable headroom (§ 4.1), so headroom can arise from answer changes alone. Systems therefore need a mechanism that prunes redundant messages based on actual behavior.

We formalize the decision to skip a message based on two criteria: safety and value (Figure 1). We treat the system that always sends the message as a fixed, deployed reference, and consider skipping safe if the receiver agent alone rarely gives a different answer from this reference. Bounding such answer changes is vital, as users notice changed answers even when overall accuracy remains constant Yan et al. [48]; Dutta et al. [13]. This agreement-based formulation offers a practical advantage. Unlike accuracy, agreement can be checked without ground-truth labels, allowing operators to calibrate the skipping mechanism directly on abundant unlabeled deployment traffic. To achieve this, we read the receiver agent’s confidence from one forward pass and apply a statistical test on a finite calibration sample, building on risk-controlled early exiting Schuster et al. [42]; Jazbec et al. [23] and Learn then Test Angelopoulos et al. [2]. This test certifies a threshold at which, with high probability, at most 5% of skipped answers change. If no threshold passes, the system conservatively keeps communicating. Furthermore, skipping is worth it only if the latency saved on skipped queries exceeds the computational cost of the confidence probe paid across all queries. Unlike an early-exit gate, which reads intermediate states of the computation it may cut short, our probe is a separate forward pass paid on every query, and the message it decides about comes from another agent and can make the answer less accurate.

Figure 1. Dynamic message skipping in collaborative LLM agents. (a) The deployed system always waits for the helper agent’s message, natural-language text (Text) or a projection of its KV cache (C2C), before the receiver agent answers. (b) Our method reads the receiver agent’s uncertainty u(x)u(x) from one forward pass (the probe): if u(x)≤τu(x) \le\tau, the receiver agent answers alone (green); otherwise the deployed system runs (orange). τ\tau is certified on unlabeled calibration questions so that, with high probability, at most 5% of skipped answers differ from the deployed system’s (red strip); if no threshold passes, the system always communicates. Inset: distribution of u(x)u(x).

Our main contributions are summarized as follows:

  • Oracle headroom is weak evidence of message utility. Control inputs without answer-specific content reproduce about three quarters of the oracle’s gain over the receiver agent. Whether this null baseline separates text messages from latent representations depends on how the match is designed (§ 4.1).

  • Skippability is a behavioral property. Whether a message can be skipped depends on how often and how predictably it alters the receiver agent’s answer, not on its overall accuracy. In our settings, the probe’s AUROC separates every certified setting from every fallback, and a label-free rule on fit-split counts reproduces all 25 outcomes it covers and predicted two later fallbacks in advance (Appendix D). The certified C2C paths are all less accurate than the receiver agent alone under our answer parser, so certification here identifies redundant paths. Certification applies to multiple-choice tasks and one extractive setting (SQuAD; § 4.2).

  • Skipping can keep most of a helpful message’s gain. In four settings where Text helps, the deployed policies keep an estimated 76 to 95% of its gain on unseen questions, with the retained gain’s interval above zero in three (§ 4.3). The certificate bounds the accuracy loss by κα\kappa\alpha, which can exceed the gain; post hoc, labels on 7 to 12 skipped disagreements bounded the loss by one point for three of six helpful Text paths. In two prospective tests with every rule fixed in advance, on a three-agent system and on a 32B pair, the primary policy meets every pass rule (§ 4.3).

  • System efficiency and latency analysis. Safety does not determine value: at a batch size of one, skipping text messages saves a mean of hundreds of milliseconds per query, whereas skipping latent representations saves far less (§ 4.4). In tests on unseen data, every deployed policy that could be tested keeps its change rate’s 95% upper bound below 5% (§ 4.5).

A framework for dynamic message skipping

We frame the decision to skip communication as a risk-controlled routing problem. Given a question xx, a deployed reference path bb (e.g., passing a Text or KV cache message from the helper agent to the receiver agent) produces an answer ob(x)o_b(x) with latency cb(x)c_b(x). The receiver agent answering alone, path RR, produces an answer oR(x)o_R(x) with latency cR(x)c_R(x). An answer is the label that one fixed parser reads from the receiver agent’s output, so the system outputs this parsed label, not raw text; outputs the parser cannot read share one INVALID label and so agree.

A routing policy πτ\pi_\tau computes a confidence score s(x)s(x) at cost cs(x)c_s(x), skips the message and routes to RR when s(x)≤τs(x) \le\tau, and runs bb otherwise. The overall latency of the policy is Cπτ(x)=cs(x)+cπτ(x)(x)C_{\pi_\tau}(x) = c_s(x) + c_{\pi_\tau(x)}(x), and its coverage κ(τ)=Pr⁡[s(x)≤τ]\kappa(\tau) = \Pr[s(x) \le\tau] is the fraction of queries that skip the message.

Bounding the consistency risk. Skipping should rarely change the deployed system’s answer. The change rate ρ(τ)\rho(\tau) is the conditional probability of disagreement on skipped queries, relative to the deployed answer rather than the correct one:

ρ(τ)=Pr⁡[oR(x)≠ob(x)∣s(x)≤τ].(1)\rho(\tau) = \Pr[o_R(x) \ne o_b(x) \mid s(x) \le\tau]. \tag*{(1)}

Skipping is safe at a tolerance α\alpha if ρ(τ)≤α\rho(\tau) \le\alpha. Because the policy and the reference differ only on skipped queries whose answers change, the accuracy change is bounded by the probability of skipping and changing the answer:

∣acc⁡(πτ)−acc⁡(b)∣≤κ(τ)ρ(τ).(2)\lvert\operatorname{acc}(\pi_\tau) - \operatorname{acc}(b) \rvert\le\kappa(\tau)\rho(\tau). \tag*{(2)}

When ρ(τ)≤α\rho(\tau) \le\alpha, the accuracy change is at most κ(τ)α\kappa(\tau)\alpha, which can exceed a helpful path’s accuracy gain over the receiver agent (§ 4.3).

Statistical certification via finite-sample testing. To certify ρ(τ)≤α\rho(\tau) \le\alpha without ground-truth labels, we use Learn then Test ([2]). The score s(x)s(x) is the receiver agent’s uncertainty: one minus its maximum predicted answer probability from one forward pass. The test needs no continuous score, since ties only merge candidate thresholds.

The test uses mm candidate thresholds τq\tau_q, set on a separate fit split as qq-quantiles of the scores before any calibration outcome is read. On a disjoint, unlabeled calibration split, we observe the number of queries nqn_q that fall at or below τq\tau_q, out of which kqk_q queries produce a changed answer oR(x)≠ob(x)o_R(x) \ne o_b(x). An exact binomial test of H0:ρ(τq)≥αH_0:\rho(\tau_q) \ge\alpha accepts a threshold if kqk_q is improbable under H0H_0:

Pr⁡[Bin⁡(nq,α)≤kq]≤δ.(3)\Pr[\operatorname{Bin}(n_q,\alpha) \le k_q] \le\delta. \tag*{(3)}

We deploy the largest accepted threshold τq\tau_q. If no candidate threshold satisfies this bound, the policy always runs bb; such a fallback means only that the test could not certify skipping with the available calibration data. Testing each of mm predetermined candidates at level δ\delta bounds the probability of deploying a threshold whose true change rate is at least α\alpha (a Bonferroni correction; Dunn, [12]), assuming i.i.d. calibration questions and a parser fixed independently of them. The certificate covers a threshold together with the parser, so an operator should certify the answer form its application consumes.

Evaluating the empirical value. Skipping is valuable only if the latency saved on skipped queries outweighs the probe’s cost, paid on every query. The expected saving over always running the reference path is:

E[cb(x)−Cπ(x)]=κ(τ)E[cb(x)−cR(x)∣s(x)≤τ]−E[cs(x)].(4)\mathbb{E}[c_b(x)-C_{\pi}(x)] = \kappa(\tau)\mathbb{E}[c_b(x)-c_R(x)\mid s(x)\leq\tau]-\mathbb{E}[c_s(x)]. \tag*{(4)}

A deployed policy is thus considered worth omitting when the 95% bootstrap interval of the paired mean end-to-end saving lies above zero.

Experimental setup

Models and communication paths. We evaluate three helper and receiver pairs from the Qwen family (Qwen et al., [40]; Yang et al., [49]): small (Qwen2.5-0.5B-Instruct with Qwen3-0.6B), medium (Qwen2.5-1.5B-Instruct with Qwen3-1.7B), and large (Qwen2.5-7B-Instruct with Qwen3-8B). On the Text path, the helper agent generates up to 256 tokens describing the essential background knowledge for the question without answering it (Appendix § 22), which the receiver agent reads before generating its answer. As latent communication we use Cache-to-Cache (C2C; Fu et al., [15]), which projects the helper agent’s key–value cache into the receiver agent, as do LatentMAS (Zou et al., [55]) and cross-model KV transfer (Heo et al., [21]). Its official general-purpose fusers exist for all three pairs, and we use them as released, reproducing official outputs (Appendix § 18). All models decode greedily with thinking disabled, as in the official C2C evaluator.

Certification setup and answer parsing. For the test of § 2, we set α=0.05\alpha= 0.05 and δ=0.001\delta= 0.001. We define the confidence score, which we call ProbeMax (Hendrycks & Gimpel, [20]; Geng et al., [17]), by running one receiver agent prefill over the question and options followed by The correct answer is. We extract the probabilities of the option labels, renormalize them, and set s(x)=u(x)=1−max⁡lpls(x)=u(x)=1-\max_l p_l, the receiver agent’s uncertainty. On SQuAD, the score is one minus the probability of the receiver agent’s own answer, so the probe generates that answer. The candidate grid consists of 20 thresholds: 19 quantiles (q∈{0.05,0.10,…,0.95}q\in\{0.05,0.10,\ldots,0.95\}) extracted from the fit split, plus q=1q=1 (always skip). Testing these 20 candidates at δ=0.001\delta=0.001 yields a Bonferroni bound of 0.02 per setting, so with probability at least 0.98 the bound in Eq. (2) is at most κα\kappa\alpha. To evaluate answer agreement, we use one fixed rule-based parser that extracts a single final choice from the raw receiver agent output (on SQuAD, a normalized answer string; Appendix § 22). If the output format is unreadable or ambiguous, the parser returns a shared INVALID label, meaning two unparsable outputs agree with each other. As a second convention, we also relabel outputs with the official C2C evaluator’s answer extraction. It reads only labels A to D and differs from our parser on hedged answers and on options whose text it mistakes for a math expression (Appendix § 12).

Benchmarks and data splits. Certification requires answer changes that can be checked without labels, which multiple choice makes exact (two parsed labels differ). Our primary evaluation therefore uses OpenBookQA (OBQA; Mihaylov et al., [35]) and ARC-Challenge (ARC; Clark et al., [6]), two knowledge-dependent science benchmarks from C2C’s published evaluation. We partition OBQA into 2,100, 1,366, and 742 questions for fit, calibration, and development, all drawn from its official training set. ARC is partitioned into 670, 448, and 299 questions. All three pairs also run on the harder MMLU-Pro benchmark (Wang et al., [46]), which features questions with 3 to 10 options across 14 disciplines. Its 12,032 test rows were grouped by normalized text into 3,000, 6,000, and 2,641 questions for our three splits. To test answers outside a fixed label set, we include the SQuAD reading comprehension benchmark (Rajpurkar et al., [41]), where answers are spans of a passage, and the GSM8K benchmark of math word problems (Cobbe et al., [8]), where the answer is a number derived by reasoning. Data are development if observed during the study, held out if never used by the pair tested, and sealed if untouched until a frozen policy ran (Appendix § 12).

Evaluation settings and measurement. A setting is a model pair, benchmark, and reference path; the main study has 18 Qwen-pair settings. Thirteen extension settings test other models and another swer forms, for 31 certification outcomes (Figure 2). Nine use cross-family pairs: the large helper agent with an OLMo-2-1124-7B-Instruct receiver agent (Team OLMo et al., [44]), a Llama-3.2-1B-Instruct helper agent with the small receiver agent (Grattafiori et al., [18]), and, after both fell back, the large helper agent with a Llama-3.1-8B-Instruct receiver agent (the Llama-8B pair). Three use informed or stronger helper agents, and one uses SQuAD. Every extension protocol was frozen before its outcomes were generated, committing to report every setting. Two prospective tests of § 4.3 add a three-agent pipeline on QASC (Khot et al., [29]) and a 32B Qwen pair on OBQA. We measure latency as the end-to-end wall-clock time per query at a batch size of one. Each deployed policy is replayed, interleaved with its reference path, on a fixed 128-question development panel per benchmark (the first in a fixed pseudorandom order of question IDs), on two A100 GPUs, and repeated on a second hardware configuration (Appendix H). Latency and accuracy intervals are paired bootstrap 95% intervals; change-rate intervals are exact two-sided Clopper–Pearson ones (Clopper & Pearson, [7]). We summarize how well the score predicts answer changes by its development AUROC: the probability that a random question on which the receiver agent alone disagrees with the reference gets a higher uncertainty than a random question on which they agree (ties count one half).

Figure 2. Messages that rarely and predictably change the receiver agent’s answer can be skipped. Each circle is one of the 31 settings (model pair, reference path, and benchmark); diamonds are the two prospective tests (§ 4.3), the 32B pair on OBQA development questions (AUROC 0.915) and the three-agent system on its 2,000 sealed QASC test questions (0.858). x-axis: how often the receiver agent alone disagrees with the deployed reference on development questions; y-axis: how well the probe ranks those disagreements (AUROC, § 3; ties count one half). Filled: a threshold is certified and deployed; open: the system falls back to always communicating. The shaded region marks the observed separation: every deployed setting has AUROC at least 0.81, every fallback at most 0.80. Arrow: on OBQA, the medium pair’s more accurate Text path is not certified, while C2C is. Nominal certificates: large/OBQA/Text, large/OBQA/C2C, and Text+fact (§ 4.2). Out-of-sample results, including the one policy that fails under the official C2C extraction (large-pair ARC C2C), are in Table 4. Inset: three close settings. Text+fact: the helper also sees the OBQA science fact.

Results

Oracle headroom is weak evidence of communication utility

What oracle headroom shows. In each of our seven populations (a model pair on one benchmark), an oracle that picks a correct path among the receiver agent alone, Text, and C2C beats the best single path by 2.01 to 17.79 points (Table 1). High headroom shows only that different paths answer different questions correctly, not that the helper agent’s messages carry indispensable content.

PairBenchmark (NN)RRTextC2Cheadroom (pp)UnstratifiedStratifiedover RRTextC2CTextC2C
smallOBQA (742)28534636617.791.050.790.660.72*0.69*0.80*0.80*
smallARC (299)11012615516.391.150.730.740.850.75*0.840.86
mediumOBQA (742)4905234869.431.301.00†0.900.80*1.130.971.20
mediumARC (299)2172212029.030.95†0.94†0.85†0.741.71*0.791.48
largeOBQA (742)6176455983.371.601.080.760.55*1.310.81*1.08
largeARC (299)2682742662.011.270.97†0.640.50*0.990.90§1.35
largeMMLU-Pro (2641)1282131912339.051.091.020.970.86*1.25*0.891.22*
Median over the seven populations1.150.970.760.741.130.841.20

Table 1. Input changes without answer-specific content create comparable oracle headroom and reproduce about three quarters of the real gain. Correct answers: development questions each path answers correctly, out of NN. Headroom: oracle over RR, Text, and C2C minus the best single path (points). Null/real: the content-free null’s headroom or oracle gain over RR divided by the real one; near or above 1, the null reproduces the effect. Stratified: the null also matches how many changes fall on questions RR answers correctly and incorrectly. *95% interval excludes 1 (per-path columns); †not fully matched (too few questions); §lower bound (0.99 rescaled). All intervals: Appendix F.

Constructing a content-free null baseline. To test whether headroom stems from message content, we rerun the receiver agent under two inputs without answer-specific content: options rotated by one position (a format change LLMs are sensitive to; Zheng et al., [53]; Pezeshkpour & Hruschka, [38]) and a message written for another question. Unmatched, an oracle over the receiver agent and these two inputs already matches or exceeds the real headroom in six of seven populations. Because an oracle gains more when answers are volatile, we resample these perturbed answers so that the null changes exactly as many answers as the real paths (1,000 draws; Appendix F). It matches all paths jointly or one at a time, in total (unstratified) or separately on questions the receiver agent answers correctly and incorrectly (stratified).

Content-free changes create comparable headroom. In the median population, an oracle operating over the receiver agent and the jointly matched null beats its own best single path by about as much as the real oracle does, with a headroom ratio of 1.15 [1.00, 1.32], and 0.97 [0.80, 1.04] stratified. Measured as gain over the receiver agent alone, this null reproduces a median 76% [?, ?], or 85% [?, ?] stratified, of the real gain. The comparison is descriptive: it shows that headroom arises without answer-specific content, not how much of the real gain content explains.

Whether the null separates Text from C2C depends on how it is matched. We compare each real path with its matched null by the gain ratio (null over real; Table 1). A ratio significantly below 1 means the path’s complementary answers go beyond answer volatility:

  • C2C: For the five medium- and large-pair populations, the C2C ratio is never significantly below 1 under either match (0.99 to 1.71 unstratified, 1.08 to 1.48 stratified). The small pair, whose receiver agent is the one in the C2C authors’ main evaluation, separates from the null: its ratios are 0.69 and 0.75 unstratified (both significant) and 0.80 and 0.86 stratified (significant on OBQA).

  • Text: Text’s ratio is below 1 in every determined population under both matches (0.50 to 0.86 unstratified, 0.79 to 0.97 stratified). It is significantly below 1 in five of seven populations unstratified, but in only two of six stratified, where large-pair ARC (at least 0.90) is undetermined.

Messages that rarely and predictably change the answer can be skipped

Which settings are certified. The large pair can skip both messages on every benchmark, the medium pair only C2C on OBQA and ARC, and the small pair neither (Figure 2). Of the 18 main settings, 8 are certified and 10 fall back; on MMLU-Pro only the large pair is certified (disagreement 21.2 and 28.6%, AUROC 0.821 and 0.846). Three development certificates are nominal, meaning their error bound is not established because the calibration questions were also used to choose the probe: large/OBQA/Text, large/OBQA/C2C, and Text+fact (below); for them, the held-out test of § 4.5 is the evidence we rely on. Two quantities explain the pattern: how often a message changes the answer, and how well the probe predicts it. The small pair’s receiver agent disagrees with its references on 46 to 54% of OBQA and ARC questions, which the probe ranks poorly, whereas the large pair disagrees on 4 to 10% with AUROC 0.87 to 0.95. Sample size matters too: a certified threshold needs at least 135 skipped calibration questions to pass the statistical test, and the small pair’s lowest quantiles hold 18 to 65 (Appendix § 13). The certificate covers the skipped questions as a whole, not each subgroup: on MMLU-Pro, 12 of 72 skipped engineering C2C answers change (Appendix § 13). Certifying each of the 14 disciplines separately trades coverage for per-discipline guarantees: it skips 11.1% of development questions instead of 40.5% for either reference, and 0.0% (Text) or 3.7% (C2C) with δ\delta also divided by 14 (Table 11). The boundary is stable across 200 re-drawn splits and tested (post hoc) alternatives, but depends on α\alpha and the parser (Appendices § 13 and § 12).

Table 11: Certifying each MMLU-Pro discipline separately trades most of the pooled coverage for per-discipline guarantees (development; post-hoc aggregation of the per-discipline tests). Each discipline uses its own fit quantiles and calibration questions; a discipline without a certified threshold always runs the reference. Coverage: total skipped questions over all 2,641 development questions. Changed: changed among skipped, with Clopper–Pearson 95% CI for certified disciplines. Per discipline at δ=0.001\delta= 0.001, the nominal family-wise error over the 14 disciplines is at most 0.28 per reference; with δ/14\delta/14 it is at most 0.02.

(a) Overall

ReferenceStrategySkippedCoverage (%)Changed
Textpooled certificate1,06940.540/1,069
Textper discipline, δ=0.001\delta= 0.00129211.16/292
Textper discipline, δ=0.001/14\delta= 0.001/1400.0–
C2Cpooled certificate1,06940.537/1,069
C2Cper discipline, δ=0.001\delta= 0.00129311.13/293
C2Cper discipline, δ=0.001/14\delta= 0.001/14973.70/97

(b) Certified disciplines

ReferenceDiscipline (δ\delta)qqSkipped / NNChanged [95% CI, %]
Textbiology (0.001)0.6093/1441/93 [0.03, 5.85]
Texteconomics (0.001)0.4597/1900/97 [0.00, 3.73]
Textengineering (0.001)0.35102/2205/102 [1.61, 11.07]
C2Chealth (0.001)0.4574/1550/74 [0.00, 4.86]
C2Cother (0.001)0.50101/2102/101 [0.24, 6.97]
C2Cpsychology (0.001)0.65118/1811/118 [0.02, 4.63]
C2Cpsychology (0.001/14)0.6097/1810/97 [0.00, 3.73]

Table 11.

Accuracy does not decide. The medium pair is certified for C2C on OBQA and ARC but not for Text, the more accurate path. On OBQA its receiver agent disagrees with Text on 23.5% of questions but with C2C on 17.7%; on ARC, where the rates are similar (17.4 and 16.1%), the probe ranks C2C’s changes far better (AUROC 0.910 against 0.742). The probe cannot see the message, so a Text message that overturns a confident answer is invisible to it. A helper-agreement term favors Text over C2C in every population but certifies no new setting (post hoc; Appendix § 14). Every C2C path that can be skipped is less accurate than the receiver agent alone (for example, 598 against 617 of 742 on large-pair OBQA), a property of the released fusers, whose official outputs we reproduce (Appendix § 18). For C2C, the certificate therefore identifies redundancy rather than usefulness. An operator with labels would simply retire such a path. For medium/ARC/C2C, 2 of 4 development and 7 of 15 out-of-sample changes come from unparsed C2C answers.

Other receiver and helper agents fit the same boundary. An OLMo-2-7B receiver agent and a Llama-3.2-1B helper agent fall back. Part of the OLMo fallback is answer format, as 4 to 17% of its calibration answers do not parse, and no deployable parse rule or argmax answering (which needs no parser) certifies any of its settings (Appendix § 17.§ 1). A Llama-3.1-8B receiver agent, added afterwards, reads the large helper agent’s messages and is certified on OBQA and ARC. It skips 59 and 69% of development questions and changes 1.6% [0.6, 3.3] and 1.9% [0.5, 4.9] of them, deploys in all 200 re-drawn splits, and passes out of sample (Appendix § 17.§ 1). A stronger helper, the large one, raises the medium receiver agent’s accuracy but changes 21 and 13% of its answers, and both settings fall back (Appendix § 17.§ 2). Given the science fact behind each OBQA question (Text+fact; post hoc), the large helper raises accuracy from 617 to 670 of 742. Yet the receiver agent alone already gives the path’s answer on 90% of questions (AUROC 0.889), so skipping is

Table 2: Where Text helps, certified skipping keeps most of its gain on unseen questions. Unseen: questions never used to fix the policy. Accuracy (%) of the receiver agent alone (RR), the always-Text reference, and the certified policy; gains in points with paired-bootstrap 95% CIs (2,000 resamples over questions). Kept: policy gain over Text gain (point estimate). κ^α\hat{\kappa}\alpha: unseen coverage times α=0.05\alpha= 0.05, in points, a magnitude comparison for Eq. (2), not an additional guarantee. Llama-8B/OBQA’s lower bound −0.00-0.00 is −0.003-0.003. ‡Nominal certificate.

certified at q=0.75q = 0.75 (nominally). The policy answers 660 correctly, 1.35 points [0.54, 2.29] below the reference and within κα=3.6\kappa\alpha= 3.6 (Appendix § 17.§ 2).

Certification carries over to one extractive setting. A label-free pilot scored eight short-answer settings, and a rule fixed before any pilot output selected SQuAD with the Llama-8B pair (predicted certification probability 0.975, inflated by choosing the best of eight; Appendix § 17.§ 3). It is certified with normalized answer strings and one minus the answer’s probability as the score. On 1,000 new development questions, the receiver agent disagrees with Text on 15.4% (AUROC 0.811), and 4.0% [2.3, 6.5] of skipped answers change. Text adds only 1.0 point [−0.9, +2.9], below κα=2.0\kappa\alpha= 2.0, so the message is redundant here; its saving in Table 4 is estimated, not replayed.

PairTaskPathSkipped (%)Changed Dev.Changed UnseenLevelΔ\Deltaacc (pp) [95% CI]Gain (pp)Gain Kept (%)Saving (ms) [95% CI]
mediumOBQAC2C∗52.611/390held out 8/3880.025+0.27 [−0.54, +1.08]−0.5–79.7 [48.1, 129.1]
mediumARCC2C63.24/189held out 15/733◦0.001+0.34 [−0.26, +0.94]−5.0–100.7 [74.5, 133.1]
largeOBQAText‡77.119/572held out 12/5950.001−0.94 [−1.88, −0.13]3.875517.1 [447.8, 583.6]
largeOBQAC2C‡77.113/572held out 13/5950.001+1.21 [+0.40, +2.15]−2.6–36.6 [18.0, 56.8]
largeARCText95.09/284sealed 20/1,1000.001−0.43 [−1.19, +0.26]2.033717.6 [670.6, 763.2]
largeARCC2C90.33/270sealed 11/1,047§0.001+0.34 [−0.17, +0.85]−0.7–110.9 [82.0, 142.8]
largeMMLU-ProText40.540/1,069none––1.4105346.3 [249.8, 443.0]
largeMMLU-ProC2C40.537/1,069none––−1.9–8.2 [−20.5, +44.0]
largeOBQAText+fact‡72.213/536held out 16/5580.025−1.48 [−2.55, −0.54]7.181343.4 [297.7, 386.0]b
Llama-8BOBQAText∗59.07/438held out 3/440◦0.001−0.40 [−0.94, 0.00]4.4106318.6 [261.1, 372.1]b
Llama-8BARCText∗68.94/206held out 15/794◦0.001−0.17 [−0.77, +0.43]7.090418.9 [361.7, 473.7]b
Llama-8BSQuADText39.616/396sealed 9/364◦0.025+0.41.0140198.5 [167.2, 229.9]e

Table 4. Every deployed policy tested on unseen questions keeps the change rate’s 95% upper bound below 5% under our parsers; skipping Text saves hundreds of milliseconds per query, skipping C2C much less. Skipped: % of development questions answered by the receiver agent alone. Changed: changed skipped answers on development and on sealed or held-out (unseen) questions. Level: the test rejects ρ≥0.05\rho\ge0.05 at 0.001, or only the upper bound is below 5% (0.025). Δ\Deltaacc: policy minus reference accuracy on unseen questions (SQuAD: point estimate). Gain (G): development reference minus receiver-alone accuracy; Gain Kept: % of it retained (–: G ≤ 0). Saving: paired mean per query on 128-question panels, cold requests included. ‡Nominal certificate. ◦Pass rule prespecified. §Official C2C extraction: 54/1,044. ∗Certified only at a lower quantile qq without the parser-review questions. bSecond hardware configuration. eEstimated, not replayed. Full results: Table 26 and Table 6.

Agreement and accuracy

Where Text helps, skipping can keep most of the gain. Four deployed policies for helpful Text paths (two nominal) have unseen questions on which the receiver agent also ran (Table 2). They skip 59 to 80% of these questions and keep an estimated 76 to 95% of Text’s gain over the receiver agent; the retained gain’s interval excludes zero for three and includes zero for Llama-8B/OBQA. Removing the helper agent would forgo 2.7 to 8.1 points (single models: Appendix § 15).

PolicyUnseen questionsSkipped (%)ChangedAccuracy (%) RRAccuracy (%) TextAccuracy (%) PolicyText − RR [95% CI]Policy − RR [95% CI]Kept (%)κ^α\hat{\kappa}\alpha
Llama-8B/OBQA/Textheld out (744)59.13/44080.3883.0682.662.69 [0.40, 4.97]2.28 [−0.00, 4.44]852.96
Llama-8B/ARC/Texttest split (1,172)67.715/79481.3184.7384.563.41 [1.71, 5.12]3.24 [1.62, 4.86]953.39
large/OBQA/Text‡held out (744)80.012/59583.0686.9686.023.90 [1.88, 5.78]2.96 [1.21, 4.57]764.00
large/OBQA/Text+fact‡held out (744)75.016/55883.0691.1389.658.06 [5.91, 10.22]6.59 [4.57, 8.47]823.75

Table 2.

The bound protects a helpful path’s gain only when κα\kappa\alpha is below the gain. The certificate bounds answer changes, not accuracy. Across the eight deployed Qwen-pair policies, development accuracy moves by −1.34 to +0.94 points (Appendix § 13). The accuracy change is at most the probability of skipping and changing the answer, which the certificate bounds by κα\kappa\alpha with probability at least 0.98 (Eq. (2)). At α=0.05\alpha= 0.05, κα\kappa\alpha exceeds Text’s gain over the receiver agent in all three large-pair Text settings (e.g., 3.9 against 3.8 points on OBQA), yet the policies keep 33 to 105% of the gain (on ARC the gain is uncertain; Appendix § 15). The retentions in Table 2 are measured, not guaranteed: on unseen questions, κ^α\hat{\kappa}\alpha exceeds the gain for Llama-8B/OBQA and large/OBQA/Text, roughly equals it for Llama-8B/ARC, and lies well below it only for Text+fact, whose certificate is nominal. Out of sample, the large/OBQA/Text policy loses 0.94 points [0.13, 1.88] against the reference. A helpful path therefore needs an accuracy check.

Labels on a few skipped disagreements bound the accuracy loss. A stricter α\alpha protects more of the gain but must be chosen with labels (Appendix § 15), though only on skipped disagreements, because accuracy can change only where the receiver agent alone and the reference disagree. With gold labels on 7 to 12 calibration questions per path (the lowest-uncertainty receiver–reference disagreements), the same test also certifies (post hoc; Appendix § 15), at a threshold no higher than the deployed one, an accuracy loss of at most one point for three of the six helpful Text paths (two of them nominal). This achieves 45 to 62% development coverage while retaining 98 to 106% of the gain. On the 744 held-out OBQA questions, these three policies lose at most two corrections each (skipped questions where the reference is right and the receiver agent alone is wrong; 95% upper bounds 0.75 to 0.97%). Labels do not rescue the medium pair’s OBQA Text path, which fails the change-rate test at every threshold: run alone after labels on 36 calibration disagreements, the lost-correction test accepts no threshold at ε=0.01\varepsilon= 0.01 or 0.02 (Appendix § 15).

Two prospective tests confirm the label-light certificate. Because the analysis above is post hoc, we froze the whole procedure of this label-light certificate (prompts, score, candidate thresholds, tests, label rule, policies, and pass rules) before any calibration output and ran it on two new systems, each evaluated once on unseen questions (Table 3; Appendix § 17.§ 6). The first is a three-agent pipeline on QASC: agent A writes background knowledge, a verifier agent B, which alone sees the two supporting facts, checks and extends A’s note, and the receiver agent decides after A’s message whether to wait for B. The second scales both agents of the Text path to 32B parameters on OBQA. Calibration queried gold labels on 40 of 3,000 and 21 of 1,366 questions. On the 2,000 sealed QASC questions, the verifier agent raises accuracy by 12.80 points and changes 16.4% of the receiver agent’s answers, yet the primary policy (ε=0.01\varepsilon= 0.01) skips it on 39.9% of questions, loses 5 corrections (95% upper bound 0.58%), and keeps 12.55 points. On the 744 held-out OBQA questions, the primary policy skips the 32B helper agent on 68.4% of questions, changes 2 answers, and keeps the whole 2.55-point gain. Both primary policies meet every pass rule fixed in advance.

SystemPolicySkipped (%)Changed / skipped (bound, %)Lost corrections (bound, %)Accuracy (%)Gain kept (points) [95% CI]Saving (ms) [95% CI]
three agentsjoint, ε=0.01\varepsilon= 0.0139.95/798 (1.46)5 (0.58)76.3512.55 [11.10, 14.15]483 [365, 604]
three agentsjoint, ε=0.02\varepsilon= 0.0249.615/991 (2.48)14 (1.17)75.9012.10 [10.65, 13.65]618 [501, 732]
three agentslabel-free54.729/1,094 (3.78)26 (1.90)75.3511.55 [10.15, 13.05]697 [581, 818]
32B pairjoint, ε=0.01\varepsilon= 0.0168.42/509 (1.41)1 (0.75)92.882.55 [0.94, 4.17]1,377 [1,219, 1,533]
32B pairjoint, ε=0.02\varepsilon= 0.0279.86/594 (2.19)4 (1.37)92.612.28 [0.67, 3.90]1,525 [1,377, 1,667]
32B pairlabel-free84.99/632 (2.69)6 (1.75)92.472.15 [0.67, 3.76]1,618 [1,485, 1,750]

Table 3. With every rule fixed before calibration, the label-light certificate holds on unseen questions for a three-agent system and for a 32B pair. Three agents (QASC, 2,000 sealed questions): the verifier agent raises accuracy from 63.80% to 76.60%, a gain of 12.80 points [11.30, 14.40]. 32B pair (OBQA, 744 held-out questions): Text raises accuracy from 90.32% to 92.88%, a gain of 2.55 points [0.94, 4.30]. Joint: largest threshold passing the change-rate and lost-correction tests at ε\varepsilon (ε=0.01\varepsilon= 0.01 is the primary policy); label-free: change-rate test only. Bounds: upper ends of two-sided 95% Clopper–Pearson intervals. Lost corrections: skipped questions that only the reference answers correctly, over all questions. Gain kept: policy accuracy minus accuracy without the skipped agent, in points, with a paired-bootstrap 95% CI. Saving: paired mean per query on 128-question development panels. Details: Appendix § 17.§ 6.

Skipping pays when the message is slow

Latency savings track the message generation cost. Every certified policy on OBQA and ARC saves time (Table 4): skipping Text, whose helper agent must generate a message, saves 319 to 718 ms per query on the 128-question panels, while skipping C2C, which needs only a prefill and a projection, saves 37 to 111 ms. The prospective systems follow the same pattern: skipping the verifier agent saves 483 ms [365, 604] per query, and skipping the 32B helper agent 1,377 ms [1,219, 1,533] (Table 3).

Queries that are not skipped pay the probe. By Eq. (4), the margin is wide for Text, but for the MMLU-Pro C2C policy, coverage times the latency gap barely exceeds 46.1 ms probe. On MMLU-Pro, the two large-pair policies skip the same 48 of 128 panel questions, yet only the Text saving is established (panel coverage is 3 points below the full set’s; Appendix § 18). Over all 2,641 development questions (second configuration), the C2C saving is small but positive, 23.3 ms [14.5, 32.2]. Both MMLU-Pro policies have negative paired medians. In three repeats on other hardware, which kept every saving class, large/MMLU-Pro/Text saves a mean of 261 ms (median −21 to −32 ms), but it skips fewer than half of its panel questions, and 84.6 to 100% of routed queries are slower in every replay. No policy’s 99th-percentile latency exceeds its reference’s in any replay (Appendix § 18). Our policy reruns the receiver agent so a skipped query returns exactly its usual answer. Answering from the probe’s prediction needs its own certificate (every threshold but one is kept) and raised the panel C2C saving from 4.6 to 85.0 ms [50.3, 122.8] in one run on the second configuration (Appendix § 18).

Savings persist under batching. The replays above run at batch size one. We replayed the four deployed Text policies of the large and Llama-8B pairs, with their thresholds unchanged, on the full OBQA and ARC development sets in static batches of 1, 8, and 32 (Appendix § 18). When skipped requests return as soon as they are answered, skipping lowers mean request latency in all twelve settings, by 310 to 1,728 ms (35 to 67% of the reference latency), and the absolute saving grows with the batch size. When a batch must return together, the saving is established in ten of the twelve settings. Batching changes a few probe scores and answers, so an operator should calibrate at the batch size it serves.

The tested policies hold on unseen questions

Sealed and held-out tests. Every frozen out-of-sample test keeps the change rate’s 95% upper bound below 5% under our answer parsers (Table 4; Appendix § 19). Each policy ran once, with routes and outputs saved before the gold labels were decoded. On the 1,172 sealed ARC-Challenge test questions, the two large-pair ARC policies change under 2% of skipped answers and run faster than their references. The C2C result holds under our parser only: the official extraction gives 5.2% [3.9, 6.7], but all 44 outputs it reads differently are an option letter plus that option’s text, rejected for strings such as tan. The Text result is identical.

The four OBQA policies ran on 744 held-out questions (medium-pair C2C also at q=0.50q = 0.50). All five OBQA tests have 95% upper bounds below 5%, which rejects ρ≥0.05\rho\ge0.05 at level 0.025, and large-pair Text and C2C also reject at δ=0.001\delta= 0.001. For the three nominal certificates, this test is the primary evidence; for Text+fact it rejects only at 0.025. Three later policies (the two Llama-8B policies and medium/ARC/C2C) each passed its protocol’s rule on held-out questions (a 95% upper bound below 5%) and rejected at 0.001, also under the official extraction. On SQuAD’s sealed test split, 2.5% [1.1, 4.6] of skipped answers change, which passes its protocol’s rule but does not reject at 0.001 (p=0.012p = 0.012). In all, eleven tests of ten of the twelve deployed policies have 95% upper bounds below 5% under our parsers and seven also reject at δ=0.001\delta= 0.001. A pass rule was prespecified only for the last four, and the MMLU-Pro policies have no unused questions.

Related work

Communication, routing, and efficiency. Agents communicate in natural language [11, 32] or through internal states [15, 21, 55]; learned gates [43] and AgentPrune [52] drop messages to improve task performance. Systems frequently trade quality against cost through routers [10, 37], cascades [4, 51, 19], and adaptive retrieval [34, 26]. These choices are often motivated by oracle headroom [25, 22]. We show that it is weak evidence of communication utility, because inputs without answer-specific content reproduce comparable headroom (§ 4.1).

Statistical risk control. Finite-sample risk is bounded in selective classification [16], Learn then Test [2], and conformal risk control [1, 39]. Our target is closest to the consistency risk of CALM [42] and risk-controlled early exiting [23], whose risk-control framework we share. Uddin & Bauer (2026) certify LLM routing with a Clopper–Pearson test and find routability model- and task-dependent, and Uddin et al. (2026) give a population condition for it that holds in every multiple-choice setting certified here and in eight that fall back (Appendix § 14). While both target the correctness of a routed answer, we certify agreement with a deployed path that may be less accurate than the receiver agent alone.

What changes when the skipped computation is a message. We claim no new test, and neither guarantee requires the reference to be the more accurate one. CALM bounds the expected inconsistency with the full model over all inputs; we bound the change rate among skipped questions, with an exact binomial test per candidate threshold. The gates read different states. Within each decoding step, an early-exit gate reads intermediate states of the computation it may cut short; our gate reads a separate message-free prefill of the receiver agent, which in our implementation adds a forward pass to every query, so we measure whether skipping pays (§ 4.4; Appendix § 21). The message comes from another agent: it may carry no answer-specific content (§ 4.1) or make the system less accurate (§ 4.2), so we measure agreement with the reference and retention of its gain separately (§ 4.3).

Discussion and limitations

Practical value and predictability. The certificate needs only unlabeled traffic. Without labels, retiring a path is a blind change, since answering alone fails the test in every setting. A binormal model using only the disagreement rate and AUROC reproduces all 30 earlier certification outcomes at their calibration sizes, and a label-free plug-in rule applied to fit-split counts reproduces all outcomes it covers (not an independent check). It also predicted the one of ten later settings that an AUROC cut near 0.80, fixed in advance, misclassified: the stronger helper agent on ARC (Appendix § 14).

Scope and failure modes. The main experiments study two agents exchanging one message before a multiple-choice answer, plus one extractive setting chosen from eight (an existence result); a prospective test adds a three-agent pipeline (§ 4.3). In the main settings the helper and receiver agents read the same question; only Text+fact and the three-agent pipeline give the helper or verifier agent information the receiver agent lacks, and no setting uses tools or several rounds. Receiver agents have at most 8B parameters in the main settings; a 32B pair repeats the label-light certificate on OBQA (§ 4.3). We have not run closed models: a closed helper agent must fit once the policy is recalibrated, a closed receiver agent must expose option-label probabilities, and C2C needs the helper agent’s KV cache. A prespecified GSM8K check fell back: the receiver agent disagrees with Text on 19.3% of development questions, a rate at which multiple-choice settings certify, but the score ranks these changes poorly (AUROC 0.58). A number restated after a derivation carries no residual uncertainty, so such tasks need a score read before the answer is committed (Appendix § 17.§ 4). We decode greedily without reasoning tokens. Sampling at temperature 0.7 keeps large-pair Text skipping certified on OBQA (nominally) and ARC (Appendix § 17.§ 5). Latency is measured at batch size one, the only mode the C2C release supports; in static batches of up to 32, skipping Text saves 35 to 67% of the reference latency when skipped requests return early (§ 4.4).

Exposure. The score, α\alpha, δ\delta, grid, and parser were fixed after OBQA and ARC development accuracy was seen, and ProbeMax was chosen on large/OBQA/Text calibration labels (Appendix § 12). The parser was written after reviewing 341 later calibration questions with gold labels, but an earlier one written without them gives identical labels in every certified Qwen setting. An earlier project stage scored small-pair outputs on the held-out OBQA questions, so they are not sealed (Appendix § 19).

Reliance on zero-uncertainty predictions. Certification does not require zeros: the Llama-8B receiver agent has none, yet is certified and holds out of sample. For Qwen receiver agents, 60 to 92% of skipped development questions have zero uncertainty (each other label <2−24< 2^{-24}). On questions with ν>0\nu> 0 alone, the test certifies only 1 of 9 settings, meaning these certificates rest on the zeros. Recomputing the probe in FP32, or breaking ties by the log of the non-top mass, changes no certification decision in 23 settings (Appendix § 20). Under option rotation, the receiver agent’s development answer changes 1.8 to 6.3% of the time at ν=0\nu= 0, so part of its confident answers follow option position. The certificate covers the deployed prompt format (Appendix § 20), and under rotation the five frozen large-pair policies still change 1.7 to 2.6% of skipped sealed and held-out answers (upper bounds below 5%; Appendix § 19). Pretraining exposure does not by itself invalidate the certificate, which assumes that deployment queries come from the calibration distribution; it matters when that assumption fails, for instance when the receiver agent has seen the benchmark questions but not the deployment traffic. Our OBQA splits come from its official training split, so an operator should recertify on its own traffic. The Llama-8B result does not rule out contamination.

Conclusion

A receiver agent often does not need the message a helper agent sends. A lightweight probe and a finite-sample test find these queries and bound how often skipping alters the answer. In the two-agent systems we study, on multiple-choice questions and one extractive task, headroom, redundancy, and net latency value are distinct properties of a communication path. Oracle headroom is weak evidence of useful content. Control inputs without answer-specific content yield comparable headroom and reproduce about three quarters of the gain over the receiver agent. Matched one path at a time, whether they separate C2C from Text depends on the match design (§ 4.1). Whether a message can be skipped follows how often and how predictably it changes the answer, not its accuracy. In four settings where Text helps, skipping kept an estimated 76 to 95% of its gain on unseen questions. Post hoc, labels on a few skipped disagreements bounded the accuracy loss of three helpful Text paths (two nominally) by one point. With every rule fixed in advance, the same certificate held on a three-agent system and on a 32B pair, whose primary policies kept 12.55 of 12.80 points and all 2.55 points of the gain. At batch size one, skipping Text saves a mean of hundreds of milliseconds per query, whereas skipping C2C saves far less unless the receiver agent answers from the probe. All eleven frozen out-of-sample tests kept the change rate’s 95% upper bound below 5% under our answer parsers (a pass rule was prespecified for four).

Acknowledgments

This research used resources of the Argonne Leadership Computing Facility, a U.S. Department of Energy (DOE) Office of Science user facility at Argonne National Laboratory (ANL) operated under Contract No. DE-AC02-06CH11357. The work was also supported under the same contract by the DOE Office of Science’s Advanced Scientific Computing Research Program and by Laboratory Directed Research and Development (LDRD) funding from ANL, provided by the Director, DOE Office of Science. The work of the University of Houston coauthors was supported in part by the National Science Foundation under grants CAREER-2305491 and CNS-2504762.

AI Use Statement

In this work, we used generative AI tools for three purposes: to edit the manuscript for readability and polish its wording; to search for and identify relevant literature and to format references; and to assist with data processing and analysis, including drafting and refining the code for these tasks and for the figures. All AI-generated text and code were reviewed by the authors, and LLM-generated code was verified and tested for correctness by the authors. We have not used generative AI tools to develop the theoretical framework, formulate mathematical claims or proofs, propose hypotheses, design the methodology or experiments, or interpret results; generating synthetic data, translation, and qualitative data analysis are not applicable to this work. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Reproducibility Statement

Sections 2 and 3 and Appendix A specify the score, label tokens, splits, threshold grid, and acceptance rule, and Appendix L gives every prompt verbatim and the answer parser’s rules. Section 3 and Appendices A, B, and H specify models, decoding, references, the timing boundary, freeze order, and statistical procedures; Appendix C summarizes the development results of all settings and their sensitivity, Appendix I the out-of-sample tests, and Appendix H the port check and repeated replays. The code repository (https://github.com/Piedpiperrr/safe-to-omit) contains the code, frozen configurations, exact partition and question IDs, model revisions, the parser’s exact expressions, scripts that regenerate every figure, and a list of the details that the appendix leaves out for length. The released dataset (https://huggingface.co/datasets/AgentsSci/ICLR2027-SafeToOmit) contains the per-question confidence scores and routes, raw action outputs, per-request timing records, and every calibration test, including rejected grid points and fallback settings. Model weights and full-vocabulary logits are not redistributed; all models and datasets are publicly available.

Method and protocol

Appendices A–L give the records of every setting and check. The code repository holds the code and configurations, and the released dataset holds the per-question records. As in the main text, a policy skips the message when the receiver answers alone, and “helper” and “receiver” abbreviate the two agents. Every setting uses one frozen procedure: a probe score from one receiver prefill, 20 thresholds from the fit split, and one exact binomial test per threshold on the calibration split (parser exposure: Appendix § 12; what the certificates rest on: Appendix § 20).

Decision, answer change, and parsing. Each setting is a binary choice between receiver-only RR and one prespecified reference bb (Text or C2C); no Text/C2C selector is learned, and C2C is each pair’s official fuser, not one of the latent protocols of earlier project stages. An answer change is db=1[oR≠ob]d_b = \mathbb{1}[o_R \ne o_b] from frozen parsed outputs. The rule-based parser reads only the raw output and displayed labels (never gold labels, IDs, or option texts), ignoring quoted, hedged, conditional, and math-like contexts (rules: Appendix § 22). Unparseable outputs share one INVALID label (two INVALID outputs agree, and each is scored as incorrect). Runtime failures are recorded separately, and gold labels are joined only after routes are frozen.

Models and prompts. All models run in bflo16 with SDPA attention, greedily, with thinking disabled, at batch size one. C2C uses each pair’s official pretrained fuser, and, as in the official pipeline, the helper runs on the receiver’s token IDs so that transferred cache positions align. The Text helper writes at most 256 tokens and the receiver at most 64 (prompts: Appendix § 22). No two paths share a helper message, KV cache, or prefill.

Score. ProbeMax uses one receiver prefill of the receiver-alone prompt followed by The correct answer is. We renormalize (in FP32) the predicted probabilities of the question’s KK displayed labels (A–J on MMLU-Pro), and define the confidence score as s(x)=u=1−max⁡lpls(x) = u = 1 - \max_l p_l. A label’s probability uses only the non-special single tokens whose surface decoding, stripped of whitespace, equals the displayed label. The probe’s argmax is not RR’s answer oRo_R and serves only as a diagnostic that affects no decision, although it matches oRo_R on 740 of 742 (large/OBQA), 299 of 299 (large/ARC), 720 of 742 (medium/OBQA), 294 of 299 (medium/ARC), 711 of 742 (small/OBQA), and 264 of 299 (small/ARC) development questions.

Grid, acceptance, and ties. Candidate j=1,…,19j = 1,\ldots,19 (q=j/20q = j/20) is the ⌈jNfit/20⌉\lceil jN_{\mathrm{fit}}/20\rceil-th smallest fit-split score (ties included; q=1q = 1 is fixed RR), frozen before reading calibration data. A threshold is accepted if Pr⁡[Bin⁡(nq,0.05)≤kq]≤0.001\Pr[\operatorname{Bin}(n_q, 0.05) \le k_q] \le0.001 on its nqn_q skipped calibration groups, of which kqk_q changed. We deploy the largest accepted candidate qq; otherwise, the system falls back to fixed bb. Clopper–Pearson 0.999 bounds are diagnostics. Exact zero uncertainties are common (large receiver: 354 of 742 OBQA and 217 of 299 ARC development questions; small: none). Equality routes to RR, and ties only make neighboring candidates identical, because the test uses only nqn_q and kqk_q; every AUROC credits ties one half.

Families and assumptions. Testing 20 predetermined candidates per setting bounds by 0.02 (Bonferroni) the chance of deploying a threshold whose true change rate is ≥α\ge\alpha, assuming i.i.d. calibration groups and that the score, thresholds, and parser were fixed independently of their outcomes. Each extension is its own family (e.g., 80 tests for the medium pair and 40 for large-pair MMLU-Pro, ideal bounds 0.08 and 0.04). There is no joint guarantee across settings or for subgroups, and the records do not establish a guarantee covering the full selection process. Of the 160 small- and large-pair tests, the 20 of large/OBQA/Text were reused from an earlier 100-test family and carry only that family’s nominal 0.10 bound. The other 140 tests are new. The large/OBQA/C2C and Text+fact certificates are also nominal (Appendix § 12).

Splits. Fit, calibration, and development hold 2,100, 1,366, and 742 OBQA questions (a partition, inherited from an earlier project stage, of the first 4,208 of the 4,952 deduplicated training questions in a fixed pseudorandom order, drawn without using labels; the other 744 are held out; all development evidence), 670, 448, and 299 ARC groups (1,119 training rows in 1,118 groups, and the validation split; the 1,172-question test split stayed sealed), and 3,000, 6,000, and 2,641 MMLU-Pro groups (all 12,032 test rows in 14 categories with 3 to 10 options, grouped by normalized text; all exposed development evidence). Because ARC’s calibration and development sets originate from different official splits, we compared them under a rule fixed before computing: calibration disagreement is lower in 5 of 10 settings, the Mantel–Haenszel odds ratio (development vs. calibration) is 0.98 [0.87, 1.10], and a permutation test gives p=0.82p = 0.82.

Timing boundary. Unless marked as the second configuration (Appendix § 18), latency is synchronized wall-clock time per question from input preparation to parsed answer on two A100 40GB GPUs of a shared node, with the helper on one GPU and the receiver and fuser on the other (startup excluded; no warmup, retry, or probe reuse; cold requests kept). Each policy runs interleaved with its fixed reference on a fixed 128-question development panel, taking the first questions in a fixed pseudorandom order of question IDs (stratified by category on MMLU-Pro). Fallback settings run no probe.

Statistics. Answer changes among skipped questions are reported as counts. Paired differences in accuracy and latency use 2,000 seed-0 paired bootstrap resamples over question groups. Change-rate intervals are exact two-sided 95% Clopper–Pearson intervals. All intervals are descriptive: they exclude model-training uncertainty and support neither equivalence nor noninferiority claims.

Provenance and exposure

Different parts of this paper rest on evidence of varying strength. Table 5 specifies when each protocol was fixed relative to its reported outcomes. Sealed data remained untouched until the frozen policy was evaluated. Held out data were never used by the tested model pair, although another pair or an earlier project stage had used them. Development data were observed during the study. A protocol frozen before outcomes was written, with a commitment to report every outcome, before any output it covers existed; this does not imply public preregistration. Post hoc denotes analyses designed after observing the results.

Four entries carry known exposure. First, the reused large/OBQA/Text certification shares its calibration labels with 440 other tests (460 total, 360 outside its declared 100-test family). Consequently, its family bound is nominal rather than established. Because large/OBQA/C2C and Text+fact reuse these calibration questions and receiver outputs, and Text and C2C change the same development questions much more frequently than independent changes would (17 against 5.8; Appendix § 16), all three bounds are nominal and rest on their held-out tests instead. Second, the parser’s rules were formulated after reviewing raw outputs and gold labels for 341 questions that later entered calibration

EvidenceSettings or policiesProtocol fixedAppendix
Certification of settings
Original settingssmall and large pairs, OBQA and ARC (6)before calibration statisticsA, C
Reused certification{sup}‡`large/OBQA/Textearlier 100-test family; exposedB
Coupled labels{sup}‡`large/OBQA/C2Cbefore calibration statisticsB
Medium pair4 settingsbefore outputsB, C
MMLU-Prolarge (2), then small and medium pairs (4)before outputsC
Informed helper{sup}‡`large/OBQA/Text+factbefore outputsG.2
Cross-family pairsOLMo and Llama pairs (7)before outputsG.1
Llama-8B receiver2 settings, added after the OLMo and Llama pairs fell backbefore outputs; post hocG.1
Stronger helpermedium receiver, OBQA and ARC Textbefore outputs; post hocG.2
Sampled answerslarge pair, OBQA and ARC Textbefore outputs; post hocG.5
Numeric-answer checklarge/GSM8K/Textbefore outputsG.4
Derivation-prefix scorelarge/GSM8K/Text; not completedbefore prefix scores; post hocG.4
SQuADLlama-8B pair (1), chosen label-free from 8 pilot settings after the GSM8K fallbackbefore outputs; selection rule before any pilot outputG.3
Out-of-sample tests
Sealed ARC testlarge/ARC, 2 policiesbefore test accessI
Held-out OBQA test4 policies, 5 testsbefore outputs; gate amendedI
Later held-out testsLlama-8B (2), medium/ARC/C2Cbefore outputs, with a pass ruleI
Sealed SQuAD test1 policybefore outputs, with a pass ruleG.3
Option rotation5 large-pair policies; post hocbefore any rotated outputI

Table 5. Most evidence rests on protocols fixed before the outcomes they cover, and exposed entries say so. Every certification outcome uses development data. Before outputs: written, with a commitment to report every outcome, before any output it covers existed. Before computing: rules fixed on stored outputs before any number was computed. Post hoc: designed after observing the results. Exposed: gold labels or scored outputs had been read before the rule or test was fixed. Gate amended: a gate found earlier use, so the questions count as held out. Pass rule: a 95% upper bound below 5%. ‡Nominal certificate: the calibration labels are exposed or coupled to the score choice (Appendix § 12). Appendix: where the full record is. (a) Certification of settings and out-of-sample tests.

Rules fixedAnalysisAppendix
Before any outputend-to-end replays and second-hardware repeatsH
probe precision and tiesJ
helper-aware score, as a hypothesisD
Before computingnull control, null headroom, and stratified nullF
argmax recertificationG.1
gain retention and tolerance gridE
busy GPU time and panel representativenessH
sealed official extractionI
zero-uncertainty split, saving by source, and unparseable changesJ
small-sample check and fit-split predictionD
Before computing; post hoclabel-light loss certificate, on calibration labels read earlierE
unsaturated AUROC and receiver rotationJ
Post hoctest and score variants, and re-split stabilityC
port check on the 500 public OBQA test questionsH
Exposedanswer parser, written after reviewing 341 later calibration questionsB

Table 5. (b) Supporting analyses, by when their rules were fixed.

splits. However, an earlier parser developed without gold labels yields identical calibration labels for every certified Qwen setting. Removing the reviewed questions leaves all deployed thresholds certified, except medium/OBQA/C2C (which passes at q=0.50q = 0.50) and the two later Llama-8B policies (which pass at 0.50 and 0.55). Third, the held-out OBQA test is the only place where a protocol was amended: after a gate detected prior use of the questions by an earlier project stage, the run was stopped. The amendment, written before any output existed, reports them as held out rather than sealed (Appendix § 19). Finally, the label-light loss certificate reuses calibration labels read earlier, and a lost-correction count for Text+fact existed prior to its protocol, which stated that none was known (Appendix § 15).

The reused large/OBQA/Text certification. The 20 tests for large/OBQA/Text originated from an earlier stage declaring a 100-test family: 60 new tests using ProbeMax, a probe-entropy score, and a word-feature disagreement classifier, and 40 reused tests, whose outcomes were already known, of a learned hidden-state predictor and a correctness-trained head. Across three project stages, the same 1,366 calibration questions and receiver–Text labels served 460 tests, and the 360 outside the declared family preceded it (60 for three other score families and 300 for correctness-trained heads under label budgets), so the records cannot exclude that the family’s design depended on earlier calibration outcomes. The computation itself uses no correctness labels: thresholds rely on fit-split scores frozen before that stage read the calibration disagreement labels, and all 20 tests recompute from the saved fit scores, calibration scores, and answer-only disagreement labels. For the three nominal bounds, the evidence we rely on is the held-out test on 744 questions (Table 33), which rejects ρ≥0.05\rho\ge0.05 at δ=0.001\delta= 0.001 for large/OBQA/Text and C2C, and only at level 0.025 for Text+fact (p=9.2×10−3p = 9.2 \times10^{-3}). No statistic restricted to the calibration split of any of the other 13 settings was computed before freezing ProbeMax, α\alpha, δ\delta, and the grid, although aggregate statistics over the full training populations that contain these splits had been. The ARC calibration split was drawn, and medium-pair and MMLU-Pro outputs produced, only after freezing these choices; the reused probe file, which included old development scores, was loaded before the C2C calibration, but only fit scores and calibration disagreement enter it.

Table 33: All five held-out OBQA tests keep the 95% upper bound below 5%, and large Text and C2C also reject at δ=0.001\delta= 0.001 (744 questions). Change rate with its two-sided Clopper–Pearson 95% CI; pp: single prespecified test of H0:ρ≥0.05H_0 : \rho\ge0.05; Level: stricter level at which H0H_0 is rejected (0.025: upper end below 5%); Always skip: change rate of answering alone; (b) accuracy of the policy and the fixed reference, and their difference Δacc\Delta\mathrm{acc} with its paired-bootstrap 95% CI; pp: percentage points. The medium q=0.50q = 0.50 row is a sensitivity threshold (deployed q=0.55q = 0.55).

(a) Changed answers
PolicyChanged / skippedChange rate (%) [95% CI]ppLevelAlways skip (%)
medium/OBQA/C2C, q=0.55q = 0.558/3882.06 [0.89, 4.02]2.5×10−32.5 \times10^{-3}0.02517.47
medium/OBQA/C2C, q=0.50q = 0.506/3461.73 [0.64, 3.74]1.4×10−31.4 \times10^{-3}0.02517.47
large/OBQA/Text, q=0.80q = 0.8012/5952.02 [1.05, 3.50]1.5×10−41.5 \times10^{-4}0.0018.20
large/OBQA/C2C, q=0.80q = 0.8013/5952.18 [1.17, 3.71]3.7×10−43.7 \times10^{-4}0.0018.33
large/OBQA/Text+fact, q=0.75q = 0.7516/5582.87 [1.65, 4.61]9.2×10−39.2 \times10^{-3}0.02511.02
(b) Accuracy
PolicyAccuracy (%) PolicyAccuracy (%) ReferenceΔacc\Delta\mathrm{acc}, policy − reference (pp)95% CI
medium/OBQA/C2C, q=0.55q = 0.5561.1660.89+0.27[−0.54, +1.08]
medium/OBQA/C2C, q=0.50q = 0.5061.1660.89+0.27[−0.40, +0.94]
large/OBQA/Text, q=0.80q = 0.8086.0286.96−0.94[−1.88, −0.13]
large/OBQA/C2C, q=0.80q = 0.8081.4580.24+1.21[+0.40, +2.15]
large/OBQA/Text+fact, q=0.75q = 0.7589.6591.13−1.48[−2.55, −0.54]

Table 33.

Parser exposure. The parser was frozen before computing calibration disagreement, but after reviewing raw outputs with gold labels for 341 questions later assigned to calibration splits (208 OBQA and 133 ARC in 355 reviews across small and large pairs; no MMLU-Pro). Questions entered review because two parser versions disagreed, or the earlier version flagged ambiguity; thus, they are not a random sample and exhibit higher disagreement (0.135 vs. 0.099 for large/OBQA/Text, 0.075 vs. 0.035 for large/ARC/C2C). We prespecified three checks. (i) The primary check removes the 341 questions from every calibration split and reruns all 20 tests with unchanged fit thresholds. The check conditions on the questions that were not reviewed. Every deployed Qwen threshold remains certified except medium/OBQA/C2C, where q=0.55q = 0.55 gives 18 changes among 640 skipped (p=0.0044p = 0.0044), while q=0.50q = 0.50 passes (skipping 355 of 742 development questions, changing 7, and answering 490 correctly vs. 486 for fixed C2C). No fallback gains a threshold, and the two Llama-8B policies (added later and never reviewed) pass at 0.50 and 0.55 instead of 0.60 and 0.70 (post hoc). The original tests remain the certificates for these three policies. (ii) The earlier parser, written from 24 synthetic outputs and 1 real output without gold labels and frozen before the review, lacks rules for embedded answers, choice verbs, reversed phrasing, scope, refusals, and some hedge words. Yet, it parses every calibration, development, held-out, sealed, and replay output of the 14 Qwen OBQA, ARC, and large-pair MMLU-Pro settings, the 7 OLMo/Llama settings, and Text+fact exactly as the final parser does, so for the nine certified Qwen-receiver policies every test statistic is what a parser fixed independently of the calibration data would give. The two differ only on some small- and medium-pair MMLU-Pro outputs (which fall back) and five small-pair C2C fit/panel outputs (where the earlier parser returns INVALID). (iii) The official C2C evaluator extracts only labels A to D, so it cannot score MMLU-Pro. It reads hedges and repeated phrases as their first letter (INVALID under our parser), marks an output unparseable if its option text contains a string it treats as a math term (tan, sin, /), and reads “A. 0°C” as the last capital letter. Under its labels, medium C2C and large Text thresholds pass, large/OBQA/C2C passes only at q=0.70q = 0.70 (its labels differ on 34 labels inside the 341 reviewed questions), and large/ARC/C2C falls back. Of the 19 settings in (ii) with labels A to D, 6 certify under both conventions and none under the official extraction alone. One five-option ARC development question lies outside the official range and is left out of that relabeling. The certified risk is defined by the frozen parser, so another notion of answer change can move the boundary.

Extensions and their freeze order. The medium pair, MMLU-Pro (large pair first; small and medium later as an 80-test family), cross-family pairs, Text+fact, stronger helper, GSM8K, SQuAD, and sampled answers were each added after earlier results (post-hoc motivated). Each used a protocol frozen before any of its outputs, committing to report every setting (fixing models, fusers, populations, paths, parser, score, grid, acceptance, and analysis). The medium pair’s first execution stopped on an engineering guard before any action or probe and was rerun unchanged. For medium-pair and MMLU-Pro replays, thresholds and panel identities were frozen before any request. MMLU-Pro went first to the large pair because the compute budget allowed one pair with the full 6,000-question calibration split; its thresholds, calibration decisions, and deployments were written before gold labels were read for accuracy. Before collecting new outputs, later runs reproduced saved outputs bit for bit: cross-family code on 20 large and 16 small fit questions, small and medium MMLU-Pro and GSM8K on saved OBQA fit rows, sampling on saved greedy outputs, and SQuAD on 16 pilot questions. Smoke tests read no gold labels.

All settings and robustness of the boundary

The certification boundary is stable: each of the 27 re-split settings (Table 6 and Table 9) retains its outcome in at least 96% of 200 re-drawn splits. At α=0.05\alpha= 0.05, none of the tested post-hoc alternative tests, scores, or learned predictors (Table 8) certifies a setting that ProbeMax rejects. Figure 3 shows every grid point of the large and small pairs, and Figure 4 compares the policies with simple baselines under the same target. All grid tests are provided in the released dataset; gain retention and the label-light certificate are in Appendix § 15, and boundary predictions in Appendix § 14.

PairTaskReferenceDisagreements / questionsAUROCSmallest 0.999 boundDeployed qqCoverage (%)Changed / skippedCorrect: policy / reference
smallOBQAText339/7420.6410.232fallback0.0–346/346
smallOBQAC2C350/7420.7000.161fallback0.0–366/366
smallARCText145/2990.6340.394fallback0.0–126/126
smallARCC2C160/2990.6410.296fallback0.0–155/155
mediumOBQAText174/7420.7870.096fallback0.0–523/523
mediumOBQAC2C*131/7420.8400.0320.5552.611/390492/486
mediumARCText52/2990.7420.102fallback0.0–221/221
mediumARCC2C48/2990.9100.0310.6063.24/189203/202
largeOBQAText‡72/7420.8730.0130.8077.119/572638/645
largeOBQAC2C‡60/7420.8830.0160.8077.113/572605/598
largeARCText15/2990.9520.0190.9595.09/284270/274
largeARCC2C12/2990.9090.0310.9090.33/270267/266
largeMMLU-ProText561/26410.8210.0210.4040.540/10691321/1319
largeMMLU-ProC2C756/26410.8460.0150.4040.537/10691257/1233
OLMoOBQAText172/7420.7900.118fallback0.0–516/516
OLMoARCText67/2990.7490.169fallback0.0–222/222
OLMoMMLU-ProText1112/26410.6330.252fallback0.0–664/664
LlamaOBQAText191/7420.6930.095fallback0.0–333/333
LlamaOBQAC2C427/7420.6180.325fallback0.0–358/358
LlamaARCText88/2990.6780.296fallback0.0–145/145
LlamaARCC2C174/2990.6150.247fallback0.0–146/146
Llama-8BOBQAText*86/7420.8740.0220.6059.07/438638/636
Llama-8BARCText*37/2990.8800.0330.7068.94/206258/260

Table 6. Ten of these 23 settings deploy and thirteen fall back (development). OLMo: large helper, OLMo-2-1124-7B-Instruct receiver; Llama: Llama-3.2-1B-Instruct helper, small receiver; Llama8B: large helper, Llama-3.1-8B-Instruct receiver. Disagreements: RR and the reference answer differently; smallest 0.999 bound: smallest calibration Clopper–Pearson 0.999 bound over the grid (diagnostic); fallback: no threshold passes; –: no question skipped; correct: development answers correct under the policy and under the reference. ‡Nominal certificate (Appendix § 12). *Without the 341 parser-review questions, only a lower qq is certified: 0.50 for medium/OBQA/C2C (47.8% coverage, 7 of 355 changed), 0.50 for Llama-8B/OBQA, and 0.55 for Llama-8B/ARC. Parser-review questions: 341 calibration questions whose outputs were read with gold labels before the parser was frozen.

Figure 3. The large pair passes the test over a wide range of thresholds under both references, whereas no threshold passes for the small pair. Each curve runs over the 20 predetermined quantiles of the same ProbeMax score, plotted against the share of questions skipped on its own population. Solid: calibration Clopper–Pearson 0.999 upper bound on the change rate among skipped questions; a grid point passes the exact test (p≤0.001p \le0.001) exactly when this bound is below the 5% target (green band; filled markers). Dashed: observed change rate among skipped development questions, which does not certify a threshold. Star and ring: deployed threshold on calibration and development. Low quantiles that share the zero threshold coincide. Vertical ranges differ by pair; medium-pair summaries are in Table 6. Both large/OBQA certificates are nominal (Appendix § 12).

Table 8. At α=0.05\alpha= 0.05, no post-hoc variant certifies a setting that the prespecified procedure rejects. (a) Deployed qq, the quantile that sets the skip threshold (a larger qq skips more questions, and 1 skips all); –: no threshold passes, so the setting keeps communicating (fallback). (b) At the deployed qq, skipped development questions whose answer changes, out of all skipped ones. Primary: ProbeMax, α=0.05\alpha= 0.05, δ=0.001\delta= 0.001. Row small/both/both: the four small-pair OBQA and ARC settings. Fixed sequence: candidates tested in ascending order of qq, each at 0.02 instead of 0.001, until one fails. Entropy, Margin: probe entropy, or one minus the top-two probability margin, replaces ProbeMax. Learned: learned predictor (receiver hidden-state logistic regression refitted on each fit split). ‡Nominal certificate (Text: reused from an earlier 100-test family; Appendix B); §tested on its calibration labels before.

PairTaskReferencePrimaryα=0.02\alpha= 0.02α=0.10\alpha= 0.10Fixed sequenceEntropyMarginLearned
smallbothboth–––––––
mediumOBQAText––0.45––––
mediumOBQAC2C0.55–0.650.550.550.55–
mediumARCText–––––––
mediumARCC2C0.60–0.750.600.600.60–
largeOBQAText‡0.800.650.900.800.80§0.800.30§
largeOBQAC2C‡0.800.650.950.850.800.800.55
largeARCText0.950.80110.950.950.90
largeARCC2C0.90–10.950.900.900.65
largeMMLU-ProText0.40–0.600.450.400.400.05
largeMMLU-ProC2C0.400.250.500.400.400.40–

caption:

Table 9. All four small- and medium-pair MMLU-Pro settings fall back, and none certifies in any of 200 re-drawn splits (N=2641N = 2641 development questions). Disagreement: share of development questions on which RR and the reference answer differ. Best candidate: smallest exact-test p-value over the 20 candidates, with its k/nk/n (changed/skipped calibration questions). Correct and INVALID: development counts of RR and the reference. Re-split: share of the 200 re-drawn splits that deploy.

PairReferenceDisagreement (%)AUROCppk/nk/nDeployed qqRRReferenceRRReferenceRe-split
smallText46.760.6441102/280fallback4035161512290.000
smallC2C60.240.6261106/280fallback4034621514200.000
mediumText42.710.717164/690fallback813913160180.000
mediumC2C53.390.7650.19329/690fallback81353016011240.000

caption:

Figure 4. Receiver-alone execution fails the test everywhere and random skipping carries no certificate, although both can answer as well as the certified policy or run faster. (a) Fixed receiver-only execution: calibration change rate in the 14 OBQA, ARC, and large-pair MMLU-Pro settings; none passes the test, including the two large-pair ARC settings below the 5% target (green band; pp: exact-test pp-value). (b) The eight deployed settings, one panel each, in the accuracy–latency plane relative to the fixed reference (+, dashed lines). Horizontal: mean latency as a share of the reference’s, recomposed from per-question development latencies (component estimates, not end-to-end measurements); the policy includes its probe, and random skipping has none. Vertical: development correct answers minus the reference, in percentage points. Filled circle or triangle: the certified policy (Text or C2C); open diamond: receiver alone; dotted line: random skipping at any coverage, which lies on the line from receiver alone to the reference, with a ring at the policy’s coverage (expected count); black square: the historical learned predictor (large/OBQA/Text), an earlier stage’s receiver hidden-state logistic regression, at its latency in that stage’s accounting.

Calibration statistics. Table 7 lists the calibration test at the deployed quantile for the 14 OBQA, ARC, and large-pair MMLU-Pro settings (Table 6), or at the grid point with the smallest pp-value for fallbacks. With α=0.05\alpha= 0.05 and δ=0.001\delta= 0.001, a skipped subset with no observed changes passes only if it contains at least 135 calibration questions, since 0.95135<0.001<0.951340.95^{135} < 0.001 < 0.95^{134}. At q=0.05q = 0.05, the small pair skips only 65 OBQA and 18 ARC calibration questions. Figure 3 shows the full grid: the large pair’s 0.999 upper bound stays below 5% until 78 to 94% of calibration questions are skipped, whereas the small pair’s never falls below 16%.

PairTaskReferenceDeployed qqTested qqnnkkppk/nk/nppRe-splitNeeded NcalN_{\mathrm{cal}}
smallOBQATextfallback0.056550.894––0.00never
smallOBQAC2Cfallback0.056520.363––0.0021,793
smallARCTextfallback0.103350.995––0.00never
smallARCC2Cfallback0.103320.773––0.00never
mediumOBQATextfallback0.05601360.884––0.00never
mediumOBQAC2C0.550.55739196.9×10−46.9 \times10^{-4}18/6404.4×10−34.4 \times10^{-3}1.00–
mediumARCTextfallback0.05256120.483––0.0079,333
mediumARCC2C0.600.6029015.6×10−65.6 \times10^{-6}1/2102.5×10−42.5 \times10^{-4}0.96–
largeOBQAText{sup}‡`0.800.801064181.2×10−81.2 \times10^{-8}17/9087.8×10−77.8 \times10^{-7}1.00–
largeOBQAC2C{sup}‡`0.800.801064226.7×10−76.7 \times10^{-7}19/9085.4×10−65.4 \times10^{-6}1.00–
largeARCText0.950.9542244.8×10−64.8 \times10^{-6}2/2973.4×10−53.4 \times10^{-5}1.00–
largeARCC2C0.900.9040475.4×10−45.4 \times10^{-4}2/2826.6×10−56.6 \times10^{-5}1.00–
largeMMLU-ProText0.400.402358671.3×10−71.3 \times10^{-7}67/23581.3×10−71.3 \times10^{-7}1.00–
largeMMLU-ProC2C0.400.402358751.0×10−51.0 \times10^{-5}75/23581.0×10−51.0 \times10^{-5}1.00–

Table 7. Deployed settings pass the exact test with 290 to 2,358 skipped calibration questions, whereas the small pair’s best candidates skip at most 65. Rows: small, medium, and large pairs only. Tested qq: the deployed quantile, or for a fallback the grid point with the smallest pp; nn, kk: skipped and changed calibration questions at the tested qq; pp: exact-test p-value; –: not applicable. Reviewed out: the test without the 341 parser-review questions. Parser-review questions: 341 calibration questions whose outputs were read with gold labels before the parser was frozen (none on MMLU-Pro). Re-split: share of 200 re-drawn splits that deploy. Needed NcalN_{\mathrm{cal}}: calibration size at which the observed rate would pass (never: the observed rate is at least 0.05). {sup}‡Nominal certificate (Text: reused from an earlier 100-test family; {xref}Appendix B`).

Stability and sensitivity. We re-drew the fit and calibration splits of these 27 settings 200 times using the original grouping, sizes, and shuffle rule (for MMLU-Pro, the seed of its fixed pseudorandom order), recomputed the 20 thresholds, and reran the test; development sets remained unchanged. Medium/ARC/C2C deploys in 96% of re-draws, every other deployment in all 200, and no fallback setting deploys in any (the stronger-helper fallbacks certify in none and 15.5%, Appendix § 17.§ 2; Text+fact and GSM8K were not re-split). Among deploying re-draws, the median deployed qq matches the original everywhere except large/ARC/Text (0.90 instead of 0.95). Five fallback settings observe a change rate below 0.05 but would need 3,975 (OLMo/ARC/Text), 7,494 (Llama/OBQA/Text), 13,386 (Llama/ARC/C2C), 21,793 (small/OBQA/C2C), and 79,333 (medium/ARC/Text) calibration questions to pass. The medium pair’s disagreement rates remain similar across its fit, calibration, and development splits (OBQA: 24.4, 25.5, and 23.5% with Text, 16.8, 17.7, and 17.7% with C2C; ARC: 20.0, 15.0, and 17.4%, and 16.0, 12.3, and 16.1%), and its probe produced no INVALID score.

Post-hoc variants. Table 8 recomputes every setting from saved records under post-hoc variants modifying one element each. The variants are: setting α=0.02\alpha= 0.02 or 0.10 with the same per-candidate cutoff; fixed-sequence testing (evaluating candidates in ascending order of qq, each at the family’s 0.02 Bonferroni bound instead of 0.001, deploying the last accepted qq); and using ProbeMax with probe entropy or one minus the top-two probability margin (read from the same prefll); and using the learned predictor below. This predictor is refitted per setting on its fit split using historical features and hyperparameters (fixed on large/OBQA/Text, not tuned per setting). Thresholds always derive from fit-split quantiles. At α=0.05\alpha= 0.05, no variant certifies a setting that ProbeMax rejects, and every small-pair setting falls back under all variants. Setting α=0.02\alpha= 0.02 removes the medium pair, large/ARC/C2C, and MMLU-Pro/Text, while α=0.10\alpha= 0.10 adds medium/OBQA/Text. Fixed receiver-only execution (q=1q = 1), which fails the prespecified test everywhere, is accepted for largepair ARC at α=0.10\alpha= 0.10 (both references) and under fixed-sequence testing (Text; 12 changes among 448 calibration questions, p=0.011p = 0.011). The learned predictor’s development AUROC for receiver–reference disagreement trails ProbeMax’s in every setting (0.55–0.64 for the small pair, 0.69–0.75 for the medium pair, 0.64–0.88 for the large pair); its lower coverage does not imply learned skipping is useless. Fixed-sequence testing yields an equal or larger qq across all settings here, but it depends on testing order and stops at the first failure. Bonferroni remains primary because it was prespecified before choosing these variants.

Same-target baselines (post hoc). Figure 4 compares each policy with two simple skipping baselines under the same target; no threshold, reference, or split was changed, and neither baseline was replayed end-to-end. Fixed receiver-only execution (q=1q = 1) fails the test in all 14 OBQA, ARC, and large-pair MMLU-Pro settings (calibration change rate 2.7 to 56.7%). The two large-pair ARC rates fall below 5% but yield p=0.011p = 0.011 (Text, 12 of 448) and p=0.435p = 0.435 (C2C, 21 of 448); there, fixed RR answers about as many development questions correctly as the certified policies, at a lower component-estimated latency. A rule skipping questions at random, independently of content, changes answers at the receiver–reference disagreement rate in expectation regardless of coverage (2.7 to 29.1% on calibration for the eight deployed settings). At the policy’s development coverage κ\kappa, its expected correct count (1−κ) ab+κ aR(1-\kappa)\,a_b + \kappa\,a_R (where aba_b and aRa_R are the correct counts of the reference and RR) exceeds the policy’s in three C2C settings (medium ARC, large OBQA, large ARC) and trails it in the other five. Its component-estimated latency is 18 to 67 ms lower (it runs no probe, and its decision overhead was not measured). The probability that a random subset of a grid point’s size, drawn without replacement from the fixed calibration split, passes the test remains below 10−410^{-4} at every grid point in every setting except large ARC with Text, peaking at 0.017 (at q=0.80q = 0.80). These are per-grid-point probabilities, not the probability that a search over the grid passes.

Historical learned predictor. An earlier stage evaluated ProbeMax and a learned predictor on large/OBQA/Text using the same grid, α\alpha, and cutoff, within its declared 100-test family (Bonferroni level 0.10; Appendix § 12), where both passed. The predictor is an L2-regularized logistic regression on 8,192-dimensional receiver hidden states (last-token outputs of an intermediate layer and the final normalization layer of Qwen3-8B), fitted on the 2,100 fit questions to predict receiver–Text disagreement. It requires reference executions and offline feature extraction but no correctness labels. It skipped 230 of 742 development questions with 6 changes and 643 correct answers (component latency 797.2 ms in that stage’s accounting), against 572 skipped, 19 changes, 638 correct, and 460.9 ms for ProbeMax (462.9 ms in the unified records of Figure 4).

PairTaskReferencePrimaryα=0.02\alpha= 0.02α=0.10\alpha= 0.10Fixed sequenceEntropyMarginLearned
smallbothboth–––––––
mediumOBQAText––18/321––––
mediumOBQAC2C11/390–25/48111/39011/38911/391–
mediumARCText–––––––
mediumARCC2C4/189–7/2194/1893/1873/187–
largeOBQAText‡19/5726/46044/67619/57218/57219/5726/230
largeOBQAC2C‡13/5721/46045/70322/61313/57213/57213/399
largeARCText9/2840/23315/29915/2999/2849/2847/272
largeARCC2C3/270–12/2996/2843/2703/2704/194
largeMMLU-ProText40/1069–109/159656/120540/106840/10695/128
largeMMLU-ProC2C37/10696/69785/134137/106937/106838/1069–

Table 8b. Changed / skipped development questions at the deployed qq

MMLU-Pro. The large receiver shows exact zero uncertainty on 713 of 3,000 fit groups (1,442 of 6,000 calibration groups), merging quantiles q≤0.20q \le0.20 into one threshold. Both large-pair references pass at q=0.40q = 0.40 (67 and 75 changes among 2,358 skipped calibration groups) and fail at q=0.45q = 0.45 (105 and 118 of 2,654; p=0.0063p = 0.0063 and 0.101). On development, ProbeMax’s average precision for disagreement is 0.522 (Text) and 0.629 (C2C). Of the policies’ changed answers, 14 (Text) and 27 (C2C) became correct, 12 and 3 became incorrect, and 14 and 7 stayed incorrect. INVALID counts are 6 (R), 2 (Text), and 20 (C2C), and 2 and 19 for the two policies. The pooled certificate gives no subgroup guarantee: some development categories exceed 5% changed answers (e.g., engineering C2C, 12 of 72; Table 10). Run within each of the 14 categories under a rule fixed before computing, using category-specific fit quantiles and calibration questions, the test certifies three categories for Text (biology, economics, and engineering; the latter changing 5 of 102 skipped development answers under its own threshold) and three for C2C (health, other, and psychology). With δ\delta further divided by 14, it certifies none for Text and one (psychology) for C2C. Skipping only within certified disciplines lowers development coverage from 40.5% to 11.1% for both references, and to 0.0% (Text) and 3.7% (C2C) with δ/14\delta/14 (Table 11). All four small- and medium-pair settings fall back (smallest p=0.193p = 0.193 over 80 tests; disagreement 42.7 to 60.2%, AUROC 0.626 to 0.765; Table 9).

Table 10: The pooled MMLU-Pro certificate gives no subgroup guarantee: some categories exceed 5% changed answers (development group representatives; descriptive). Both references skip the same questions. Policy: large pair at its deployed qq; NN: development questions in the category.

CategoryNNSkippedCoverage (%)TextC2C
biology14410069.41/1000/100
business1774726.61/472/47
chemistry2576023.34/606/60
computer science934245.24/421/42
economics19011661.12/1160/116
engineering2207232.74/7212/72
health1559561.35/950/95
history864754.71/470/47
law2176228.64/622/62
math3059731.84/979/97
other2109645.75/961/96
philosophy1133934.52/390/39
physics2938228.00/823/82
psychology18111463.03/1141/114

Table 10.

Their answers fail to parse more often, above all with C2C: INVALID shares of development answers are 5.7 (RR), 8.7 (Text), and 15.9% (C2C) for the small pair, and 6.1, 0.7, and 42.6% for the medium pair, compared to 0.2, 0.1, and 0.8% for the large pair. Consequently, medium-pair C2C answers far fewer questions correctly than the receiver alone (530 against 813). Their three-action headroom is 11.59 (small) and 7.91 points (medium) over the best fixed action, against 9.05 for the large pair, and 15.87 and 11.70 points over the receiver alone (no null control was run for them).

Predicting the boundary

Given its calibration size, each setting’s disagreement rate and AUROC predict its certification outcome, allowing a small shadow sample to anticipate it. However, the bound still comes from the calibration test. The rules of every analysis below were fixed before computing.

A small-sample check. Before the four small- and medium-pair MMLU-Pro settings and GSM8K had calibration results, a manuscript draft already stated the pattern among the 22 settings then run: deployed settings had development AUROC of at least 0.82 and fallbacks at most 0.79. All five later settings have AUROC below 0.79 and fall back (the timestamps of their result files are in the code repository). Because the cut near 0.80 was read off the resulting 26 multiple-choice settings, a protocol fixed before computing drew 200 samples of nn fit questions per setting. It predicted deployment when the sample AUROC is at least 0.80. This prediction matches the outcome in 0.880, 0.911, and 0.924 of draws on average for n=100n = 100, 200, and 400, and at n=200n = 200 in at least 0.80 of draws for 21 of 26 settings (0.950 over deployed and 0.890 over fallback settings). Misses concentrate near the cut and where the fit and development splits differ: medium/ARC/Text matches in 14.5% of draws (fit-sample AUROC median 0.829, development 0.742), while medium/OBQA/Text, large/MMLU-Pro/Text, and OLMo/OBQA match in 53.0, 66.5, and 78.5%. This cut, fixed in that protocol, sorts four of the five settings that ran afterwards (the two stronger-helper settings, the two Llama-8B settings, and SQuAD). The exception is the stronger helper’s ARC setting (AUROC 0.800), a fallback that the plug-in rule and the binormal model below, both fixed before it ran, predicted.

Predicting certification from the fit split. A label-free plug-in rule applies the frozen test to fit-split counts rescaled to the calibration size. For candidate jj, with mjm_j fit questions at or below its threshold and djd_j of them disagreeing, it tests n^j=round⁡(Ncalmj/Nfit)\hat{n}_j = \operatorname{round}(N_{\mathrm{cal}}m_j/N_{\mathrm{fit}}) and k^j=round⁡(n^jdj/mj)\hat{k}_j = \operatorname{round}(\hat{n}_jd_j/m_j) and predicts the largest accepted qq, with coverage mj/Nfitm_j/N_{\mathrm{fit}}. The rule matches all 8 deployments and 17 fallbacks of the 25 settings with fit-split reference outputs (qq exact in 5 of 8, coverage off by a median of 2.7 points; Table 12). This is a plug-in estimate of the test’s outcome, not an independent check. The predicted saving (the predicted coverage times the gap between the mean development latencies of the reference and of the receiver alone, minus the mean probe cost) is off by a median of 15.4 ms (12.6%), and by 15.3 ms with each panel’s actual coverage. Thus, the error comes mainly from the latency model. The largest relative miss is large/MMLU-Pro/C2C, whose measured saving has an interval that includes zero. Run later, byte-identical, the rule predicted both stronger-helper fallbacks prospectively (2 of 2) and both Llama-8B deployments. The Llama-8B predictions are not prospective because those settings had begun producing outputs before the rule was fixed (q=0.65q = 0.65 against 0.60 and 0.70, coverage off by 6.0 and 3.8 points).

SettingDeployed qq certifiedDeployed qq predictedCoverage (%) development (certified)Coverage (%) fit (predicted)Saving (ms) measuredSaving (ms) predicted, with predicted coverageSaving (ms) predicted, with panel coverage
medium/OBQA/C2C0.550.5552.655.179.764.758.4
medium/ARC/C2C0.600.6063.261.0100.798.997.0
large/OBQA/Text0.800.8077.180.0517.1536.6506.1
large/OBQA/C2C0.800.8577.185.036.626.619.0
large/ARC/Text0.950.9095.090.0717.6670.6700.4
large/ARC/C2C0.900.9090.390.0110.988.988.6
large/MMLU-Pro/Text0.400.4040.540.0346.3360.9335.4
large/MMLU-Pro/C2C0.400.3540.535.28.2−7.7−5.1

Table 12.

Table 12: The fit split alone predicts every certification outcome of the 25 settings, with the deployed qq exact in five of the eight deployments. Predicted: the frozen test applied to fit-split counts rescaled to the calibration size. Coverage: development (certified) and fit (predicted); saving predicted with the predicted or the panel’s coverage. Measured: saving of the end-to-end replay on the 128-question panel; predicted saving: coverage times the gap between the reference’s and RR’s mean development latencies, minus the mean probe cost. OLMo: large helper, OLMo-2-112-7B-Instruct receiver; Llama: Llama-3.2-1B-Instruct helper, small receiver.

Certified fallback and predicted fallback (17 settings): small/OBQA/Text, small/OBQA/C2C, small/ARC/Text, small/ARC/C2C, medium/OBQA/Text, medium/ARC/Text, small/MMLU-Pro/Text, small/MMLU-Pro/C2C, medium/MMLU-Pro/Text, medium/MMLU-Pro/C2C, OLMo/OBQA/Text, OLMo/ARC/Text, OLMo/MMLU-Pro/Text, Llama/OBQA/Text, Llama/OBQA/C2C, Llama/ARC/Text, Llama/ARC/C2C.

A binormal model of the boundary. We simulated the frozen procedure with binormal scores: agreeing questions draw from N(0,1)\mathcal{N}(0,1) and disagreeing ones from N(μ,1)\mathcal{N}(\mu,1), μ=2 Φ−1(AUROC)\mu= \sqrt{2}\,\Phi^{-1}(\mathrm{AUROC}), at each setting’s calibration disagreement rate and split sizes. Each of 2,000 simulations draws a fit split, sets the 20 thresholds, draws a calibration split, and applies the frozen test and selection rule. Predicting deployment when the simulated probability is at least 0.5 reproduces all 28 outcomes, using either the development or the calibration AUROC. Certified settings have probabilities from 0.61 (medium/OBQA/C2C) to 1, and no fallback exceeds 0.05. Run later, the same code predicted both stronger-helper fallbacks (probability 0.040 and 0.049; AUROC needed 0.853 and 0.885). Therefore, from each setting’s disagreement rate and AUROC, the model accounts for all 30 outcomes that preceded SQuAD: the boundary is what the test implies given these two numbers, and the empirical content is where each path falls. From the pilot, it predicted the SQuAD certification (0.975, inflated by selecting the best of eight; conservative 0.081), a prospective test, and 0.749 from SQuAD’s development statistics. The model has no point mass at zero uncertainty yet reproduces every outcome. The AUROC needed for probability 0.5 ranges from 0.63 (large/ARC/Text) to 0.996 (the Llama pair’s ARC/C2C; Figure 5). The population condition of Uddin et al. [45], a candidate whose true-to-false-positive ratio on calibration reaches (1−π)(1−α)/(πα)(1-\pi)(1-\alpha)/(\pi\alpha), holds in all 11 certified multiple-choice settings, in SQuAD (18.0 against 3.38), and in eight fallback settings, including both stronger-helper ones, whose calibration splits are too small for the finite-sample test.

Figure 5. The frozen test certifies a binormal score only above an AUROC that rises with the disagreement rate and falls with the calibration size, which reproduces all 28 certification outcomes. Binormal score: agreeing questions score N(0,1)\mathcal{N}(0,1), disagreeing ones N(μ,1)\mathcal{N}(\mu,1), μ=2Φ−1(AUROC)\mu= \sqrt{2}\Phi^{-1}(\mathrm{AUROC}). Probability that the frozen test certifies (1,000 simulations per cell) at the fit and calibration sizes of ARC, OBQA, and MMLU-Pro; black line: probability 0.5, not drawn where it falls below AUROC 0.60 or above 0.98. Points: the 28 settings (18 main, 9 cross-family, and Text+fact) at their calibration disagreement rate and development AUROC; filled if deployed; Text+fact is drawn as Text. Llama: Llama-3.2-1B-Instruct helper, small receiver. The Llama pair’s ARC/C2C setting (disagreement 0.61) sits on the right edge of the first panel.

A helper-aware score. Under a hypothesis fixed before any output, we added a helper-agreement term to ProbeMax: sG(x)=uR(x)+I[arg⁡max⁡H(x)≠arg⁡max⁡R(x)]s_G(x) = u_R(x) + \mathbb{I}[\arg\max_H(x) \ne\arg\max_R(x)]. Here, arg⁡max⁡H\arg\max_H comes from one helper prefill of the question and options with the same answer phrase and label-token rule. The extracted prompt construction reproduced the frozen receiver probes bit for bit, and setting arg⁡max⁡H:=arg⁡max⁡R\arg\max_H := \arg\max_R reproduced the deployed qq of all 14 OBQA, ARC, and large-pair MMLU-Pro settings. The term raises development AUROC by 0.021 on average for Text and lowers it by 0.049 for C2C, favoring Text in all seven populations (at least five were required). The four later small- and medium-pair MMLU-Pro settings point the same way (Text larger in all nine populations, +0.019 against −0.043), which extends the prespecified rule rather than replacing it. No setting crosses the boundary in either direction. Where the helper and receiver often differ, the term is counterproductive: on MMLU-Pro, it pushes a third of the small pair’s and about 60% of the medium pair’s questions above every threshold. The helper prefill costs 17.4 ms (small pair), 20.2 to 20.8 ms (medium), and 25.9 to 35.5 ms (large), and it runs the helper that skipping avoids. This analysis covers the 18 main settings and Text+fact, is post hoc, and needed one key-naming fix to the script, which then reproduced every earlier number and the verdict.

Accuracy under skipping

The change-rate certificate bounds accuracy only through kαk\alpha, which can exceed a helpful path’s gain. However, gold labels on a few skipped disagreements bound the loss directly: skipping can lower accuracy only on skipped questions where the reference is correct and the receiver alone is wrong. At α=0.05\alpha= 0.05, the three large-pair Text policies retrial 33 to 105% of the reference’s gain. Stricter tolerances (α≤0.03\alpha\le0.03), which require labels to set, retain more gain on OBQA, ARC, and Text+fact, though not on MMLU-Pro.

Gain retention. Under rules fixed before computing, we measured the reference’s gain over the receiver (G=acc⁡(b)−acc⁡(R)G = \operatorname{acc}(b) - \operatorname{acc}(R)), the policy’s gain (acc⁡(πτ)−acc⁡(R)\operatorname{acc}(\pi_\tau) - \operatorname{acc}(R)), and the share of GG retained, using paired bootstrap intervals. We repeated the certification with α\alpha from 0.01 to 0.05 at the same per-candidate cutoff (Table 13; Figure 6b). At α=0.05\alpha= 0.05, policies retain 0.750 [0.467, 1.067] of Text’s gain on OBQA, 1.054 [0.717, 1.858] on MMLU-Pro, and 0.333 [−1.0-1.0, 1.4] on ARC, where the gain itself is uncertain (G=2.01G = 2.01 points [−0.33-0.33, +4.68+4.68]; G≤0G \le0 in 6.8% of resamples), and Text+fact retains 0.811 [0.686, 0.927]. On the 744 held-out OBQA questions, where RR ran on every question, large Text retains 0.759 [0.500, 0.958] and Text+fact retains 0.817 [0.706, 0.933]; the sealed ARC run did not execute RR on routed questions. The largest α\alpha with κα<G\kappa_\alpha< G is 0.04 for OBQA/Text (threshold deployed at 0.05), 0.02 for ARC/Text, 0.03 for MMLU-Pro/Text, and 0.05 for Text+fact; any such α\alpha must be chosen with labels. For these thresholds, we recomposed the panel saving from the per-request timings of each existing replay, excluding cold requests: a question the stricter policy still skips retains its recorded policy latency, and a routed question is charged its recorded probe latency plus the fixed arm’s latency on that question; at the original thresholds this reproduces the measured saving within 1 ms. The recomposed savings are 532.9, 587.3, and 247.0 ms on the original configuration and 420.3 to 429.5, 469.3 to 479.5, and 197.1 to 199.8 ms in the three repeats on the second one (OBQA, ARC, and MMLU-Pro), and 342.7 ms for Text+fact on the second configuration; these are not new replays.

α\alphaqqCoverage (%)Changedκα\kappa_\alphaRetainedqqCoverage (%)Changedκα\kappa_\alphaRetained
large/OBQA/Textlarge/ARC/Text
0.01fallback0———fallback0———
0.020.6562.06/4601.241.0000.8077.90/2331.561.000
0.030.7572.215/5362.170.8930.8584.91/2542.550.833
0.040.8077.119/5723.080.7500.9595.09/2843.800.333
0.050.8077.119/5723.850.7500.9595.09/2844.750.333
large/MMLU-Pro/Textlarge/OBQA/Text+fact
0.01fallback0———fallback0———
0.02fallback0———0.5553.61/3981.070.981
0.030.3030.716/8120.920.9730.6562.03/4601.860.943
0.040.3536.031/9521.441.0540.7066.47/4932.660.887
0.050.4040.540/10692.021.0540.7572.213/5363.610.811

Table 13.

Figure 6. Skipping loses few reference corrections under C2C but more under Text, and a stricter tolerance keeps more of Text’s gain except on MMLU-Pro. (a) Reference corrections (development questions the reference answers correctly and the receiver alone does not) that each deployed Qwen-pair policy loses by skipping or keeps by running the reference, with the lost count out of all the reference’s corrections and the net change in correct answers (policy minus reference, gained minus lost). (b) Share of the reference’s gain over the receiver that the large-pair Text policies keep at each tolerance α\alpha (Table 13); missing points fall back, and the dashed line marks the whole gain. Development populations; descriptive and gold-dependent; large/OBQA/Text, large/OBQA/C2C, and Text+fact are nominal (Appendix § 12).

Lost reference corrections. On skipped questions a policy replaces the reference’s answer by the receiver’s own, so for the skip set Oτ={x:s(x)≤τ}O_\tau= \{x : s(x) \le\tau\} and gold label yy, acc⁡(πτ)−acc⁡(b)=Pr⁡(x∈Oτ,oR=y,ob≠y)−Pr⁡(x∈Oτ,oR≠y,ob=y)\operatorname{acc}(\pi_\tau) - \operatorname{acc}(b) = \Pr(x \in O_\tau, o_R = y, o_b \ne y) - \Pr(x \in O_\tau, o_R \ne y, o_b = y): the receiver’s successes that skipping recovers minus the reference corrections (questions the reference answers correctly and RR does not) that it loses. The identity holds in all fourteen OBQA, ARC, and large-pair MMLU-Pro settings. No C2C policy loses more than 3 reference corrections, whereas large/OBQA/Text loses 13 of 46 and large/ARC/Text 6 of 10 (Figure 6a); net of the successes they recover, the C2C policies answer 1 to 24 more questions correctly than their references, large/MMLU-Pro/Text answers 2 more, and large/OBQA/Text and large/ARC/Text answer 7 and 4 fewer. These are descriptive development outcomes that the risk test does not target. Questions where some action is correct but the policy is not (unused three-action opportunity) are more common on the kept side in seven of the eight settings; this does not show that the probe identifies them or that a three-action router could realize the oracle.

Table 13: Lowering α\alpha never raises coverage; at α≤0.03\alpha\le0.03 it keeps more of the reference’s gain over the receiver on OBQA, ARC, and Text+fact, but not on MMLU-Pro (development sets). qq: deployed quantile (fallback: no threshold passed; —: not applicable). Changed: changed/skipped development questions. κα\kappa_\alpha: worst-case accuracy loss (points). Retained: share of GG kept; GG is 3.77 (OBQA), 2.01 (ARC), 1.40 (MMLU-Pro), and 7.14 points (Text+fact).

Label-light accuracy loss certificate. Let kLk_L count the lost corrections (skipped questions where the reference is right and RR is wrong) among the NN calibration questions. Because acc⁡(b)−acc⁡(πτ)≤Pr⁡[L]\operatorname{acc}(b) - \operatorname{acc}(\pi_\tau) \le\Pr[L], the exact binomial test of H0:Pr⁡[L]≥εH_0 : \Pr[L] \ge\varepsilon certifies an accuracy loss of at most ε\varepsilon, and it needs gold labels only on calibration disagreements, taken in order of increasing uu up to the first rejected candidate or the deployed threshold. After the main results, under rules fixed before computing, we ran this test for the six helpful Text policies at ε=0.01\varepsilon= 0.01 and 0.020.02 over the same 20 frozen candidates (δ=0.001\delta= 0.001 each) and intersected it with the change-rate test: the joint certificate is the largest candidate that both accept, with family-wise error of at most 0.04 per setting and ε\varepsilon (0.06 per setting for both ε\varepsilon). A setting counts if the joint certificate skips at least 20% of development questions and ε\varepsilon is at most half the development gain GG. At ε=0.01\varepsilon= 0.01, large/OBQA/Text, Text+fact, and Llama-8B/OBQA/Text count (Table 14 and Table 15): with labels on 8, 7, and 12 of their 143, 170, and 217 calibration disagreements they skip 62.0, 53.6, and 45.0% of development questions and keep 100, 98.1, and 106.1% of the gain. On the 744 held-out questions they lose 2, 2, and 1 corrections (upper bounds 0.97, 0.97, and 0.75%). Large/MMLU-Pro/Text certifies a loss of at most one point at its deployed threshold, but its gain of 1.40 points is less than twice ε\varepsilon; the two ARC settings have 448 calibration questions, fewer than the 688 that ε=0.01\varepsilon= 0.01 needs. At ε=0.02\varepsilon= 0.02, Text+fact’s held-out upper bound (2.11%) exceeds ε\varepsilon, and its development accuracy falls by 0.81 points [0.27, 1.48]. The analysis is post hoc and reuses calibration labels read earlier; two of the three counted settings are nominal (Appendix § 12). Our protocol stated that no calibration lost-correction count was known to us, but an earlier analysis had reported two for Text+fact (18 among the 999 questions skipped at q=0.75q = 0.75; 122 over the whole split). Neither was used to fix ε\varepsilon, the 20% coverage bar, or the gain condition, which were fixed with development lost-correction counts known for some settings. Without the 341 parser-review questions (Appendix § 12), the joint threshold falls or disappears in 6 of the 10 joint certificates; for the three counted settings it is 0.60, 0.55, and 0.35.

Settingε\varepsilonNNCandidates accepted (of 20): lost-correctionCandidates accepted (of 20): change-rateCandidates accepted (of 20): bothLargest accepted qq: lost-correction (qLq_L)Largest accepted qq: both (qjointq_{\mathrm{joint}})Largest accepted qq: change-rate (deployed)qjointq_{\mathrm{joint}} without reviewed
large/OBQA/Text‡0.011,3661316130.650.650.800.60
large/OBQA/Text‡0.021,3661516150.750.750.800.75
large/ARC/Text0.014480190infeasibleinfeasible0.95infeasible
large/ARC/Text0.024481619160.800.800.95infeasible
large/MMLU-Pro/Text0.016,0009880.450.400.400.40
large/MMLU-Pro/Text0.026,00012880.600.400.400.40
large/OBQA/Text+fact‡0.011,3661115110.550.550.750.55
large/OBQA/Text+fact‡0.021,3661415140.700.700.750.65
Llama-8B/OBQA/Text0.011,36691070.450.450.600.35
Llama-8B/OBQA/Text0.021,3661210100.600.600.600.50
Llama-8B/ARC/Text0.01448080infeasibleinfeasible0.70infeasible
Llama-8B/ARC/Text0.024489830.450.450.70infeasible

Table 14. (a) Accepted candidates and certified thresholds

Settingε\varepsilonqjointq_{\mathrm{joint}}Coverage (%)Lost correctionsGained answersΔ\Deltaacc (points) 95% CIGG (points)Kept (of GG)Saving (ms) 95% CI
large/OBQA/Text‡0.010.6562.033+0.00 [−0.67, +0.67]3.771.000393.4 [310.3, 468.8]
large/OBQA/Text‡0.020.7572.296−0.40 [−1.48, +0.54]3.770.893490.0 [411.9, 560.5]
large/ARC/Text0.01infeasible––––2.01––
large/ARC/Text0.020.8077.900+0.00 [0.00, 0.00]2.011.000588.6 [522.4, 656.1]
large/MMLU-Pro/Text0.010.4040.51214+0.08 [−0.30, +0.45]1.401.054346.3 [249.8, 443.0]
large/MMLU-Pro/Text0.020.4040.51214+0.08 [−0.30, +0.45]1.401.054346.3 [249.8, 443.0]
large/OBQA/Text+fact‡0.010.5553.610−0.13 [−0.40, 0.00]7.140.981245.1 [195.7, 293.1]
large/OBQA/Text+fact‡0.020.7066.460−0.81 [−1.48, −0.27]7.140.887306.2 [259.2, 352.0]
Llama-8B/OBQA/Text0.010.4545.013+0.27 [−0.27, +0.81]4.451.061218.1 [164.1, 273.4]
Llama-8B/OBQA/Text0.020.6059.024+0.27 [−0.40, +0.94]4.451.061318.6 [261.1, 372.1]
Llama-8B/ARC/Text0.01infeasible––––7.02––
Llama-8B/ARC/Text0.020.4540.800+0.00 [0.00, 0.00]7.021.000199.0 [144.6, 254.9]

Table 15. At ε=0.01\varepsilon= 0.01 the three counted settings keep 100, 98.1, and 106.1% of the gain on development and lose at most two corrections on the held-out questions. ε\varepsilon: certified accuracy-loss bound; counted: coverage at least 20% and ε\varepsilon at most half of GG. Values at qjointq_{\mathrm{joint}}, the largest candidate that the lost-correction and change-rate tests both accept (Table 14); infeasible: too few calibration questions (448; 688 needed); –: no joint certificate. Lost corrections: skipped questions that the reference answers correctly and the receiver alone does not; gained answers: the reverse. Δ\Deltaacc: policy minus reference accuracy (points) with its paired-bootstrap 95% CI. GG: reference minus receiver (points); Kept: share of GG. (a) Saving: panel saving over all requests with its paired-bootstrap 95% CI, recomposed from the per-request timings of the existing replays with the same configurations (a newly routed question costs its recorded probe latency plus the reference’s); in row order, the deployed policies save 517.1, 717.6, and 346.3 ms (original configuration) and 343.4, 318.6, and 418.9 ms (second configuration). (b) Lost corrections over all held-out or sealed questions with a Clopper–Pearson 95% CI for their rate, and coverage, gained answers, and Δ\Deltaacc on the same questions; MMLU-Pro has no unused questions. ‡Nominal certificate.

The medium pair’s Text path. On OBQA, the medium pair’s Text path helps (G=4.45G = 4.45 points on development; 523 against 490 of 742 correct) but fails the change-rate test at every threshold (AUROC 0.787), so no joint certificate can include it. Run alone after the main analysis, with the same 20 frozen candidates, δ=0.001\delta= 0.001, and the whole calibration split (1,366 questions) as denominator, the lost-correction test accepts no candidate at ε=0.01\varepsilon= 0.01 or 0.020.02. Its first candidate (q=0.05q = 0.05) already skips 601 calibration questions, because ties at zero uncertainty merge the lowest quantiles, and loses 21 corrections where at most 3 and 12 would pass (p=0.978p = 0.978 and 0.128). Reaching this decision takes labels on 36 of the 348 calibration disagreements, and the system keeps communicating. A loss-only certificate would also give up the label-free bound on answer changes.

Table 14: The lost-correction test needs gold labels only on the calibration disagreements up to its first rejected candidate or the deployed threshold, whichever is lower (DneedD_{\mathrm{need}}), and at ε=0.01\varepsilon= 0.01 it is infeasible on the two ARC settings. Lost-correction test: exact binomial test of H0H_0: rate of lost corrections (skipped questions the reference gets right and RR wrong) ≥ε\ge\varepsilon, the certified accuracy-loss bound. Accepted: candidates (of 20) accepted by the lost-correction test, the change-rate test, and both. qLq_L: largest candidate the lost-correction test accepts; qjointq_{\mathrm{joint}}: largest candidate both accept (deployed by the joint certificate); NN: calibration questions; infeasible: NN below the 688 that ε=0.01\varepsilon= 0.01 needs. Disagreements: all (DallD_{\mathrm{all}}), up to the first rejected candidate (DseqD_{\mathrm{seq}}), at qjointq_{\mathrm{joint}} (DdepD_{\mathrm{dep}}), and up to that candidate or the deployed threshold, whichever is lower (DneedD_{\mathrm{need}}). Without reviewed: qjointq_{\mathrm{joint}} without the parser-review questions (1,158 OBQA and 315 ARC remain; infeasible: 315 is below the 342 that ε=0.02\varepsilon= 0.02 needs); parser-review questions: 341 calibration questions whose outputs were read with gold labels before the parser was frozen. –: not applicable. ‡Nominal certificate.

Settingε\varepsilonCalibration disagreements: all (DallD_{\mathrm{all}})Calibration disagreements: up to the first rejection (DseqD_{\mathrm{seq}})Calibration disagreements: at qjointq_{\mathrm{joint}} (DdepD_{\mathrm{dep}})Calibration disagreements: labels needed (DneedD_{\mathrm{need}})
large/OBQA/Text‡0.01143848
large/OBQA/Text‡0.02143181218
large/ARC/Text0.01120–0
large/ARC/Text0.0212101
large/MMLU-Pro/Text0.011,2951536767
large/MMLU-Pro/Text0.021,2953616767
large/OBQA/Text+fact‡0.01170737
large/OBQA/Text+fact‡0.02170221622
Llama-8B/OBQA/Text0.0121712712
Llama-8B/OBQA/Text0.02217322121
Llama-8B/ARC/Text0.01400–0
Llama-8B/ARC/Text0.0240101

Table 14b. (b) Calibration disagreements that need gold labels

Single models. Table 16 compares the deployed paths with each model answering alone on development questions. The large pair with Text is the most accurate system on both benchmarks: 0.7 (OBQA) and 4.0 points (ARC) above the large helper agent alone, and 3.8 and 2.0 points above the large receiver agent alone. Large-pair C2C falls below its receiver agent alone. The helper agent’s accuracy comes from the argmax over option-label probabilities under a fixed answer prefix, not from free generation and parsing, so that row is not scored under the same protocol as the others. The comparison holds the models fixed and does not equalize compute.

SystemOBQAARC
Qwen3-1.7B alone (medium receiver)490 (66.0)217 (72.6)
medium pair, Text523 (70.5)221 (73.9)
medium pair, C2C486 (65.5)202 (67.6)
Qwen3-8B alone (large receiver)617 (83.2)268 (89.6)
Qwen2.5-7B-Instruct alone (large helper)a^{a}640 (86.3)262 (87.6)
large pair, C2C598 (80.6)266 (89.0)
large pair, Text645 (86.9)274 (91.6)

Table 16. Development accuracy of the deployed paths and of each model alone. Correct answers out of 742 (OBQA) and 299 (ARC), with accuracy in %. a^{a} Option-label argmax under a fixed answer prefix; the other rows use free generation and our parser.

Headroom and the null control

Input changes without answer-specific content create about as much oracle headroom as the real paths, so headroom alone does not attribute complementarity to communication content. This post-

Settingε\varepsilonTest questionsLost corrections countLost corrections 95% CI (%)Coverage (%)Gained answersΔ\Deltaacc (points) 95% CI
large/OBQA/Text‡0.01held out (744)2[0.03, 0.97]65.21−0.13 [−0.54, +0.27]
large/OBQA/Text‡0.02held out (744)4[0.15, 1.37]75.02−0.27 [−0.94, +0.40]
large/ARC/Text0.02sealed (1,172)0[0.00, 0.31]79.00+0.00 [0.00, 0.00]
large/OBQA/Text+fact‡0.01held out (744)2[0.03, 0.97]54.30−0.27 [−0.67, 0.00]
large/OBQA/Text+fact‡0.02held out (744)8[0.47, 2.11]70.22−0.81 [−1.61, 0.00]
Llama-8B/OBQA/Text0.01held out (744)1[0.00, 0.75]44.00−0.13 [−0.40, 0.00]
Llama-8B/OBQA/Text0.02held out (744)3[0.08, 1.17]59.10−0.40 [−0.94, 0.00]
Llama-8B/ARC/Text0.02held out (1,172)2[0.02, 0.62]44.32+0.00 [−0.34, +0.34]

Table 42.

hoc audit of saved development outputs (9,369 action observations on OBQA and ARC, 7,923 on large-pair MMLU-Pro) ran no router. Oracle quantities use gold labels and are non-deployable upper bounds, with INVALID scored as incorrect. Figure 7 summarizes each path against the null.

Figure 7. Matched one path at a time, the content-free null never falls significantly below the medium- and large-pair C2C paths, whereas whether it falls significantly below Text depends on the match: in five of seven populations unstratified and in two of six stratified. Bar: the null’s gain over the receiver divided by the real path’s gain (mean of 1,000 draws), drawn from 1 on a log scale, so a bar below 1 means that the null gained less than the real path; whisker: 95% question-bootstrap interval; filled: the interval excludes 1; outlined: it includes 1. Top: unstratified; bottom: stratified. Large-pair ARC Text under the stratified match is a lower bound. Every value is in Tables 17 and Table 18.

Null control. Each receiver agent was rerun alone with its options rotated by one position (with the answer mapped back) and with the Text prompt carrying the saved helper message of another question from the same population (the question ⌊N/2⌋\lfloor N/2 \rfloor positions later in the frozen order). An unchanged rerun reproduced every receiver-only output, confirming these changes are not run-to-run noise. Option rotation changes 6.69 to 75.20% of the receiver’s answers, and another question’s message changes 4.01 to 29.77% (Table 17). Before matching, an oracle over the receiver and these two inputs achieves at least as much headroom as the oracle over RR, Text, and C2C in six of seven populations. Text answers more questions correctly than another question’s message in six of seven populations, by 1 to 30 (small-pair ARC is the exception, 126 against 130). Because an oracle gains more when more answers change, a rate-matched null action replaces oR(x)o_R(x) with a random differing perturbed answer on randomly drawn questions. It changes exactly as many answers as Text only (a), C2C only (b), and both jointly (c) (joint design; the per-reference and independent designs match one reference at a time). Ratios average 1,000 draws with 2,000-resample question bootstraps, all fixed before any of these numbers was computed. For medium ARC, the joint ratio is a lower bound due to under-matching (73 eligible questions for 77). Another question’s message alone cannot serve as a rate-matched null, because it changes fewer answers than Text across all populations. When Text and C2C change a question to different answers, the two joint nulls must still agree on it if only one alternative answer exists (on average 1.0 to 133.3 questions per draw). This lowers the null gain, so the joint ratio, if anything, understates what the null reproduces.

Gain and headroom. As gain over RR, the joint null reproduces a median 76.1% [67.6, 88.5] of the real gain (64.1 to 97.0%), and the independent design a median 73.8%. Matched one reference at a time, the null gains less than Text in all seven populations, but more than C2C in four. This is descriptive and does not split the real gain into a part due to content and a remainder, because real and null changes occur on different questions. Joint-null headroom over the best fixed action equals or exceeds the real one in six of seven populations (median ratio 1.149 [1.000, 1.316]; independent 1.111, Text only 1.235, C2C only 1.306). This is partly because Text outperforms RR by 4 to 61 questions, whereas each null action is on average below RR in six of seven, so RR remains the best fixed action of the null triple in 0.5 to 95% of draws. Matching separately on questions RR answers correctly and incorrectly gives a headroom ratio of 0.967 [0.800, 1.040] (0.905 over the four populations where both strata match fully) and a gain-over-RR ratio of 0.849 [0.733, 0.937].

PairTaskRealNullIndependentTextC2CJoint
smallOBQA2132030.686 [0.595, 0.782]0.719 [0.606, 0.845]0.686 [0.584, 0.804]0.664 [0.573, 0.759]
smallARC94900.738 [0.608, 0.889]0.851 [0.671, 1.080]0.747 [0.598, 0.953]0.737 [0.605, 0.893]
mediumOBQA1031060.805 [0.692, 0.937]0.800 [0.680, 0.945]1.125 [0.880, 1.464]0.903 [0.769, 1.048]
mediumARC†31280.772 [0.535, 1.060]0.742 [0.505, 1.057]1.705 [1.019, 3.449]0.84 …
largeOBQA53620.701 [0.578, 0.856]0.547 [0.443, 0.665]1.313 [0.882, 2.175]0.761 [0.613, 0.935]
largeARC1290.562 [0.334, 0.803]0.498 [0.263, 0.778]0.986 [0.490, 2.970]0.641 [0.369, 0.900]
largeMMLU-Pro2763170.884 [0.800, 0.981]0.861 [0.769, 0.969]1.246 [1.071, 1.447]0.970 [0.877, 1.079]
Median over populations0.738 [0.659, 0.825]0.742 [0.644, 0.840]1.125 [0.895, 1.349]0.761 [0.676, 0.885]

Table 17. Rate-matched nulls without answer-specific content reproduce a median 0.761 of the real gain over RR and a median 1.149 of the real headroom over the best fixed action (development sets). Ratios: matched null over real (mean of 1,000 draws) [question bootstrap 95% CI]. Best fixed action: the best single path (for the null, of RR and the two null actions). Joint: two nulls changing as many answers as Text only, C2C only, and both (all paths in Table 1); Independent: one null per reference, each matched to that reference alone. (a) Real, Null: oracle gain over RR in questions for {RR, Text, C2C} and for the unmatched null over RR and the two perturbed inputs (option rotation: each option’s text moved one position down, labels kept; other message: the Text prompt carrying another question’s helper message); Text, C2C: per-reference design, each reference against its own matched null (per path in Table 1). (b) Rotated, Other message: share of RR’s answers that rotating the options or another question’s message changes; Real, Null: headroom in questions (null: joint design, mean over draws); Stratified: joint design run separately on questions RR answers correctly and incorrectly. †Not fully matched: in (a) the joint ratio, a lower bound; in (b) the stratified ratios (for medium ARC, all three).

PairTaskRotatedOther messageRealNullHeadroom, unstratifiedHeadroom, stratifiedGain over RR, stratified
smallOBQA75.2028.71132138.11.046 [0.883, 1.211]0.787 [0.709, 0.873]0.729 [0.640, 0.826]
smallARC69.9029.774956.31.149 [0.922, 1.490]0.725 [0.592, 0.873]0.754 [0.626, 0.904]
mediumOBQA†24.6620.497090.91.298 [0.994, 1.587]0.996 [0.884, 1.124]0.938 [0.791, 1.083]
mediumARC†17.7312.042725.60.950 [0.630, 1.405]0.939 [0.626, 1.126]0.849 [0.583, 1.058]
largeOBQA18.469.162540.11.603 [1.071, 2.498]1.084 [0.946, 1.280]0.877 [0.755, 1.005]
largeARC†6.694.0167.61.267 [0.513, 3.764]0.967 [0.537, 1.500]0.750 [0.429, 1.084]
largeMMLU-Pro34.6120.26239260.81.091 [0.956, 1.217]1.023 [0.923, 1.108]0.956 [0.866, 1.058]
Median over populations1.149 [1.000, 1.316]0.967 [0.800, 1.040]0.849 [0.733, 0.937]

Table 17 (PDF p. 34). Headroom over the best fixed action.

Matching one path at a time, on right and wrong answers separately. After the main results, under rules fixed before computing (post hoc), we evaluated the per-reference design by matching the reference’s change rate separately on questions RR answers correctly (S+S^{+}) and incorrectly (S−S^{-}), then pooling the two draws. Because only changes on S−S^{-} affect the gain over RR, this stratified ratio equals the share of changed wrong answers that the null turns right (denoted g−g_{-}) divided by the reference’s share (Table 19). If S−S^{-} contains fewer eligible questions than the reference changes, the cell is undetermined and its ratio serves as a lower bound. Intervals follow the prior protocol

Table 19: Direction of the changes, development sets. For each path: its changes on questions RR answers incorrectly (to right, to another wrong answer, and the share that become right) and on questions RR answers correctly (to wrong). g−g_{-}: expected share of a null change on a wrong answer that becomes right. (a) The two references; (b) the two perturbed inputs (option rotation: each option’s text moved one position down, labels kept; other question’s message: the Text prompt carrying another question’s helper message).

(a) References

PairTaskNull share right (g−g_{-})Text RR wrong →\torightText RR wrong →\towrongText share rightText RR right →\towrongC2C RR wrong →\torightC2C RR wrong →\towrongC2C share rightC2C RR right →\towrong
smallOBQA0.4141361280.515751471370.51866
smallARC0.43655510.5193969670.50724
mediumOBQA0.63782430.6564944390.53048
mediumARC0.6342560.8062110130.43525
largeOBQA0.6904680.852181690.64035
largeARC0.9001010.9094420.6676
largeMMLU-Pro0.3431662660.3841291553970.281204

(b) Perturbed inputs

PairTaskNull share right (g−g_{-})Rotated options RR wrong →\torightRotated options RR wrong →\towrongRotated options share rightRotated options RR right →\towrongOther question’s message RR wrong →\torightOther question’s message RR wrong →\towrongOther question’s message share rightOther question’s message RR right →\towrong
smallOBQA0.4141642350.41115978880.47047
smallARC0.43665920.4145238330.53518
mediumOBQA0.63766430.6067466370.64149
mediumARC0.63419140.576201670.69613
largeOBQA0.69048230.6766631130.70524
largeARC0.900601.00014710.8754
largeMMLU-Pro0.3432324540.3382281512570.370127

Table 19.

(2,000 resamples, 100 draws each, seed 0). As detailed in Table 18, Text’s ratio is below 1 in all six determined populations, with the interval below 1 in two (small-pair and large-pair OBQA). The large-pair ARC setting is undetermined (0.900, or 0.990 after rescaling to the reference’s count). The five medium- and large-pair C2C ratios range from 1.077 to 1.475; none has an interval below 1, and the interval for large-pair MMLU-Pro exceeds 1. A prespecified Monte Carlo check (mean null gain deviating by more than three standard errors from min⁡(n−,elig−)g−\min(n_{-}, \mathrm{elig}_{-})g_{-}) flagged medium/ARC/C2C at 3.28 standard errors. A diagnostic across 400 seeds found no systematic bias (mean deviation −0.026 standard errors), and seed 0 was the most extreme of the 400. After observing all results, we lifted the stop and report the prespecified numbers unchanged. This analysis reveals the direction of every change (Table 19): relative to g−g_{-}, the share of changed wrong answers that become right differs by +0.9 to +17.2 points for Text, −23.3 to +10.4 for C2C, −5.8 to +10.0 for rotated options, and −2.5 to +9.9 for another question’s message across the seven populations.

Table 18: The per-reference null matched on right and wrong answers separately (stratified) next to the original per-reference null (unstratified), development sets. Per-reference null: a null matched to one reference alone (per path in Table 1). Changed by reference: answers the reference changes, on questions RR answers correctly (n+n_{+}) or incorrectly (n−n_{-}); eligible for null: questions with a perturbed alternative. Oracle gain over RR: real and null (null: mean of 1,000 draws). §Undetermined: fewer eligible wrong answers than changes; the ratio is a lower bound (0.990 rescaled to n−n_{-} changes).

PairTaskReferenceRR right (n+)(n_{+})RR wrong (n−)(n_{-})RR rightRR wrongRealNullStratifiedUnstratified
smallOBQAText75264179411136109.190.803 [0.687, 0.940]0.719 [0.606, 0.845]
smallARCText39106601715546.180.840 [0.680, 1.074]0.851 [0.671, 1.080]
mediumOBQAText491251031508279.460.969 [0.859, 1.115]0.800 [0.680, 0.945]
mediumARCText213132412519.670.787 [0.601, 1.004]0.742 [0.505, 1.057]
largeOBQAText185482844637.240.809 [0.702, 0.924]0.547 [0.443, 0.665]
largeARCText4111710109.000.900§ [0.538, 1.056]0.498 [0.263, 0.778]
largeMMLU-ProText129432288827166148.200.893 [0.806, 1.006]0.861 [0.769, 0.969]
Median over the 7 Text populations0.840 [0.769, 0.936]
smallOBQAC2C66284179411147117.720.801 [0.694, 0.923]0.686 [0.584, 0.804]
smallARCC2C24136601716959.080.856 [0.691, 1.078]0.747 [0.598, 0.953]
mediumOBQAC2C48831031504452.831.201 [0.984, 1.517]1.125 [0.880, 1.464]
mediumARCC2C252332411014.751.475 [0.964, 2.844]1.705 [1.019, 3.449]
largeOBQAC2C352582841617.241.077 [0.823, 1.558]1.313 [0.882, 2.175]
largeARCC2C66171045.411.351 [0.800, 3.151]0.986 [0.490, 2.970]
largeMMLU-ProC2C204552288827155189.481.222 [1.067, 1.406]1.246 [1.071, 1.447]
Median over the 7 C2C populations1.201 [0.947, 1.331]
Median over the 5 medium- and large-pair C2C populations1.222 [1.059, 1.448]

Table 18.

Overlap, confidence, and which path is right. Each path is uniquely correct on certain questions (Figure 8(a)). Text and C2C change the same questions 1.3 to 6.6 times as often as independent changes at their rates would (Figure 8(b)). Both references mostly change answers when the receiver is unsure: 0 to 40.4% of changed answers have an uncertainty uu at or below the fit-split median, against 45.5 to 72.6% of all questions. For the medium pair, 14.9 and 26.9% of Text’s changes are at or below it, compared to 5.3 and 2.1% of C2C’s changes (Figure 8(c)). Where exactly one of Text and C2C is right on the medium pair (155 OBQA questions, 96 with Text right; 53 ARC questions, 36 with Text right), ProbeMax separates the two with AUROC 0.569 [0.472, 0.667] (OBQA) and 0.608 [0.438, 0.753] (ARC), with bootstrap resamples stratified by class. Since both intervals include 0.5, this descriptive check establishes no signal for which path is right. Small- and medium-pair MMLU-Pro headroom is in Appendix § 13.

Figure 8. Each path is the only one right on some questions, the two references change largely the same questions, and both mostly change answers on which the receiver is unsure (development sets). (a) Questions that only the receiver alone, only Text, or only C2C answers correctly, as a share of the population. (b) Questions changed by both references, divided by the number that independent changes at the observed rates would give. (c) Share of the answers each reference changes whose receiver uncertainty uu is at or below the fit-split median; tick: share of all questions.

Extensions

The same pattern holds across all extensions: messages that rarely and predictably change the receiver’s answer are certified (the Llama-8B receiver, Text+fact nominally, SQuAD, and sampled answers, on OBQA nominally). A stronger helper for the medium receiver falls back because its calibration splits are too small, while the remaining settings fall back (the OLMo and Llama pairs, GSM8K). Each ran under a protocol fixed before its outputs (Appendix § 12). Two prospective tests of the label-light certificate, on a three-agent system and on a 32B pair, close this appendix (Appendix § 17.§ 6).

Cross-family receivers and helpers

Each cross-family pair ran the main study’s protocol, using prompts in each model’s native chat template, under its own frozen protocol, with every setting reported. The OLMo pair combines the large helper (whose saved messages serve as Text messages, matching token counts on every row) with an OLMo-2-1124-7B-Instruct receiver ([44]), for which no C2C fuser exists (a 60-test family). The Llama pair combines a Llama-3.2-1B-Instruct helper ([18]) with the small receiver on OBQA and ARC (an 80-test family). Its C2C outputs come from the official evaluator and cross-family fuser run on our rows, with receiver prompts byte-identical to ours across all 5,626 rows and 16 saved small-pair C2C outputs reproduced bit for bit. All seven settings fall back (Table 6). OLMo disagrees with Text on 23.2, 22.4, and 42.1% of OBQA, ARC, and MMLU-Pro development questions (Qwen3-8B: 9.7, 5.0, and 21.2%), at AUROC 0.790, 0.749, and 0.633 (0.873, 0.952, and 0.821), and answers 71.6, 68.2, and 22.4% correctly alone. The Llama helper raises the small receiver’s accuracy from 38.4 to 44.9 (Text) and 48.2% (C2C) on OBQA, and from 36.8 to 48.5 and 48.8% on ARC. Its Text messages change fewer answers than the Qwen helper’s (25.7 and 29.4% against 45.7 and 48.5%), but they are ranked weakly (AUROC 0.693 and 0.678), and the fuser changes 57.5 and 58.2%.

Part of the OLMo fallback stems from answer format, as unparseable outputs count as changes: 4.3, 9.2, and 17.1% of OLMo’s receiver-only and 5.9, 8.0, and 13.1% of its Text calibration answers are INVALID (on development 2.8, 12.0, and 17.0% and 3.5, 6.4, and 13.7%; Llama-pair receiver 3.4 and 10.7%, references 2.7 to 6.9%; Llama-8B pair at most 0.1%). Furthermore, 72 of 320, 47 of 95, and 968 of 2,457 calibration disagreements involve exactly one INVALID output. A non-deployable, post-hoc filter retaining only questions where both answers parse certifies all three OLMo settings (q=0.45q = 0.45, 0.40, and 0.05; development disagreement 18.8, 15.5, and 33.5%, AUROC 0.855, 0.881, and 0.701) but no Llama setting. No deployable rule we tested certifies OLMo or any Llama setting. Post hoc, answering skipped questions with the probe’s argmax agrees with OLMo’s own answer on 95.2, 90.2, and 81.1% of calibration questions but passes no threshold (smallest p=0.901p = 0.901, 0.377, and 1.000; AUROC 0.808, 0.765, and 0.863), because the reference’s unparseable answers (15.8, 29.2, and 31.2% of development disagreements) remain changes; routing unparseable receiver answers to the reference also fails (Appendix § 20).

Llama-8B pair. A Llama-3.1-8B-Instruct receiver reading the large helper’s saved messages was added after these fell back, as the first of three candidates in an order fixed before any output (followed by Mistral-7B-Instruct-v0.3 and Granite-3.1-8B-Instruct, which produced no outputs, making OLMo-2 the only other non-Qwen receiver run). It used its default chat template and a 40-test family. It disagrees with Text on 11.6 and 12.4% of OBQA and ARC development questions (AUROC 0.874 and 0.880) and is certified at q=0.60q = 0.60 (21 of 800 skipped calibration answers change, p=5.7×10−4p = 5.7 \times10^{-4}) and q=0.70q = 0.70 (4 of 320, p=3.2×10−4p = 3.2 \times10^{-4}). On development, 7 of 438 (1.60% [0.64, 3.27]) and 4 of 206 (1.94% [0.53, 4.90]) skipped answers change. Both deploy in all 200 re-drawn splits and keep their quantile on questions whose two answers parse and with argmax answers; no score is exactly zero. Text raises accuracy from 603 to 636 of 742 and from 239 to 260 of 299, with the policies answering 638 and 258 correctly.

Informed and stronger helpers

Text+fact. Post hoc, the large helper’s input gains one line before the question, Useful fact: <fact1>, providing the OpenBookQA science fact behind the question. Consequently, the receiver sees the fact only through the message. Receiver-only outputs, scores, and thresholds are reused in a separate 20-test family. The message changes 12.4% of calibration answers (170 of 1,366; plain Text 10.5%) and 10.2% of development answers (AUROC 0.889). The test deploys at q=0.75q = 0.75 (22 of 999, p=4.8×10−6p = 4.8 \times10^{-6}; q=0.80q = 0.80 fails, 33 of 1,064, p=0.0016p = 0.0016), also without the 208 reviewed questions (20 of 852). The certificate is nominal because ProbeMax was chosen on the plain Text labels of the same questions; held out, 16 of 558 skipped answers change (p=9.2×10−3p = 9.2 \times10^{-3}). On development, the policy skips 536 of 742 questions and changes 13. The message correctly predicts 62 development answers and breaks 9; the policy skips 11 of the corrections and 1 of the breaks, answering 660 of 742 correctly against 670 for Text+fact and 617 for RR (−1.35 points [−2.29, −0.54], within κα≈3.6\kappa\alpha\approx3.6). This is the only deployed policy whose development accuracy interval lies below zero.

A stronger helper. Under a protocol fixed before its outputs, the large helper’s stored messages were given to the medium receiver on OBQA and ARC (40 tests). The helper alone answers 86.3 and 87.6% of development questions correctly (receiver: 66.0 and 72.6%). Its messages raise the receiver’s accuracy by 7.3 and 4.3 points but change 21.2 and 13.4% of its development answers (calibration: 297 of 1,366 and 55 of 448; AUROC 0.792 and 0.800). Both settings fall back (smallest p=0.26p = 0.26 and 0.026); they certify in none and 15.5% of 200 re-drawn splits, and would need about 21,500 and 880 calibration questions to pass.

Extractive questions: SQuAD

Table 20: The label-free pilot makes three Llama-8B settings eligible and selects SQuAD with the Llama-8B receiver, whose predicted probability of certification is the highest (400 questions per setting). Helper: the large helper; TriviaQA and NQ-Open: closed-book. Disagreement: receiver-only versus Text. AUROC of the first-token, whole-answer, and per-token-mean scores (one minus the top probability at the first alphanumeric token, one minus the whole answer’s probability, and one minus its per-token geometric mean). Changes in lowest 25%: changes among the 100 lowest-score questions (ranked by the score with the highest pilot AUROC, the whole-answer score in all eight settings); Eligible: at most 4% of these change. P(certify): certification probability predicted by the frozen binormal model (Appendix § 14) at 1,000, 2,000, and 3,000 calibration questions. Contain: share of changes in which one normalized answer contains the other. No answer was INVALID.

DatasetReceiverDisagreement (%) [95% CI]First tokenWhole answerPer-token meanChanges in lowest 25%Eligible1,0002,0003,000Contain (%)
SQuADQwen3-8B24.5 [20.4, 29.0]0.7510.8050.7917 of 100no0.0220.0750.19268
SQuADLlama-8B14.3 [11.0, 18.1]0.7730.8360.7381 of 100yes0.5980.9751.00060
NQ (passage)Qwen3-8B28.3 [23.9, 32.9]0.7190.7450.74312 of 100no0.0000.0000.00251
NQ (passage)Llama-8B22.3 [18.3, 26.6]0.7420.8080.7543 of 100yes0.0390.1790.39344
TriviaQAQwen3-8B40.5 [35.7, 45.5]0.7460.7920.7849 of 100no0.0000.0000.00216
TriviaQALlama-8B29.0 [24.6, 33.7]0.8030.8380.8214 of 100yes0.0350.1620.41315
NQ-OpenQwen3-8B62.3 [57.3, 67.0]0.7070.7930.76030 of 100no0.0000.0000.00010
NQ-OpenLlama-8B48.3 [43.3, 53.3]0.7920.8560.8069 of 100no0.0000.0060.02417

Table 20.

Table 21: SQuAD certifies: the test accepts q=0.15q = 0.15 to 0.35 and deploys q=0.35q = 0.35. Threshold τ\tau on the whole-answer score; skipped (nn) and changed (kk) calibration questions; exact binomial pp for H0:ρ≥0.05H_0: \rho\ge0.05; CP: Clopper–Pearson 0.999 upper bound on ρ\rho (diagnostic). Accepted in bold.

qqτ\taunnkkppCPqqτ\taunnkkppCP
0.050.00018811120.0800.0970.550.3501,166580.5190.073
0.100.00065521230.00570.0600.600.3981,261720.8870.080
0.150.0016130731.2 × 10⁻⁴0.0420.650.4331,322760.9030.080
0.200.0049240943.8 × 10⁻⁶0.0360.700.4691,390850.9730.084
0.250.016151977.9 × 10⁻⁶0.0370.750.5111,5011070.99990.094
0.300.0421592116.1 × 10⁻⁵0.0430.800.5591,6071381.0000.110
0.350.0992721172.5 × 10⁻⁴0.0470.850.6151,6961581.0000.117
0.400.167847290.0170.0580.900.6861,8061961.0000.133
0.450.230952400.1450.0660.950.7921,9082361.0000.149
0.500.2981,063480.2600.0681∞2,0003021.0000.177

Table 21.

Design and selection. A pilot protocol fixed before any pilot output defined eight candidates (SQuAD and NQ with passages, TriviaQA and NQ-Open closed-book) ([41, 30, 14, 27, 31]), each with a Qwen3-8B and a Llama-8B receiver and the large helper, three scores, and a label-free selection rule. A full-run protocol fixed before any full-run output added a 1,000-question test split and narrowed the reporting rule, so that an unqualified claim for extractive questions required a passing test as well as a certificate (recorded deviations from the pilot protocol); the certificate was fixed before any development or test output existed. For SQuAD, the training pool is deduplicated and limited to passages of at most 300 words (86,830 questions), and all splits are disjoint blocks of a fixed pseudorandom order. The receiver answers directly with at most 32 new tokens, and answers are compared using official SQuAD normalization. The three scores are one minus the top probability at the first generated token containing a letter or digit (analogous to ProbeMax), one minus the whole answer’s probability, and one minus its per-token geometric mean; the whole-answer score had the best pilot AUROC in all eight settings. It needs the receiver-only answer, which is served when the message is skipped and wasted otherwise. A setting was eligible if at most 4% of its 25% lowest-score pilot questions changed; eligible settings were ranked by the certification probability PP that the frozen binormal model (Appendix § 14) predicts from the pilot’s disagreement rate and AUROC, and at most two with P≥0.5P \ge0.5 would run, each at the smallest calibration size with P≥0.8P \ge0.8. The four Qwen3-8B settings and NQ-Open with the Llama-8B receiver are ineligible (7 to 30 of 100 changed), and the rule selected SQuAD with the Llama-8B receiver at 2,000 calibration questions (P=0.975P = 0.975, inflated by taking the best of eight; conservative 0.081) and no second setting (next: 0.413 and 0.393; Table 20).

Results. Before production, the runtime reproduced 16 pilot questions exactly. On calibration, the receiver disagrees with Text on 302 of 2,000 questions (15.1%), and the test deploys q=0.35q = 0.35 (17 of 721, p=2.5×10−4p = 2.5 \times10^{-4}; Table 21). All 200 re-drawn splits certify (median q=0.375q = 0.375, median development coverage 41.5%). On development (disagreement 15.4% [13.2, 17.8], AUROC 0.811 [0.771, 0.848]), the policy skips 39.6% of questions, and 16 of 396 skipped answers change (4.04% [2.3, 6.5]). In 15 of these, one normalized answer contains the other; counting containment as agreement leaves 1. On the sealed test, 9 of 364 skipped answers change (2.47% [1.14, 4.64], p=0.012p = 0.012); the pass rule is a two-sided 95% upper bound below 5%, so the test passes, though pp exceeds 0.001. On development and test, RR answers 659 and 664 of 1,000 questions correctly, Text 669 and 682, and the policy 673 and 686. Among the skipped changes, Text would have corrected 4 and 2 answers and broken 8 and 6. Text’s development gain, 1.0 point [−0.9, +2.9], is below κα=2.0\kappa\alpha= 2.0, and its interval includes zero, so the protocol’s redundancy rule holds. From recorded per-question times (sequential execution, on a different machine than the § 4.4 replays), skipping saves an estimated 198.5 ms [167.2, 229.9] per question, while always skipping saves 785.4 ms (helper 772.5 ms; receiver 153.5 ms alone and 166.4 ms with the message, since the receiver-only answer is always computed). This is an estimate, not a replay.

A numeric-answer check: GSM8K

A prespecified check (large pair, Text) used 400 fit, 1,000 calibration, and 400 development training questions in a fixed content-based order (a recorded deviation from the main protocol’s seeded shuffle) and raised the receiver’s limit from 64 to 320 tokens (a deviation fixed in advance), because the receiver derives before answering. A change is a mismatch of the normalized final number (after the last “answer is”, otherwise the last number; the extractor was tested on twelve hand-written strings and, as prespecified, never changed). The score is one minus the length-normalized probability of the receiver’s answer tokens under teacher forcing, which reduces to ProbeMax for a single label token. No candidate passes (every p=1.000p = 1.000), although all 3,200 outputs parse and the receiver disagrees with Text on 17.4% of calibration and 19.3% of development questions (accuracy 87.75% alone, 86.50% with Text). The score fails: 378 of 400 fit questions score exactly zero after the derivation, so 18 of the 20 thresholds coincide. Its development AUROC is 0.5795 (0.6444 for the well-spread whole-continuation probability, 0.5680 for the first answer token). The last-number rule fired on 15.2% of receiver-only outputs, tracking the 320-token cap (reached by 43 fit, 106 calibration, and 49 development outputs); the Text path is barely affected (6 of 1,000 and 2 of 400). A later protocol, written after this fallback, replaced a score over the first 32 generated tokens but stopped at its reproduction check (differences up to 0.04 and 0.06 on two capped outputs, against a tolerance of 10−410^{-4}), so no prefix score was tested (Table 5).

Sampled receiver answers

Post hoc, under a protocol fixed before its outputs, the large pair’s OBQA and ARC Text settings were rerun with receiver answers sampled on both paths (the Qwen3 model card’s settings with thinking disabled: temperature 0.7, top-pp 0.8, top-kk 20, min-pp 0; one sample per question and path, seeded independently), with messages, scores, thresholds, and the test unchanged in a 40-test family (OBQA stays nominal). Both settings deploy at the greedy thresholds (q=0.80q = 0.80: 19 of 1,064, p=3.4×10−8p = 3.4 \times10^{-8}; q=0.95q = 0.95: 4 of 422, p=4.8×10−6p = 4.8 \times10^{-6}) and in all 200 re-drawn splits, and 3.32% [2.01, 5.14] and 2.82% [1.22, 5.47] of skipped development answers change. Sampled disagreement is 9.4 and 4.7% (AUROC 0.867 and 0.956; greedy: 9.7 and 5.0%), most skipped questions still have zero uncertainty (354 of 572 and 217 of 284), and accuracy is 617 and 646 of 742 (receiver alone, Text) and 268 and 275 of 299. Sampling changes few of this receiver’s labels (6 of 742 receiver-only and 2 Text answers on OBQA, 0 and 1 of 299 on ARC), so this check says nothing about receivers whose samples vary more or that generate reasoning tokens.

Prospective tests: a three-agent system and a 32B pair

The label-light certificate of Appendix § 15 was analyzed post hoc. Two tests apply it prospectively: one on a three-agent system and a dataset used nowhere else in this work, and one on the Text path with both agents scaled to 32B parameters (OBQA is used throughout this paper, so the second test extends model scale, not data). For each test, the protocol (models, prompts, score, candidate thresholds, tests, label rule, policies, and pass rules) was committed before any calibration, development, or unseen-question output of its models existed (for the three-agent system, committed and time-stamped), and the unseen questions were evaluated once. Both primary policies meet every pass rule (Table 3).

Three-agent system. Agents A and B are Qwen2.5-7B-Instruct, and the receiver C is Qwen3-8B with thinking disabled. A (background) reads the question and choices under the helper instruction of Appendix § 22 and writes one background message (at most 256 new tokens). B (verifier) reads the question and choices, the two QASC facts, and A’s message, and writes a checked and extended background (256). C answers (64) from A’s turn on the cheap path, or from A’s and B’s turns on the reference path. The reference bb runs all three agents; the cheap path, which takes the place of RR, omits B. One decision is made per question: after A’s message, the probe on C’s cheap-path input decides whether C waits for B. C sees only the agents’ turns, in the receiver template of Appendix § 22: A’s turn (helper instruction, then A’s message) and, on the reference path, B’s turn, whose user content is B’s instruction without the two facts, A’s note, and the clauses that introduce them, followed by B’s message. As for the Text+fact receiver (Appendix § 17.§ 2), the facts reach C only through B’s message. B’s user message is the first line below, the question block of Appendix § 22, the two facts, A’s note, and the closing rule; in C’s view of B’s turn, the two bracketed clauses, the facts, and the note are absent.

In at most three clear sentences, state the background knowledge needed to answer the question below.
[ Another assistant has already written a background note for this question: check that note, correct
anything in it that is wrong, and add the knowledge it is missing. ] [ Use the two reference facts given
below as your source.]

{question}

Reference facts:
1. {fact1}
2. {fact2}

Background note from another assistant:
{A’s message}

Do NOT directly solve or give answer to the question: do not say which choice is correct, do not discuss
or rule out any of the choices, and do NOT mention any option letter.

Each question’s A and B messages are generated once and shared by both paths and the probe. All models run in bfloat16 with SDPA attention, greedy decoding, and batch size one; the parser (version 2) is extended to labels A to H. QASC’s training and development splits contain 9,060 eight-option questions, each with two supporting facts; the official test split has neither answers nor facts and is not used. Deduplicating by question id and then by normalized question and choices leaves 9,051 questions. A fixed-seed shuffle assigns 2,000 to fit, 3,000 to calibration, 1,000 to development, and 2,000 to the sealed test; the remaining 1,051 were never processed by any model. QASC is public, so the split is independent of our design process, not of the models’ pretraining data.

32B pair. The helper agent is Qwen2.5-32B-Instruct and the receiver agent is Qwen3-32B with thinking disabled. The reference is the Text path and the cheap path is the receiver agent alone, with the prompts, decoding, and parser of the main experiments (Appendices § 11 and § 22). Both models run in bfloat16 in Hugging Face Transformers with greedy decoding and batch size one, and across two A100-SXM4-40GB GPUs of one node, and the same backend is used from fit to evaluation. The splits are those of the main OBQA experiments: 2,100 fit, 1,366 calibration, 742 development, and 744 held-out questions. A pilot on the first 300 fit questions was to decide only whether gain retention would be emphasized (at least 6 more correct answers with Text): Text answered 276 and the receiver agent alone 271. The full protocol was committed to run in either case before the pilot result.

Score. Each label’s probability sums the next-token probabilities over its token set and is renormalized over the labels (A to H on QASC, A to D on OBQA). The score is the log of the non-top mass, u=log⁡∑l≠l∗plu = \log\sum_{l \ne l^*} p_l, computed as a log-sum-exp in FP32; it orders questions as 1−max⁡lpl1 - \max_l p_l does wherever that is nonzero. On the QASC fit split, 1−max⁡lpl1 - \max_l p_l is exactly zero in FP32 for 1,143 of 2,000 questions (57.15%), which would merge the first 11 candidate thresholds, whereas uu takes 1,928 distinct values. This is the tie-breaking score of Appendix § 20, chosen from the fit-split score distribution before any calibration output existed. A non-finite score routes to the reference (none occurred in either test).

Tests, labels, and pass rules. The 20 candidates and the change-rate test are those of Appendix § 11, applied to uu (a candidate that skips no question fails), and the lost-correction test is that of Appendix § 15, with all NN calibration questions as its denominator (notation and pp-values: Table 22). Gold labels are queried in increasing τ\tau, only for newly skipped disagreements; a candidate is decided after all its disagreements are labeled; for each ϵ\epsilon the scan stops at the first candidate that fails the lost-correction test; ϵ=0.02\epsilon= 0.02 runs first, and no label is queried beyond the largest candidate that passes the change-rate test. The label-free policy is the largest candidate passing the change-rate test, and the joint policies are the largest passing both tests at ϵ=0.01\epsilon= 0.01 (primary) and at ϵ=0.02\epsilon= 0.02. With 20 change-rate and 40 lost-correction tests at level 0.001, all calibration conclusions of a test hold jointly with probability at least 0.94 (Bonferroni). The pass rules on the unseen questions use the upper end of the two-sided 95% Clopper–Pearson interval, as in the main text: R1, changes among skipped questions below 5%; R2, lost corrections over all unseen questions below ϵ\epsilon (on 744 questions, at most 2 lost corrections at ϵ=0.01\epsilon= 0.01 and 7 at ϵ=0.02\epsilon= 0.02); the lower ends of the paired-bootstrap 95% intervals (2,000 resamples) of the reference’s gain GG and the policy’s gain GπG_{\pi}, both over the cheap path, above zero; and a saving on the development timing panel whose lower end is above zero. “Keeps at least half of the gain” requires the lower end of Gπ−G/2G_{\pi} - G/2 to exceed zero.

Table 22: Both prospective calibrations certify a joint threshold at each ϵ\epsilon with gold labels on 40 of 3,000 and 21 of 1,366 questions. Candidate jj skips the fit-split quantile q=j/20q = j/20. nn: skipped; mm: answer changes among them; pchange=Pr⁡[Bin⁡(n,0.05)≤m]p_{\mathrm{change}} = \Pr[\operatorname{Bin}(n,0.05) \le m]; ll: lost corrections; pϵ=Pr⁡[Bin⁡(N,ϵ)≤l]p_{\epsilon} = \Pr[\operatorname{Bin}(N,\epsilon) \le l]. A test passes at p≤0.001p \le0.001. A candidate that skips no question fails the change-rate test, so not labeled. Bold: the joint policy at ϵ=0.01\epsilon= 0.01, the joint policy at ϵ=0.02\epsilon= 0.02, and the label-free policy, at j=8j = 8, 10, and 11 in (a) and j=14j = 14, 16, and 17 in (b).

(a) Three agents, QASC (N=3,000N = 3,000)

jjnnmmpchangep_{\mathrm{change}}llp0.01p_{0.01}p0.02p_{0.02}
1116.09×10−36.09 \times10^{-3}12.52×10−122.52 \times10^{-12}2.97×10−252.97 \times10^{-25}
229314.88×10−64.88 \times10^{-6}12.52×10−122.52 \times10^{-12}2.97×10−252.97 \times10^{-25}
346411.17×10−91.17 \times10^{-9}12.52×10−122.52 \times10^{-12}2.97×10−252.97 \times10^{-25}
460011.40×10−121.40 \times10^{-12}12.52×10−122.52 \times10^{-12}2.97×10−252.97 \times10^{-25}
574422.15×10−172.15 \times10^{-17}23.94×10−113.94 \times10^{-11}9.23×10−249.23 \times10^{-24}
688422.29×10−172.29 \times10^{-17}23.94×10−113.94 \times10^{-11}9.23×10−249.23 \times10^{-24}
71,05351.65×10−171.65 \times10^{-17}52.03×10−82.03 \times10^{-8}3.70×10−183.70 \times10^{-18}
81,17176.09 \times 10^{-18}74.78 \times 10^{-7}3.41 \times 10^{-18}
91,330187.17×10−137.17 \times10^{-13}180.01261.45×10−101.45 \times10^{-10}
101,446241.70×10−111.70 \times10^{-11}230.1133.38×10−83.38 \times10^{-8}
111,612402.45×10−72.45 \times10^{-7}390.9550.00235
121,783620.0012———
131,964960.436———
142,1311370.998———
15–202,257–3,000169–488≈1\approx1———

Table 22.

(b) 32B pair, OBQA (N=1,366N = 1,366)

jjnnmmpchangep_{\mathrm{change}}llp0.01p_{0.01}p0.02p_{0.02}
17200.024901.09×10−101.09 \times10^{-10}1.03×10−121.03 \times10^{-12}
213908.01×10−48.01 \times10^{-4}01.09×10−61.09 \times10^{-6}1.03×10−121.03 \times10^{-12}
3–11187–7220≤6.83×10−5\le6.83 \times10^{-5}01.09×10−61.09 \times10^{-6}1.03×10−121.03 \times10^{-12}
1278911.13×10−161.13 \times10^{-16}11.61×10−51.61 \times10^{-5}2.99×10−112.99 \times10^{-11}
1386625.54×10−155.54 \times10^{-15}22.20×10−52.20 \times10^{-5}4.32×10−94.32 \times10^{-9}
1492456.51 \times 10^{-15}35.96 \times 10^{-4}4.16 \times 10^{-9}
1599671.58×10−141.58 \times10^{-14}50.006751.74×10−71.74 \times10^{-7}
161,079122.57 \times 10^{-12}80.07221.27×10−51.27 \times10^{-5}
171,144211.93 \times 10^{-8}130.5010.00173
181,229390.00117———
191,298530.070———
201,366850.981———

Table 22.

Calibration. Table 22 lists every candidate. For the three-agent system, the label-free policy is j=11j = 11 (calibration coverage 53.7%), the primary policy is j=8j = 8 (39.0%), and the ϵ=0.02\epsilon= 0.02 policy is j=10j = 10 (48.2%). The label rule queried 40 of the 3,000 calibration labels, all disagreements at j≤11j \le11: 39 were lost corrections and one was wrong on both paths. Before calibration, the label-free plug-in rule of Appendix § 14, applied to fit counts, predicted the label-free policy at j=12j = 12; j=12j = 12 narrowly fails (p=0.0012p = 0.0012). For the 32B pair, the label-free policy is j=17j = 17 (83.7%), the primary policy is j=14j = 14 (67.6%), and the ε=0.02\varepsilon= 0.02 policy is j=16j = 16 (79.0%). The label rule queried 21 of the 1,366 calibration labels, all disagreements at j≤17j \le17: 13 were lost corrections, 7 were questions the receiver agent alone answers correctly and Text does not, and one was wrong on both paths. On fit, Text answers 92.95% and the receiver agent alone 90.29%.

Unseen questions. Table 3 gives every policy and both reference gains. On the 2,000 sealed QASC questions, all three policies pass R1, both joint policies pass R2 at their ε\varepsilon, and all three keep more than half of the gain: Gπ−G/2G_{\pi} - G/2 is 6.15 [5.40, 6.95], 5.70 [4.97, 6.50], and 5.15 [4.40, 5.93] points for the primary, ε=0.02\varepsilon= 0.02, and label-free policies. B changes 16.4% of C’s answers (328 of 2,000), yet only 5 of these changes fall among the primary policy’s 798 skipped questions. On the 744 held-out OBQA questions, Text changes the 32B receiver agent’s answer on 45 questions (6.0%). All three policies pass R1, both joint policies pass R2 at their ε\varepsilon, and all three keep a gain whose interval excludes zero. The primary policy’s two changed answers are one lost correction and one question that only the receiver agent alone answers correctly, so its accuracy equals that of the reference. The joint policies keep at least half of the gain (Gπ−G/2G_{\pi} - G/2 of 1.28 [0.40, 2.15] and 1.01 [0.13, 1.88]); for the label-free policy the lower end is zero (0.87 [0.00, 1.75]), so we do not make that statement, and it does not pass R2 at ε=0.01\varepsilon= 0.01.

Development questions and single models (32B pair). The development questions decide nothing and are reported descriptively. The primary, ε=0.02\varepsilon= 0.02, and label-free policies skip 69.3, 80.6, and 85.4% of the 742 questions, change 1 of 514, 5 of 598, and 10 of 634 skipped answers (upper bounds 1.08, 1.94, and 2.88%), and lose 1, 2, and 6 corrections (upper bounds 0.75, 0.97, and 1.75%). Text’s gain is smaller there, G=1.08G = 1.08 points [−0.54, 2.70] (91.91% for the receiver agent alone against 92.99%), and neither it nor the policies’ gains exclude zero: 0.94 [−0.67, 2.43], 1.21 [−0.27, 2.70], and 0.81 [−0.54, 2.16] points, at accuracies of 92.86, 93.13, and 92.72%. The helper agent answering alone, with the same free-generation protocol as the other paths, reaches 91.26% on the held-out questions and 91.91% on development; the pair with Text is 1.61 points [−0.27, 3.63] above it on the held-out questions.

Latency. Latency is measured live at batch size one on 128-question development panels, on exclusively allocated nodes, and every policy saves time (Table 23). For the three-agent system the panel is the first 128 development questions in a fixed pseudorandom order. A and B generate on one GPU and C and the probe run on another; the policy and the reference alternate per request in counterbalanced order; there is no warm-up, cold requests are kept, and each policy ran once. The replay reproduced the offline records (A’s messages and both answers on all 256 requests, B’s messages on all 205 requests that ran B, and uu on all 128 questions). Skipped queries avoid B (about 1.3 s each), whereas queries that wait for B pay the probe (about 40 ms), so 56% of the primary policy’s panel queries run slightly slower and its paired median saving is −22 ms. For the 32B pair the panel is the OBQA development panel of the main experiments. The policy, the Text reference, and the receiver agent alone run in rotating order, in four segments of 32 questions, the last three after untimed warm-up requests on questions outside the panel. The replayed policy answers equal the offline answers on every request, and dropping the first question of each segment changes no conclusion (primary policy: 1,377 ms [1,213, 1,538] on 124 questions). The replay records whole-request latency, not component times, so the table splits the paired difference by route: a skipped question saves about 1.86 s, and a question that waits pays about 90 ms for the probe. The receiver agent alone takes 486 ms.

(a) Three agents

PolicySkipped on panel (%)Mean latency PolicyMean latency ReferenceMean saving [95% CI]Relative saving (%) [95% CI]Components PolicyComponents Reference
joint, ε=0.01\varepsilon= 0.0139.81,5732,056483 [365, 604]23.5 [17.6, 29.5]538 / 39 / 797 / 199534 / – / 1,283 / 239
joint, ε=0.02\varepsilon= 0.0250.81,4002,017618 [501, 732]30.6 [24.7, 36.6]526 / 39 / 644 / 190524 / – / 1,255 / 238
label-free56.21,3552,051697 [581, 818]34.0 [28.1, 40.0]536 / 40 / 588 / 190532 / – / 1,278 / 241

(b) 32B pair

PolicySkipped on panel (%)Mean latency PolicyMean latency ReferenceMean saving [95% CI]Relative saving (%)Route split Saved if skippedRoute split Extra if waiting
joint, ε=0.01\varepsilon= 0.0175.01,0642,4411,377 [1,219, 1,533]56.41,86693
joint, ε=0.02\varepsilon= 0.0282.89172,4421,525 [1,377, 1,667]62.51,86088
label-free87.58172,4351,618 [1,485, 1,750]66.41,86291

Table 23. Every prospective policy lowers mean latency on its 128-question development panel, by 23.5 to 34.0% for the three-agent system and by 56.4 to 66.4% for the 32B pair (ms; savings: reference minus policy, paired-bootstrap 95% CIs). (a) Components: means over all 128 requests of each arm, with steps not run counted as zero, as TA/Tprobe/TB/TCT_A / T_{\mathrm{probe}} / T_B / T_C. (b) Route split: mean paired saving on skipped questions, and mean extra latency on questions that wait.

Controls fixed in the three-agent protocol. On development (descriptive; Table 24), G=14.4G = 14.4 points [12.2, 16.7], and the label-free, primary, and ε=0.02\varepsilon= 0.02 policies skip 54.9, 39.1, and 48.4% of questions. C reading the two facts directly is more accurate than the pipeline (+15.7 points [13.3, 18.2]) and faster, so when the facts can be handed to C, the pipeline can be simplified. The experiment tests skipping within a fixed information flow, not the choice of architecture. B without A is also more accurate than A+B+C, and A with a fact-free B matches A+C (63.6 vs. 63.1%). On fit, C alone answers 60.1% and C reading the facts 92.7%. A’s message names an option in 9.55,

PathAccuracy (%) [95% CI]Mean generation time (ms)
C alone61.7 [58.7, 64.7]197
A+C (cheap path)63.1 [60.2, 66.2]715
A + B without facts + C63.6 [60.7, 66.6]2,188
A+B+C (reference)77.5 [74.9, 80.1]2,092
B+C (no A)85.2 [83.0, 87.3]1,205
C reading the two facts93.2 [91.7, 94.8]195

Table 24. The verifier’s gain comes from the two facts: a fact-free verifier matches the cheap path, and the receiver reading the facts directly is the most accurate and fastest path (1,000 development questions; descriptive). Accuracy with question-bootstrap 95% CIs; generation time: mean per question on one A100-SXM4-40GB in the offline runs.

11.1, and 9.85% of fit, development, and sealed questions, and B’s message contains the correct option’s text in 72.0, 72.4, and 72.2%.

Deviations. Every difference between a protocol and its execution was recorded as it occurred; the full records are released with the dataset. Table 25 lists all thirteen for the three-agent system. For the 32B pair, five entries change no model input, score, test, policy, or pass rule. Before any formal output, two option strings that an anonymization step had altered in our copy of the inputs were restored to the original OBQA text. The label log-probabilities behind the score and the component times of the replay were not saved; the score code, which was fixed before the run, and the route split above stand in for them. The other three concern job scheduling: one-hour shards, each gated by an exact match on two regenerated pilot questions, shard sizes and a submission rule reset from measured times after the pilot, and one resubmission after a refusal for lack of disk space.

#DeviationWhen
1Leak shares quoted in the protocol were measured on 96 prompt-tuning questions; full-split values are reported (superseded by 9).before all outputs
2Generation ran on single GPUs (A, B, and C on one A100); the timing layout is unchanged.before all outputs
3Scheduling fix (GPU lock, chained submission) after two jobs collided before writing any record.before all outputs
4Fit-split controls (C alone, C reading the facts) added; descriptive.before all outputs
5The freeze record names its parent commit; the freeze commit is identified by a tag.before all outputs
6Remaining shards ran on a second cluster with the same GPU model and environment, behind an exact-match gate on four fit questions, with a per-shard lock, scheduling-only extensions applied.before all outputs; extended after Cal and after Dev
7The calibration analysis need not wait for the fit controls of entry 4, whose accuracies may be committed after calibration outputs are read; descriptive.before all outputs
8Calibration tests ran before the development outputs and fit controls were complete.after Cal and Dev, before Test
9Correction of the protocol’s leak shares (pre-freeze subsample); full-split values are used.after Cal and Dev, before Test
10Clarification: C’s reference history contains B’s instruction without the facts and A’s note, as in Text+fact; the code is unchanged since the freeze.during test generation, before Test
11Correction: the protocol’s 60.7% zero share and 12 merged candidates came from the first 112 fit questions; the full fit split gives 57.15% and 11.after Test, before timing
12The timing panel ran on the second cluster (same GPU model, exclusive node), one run per policy enforced by a lock; failure handling fixed before timing.before timing
13Clarification: the 1,051 deduplicated questions outside the four splits were never processed by any model.after all outputs

Table 25. Seven of the thirteen deviations from the three-agent protocol precede every output, and none changes a model input, the score, a primary endpoint, or a sample rule. When: timing relative to the first reading of calibration (Cal), development (Dev), sealed-test (Test), and timing outputs.

Latency

Every replayed OBQA and ARC policy saves time, all of it on skipped queries, and every saving class holds in three repeats on a second hardware configuration (Figure 9). Replays reuse the frozen thresholds and 128-question panels without recalibration, and all replayed outputs matched their development or same-replay fixed-reference counterparts (including every route and output of the medium-pair C2C replays), so only latency is re-measured (Table 26).

PopulationReferenceSkippedPolicyReferenceSaving (ms) [95% CI]PolicyReference∆acc (points) [95% CI]
(a) Development end-to-end replay on 128-question panels, OBQA and ARC
medium/OBQAC2C66/128335.6415.279.7 [48.1, 129.1]9493+0.78 [0.00, +2.34]
medium/ARCC2C77/128352.4453.1100.7 [74.5, 133.1]8585+0.00 [0.00, 0.00]
large/OBQAText97/128505.11022.2517.1 [447.8, 583.6]112113−0.78 [−3.91, +1.56]
large/OBQAC2C97/128325.9362.436.6 [18.0, 56.8]102101+0.78 [0.00, +2.34]
large/ARCText120/128372.71090.3717.6 [670.6, 763.2]117117+0.00 [−2.34, +2.34]
large/ARCC2C115/128329.9440.8110.9 [82.0, 142.8]115114+0.78 [−1.56, +3.91]
(b) Development replay, MMLU-Pro (same threshold and route set for both references)
large/MMLU-ProText48/128984.3130.7346.3 [249.9, 443.0]6563+1.56 [0.00, +3.91]
large/MMLU-ProC2C48/128332.6340.88.2 [−20.5, +44.0]5555+0.00 [0.00, 0.00]
(c) Sealed ARC-Challenge test
large/ARCText1100/1172362.01047.7685.6 [669.6, 700.8]10801085−0.43 [−1.19, +0.26]
large/ARCC2C1047/1172323.6414.590.9 [81.0, 101.0]10611057+0.34 [−0.17, +0.85]

Table 26. Every deployed Qwen-pair policy saves time on its OBQA or ARC panel and on the sealed ARC test; on MMLU-Pro only the Text saving is established. (a), (b): development replays on 128-question panels; (c): sealed test (1,172 questions); all on the first configuration. Saving: paired mean of fixed reference minus policy, every probe and cold request included [paired bootstrap 95% CI]; not comparable across pairs or benchmarks. Cold request: a path’s first request after model loading. ∆acc: policy minus reference [paired bootstrap 95% CI].

Figure 9. Every saving class holds across configurations, repeats, and probe-cache reuse (under argmax answering, large/MMLU-Pro/C2C’s interval moves above zero), and the component estimates fixed before the replays are close to the complete replays. Mean saving per query of each replayed policy, Text-reference policies left and C2C-reference policies right (separate scales), with paired bootstrap 95% intervals as whiskers: solid bar, the original run (first configuration; Table 26), its mean printed above; markers on it, the means of the three repeats on the second configuration (Table 28); outlined bar, the single later run of Text+fact or a Llama-8B policy on the second configuration (Table 39); hatched bar, answering skipped questions from the probe’s argmax (second configuration; Table 29); black tick, the component estimate (large-pair OBQA and ARC; from timed probe stages and historical action requests) or CPU recombination (medium pair; from development timings) fixed before the replays. Saving class: interval above, below, or including zero. Savings from the two configurations are not comparable in absolute terms; only large/MMLU-Pro/C2C has an interval containing zero, in every original and repeat run.

Per-request distribution. The policy’s 99th-percentile latency is never above the reference’s in any replay; its 90th percentile is above it only for medium/OBQA/C2C and, on the second configuration, large/MMLU-Pro/Text. Routed queries are slower in 84.6 to 100% of cases (median −25 to −45 ms, the probe cost), meaning the entire saving stems from skipped queries (also under argmax answering), and the paired median is negative in 16 of 58 arm-streams. The probe costs 32.88 and 43.37 ms on the small- and large-pair OBQA panels, of which the prefill with its last-position projection takes 31.25 and 41.79 ms (95 to 96%), and 48.4 and 46.1 ms on the MMLU-Pro replay, whose paired median savings are −41.3 and −43.6 ms (median latencies 1170.6 vs. 1273.1 ms for Text, and 293.6 vs. 250.9 ms for C2C). The probe, vendor runner, and official fuser are single-row, so no batched comparison is reported.

Component estimates and complete replays. Before any replay, each policy’s cost was recomposed from timed components (the probe’s stages plus the selected historical action request), without cache or prefill discounts. For the four large-pair policies, the complete replay saved 1.76 ms less than this estimate for OBQA/Text and 8.40, 20.50, and 9.74 ms more for OBQA/C2C, ARC/Text, and ARC/C2C (Figure 9). A CPU recombination of development timings had projected 68.3 105.4 ms for the medium pair, whose replays saved 79.7 and 100.7 ms (median latencies 334.6 vs. 388.2 ms on OBQA, and 287.8 vs. 412.4 ms on ARC; paired median savings 79.3 and 81.6 ms).

Busy GPU time. Under rules fixed before computing, busy GPU time (summed wall time of GPU-side stages on both GPUs) can be computed only for replays timing each stage separately; thus, it is available for four of the eight policies (large-pair OBQA and ARC replays), where the busy-time saving accounts for 88 to 99% of the latency saving (Table 27).

PairTaskReferenceBusy GPU timeLatencyBusy minus latencyBusy / latency
largeOBQAText513.5 [448.7, 574.2]517.1 [447.8, 583.6]−3.6 [−9.9, +8.3]0.993
largeOBQAC2C32.3 [14.4, 51.8]36.6 [18.0, 56.8]−4.3 [−6.3, −3.2]0.882
largeARCText705.2 [559.1, 750.0]717.6 [670.6, 763.2]−12.4 [−13.2, −11.6]0.983
largeARCC2C106.6 [77.8, 138.2]110.9 [82.0, 142.8]−4.3 [−4.6, −3.9]0.961

Table 27. Busy GPU time saved is 88 to 99% of the latency saved on the four large-pair panels. Original runs (first configuration). Busy GPU time: summed wall time of GPU-side stages on both GPUs. Intervals: paired bootstrap 95% CI.

Second configuration and port check. Repeats and variants, added after main results under protocols fixed before outcomes, ran on exclusively allocated nodes with four A100-SXM4-40GB GPUs connected via NVLink (two used with the original placement, while original runs used two of eight A100-SXM4-40GB GPUs on a shared node). The repeats followed bit-for-bit reproduction of saved outputs: the receiver-only outputs of all seven development populations, large-pair ARC receiver-only and C2C outputs, and learned predictor features and predictions. In a post-hoc port check, our runtime and the official C2C evaluator [15], using byte-identical prompts, produce identical raw outputs on all 500 official OBQA test questions for every pair and action. Small-pair C2C answers 263/500 correctly, matching published results, whereas medium and large C2C answer fewer correctly than RR under our parser (317 vs. 327 and 405 vs. 425) and official scoring (321 vs. 326 and 395 vs. 425; small pair: 263 vs. 198). The published main evaluation uses Qwen3-0.6B as its only receiver, and the authors’ scaling study uses several Qwen3 receivers with fusers trained on MMLU’s auxiliary training split and reports smaller relative gains for larger receivers [15], so no published accuracy exists for released general-purpose medium and large fusers; for those pairs, byte-identical outputs serve as the relevant check.

Repeated replays. Under a rule fixed before any repeat, each of the eight replays ran three additional times on the second configuration, maintaining frozen thresholds, panels, arm rotation, and the timing boundary. The rule defines a policy’s class on savings excluding questions where either arm made its first request after model loading (Table 28b): ✓ if the interval lies above zero, × if below, and ? otherwise. Original classes had to be reproduced by all eight policies across all three repeats, which they were; the classes are the same over all requests (Table 28a). Online scores, inputs, and routes matched frozen records in every repeat, and every output matched its development or same-run counterpart. Absolute savings differ across configurations and are not comparable; large/MMLU-Pro/Text saves 261.4 to 263.5 ms there, and MMLU-Pro paired medians are negative (−21 to −41 ms; −21 to −32 ms for Text), mirroring the original panel.

First configurationSecond configuration
PairTaskReferenceOriginal runRepeat 1Repeat 2Repeat 3Saving class
(a) All requests
largeOBQAText517.1 [447.8, 583.6]395.5 [325.3, 458.3]427.6 [376.4, 474.4]401.8 [335.6, 460.6]✓ above 0
largeOBQAC2C36.6 [18.0, 56.8]29.4 [14.9, 45.5]30.3 [15.7, 46.5]30.4 [15.5, 46.6]✓ above 0
largeARCText717.6 [670.6, 763.2]573.5 [536.0, 608.7]585.4 [546.7, 622.9]574.2 [536.3, 611.2]✓ above 0
largeARCC2C110.9 [82.0, 142.8]91.4 [68.9, 116.2]90.5 [67.3, 115.4]90.2 [66.9, 115.8]✓ above 0
mediumOBQAC2C79.7 [48.1, 129.1]49.9 [35.8, 65.5]52.2 [37.7, 67.9]48.5 [34.3, 64.4]✓ above 0
mediumARCC2C100.7 [74.5, 133.1]79.2 [58.5, 104.5]82.4 [61.4, 108.4]76.5 [56.5, 101.2]✓ above 0
largeMMLU-ProText346.3 [249.8, 443.0]261.4 [191.1, 331.5]262.4 [192.3, 331.1]263.5 [193.5, 333.7]✓ above 0
largeMMLU-ProC2C8.2 [−20.5, +44.0]−0.1 [−23.4, +27.8]2.4 [−20.3, +30.6]2.6 [−20.1, +30.4]? includes 0
(b) Questions with a cold first request excluded (N=127N = 127); the class rule uses these
largeOBQAText532.8 [470.9, 594.7]419.6 [370.7, 468.7]430.2 [380.9, 479.9]423.1 [373.6, 471.5]✓ above 0
largeOBQAC2C33.5 [16.5, 51.7]27.4 [13.8, 42.3]29.3 [14.9, 44.2]27.8 [13.9, 42.5]✓ above 0
largeARCText717.2 [670.1, 761.8]573.2 [535.8, 609.6]585.0 [547.0, 621.8]573.8 [535.2, 610.6]✓ above 0
largeARCC2C112.3 [83.7, 142.3]92.4 [70.2, 115.8]91.6 [68.5, 115.7]91.2 [68.9, 115.1]✓ above 0
mediumOBQAC2C61.0 [43.9, 79.9]47.3 [34.0, 61.9]50.0 [36.4, 65.2]45.6 [32.5, 60.1]✓ above 0
mediumARCC2C100.8 [73.0, 132.8]79.3 [57.6, 104.3]82.5 [60.1, 108.4]76.6 [55.4, 101.1]✓ above 0
largeMMLU-ProText322.1 [234.8, 412.5]260.8 [189.9, 333.9]261.0 [189.7, 335.1]262.7 [191.1, 336.5]✓ above 0
largeMMLU-ProC2C6.5 [−23.3, +41.1]−0.0 [−24.5, +28.3]2.6 [−21.1, +30.5]2.8 [−20.7, +30.8]? includes 0

Table 28. Every replayed policy keeps its prespecified saving class in all three repeats. Paired mean saving (ms) [95% CI]. Saving class: all four intervals of the row lie above zero (✓) or all include zero (?). Cold first request: a path’s first request after model loading. Configurations are not comparable.

Cheaper implementations. Our replays rerun the receiver after the probe. In an additional run (protocol fixed before any output), four arms were interleaved per policy: the fixed reference, the original policy, a reuse policy continuing the receiver-only answer from the probe’s KV cache (the receiver prompt is an exact token prefix of the probe input), and an argmax policy answering skipped questions from the probe’s argmax without sending a receiver request. Argmax labels match native receiver answers on 88 to 100% of calibration questions, and recertifying with them retains every threshold except medium/OBQA/C2C (q=0.50q = 0.50). Argmax answering boosts every saving, for instance MMLU-Pro C2C from 4.6 to 85.0 ms [50.3, 122.8], and narrows the Text-to-C2C saving ratio from 14.0 and 6.2 to 3.2 and 2.7 on OBQA and ARC (4.2 on MMLU-Pro). Its paired medians are negative on MMLU-Pro and medium/OBQA/C2C because gains concentrate on the skipped minority. Reuse alters no saving by more than 5 ms (Table 29): it skips only the receiver’s prefill of its own prompt, which changes a skipped request’s action time by between 0.4 ms more and 6.3 ms less, while the probe is still paid on every query. Reuse outputs differed from native outputs on 13 of 668 skipped requests (exclusively via trailing periods or extended probe answer phrases) and never in parsed labels.

PairTaskReferenceOriginalReuseArgmax
largeOBQAText436.3 [378.6, 503.2]437.8 [378.1, 506.0]589.7 [518.0, 670.3]
largeOBQAC2C31.2 [16.7, 47.3]31.0 [16.5, 47.0]182.9 [158.2, 208.1]
largeARCText565.6 [528.6, 601.4]564.4 [527.5, 599.6]762.7 [717.4, 804.3]
largeARCC2C91.9 [69.5, 116.6]90.5 [68.3, 115.0]280.8 [252.7, 310.4]
mediumOBQAC2C50.1 [36.0, 65.6]49.9 [35.7, 65.6]125.2 [98.3, 154.9]
mediumARCC2C77.9 [57.4, 103.0]78.2 [57.5, 103.5]189.5 [155.4, 225.4]
largeMMLU-ProText275.2 [199.6, 348.5]280.0 [203.5, 354.2]356.5 [265.1, 446.7]
largeMMLU-ProC2C4.6 [−18.0, +32.6]6.9 [−15.8, +35.4]85.0 [50.3, 122.8]

Table 29. Answering skipped questions from the probe’s argmax raises every saving, whereas reusing the probe’s cache changes none by more than 5 ms (one run on the second configuration; paired mean saving, ms [95% CI]). Original: receiver rerun after the probe.

Full MMLU-Pro population and panel representativeness. Across all 2,641 MMLU-Pro development questions, the C2C policy saves a mean of 23.3 ms [14.5, 32.2] on the second configuration (22.0 ms [14.0, 30.5] excluding the single cold question; paired median −37.3 ms), an order of magnitude below the Text panel saving on that configuration. Under rules fixed before comparing, each panel was compared with its full development population across question, message, and output lengths, share of u=0u = 0 questions, and coverage, flagging policies whose coverage differs by over 10 points or whose length SMD exceeds 0.2. Only the two large MMLU-Pro policies are flagged due to slightly longer receiver-only outputs (5.70 vs. 5.54 tokens; SMD 0.212) and 3.0 points lower coverage (Table 30). Applying population coverage to Eq. (4) alters the saving by 0.8 to 29.0 ms; C2C saves only 5.6 ms (9.7 ms with population coverage). The 128 panel questions of the full MMLU-Pro C2C replay save 4.5 ms, versus 23.3 ms across all 2,641 questions (SMD −0.08).

PolicyCoverage panelCoverage populationLargest |SMD| valueLargest |SMD| featureEq. (4) saving (ms) with panel coverageEq. (4) saving (ms) with population coverage
large/OBQA/Text0.7580.7710.136helper message517.2527.1
large/OBQA/C2C0.7580.7710.115reference output36.638.0
large/ARC/Text0.9380.9500.053question717.8727.8
large/ARC/C2C0.8980.9030.057reference output110.9111.7
large/MMLU-Pro/Text0.3750.4050.212receiver output317.0346.0
large/MMLU-Pro/C2C0.3750.4050.212receiver output5.69.7
medium/OBQA/C2C (q=0.55q = 0.55)0.5160.5260.107receiver output56.858.6
medium/ARC/C2C0.6020.6320.053question100.6107.4
large/OBQA/Text+fact0.7190.7220.123helper message339.5341.4
Llama-8B/OBQA/Text0.5940.5900.136helper message322.4320.3
Llama-8B/ARC/Text0.6880.6890.069receiver output417.2418.2

Table 30. Only the two large MMLU-Pro policies are flagged by the prespecified rule (coverage off by more than 10 points (0.10) or an absolute length SMD (standardized mean difference) above 0.2). Population: all development questions. Eq. (4) saving (ms) with the panel’s latencies (first configuration; Text+fact and Llama-8B: second) and the panel’s or the population’s coverage.

Later replays and costs. A Text-only replay driver copy (22 changed lines, including two-arm rotation; an approved deviation) reproducing 16 saved rows byte for byte measured Llama-8B savings of 318.6 ms [261.1, 372.1] and 418.9 ms [361.7, 473.7] on the second configuration (fixed Text 770.3 and 818.4 ms, policy 451.7 and 399.5 ms; 325.5 and 417.6 ms excluding the first request), though 39 and 27% of queries run slower since queries keeping Text also pay the probe. There, the Text+fact policy skips 92 of 128 panel questions, changes 5, reduces mean latency from 726.5 to 383.1 ms, a saving of 343.4 ms [297.7, 386.0] (paired median 448.5 ms), and answers 116 panel questions correctly vs. 121 for reference. Standalone calibration requires 2Ncal2N_{\mathrm{cal}} machine requests (3Ncal3N_{\mathrm{cal}} comparing both references with shared RR). Using mean savings from the second configuration, if reference outputs come from served traffic, fit and calibration requests amortize after 230 (large/ARC/Text) to 76,205 queries (large/MMLU-Pro/C2C), within 6,415 queries for every Text reference, and after 905 to 152,171 queries if references must also run. A small-LLM controller branch (small helper producing communicate-or-skip probabilities) halted at a predetermined cost screen: the controller alone took 21.6 ms (54.5 ms with probe), and even the fastest controller plus probe (51.7 ms) exceeded an optimistic 43.0 ms saving bound, precluding signal evaluation.

Batched serving. The replays above run at batch size one. We replayed the four deployed Text policies of the large and Llama-8B pairs on OBQA and ARC, with their thresholds unchanged, in static batches of 1, 8, and 32. The development questions and the deployed thresholds were already used in the main experiments, so this is a retrospective measurement of latency; the design was fixed before any of its outputs existed, after an earlier run of the whole-batch variant on the 128-question panels. Each development set is replayed in full (742 OBQA and 299 ARC questions) in its original order, grouped into consecutive batches of B∈{1,8,32}B \in\{1, 8, 32\}. Three arms process every batch. The reference generates the helper agent’s message for all requests and then answers them. The early-return policy runs the probe for the whole batch, answers the skipped requests and returns them, and then generates messages and answers for the rest. The whole-batch policy also skips messages but returns all requests of a batch together. The arm order rotates from batch to batch. Each request’s latency is measured from the start of its batch to its own completion. Models run in Hugging Face Transformers with the probe batched and prefix caching off, the helper agent on one A100-SXM4-40GB GPU and the receiver agent on another, on exclusively allocated nodes. Each policy ran as one job; the first has no warm-up and the others start with untimed warm-up batches on questions outside the measured sets. Intervals are paired bootstraps over batches (20,000 resamples). A saving is called established when its 95% interval lies above zero.

The early-return policy establishes a saving against the reference in all twelve settings (Table 31; 35 to 67% of the reference latency), and the absolute saving grows with the batch size. The whole-batch policy establishes a saving in ten settings; for the Llama-8B receiver agent on OBQA its saving is 36 ms [−3, 82] at B=8B = 8 and −63 ms [−121, 1] at B=32B = 32, where the probe and the split of the batch offset the shorter helper call. Against the whole-batch policy, early return saves 200 to 1,120 ms at B=8B = 8 and 32 (all eight intervals above zero); at B=1B = 1 the two schedules coincide and differ by at most 5 ms. Early return answers the skipped requests first, so at B=8B = 8 the requests that wait finish 110 to 168 ms later than under the whole-batch policy, and throughput is 0.5 to 0.9 requests per second lower; at B=32B = 32 neither cost appears. Dropping the first measured question of each job changes no conclusion.

PolicyBBReferenceWhole batchEarly returnEarly returnWhole batchReference / whole batch / early return
Mean latency (ms)Saving (ms) [95% CI]Throughput
large/OBQA/Text1797380376421 [402, 441]416 [395, 437]1.26 / 2.63 / 2.66
large/OBQA/Text81,2191,044613606 [565, 647]176 [116, 239]6.55 / 7.64 / 6.75
large/OBQA/Text322,2932,1341,2321,061 [1,000, 1,120]159 [92, 224]13.73 / 14.72 / 14.75
large/ARC/Text1866285285581 [558, 604]581 [558, 604]1.15 / 3.51 / 3.51
large/ARC/Text81,420720520900 [852, 948]700 [568, 823]5.54 / 10.88 / 10.18
large/ARC/Text323,0932,1421,3651,728 [1,585, 1,873]950 [542, 1,402]9.93 / 14.34 / 14.71
Llama-8B/OBQA/Text1748438438310 [289, 331]310 [289, 331]1.34 / 2.29 / 2.28
Llama-8B/OBQA/Text81,1451,109724421 [382, 461]36 [−3, 82]6.97 / 7.20 / 6.53
Llama-8B/OBQA/Text322,2582,3211,460797 [727, 866]−63 [−121, 1]13.96 / 13.57 / 13.74
Llama-8B/ARC/Text1810401402409 [372, 445]409 [373, 445]1.23 / 2.29 / 2.49
Llama-8B/ARC/Text81,3591,223709650 [590, 717]136 [52, 235]5.79 / 6.42 / 5.92
Llama-8B/ARC/Text323,0052,6981,5781,427 [1,275, 1,592]307 [140, 488]10.21 / 11.33 / 11.75

Table 31. With early return, skipping lowers mean request latency in all twelve batched settings, by 310 to 1,728 ms; returning the whole batch together establishes a saving in ten. Batched replay of the four deployed Text policies on the full development sets. Mean latency per request (ms) for the reference, the whole-batch policy, and the early-return policy; savings against the reference with paired-batch bootstrap 95% CIs; throughput in requests per second.

At B=1B = 1 the replay reproduces the development skip and change counts of the main experiments for three policies (9 of 284, 7 of 438, and 4 of 206) and differs by one skipped question for large/OBQA/Text (19 of 571 against 19 of 572). Batching pads the inputs, which changes a few probe scores, routes, and answers (Table 32): relative to B=1B = 1, 1 to 6 routes and 0 to 9 policy answers change per setting, and the reference’s answers change on 1 to 7 questions. The share of skipped answers that differ from the reference at the same batch size stays between 1.6 and 3.3%. Because batching changes outputs, we do not claim that a certificate obtained at batch size one transfers to another batch size; an operator should calibrate at the batch size it serves.

PolicySkipped at B=1/8/32B = 1 / 8 / 32Changed among skipped (%) at B=1/8/32B = 1 / 8 / 32RoutesPolicy answersReference answers
large/OBQA/Text571 / 573 / 57219 (3.3) / 18 (3.1) / 19 (3.3)6 / 35 / 54 / 7
large/ARC/Text284 / 283 / 2839 (3.2) / 9 (3.2) / 8 (2.8)1 / 30 / 22 / 1
Llama-8B/OBQA/Text438 / 442 / 4407 (1.6) / 7 (1.6) / 7 (1.6)6 / 69 / 97 / 4
Llama-8B/ARC/Text206 / 206 / 2064 (1.9) / 5 (2.4) / 5 (2.4)2 / 45 / 33 / 2

Table 32. Batching changes at most 6 routes and 9 policy answers per setting, and 1.6 to 3.3% of skipped answers differ from the reference at every batch size (early-return policy). Skipped: requests answered without the message, of 742 (OBQA) or 299 (ARC). Changed: skipped requests whose answer differs from the reference at the same BB, with their share of skipped requests in %. Versus B=1B = 1: requests whose route, policy answer, or reference answer differs from the same arm at B=1B = 1.

Out-of-sample tests

Every frozen policy tested out of sample keeps its change rate’s 95% upper bound below 5% under our parsers (all eleven tests: Figure 10, Table 4 and Table 33). Each ran once, with routes and outputs saved before gold labels were decoded. Only the large-pair ARC and SQuAD tests are sealed, and MMLU-Pro has no unused questions. These runs are not a second calibration. The sealed and held-out protocols define no pass rule, whereas the later ones do. The contamination caveat of Appendix § 20 applies.

Figure 10. Every frozen policy tested out of sample keeps the 95% upper bound of its change rate below 5% under our parsers; only the sealed C2C policy read with the official C2C extraction exceeds it. Each marker is one frozen run on questions its pair had not used (official extraction: the sealed C2C run re-read), placed at its number of skipped questions nn (log scale) and the share of skipped answers that change, with its two-sided 95% Clopper–Pearson interval as a vertical line; the counts are in the tables below. Shaded region under the dashed curve: rates whose upper bound is below 5% at that nn (pass rule: a 95% upper bound below 5%); dotted curve: the largest rate at which the test rejects ρ≥0.05\rho\ge0.05 at δ=0.001\delta= 0.001; dashed horizontal line: the 5% target. Filled: the test rejects at δ=0.001\delta= 0.001; open: only the upper bound is below 5% (level 0.025). Official C2C extraction (gray diamond): the official C2C evaluator’s answer extraction applied to the stored outputs. q=0.50q = 0.50: sensitivity threshold for medium/OBQA/C2C (deployed q=0.55q = 0.55). Options rotated: each option’s text moved one position down, labels kept. ‡: nominal development certificate (Appendix § 12). Where a Text and a C2C run share nn, the C2C marker is drawn slightly to the right. The rotated runs set no pass rule, and their p-values are descriptive. Every statistic is in Table 4, Table 33, Table 34, and Table 35 and Appendices § 19 and § 17.§ 3.

Sealed ARC test. The population is all 1,172 ARC-Challenge test rows of allenai/ai2_arc (1,172 groups after frozen normalization), with none removed. The sealing order was: (1) a freeze recorded that the test population had not been accessed and sealed the protocol, configuration, source index, and analysis snapshot; (2) only the id, question, and choices were projected; (3) after all scores, routes, outputs, probe records, and timings were written, a second freeze locked them; (4) only then was the answer key decoded. The thresholds are the development ones (q=0.95q = 0.95 and 0.90), and no test quantile was computed. The four paths (fixed Text, Text policy, fixed C2C, C2C policy) were interleaved in one allocation, completing all 4,688 requests and 2,344 probes. The two policies’ independent probes had identical inputs and scores. All 197 reference-routed outputs match the fixed-reference path, and INVALID counts were 0/0 (Text) and 1/1 (C2C). The Text and C2C policies change 20 of 1,100 and 11 of 1,047 skipped answers ([1.11%, 2.79%] and [0.53%, 1.87%]; 1.71 and 0.94% of all questions). Relative to the reference, they gain 7 and lose 12 correct answers (Text), and gain 7 and lose 3 (C2C). The probe costs 42.5 and 40.5 ms (accuracy and latency; Table 26). On questions below the frozen thresholds, the sealed split does not significantly differ from the training split (Fisher p=0.65p = 0.65 and 0.68). Its change rates are no higher than on development (Text 1.8 vs. 3.2%, C2C 1.05 vs. 1.11%).

Sealed ARC under the official extraction. Under a rule fixed before computing, we re-read the stored sealed outputs using the official C2C extraction on the 1,169 questions lacking a fifth option (both policies skip the other three, none of which changes under our parser). Two unparsable answers count as agreement; counting them as changes leaves every count unchanged. The Text policy is unaffected (20 of 1,097 skipped answers change under both extractions, 1.82% [1.12, 2.80]). However, 54 of 1,044 skipped C2C answers change (5.17% [3.91, 6.70]; p=0.635p = 0.635), against 11 under our parser. Labels differ on 44 skipped questions, accounting for all 43 additional changes. A mechanical audit, fixed before computing, found that each of these 44 outputs starts with a single option letter followed by that option’s own text, contains no second option label, and receives that letter from our parser. The official extractor marks every one unparsable because the option text contains a string treated as a math term (tan in 17, a minus sign in 10, sin in 7, a plus sign or slash in 4 each, cos in 3, and mod in 2; some contain two). The Text policies remain nearly unaffected elsewhere: on the held-out OBQA questions, the official extraction changes 11 of 595 skipped large-pair Text answers (1.85% [0.93, 3.28], against 12 under our parser) and 16 of 558 Text+fact answers, matching our parser. As fixed in advance for a point estimate above 5%, we state that the sealed C2C result holds under our parser only. The official C2C release includes an ARC test configuration, but we could not verify whether its users were exposed to these questions.

Held-out OBQA. These are the 744 OBQA questions that the project’s partition never assigned to fit, calibration, or development. The protocol’s gate initially stopped the run because an earlier project stage had generated small-pair outputs on them under four communication channels and scored them against the answer key. Before any output existed, the protocol was amended to report them as held out rather than sealed, after two checks verified that no output, score, route, or latency of the medium or large pair existed for these questions, and that no artifact of this study reads the earlier stage’s predictions or scores (two aggregate small-pair accuracies and two rows of the earlier scoring were read during the gate check, as the amendment, fixed before the run, discloses). Only idid, question$, and choices$ (plus fact1 for Text+fact) were projected. Saved fit rows were reproduced bit for bit, and one shard was rerun after a bookkeeping failure that produced no output. With no pass rule set, all five upper bounds fall below 5% (level 0.025). Only large Text and C2C (two nominal settings) also reject at δ=0.001\delta= 0.001 (p=1.5×10−4p = 1.5 \times10^{-4} and $3.7 \times 10^{-4}$; with Bonferroni over the five tests, their family level is 0.005. Text+fact, also nominal, rejects only at 0.025. Always skipping would change 8.2 to 17.5% of answers (Table 33). The two medium thresholds reach identical accuracy: of the 42 questions whose skip status differs, only two have an answer that C2C would change, and these cancel out. No latency was measured.

Later out-of-sample tests. The two Llama-8B policies (using the large pair’s stored helper messages) and medium/ARC/C2C ran once on questions their respective pairs had never seen. They ran under a protocol fixed before any output, whose pass rule was a 95% upper bound below 5%. Before the runs, a gate confirmed that no output of the Llama-3.1-8B receiver or the medium pair’s models existed for them. All three pass and reject at 0.001, with coverage within 1.2 points of development (Table 34). This holds under the official extraction too (3 of 440, 15 of 792, and 16 of 730 changes on the A–D questions). One analysis run stopped after writing the seal receipts, so only the gold stage was recomputed from the sealed route files. The SQuAD test is in Appendix § 17.§ 3.

(a) Changed answers

PolicyTest questionsCoverage (%) TestCoverage (%) DevelopmentChanged / skippedChange rate (%) [95% CI]ppAlways skip (%)
Llama-8B/OBQA/Text, q=0.60q = 0.60held-out OBQA (744)59.159.03/4400.68 [0.14, 1.98]3.7×10−73.7 \times10^{-7}11.96
Llama-8B/ARC/Text, q=0.70q = 0.70ARC test (1,172)67.768.915/7941.89 [1.06, 3.10]4.4×10−64.4 \times10^{-6}10.75
medium/ARC/C2C, q=0.60q = 0.60ARC test (1,172)62.563.215/7332.05 [1.15, 3.35]3.1×10−53.1 \times10^{-5}14.85

Table 34.

Option rotation. With every option’s text moved one position down (labels in place, as in the null control of Appendix § 16), the five frozen large-pair policies ran once on the sealed ARC and held-out OBQA questions. They ran under a protocol fixed before any rotated output, with no recalibration and no gold label read; all outputs were written before the protocol’s cutoff. With the rotation set to the identity, the run reproduced the stored scores bit for bit and the stored outputs byte for byte on 16 questions per population. The rotation check included 28 ARC questions with numeric labels or other than four options. The policies skip 73 to 93.9% of questions and change 1.7 to 2.6% of skipped answers, and every 95% upper bound stays below 5% (Table 35). Receiver-only answers before and after rotation agree on 98.7% (ARC) and 98.0% (OBQA) of questions with u=0u = 0 in the original order, against 85.9 and 75.9% for the others. Of the u=0u = 0 questions, 90.0 and 77.7% keep u=0u = 0. References agree with their original-order answers on 79.2 to 95.7% of questions, with 0 to 2 unparseable answers per arm. All 27 rotated ARC/Text changes fall on questions with u>0u > 0, as in the original order.

Policyu=0u = 0 (%)SkippedCoverage (%) rotatedCoverage (%) originalChangedChange rate (%) [95% CI]ppChanged at u=0u = 0Changed at u>0u > 0Original order: changed / skipped
large/ARC/Text72.11,10093.993.9272.45 [1.62, 3.55]1.5×10−51.5 \times10^{-5}02720/1,100
large/ARC/C2C72.11,04989.589.3181.72 [1.02, 2.70]1.9×10−81.9 \times10^{-8}31511/1,047
large/OBQA/Text46.158278.280.0132.23 [1.19, 3.79]5.4×10−45.4 \times10^{-4}11212/595
large/OBQA/C2C46.158278.280.0122.06 [1.07, 3.57]2.2×10−42.2 \times10^{-4}11113/595
large/OBQA/Text+fact46.154573.375.0142.57 [1.41, 4.27]3.4×10−33.4 \times10^{-3}21216/558

Table 35.

Table 34: All three later out-of-sample tests pass their prespecified rule and reject at 0.001, with test coverage within 1.2 points of development coverage. Pass rule: a 95% upper bound below 5%. ARC test: held out for these pairs (Table 4). Change rate with its Clopper–Pearson 95% CI; pp: exact-test pp-value; always skip: disagreement over the whole population; (b) accuracy in percent and Δacc\Delta\mathrm{acc}, policy minus reference, with its paired-bootstrap 95% CI; pp: percentage points.

(b) Accuracy

PolicyRˉ\bar{R}ReferencePolicyΔacc\Delta\mathrm{acc}, policy − reference (pp) 95% CI
Llama-8B/OBQA/Text, q=0.60q = 0.6080.3883.0682.66−0.40 [−0.94, 0.00]
Llama-8B/ARC/Text, q=0.70q = 0.7081.3184.7384.56−0.17 [−0.77, +0.43]
medium/ARC/C2C, q=0.60q = 0.6075.9472.1872.53+0.34 [−0.26, +0.94]

Table 34b.

Table 35: With the options rotated by one position, every frozen large-pair policy keeps its 95% upper bound below 5% (one run, no recalibration; 1,172 ARC and 744 OBQA questions). Option rotation: each option’s text moved one position down, labels kept. u=0u = 0: share of questions after rotation. Changed at u=0u = 0, u>0u > 0: changes among skipped questions by their uu on the rotated input. Intervals: Clopper–Pearson 95% CI. pp: exact-test pp-value, descriptive. Original order: changed and skipped answers without rotation (Table 4 and Table 33).

What the certificates rest on

The Qwen certificates rest on questions with exactly zero uncertainty, yet certification does not require them: the Llama-8B receiver agent has none, yet is certified and holds out of sample. Unless marked post hoc, each analysis below ran under rules fixed before computing. The unsaturated-AUROC and receiver-rotation analyses and both unparseable-answer variants are post hoc (Table 5).

Zero-uncertainty questions among skipped ones. Over the nine certified Qwen-receiver thresholds, 60.3 to 92.1% of skipped development questions have u=0u = 0. These change 0 to 2.2% of the time, against 3.8 to 20.0% for the others (Table 36; Table 11). For large/ARC/Text and both large MMLU-Pro policies, the u>0u > 0 part has a Clopper–Pearson lower bound above 5%. Thus, their certificates rest on the zeros, and on calibration the u>0u > 0 part alone would pass at no frozen threshold (p=0.175p = 0.175 to 0.999). For medium/ARC/C2C and medium/OBQA/C2C at q=0.50q = 0.50, over 90% of skipped questions have u=0u = 0, so contamination (Deng et al. 2024) could inflate such coverage without voiding a certificate. Out of sample, u>0u > 0 questions also change more often. On the held-out OBQA questions, 11 of 249 (large Text and C2C), 3 of 82 and 1 of 40 (medium C2C at q=0.55q = 0.55 and 0.50), and 15 of 212 (Text+fact) of the skipped u>0u > 0 answers change, against 1 to 10 of 306 to 346 zero-uncertainty ones. On the sealed ARC test, every change falls on u>0u > 0 questions (Text 20 of 263, C2C 11 of 210); none of the 837 zero-uncertainty questions each policy skipped changed. Over all development questions, u=0u = 0 covers 43.3, 58.2, and 12.3% for the medium receiver and 47.7, 72.6, and 24.4% for the large receiver on OBQA, ARC, and MMLU-Pro.

Settingqqnnshare (%)k0/n0k_0/n_0change rate (%) [95% CI]k+/n+k_+/n_+change rate (%) [95% CI]calibration p+p_+
large/OBQA/Text0.8057261.92/3540.56 [0.07, 2.03]17/2187.80 [4.61, 12.19]0.214
large/OBQA/C2C0.8057261.91/3540.28 [0.01, 1.56]12/2185.50 [2.88, 9.42]0.46
large/ARC/Text0.9528476.40/2170.00 [0.00, 1.69]9/6713.43 [6.33, 23.97]0.427
large/ARC/C2C0.9027080.41/2170.46 [0.01, 2.54]2/533.77 [0.46, 12.98]0.765
medium/OBQA/C2C0.5539082.37/3212.18 [0.88, 4.44]4/695.80 [1.60, 14.18]0.979
medium/OBQA/C2C0.5035590.47/3212.18 [0.88, 4.44]0/340.00 [0.00, 10.28]0.601
medium/ARC/C2C0.6018992.11/1740.57 [0.01, 3.16]3/1520.00 [4.33, 48.09]0.175
large/MMLU-Pro/Text0.40106960.38/6451.24 [0.54, 2.43]32/4247.55 [5.22, 10.49]0.877
large/MMLU-Pro/C2C0.40106960.35/6450.78 [0.25, 1.80]32/4247.55 [5.22, 10.49]0.999
large/OBQA/Text+fact0.7553666.01/3540.28 [0.01, 1.56]12/1826.59 [3.45, 11.23]0.747

Table 36. At the nine deployed thresholds, skipped questions with u=0u = 0 change 0 to 2.2% of the time and the others 3.8 to 20.0% (development). nn: skipped questions; share: n0/nn_0/n; n0n_0, k0k_0: skipped questions with u=0u = 0 and their changes; n+n_+, k+k_+: the others [Clopper–Pearson 95% CI]. p+p_+: calibration p-value of the u>0u > 0 part alone (descriptive). Medium/OBQA/C2C is deployed at q=0.55q = 0.55; its q=0.50q = 0.50 row is a sensitivity threshold, accepted without the parser-review questions (341 calibration questions whose outputs were read with gold labels before the parser was frozen).

Ranking without the saturated block. Post hoc, on u>0u > 0 questions alone the development AUROC is 0.739 to 0.827 (median 0.774), against a median of 0.883 on all questions, as in Figure 2. With ties broken by the log of the non-top mass, the all-question AUROC is 0.822 to 0.952 (median 0.884; Table 37); the u>0u > 0 questions hold 91.7 to 100% of all disagreements. On the held-out OBQA questions the u>0u > 0 AUROC is 0.770 to 0.839; the tie-breaking score was not stored there.

Policyallu>0u > 0 [95% CI]disagreements / questionstie-brokenAUROCdisagreements / questionsu=0u = 0 (%)Disagreements at u>0u > 0 (%)
medium/OBQA/C2C0.8400.739 [0.690, 0.789]124/4210.8480.7777/32143.394.7
medium/ARC/C2C0.9100.748 [0.658, 0.837]47/1250.9130.6821/17458.297.9
large/OBQA/Text0.8730.769 [0.707, 0.826]70/3880.8760.7532/35447.797.2
large/OBQA/C2C0.8830.779 [0.724, 0.831]59/3880.8840.6971/35447.798.3
large/ARC/Text0.9520.796 [0.686, 0.896]15/820.952–0/21772.6100
large/ARC/C2C0.9090.827 [0.673, 0.954]11/820.9310.8611/21772.691.7
large/MMLU-Pro/Text0.8210.757 [0.734, 0.778]553/19960.8220.7238/64524.498.6
large/MMLU-Pro/C2C0.8460.774 [0.753, 0.795]751/19960.8470.9385/64524.499.3
large/OBQA/Text+fact0.8890.781 [0.728, 0.832]75/3880.8920.9831/35447.798.7

Table 37. Without the zero-uncertainty block, ProbeMax still ranks disagreements with development AUROC 0.739 to 0.827. u>0u > 0: questions with nonzero uncertainty only, with a question-bootstrap 95% CI and their disagreements / questions; tie-broken: ties broken by the log non-top mass; non-top mass: total probe probability of all labels but the top one; zero block: the u=0u = 0 questions; within u=0u = 0: that score inside the zero block (–: no disagreement there); u=0u = 0: share of questions; disagreements at u>0u > 0: share of all disagreements.

Skipping exactly the zero-uncertainty questions. In all nine settings, several low grid quantiles coincide at τ=0\tau= 0 and that threshold is accepted. Therefore, the rule that skips a question if and only if u=0u = 0 was among the tested candidates (Table 38). On development, it skips 24.4 to 72.6% of questions and changes 0 to 2.2% of them, against 40.5 to 95.0% and 1.1 to 3.7% for the deployed thresholds. With fit, calibration, and development restricted to u>0u > 0 questions and fit quantiles taken over them, the test certifies 1 of 9 settings: large/OBQA/C2C (q=0.35q = 0.35; 3 of 257 calibration questions change, p=9.7×10−4p = 9.7 \times10^{-4}; 16.4% of development questions skipped, 1 of 122 changed). The others fall back (smallest pp from 0.002 to 0.60).

Settingk0/n0k_0/n_0ppCoverage (%)Change rate (%) [95% CI]Coverage (%)Change rate (%) [95% CI]
large/OBQA/Text1/6421.7×10−131.7 \times10^{-13}47.70.56 [0.07, 2.03]77.13.32 [2.01, 5.14]
large/OBQA/C2C2/6423.0×10−123.0 \times10^{-12}47.70.28 [0.01, 1.56]77.12.27 [1.22, 3.86]
large/ARC/Text0/3217.1×10−87.1 \times10^{-8}72.60.00 [0.00, 1.69]95.03.17 [1.46, 5.93]
large/ARC/C2C2/3211.1×10−51.1 \times10^{-5}72.60.46 [0.01, 2.54]90.31.11 [0.23, 3.21]
large/MMLU-Pro/Text14/14422.1×10−172.1 \times10^{-17}24.41.24 [0.54, 2.43]40.53.74 [2.69, 5.06]
large/MMLU-Pro/C2C9/14421.9×10−211.9 \times10^{-21}24.40.78 [0.25, 1.80]40.53.46 [2.45, 4.74]
medium/OBQA/C2C (q=0.55q = 0.55)7/6013.2×10−73.2 \times10^{-7}43.32.18 [0.88, 4.44]52.62.82 [1.42, 4.99]
medium/OBQA/C2C (q=0.50q = 0.50)7/6013.2×10−73.2 \times10^{-7}43.32.18 [0.88, 4.44]47.81.97 [0.80, 4.02]
medium/ARC/C2C1/2562.9×10−52.9 \times10^{-5}58.20.57 [0.01, 3.16]63.22.12 [0.58, 5.33]
large/OBQA/Text+fact2/6423.0×10−123.0 \times10^{-12}47.70.28 [0.01, 1.56]72.22.43 [1.30, 4.11]

Table 38. Skipping exactly the u=0u = 0 questions passes in every setting (τ=0\tau= 0 is an accepted grid threshold), and the deployed thresholds add skipped questions with u>0u > 0. Calibration: changed/skipped at u=0u = 0 and exact binomial pp-value; development: coverage and change rate [Clopper–Pearson 95% CI]. The medium/OBQA/C2C q=0.50q = 0.50 row is a sensitivity threshold (deployed: q=0.55q = 0.55).

Where the replayed saving comes from. Splitting each replayed policy’s paired mean saving by source, using the scores and routes logged by each policy request, skipped questions with u>0u > 0 supply 7.3 to 41.3% of the saving on skipped questions among the eight Qwen-receiver policies whose inferer excludes zero (33.4% for large/MMLU-Pro/C2C). The Llama-8B receiver, which has no u=0u = 0 question, draws its whole saving from them (Table 39; Figure 11b). A recomposed rule that skips only the u=0u = 0 questions, charging each other skipped question its fixed latency plus its logged probe time, would keep 57 to 76% of the Text savings, 75% of the large/ARC/C2C saving, 18% of the large/OBQA/C2C saving, and 85 to 90% of the medium-pair savings. It would not save for large/MMLU-Pro/C2C (−9.8 ms).

Figure 11. The Qwen certificates rest on zero-uncertainty questions: skipped questions with u=0u = 0 rarely change, whereas those with u>0u > 0 change more often and supply 7 to 41% of the replayed saving. One column per setting in both panels, named below (b). (a) Change rate among skipped development questions with u=0u = 0 (light bars) and u>0u > 0 (dark bars), in the setting’s colour (blue: Text and Text+fact; orange: C2C), with two-sided Clopper–Pearson 95% intervals as whiskers (a rate of 0 has no bar, only its interval), at the nine certified Qwen-receiver thresholds and the medium/OBQA/C2C sensitivity threshold q=0.50q = 0.50 (Table 36); square-root scale; green band: at or below the 5% target. (b) Share of the replayed saving on skipped questions that comes from u>0u > 0 questions (dark lower part, value printed; light upper part: the u=0u = 0 share; Table 39; not replayed: no replay at q=0.50q = 0.50); the Llama-8B receiver (not plotted), which has no u=0u = 0 question, draws all of its saving from them.

Table 39: Skipped questions with u>0u > 0 supply 7.3 to 41.3% of the saving on skipped questions for the eight Qwen-receiver policies whose interval excludes zero, and all of it for the Llama-8B receiver. Paired mean saving (ms) by source, and the number nn of questions from each source, over the 128-question panels (Llama-8B and Text+fact on the second configuration, all others on the original one) and, *, over the full 2,641-question MMLU-Pro C2C replay (second configuration); ms: each source’s contribution to Saving; Routed: questions sent to the reference; Share: u>0u > 0 part over the skipped parts; Recomposed: saving if only u=0u = 0 questions were skipped.

PolicySkipped, u=0u = 0 msSkipped, u=0u = 0 nnSkipped, u>0u > 0 msSkipped, u>0u > 0 nnRouted msRouted nnSaving (ms) [95% CI]Share from u>0u > 0Recomposed (ms) [95% CI]
large/OBQA/Text332.960194.537−10.331517.1 [447.8, 583.6]0.369294.9 [211.5, 372.4]
large/OBQA/C2C29.16017.737−10.23136.6 [18.0, 56.8]0.3786.7 [−9.5, +24.8]
large/ARC/Text557.393163.127−2.88717.6 [670.6, 763.2]0.226545.6 [476.1, 614.0]
large/ARC/C2C94.79320.522−4.213110.9 [82.0, 142.8]0.17883.2 [55.2, 111.0]
large/MMLU-Pro/Text204.530144.018−2.280346.3 [249.8, 443.0]0.413196.0 [113.6, 287.0]
large/MMLU-Pro/C2C23.23011.618−26.6808.2 [−20.5, +44.0]0.334−9.8 [−33.7, +17.9]
medium/OBQA/C2C67.0559.4113.26279.7 [48.1, 129.1]0.12367.4 [35.6, 115.2]
medium/ARC/C2C105.6708.37−13.351100.7 [74.5, 133.1]0.07390.5 [63.5, 123.1]
large/OBQA/Text+fact224.560124.332−5.336343.4 [297.7, 386.0]0.356211.2 [162.0, 258.0]
Llama-8B/OBQA/Text0.00337.676−19.052318.6 [261.1, 372.1]1.000−37.7 [−48.3, −31.7]
Llama-8B/ARC/Text0.00427.388−8.440418.9 [361.7, 473.7]1.000−30.8 [−33.4, −27.8]
large/MMLU-Pro/C2C*35.964512.1424−24.6157223.3 [14.5, 32.2]0.2524.5 [−2.9, +12.1]

Table 39.

Option order. Post hoc, rotating the options as in the null control (answers mapped back) changes the medium and large receivers’ own u=0u = 0 development answers 1.8 to 6.3% of the time, against

19.5 to 44.6% at u>0u > 0. The upper bound at u=0u = 0 exceeds 5% for medium OBQA (9.1%), medium ARC (11.0%), and large MMLU-Pro (5.3%), so part of the receiver’s confident answers follow option position (Table 40). These are changes of the receiver’s own answer, not a policy’s change rate. On each policy’s skipped development questions, the receiver’s answer changes under rotation 3.3 to 10.0% of the time. In contrast, the five rotated large-pair policies change 1.7 to 2.6% of skipped answers relative to their rotated references (Table 35). The rotated receiver-only runs stored no probe score.

Table 40: At u=0u = 0, option rotation changes the medium and large receivers’ own answers 1.8 to 6.3% of the time, against 19.5 to 44.6% at u>0u > 0 (receiver alone, development) [Clopper–Pearson 95% CI]. Option rotation: each option’s text moved one position down, labels kept (answers mapped back). Change rate: Changed/nn, not a policy’s change rate. Both answers parse: changed answers out of the questions on which both the original and the rotated answer parse; –: no question.

ReceiverTaskDevelopment questions with u=0u = 0 nnChangedChange rate (%) [95% CI]Development questions with u>0u > 0 nnChangedChange rate (%) [95% CI]Both answers parse: changed / questions
smallOBQA0––74255875.2 [71.9, 78.3]521/696
smallARC0––29920969.9 [64.4, 75.0]180/258
mediumOBQA321195.9 [3.6, 9.1]42116439.0 [34.3, 43.8]176/732
mediumARC174116.3 [3.2, 11.0]1254233.6 [25.4, 42.6]51/297
largeOBQA35492.5 [1.2, 4.8]38812833.0 [28.3, 37.9]134/739
largeARC21741.8 [0.5, 4.7]821619.5 [11.6, 29.7]20/299
largeMMLU-Pro645233.6 [2.3, 5.3]199689144.6 [42.4, 46.9]906/2631

Table 40.

Precision and ties. At an exact zero, every non-top label has probability below 2−242^{-24} and all of them together below 2−232^{-23} (at most 1.08×10−71.08 \times10^{-7}). Recomputing the probe in FP32 (where 48 and 72% of the large receiver’s OBQA and ARC questions, and 45 and 57% of the medium receiver’s, stay exactly zero, and development coverage moves by at most 0.94 points), or breaking ties by the log of the non-top mass, changes no certification decision in 23 settings. By contrast, the top-two logit gap, itself tied on 98 to 100% of questions, moves medium/OBQA/C2C to q=0.50q = 0.50. No setting switches between deployment and fallback under either score.

Unparseable answers. A question counts as a disagreement when exactly one of the two answers fails to parse. These make up 0.000 to 0.707 of disagreement across the 18 main settings (pooled 0.274 on calibration and 0.264 on development) and 0.187 to 0.495 across the seven OLMo and Llama settings. Post hoc, in separate 20-test families with unchanged fit thresholds, a nondeployable filter keeping only questions where both answers parse leaves all 11 certified multiple-choice policies at their quantile. It also certifies the three OLMo settings and medium/MMLU-Pro/C2C (q=0.15q = 0.15). A deployable rule that routes to the reference whenever the receiver’s own answer fails to parse changes no outcome. Every development change of the large-pair OBQA and ARC, Text+fact, and Llama-8B policies is between parsed answers, as is every change on the sealed test, for Text+fact held out, and in the two Llama-8B tests. Exactly one INVALID answer is involved in 2 of 40 and 3 of 37 development changes for the two MMLU-Pro policies and 2 of 11 for medium/OBQA/C2C, and out of sample in 1 of 12 and 2 of 13 held-out changes for large/OBQA/Text and C2C and 2 of 8 for medium/OBQA/C2C. For medium/ARC/C2C, 2 of 4 development and 7 of 15 out-of-sample changes involve a C2C answer that does not parse.

Related approaches

Table 41 summarizes the comparison of § 5: what each family of methods controls, whether it gives a finite-sample guarantee, and whether the cheap path is a prefix of the skipped one.

Method familyControlled quantityFinite-sample guaranteeCheap path is a prefix
Risk-controlled early exitagreement with the full modelyesyes
Routing and cascadescorrectness or qualitysomeno
Online cascade learningagreement with the large modelnono
Selective LLM judgingagreement with human labelsyesno
Skipping a message (ours)agreement with a deployed pathyesno

Table 41.

Table 41: Skipping a message shares the consistency-risk framework of risk-controlled early exiting; its cheap path is not a prefix of the skipped one. Skipped path: the costlier computation avoided; cheap path: what runs instead. Cheap path is a prefix: the skipped path begins with it (for early exit, within each decoding step); some: only some methods.

Cascades and selective judges. Cascades share both structural features of message skipping: a gate that does not see the skipped computation, and a gate cost paid on every query. However, they mostly target correctness. Online cascade learning [36] has no finite-sample guarantee, and selective LLM judges [28] need human labels and assume a better escalated judge.

Pruning communication and skipping retrieval. Other work also removes communication or evidence to save cost, but optimizes task quality rather than agreement with a deployed path. Sparse communication topologies let debating agents exchange fewer messages with little loss in accuracy [33], and GPTSwarm optimizes which edges of an agent graph to keep [54], like AgentPrune [52] and AgentDropout, which also removes edges [47]; Optima trains agents to communicate with fewer tokens [5]. Self-RAG learns when to retrieve [3], and Adaptive-RAG routes queries to retrieval strategies by predicted complexity [24]. Yang et al. [50] find that predicting failure is easier than predicting which collaboration protocol pays off. None of these gives a finite-sample bound on how often skipping changes the system’s answers, and none treats the skipped path as a fixed reference.

Prompts and answer parser

This appendix gives every prompt template and the answer parser verbatim. Templates are filled by the frozen prompt code and rendered by each model’s own chat template with a generation prompt: the Qwen2.5 helper template inserts its default system message, the Qwen3 template with thinking disabled inserts an empty think block, the Llama-3.1 template inserts its default system header (knowledge-cutoff and date lines), and no system prompt is added for any receiver. Receiver inputs are tokenized without adding a second BOS token. Every rendering below was checked against saved run records (token counts, input ids, and probe ids). The released dataset contains fully rendered examples for each receiver and benchmark.

Helper instruction (Text). `{question}` is the question stem, a blank line, Choices:, and one line X. `<option>` per displayed option. The helper writes at most 256 tokens.

In one clear sentence, describe the most essential background knowledge needed to answer the question:

{question}

Do NOT directly solve or give answer to the question.

Receiver (receiver-only, C2C, and the last turn of Text). The label list in the format line is the question’s displayed labels joined by / (for example A/B/C/D, or A/.../J for ten MMLU-Pro options). ARC questions labeled 1–4 are displayed and parsed as A–D. The receiver generates at most 64 tokens.

Accurately answer the following question:

{question}

Choices:
{choices}

Instructions:
- Carefully read the question and all options.
- Select the single most correct answer.
- Respond ONLY in the following format: "The correct answer is A/B/C/D".
- Do not include any explanations, additional text, or punctuation besides the answer.

The correct answer is

Text path, C2C, and probe. The Text receiver sees three turns: the helper instruction as a user turn, the helper’s message as an assistant turn, and the receiver message above as the final user turn. The C2C receiver sees the receiver message alone, with the helper’s cache fused in. The probe is one forward pass of the receiver on the rendered receiver-only input followed by the four tokens The correct answer is (so the phrase that ends the user turn also opens the receiver’s answer); it sums the next-token probability over each displayed label’s token set, renormalizes over the displayed labels, and returns 1−max⁡lpl1-\max_l p_l. The helper-aware score of Appendix § 14 runs the same prefill with the helper model.

Text+fact. Only the helper’s input changes: the line Useful fact: <fact1> is placed immediately before the question stem inside {question}. The receiver’s first turn is the fact-free helper instruction. The verifier instruction of the three-agent system is in Appendix § 17.§ 6.

GSM8K (Appendix § 17.§ 4). The helper instruction is the one above with {question} the bare question. The receiver message, whose second line keeps the trailing spaces of the frozen template:

Accurately answer the following question:

{question}

Instructions:
- Carefully read the question.
- Let’s think step by step and explain your reasoning briefly.
- End your response with the exact string "The answer is <number>."

Short-answer tasks. For the short-answer pilot and full run, the receiver template above was edited only where the task requires, each edit applied exactly once: the options block, “and all options”, the answer-format line, and the trailing cue were removed, a task instruction was added, and for passage datasets the passage was placed before the question. The helper instruction is unchanged, with the passage (if any) and the question as {question}. The receiver generates at most 32 tokens. Passage datasets:

Accurately answer the following question:

Passage:
{passage}

{question}

Instructions:
- Carefully read the question.
- Select the single most correct answer.
- Answer with the shortest span from the passage that answers the question. Do not explain.
- Do not include any explanations, additional text, or punctuation besides the answer.

Closed-book datasets use the same template without the passage block and with this third bullet:

  • Answer with only the answer: a name, number, date or short phrase, not a sentence. Do not explain.

The answer is extracted by stripping whitespace and leading Markdown or quote characters, removing a leading Answer:, Final answer:, or The answer is, keeping the first nonempty line and removing a trailing period, then applying Unicode NFKD with accents removed and the official SQuAD normalization (lowercase, no punctuation, no articles, collapsed whitespace). An empty result is INVALID.

Multiple-choice answer parser (version 2). The parser reads the receiver’s raw output and the displayed labels. It never sees the gold answer, the question id, the method, or the option texts. The code repository lists the exact regular expressions and cue-word lists.

  • Normalization. Unicode NFKC; strip; remove one surrounding Markdown code fence; delete **, --, and backticks; map curly quotes to straight quotes; remove one pair of quotes wrapping the whole output. An empty output is INVALID.

  • Selections. A selection comes from (1) a leading label at the very start, (X), [X], or a capital letter followed by punctuation or the end of the text; (2) a declaration such as the final answer is or answer:; (3) an embedded selection of the form the correct ... is; (4) a choice verb, I choose/select/pick; or (5) a reverse selection, option X is correct. After (2)–(4) the next label is read and upper-cased.

  • Not selections. Patterns inside quotes or on quoted lines; patterns in a sentence with a hedge (if, whether, unless, assuming, suppose, hypothetically, example, quoted, says, said, claims, incorrect, not); a label followed by =, +, *, /, ^, or a degree sign; a lower-case letter followed by a word; a label followed by a conditional verb; a declaration followed by an explanation (because, since, due to, that, based on).

  • Joined labels. Labels joined by /, |, commas, or, and, or versus form a multiple selection.

  • Resolution. A final declaration or one preceded by a correction word (actually, correction, on second thought, instead, sorry) starts a new scope. If the remaining single selections agree on one displayed label, an earlier listing of exactly all labels is ignored. Otherwise the output is INVALID when a selected letter is not displayed, when a multiple selection, negated declaration, or declaration without a label remains, when two different labels remain, when several option headings follow each other with no selection after the last, when the chosen label is later called incorrect, or when the output refuses without a final selection. An output with no selection is INVALID.

  • Convention. INVALID is its own answer value: it never matches the gold label, two INVALID answers agree, and INVALID against a letter is an answer change.

References

  1. [1]Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., & Schuster, T. (2024). Conformal risk control. International Conference on Learning Representations.
  2. [2]Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. *The Annals of Applied Statistics*, 19(2), 2025. doi: 10.1214/24-AOAS1998.DOI
  3. [3]Asai, A., Wu, Z., Wang, Y., Vila, A., & Hajishirzi, H. (2024). Self-RAG: Learning to retrieve, generate, and critique through self-reflection. International Conference on Learning Representations.
  4. [4]Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance. *Transactions on Machine Learning Research*, 2024. URL https://openreview.net/forum?id=cSikmW5p6R.openreview.net/forum?id=cSikmW5p6R
  5. [5]Weize Chen, Jiarui Yuan, Chen Qian, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Optima: Optimizing effectiveness and efficiency for LLM-based multi-agent system. In *Findings of the Association for Computational Linguistics: ACL 2025*, pp. 11534–11557. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.findings-acl.601. URL https://aclanthology.org/2025.findings-acl.601/.DOI
  6. [6]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457. arXiv:1803.05457.
  7. [7]C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413, 1934. doi: 10.1093/biomet/26.4.404.DOI
  8. [8]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168. arXiv:2110.14168.
  9. [10]Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks Lakshmanan, and Ahmed H Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/file/b47d93c99fa22ac0b377578af0a1f63a-Paper-Conference.pdf.proceedings.iclr.cc/paper_files/paper/2024/file/b47d93c99fa22ac0b377578af0a1f63a-Paper-Conference.pdf
  10. [11]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 11733–11763. PMLR, 2024. URL https://proceedings.mlr.press/v235/du24e.html.proceedings.mlr.press/v235/du24e.html
  11. [12]Olive Jean Dunn. Multiple comparisons among means. Journal of the American Statistical Association, 56(293):52–64, 1961. doi: 10.1080/01621459.1961.10482090.DOI
  12. [13]Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, and Ramachandran Ramjee. Accuracy is not all you need. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2407.09141
  13. [14]Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 1–13, 2019.arxiv.org/abs/1910.09753
  14. [15]Tianyu Fu, Zihan Min, Hanling Zhang, Jiqiao Yan, Guohua Dai, Wanli Ouyang, and Yu Wang. Cache-to-cache: Direct semantic communication between large language models. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=LeatKxrBCi.openreview.net/forum?id=LeatKxrBCi
  15. [16]Geifman, Y., & El-Yaniv, R. (2017). Selective classification for deep neural networks. Advances in Neural Information Processing Systems, 30. https://proceedings.neurips.cc/paper_files/paper/2017/file/4a8423d5e91fda00bb7e46540e2b0cf1-Paper.pdfproceedings.neurips.cc/paper_files/paper/2017/file/4a8423d5e91fda00bb7e46540e2b0cf1-Paper.pdf
  16. [17]Geng, J., Cai, F., Wang, Y., Koepl, H., Nakov, P., & Guevrychev, I. (2024). A survey of confidence estimation and calibration in large language models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 6577–6595. https://doi.org/10.18653/v1/2024.naacl-long.366
  17. [18]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. The Llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. arXiv:2407.21783.
  18. [19]Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In International Conference on Learning Representations, 2024.arxiv.org/abs/2404.10136
  19. [20]Hendrycks, D., & Gimpel, K. (2017). A baseline for detecting misclassified and out-of-distribution examples in neural networks. International Conference on Learning Representations. https://arxiv.org/abs/1610.02136
  20. [21]Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, Mohammad Mahdi Kamani, Ritika Borkar, Makesh Tarun Chandran, Pantea Zardoshti, and Bita Darvish Rouhani. Cross-model KV cache transfer in LLM families: A closed-form linear mapping for prefill reuse, 2026. URL https://arxiv.org/abs/2608.03893. arXiv:2608.03893.
  21. [22]Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. RouterBench: A benchmark for multi-LLM routing system, 2024. URL https://arxiv.org/abs/2403.12031. arXiv:2403.12031.
  22. [23]Metod Jazbec, Alexander Timans, Tin Hadži Veljković, Kaspar Sakmann, Dan Zhang, Christian A. Naesseth, and Eric Nalisnick. Fast yet safe: Early-exiting with risk control. In *Advances in Neural Information Processing Systems*, volume 37, pp. 129825–129854. Curran Associates, Inc., 2024. doi: 10.52202/079017-4124. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ea5a63f7ddb82e58623693fd1f4933f7-Paper-Conference.pdf.DOI
  23. [24]Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C. Park. Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp. 7036–7050, 2024. doi: 10.18653/v1/2024.naacl-long.389.DOI
  24. [25]Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 14165–14178, 2023a.arxiv.org/abs/2306.02561
  25. [26]Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 7969–7992, 2023b. doi: 10.18653/v1/2023.emnlp-main.495.DOI
  26. [27]Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics*, pp. 1601–1611, 2017.arxiv.org/abs/1705.03551
  27. [28]Jaehun Jung, Faeze Brahman, and Yejin Choi. Trust or escalate: LLM judges with provable guarantees for human agreement. In *International Conference on Learning Representations*, 2025.arxiv.org/abs/2407.18370
  28. [29]Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. QASC: A dataset for question answering via sentence composition. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 34, pp. 8082–8090, 2020. doi: 10.1609/aaai.v34i05.6319.DOI
  29. [30]Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uskoreit, J., Le, Q., & Petrov, S. (2019). Natural Questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466.
  30. [31]Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pp. 6086–6096, 2019.arxiv.org/abs/1906.00300
  31. [32]Li, G., Hammoud, H., Itani, H., Khizbullin, D., & Ghanem, B. (2023). CAMEL: Communicative agents for “mind” exploration of large language model society. Advances in Neural Information Processing Systems, 36, 51991–52008. https://doi.org/10.52202/075280-2264
  32. [33]Li, Y., Du, Y., Zhang, J., Hou, L., Grabowski, P., Li, Y., & Le, E. (2024). Improving multi-agent debate with sparse communication topology. Findings of the Association for Computational Linguistics: EMNLP 2024, 7281–7294.
  33. [34]Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 9802–9822, 2023. doi: 10.18653/v1/2023.acl-long.546.DOI
  34. [35]Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2020). Can a suit of armor conduct electricity? A new dataset for open book question answering 2018 Conference on Empirical Methods in Natural Language Processing. Proceedings of the, 2381–2391. https://doi.org/10.18653/v1/D18-1260
  35. [36]Lunyiu Nie, Zhimin Ding, Erdong Hu, Christopher Jermaine, and Swarat Chaudhuri. Online cascade learning for efficient inference over streams. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 38071–38090, 2024. URL https://proceedings.mlr.press/v235/nie24a.html.proceedings.mlr.press/v235/nie24a.html
  36. [37]Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. In International Conference on Learning Representations, 2025.arxiv.org/abs/2406.18665
  37. [38]Pezeshkpour, P., & Hruschka, E. (2024). Large language models sensitivity to the order of options in multiple-choice questions. Findings of the Association for Computational Linguistics: NAACL 2024, 2006–2017.
  38. [39]Edoardo Pona, Milad Kazemi, Mehran Hosseini, Yali Du, David Watson, Osvaldo Simeone, and Nicola Paoletti. Calibrate-then-delegate: Safety monitoring with risk and budget guarantees via model cascades, 2026. URL https://arxiv.org/abs/2604.14251. arXiv:2604.14251.
  39. [40]Qwen, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., … al., . et . (2025). Qwen2.5 technical report. https://arxiv.org/abs/2412.15115
  40. [41]Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392, 2016.arxiv.org/abs/1606.05250
  41. [42]Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V., Tay, Y., & Metzler, D. (2022). Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35, 17456–17472. https://doi.org/10.52202/068431-1269
  42. [43]Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In International Conference on Learning Representations, 2019. URL https://arxiv.org/abs/1812.09755.
  43. [44]OLMo, Walsh, P., Soldaini, L., Groeneveld, D., & al., . et . (2025). 2 OLMo 2 furious. https://arxiv.org/abs/2501.00656
  44. [45]Iqtedar Uddin and André Bauer. Conformal LLM routing with distribution-free safety guarantees. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pp. 791–799. Association for Computational Linguistics, 2026. doi: 10.18653/v1/2026.acl-srw.70. URL https://aclanthology.org/2026.acl-srw.70/.DOI
  45. [46]Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Aururaj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Yang, R., Yue, X., & Chen, W. (2024). MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37, 95266–95290.
  46. [47]Zhexuan Wang, Yutong Wang, Xuebo Liu, Liang Ding, Miao Zhang, Jie Liu, and Min Zhang. Agent-Dropout: Dynamic agent elimination for token-efficient and high-performance LLM-based multi-agent collaboration. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 24013–24035. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.1170. URL https://aclanthology.org/2025.acl-long.1170/.DOI
  47. [48]Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Stefano Soatto. Positive-congruent training: Towards regression-free model updates. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14299–14308, 2021.arxiv.org/abs/2011.09161
  48. [49]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., … Qiu, Z. (2025). Qwen3 technical report. https://arxiv.org/abs/2505.09388
  49. [50]Chih-Hsuan Yang, Jingyan Jiang, Cheng-Hau Yang, Vikram Vasudevan, Huihuo Zheng, Venkatram Vishwanath, and Rajeev Thakur. LLMs can predict failure risk, but struggle to predict which collaboration protocol pays off: Cost-aware protocol routing across reasoning tasks, 2026. URL https://arxiv.org/abs/2608.14927. arXiv:2608.14927.
  50. [51]Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thought representations for cost-efficient reasoning. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.03094
  51. [52]Zhang, G., Yue, Y., Li, Z., Yun, S., Wan, G., Wang, K., Cheng, D., Yu, J. X., & Chen, T. (2025). Cut the crap: An economical communication pipeline for LLM-based multi-agent systems. International Conference on Learning Representations. https://arxiv.org/abs/2410.02506
  52. [53]Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations, 2024.arxiv.org/abs/2309.03882
  53. [54]Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. GPTSwarm: Language agents as optimizable graphs. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 62743–62767, 2024.arxiv.org/abs/2402.16823
  54. [55]Jiaru Zou, Ruizhong Qiu, Gaotang Li, Xiyuan Yang, Katherine Tieu, Pan Lu, Ke Shen, Hanghang Tong, Yejin Choi, Jingrui He, James Zou, Mengdi Wang, and Ling Yang. Latent collaboration in multi-agent systems. In International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2511.20639. arXiv:2511.20639.

Paper details

vlnc:W_52hdea5u
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents