betaFeedback

WorksComputer Science

Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles

Abstract

Language model benchmarks assign each question a single outcome from one fixed wording, even though the same underlying request can be expressed in many equivalent ways. We argue that the quantity of interest should therefore not be correctness on one wording, but the accessibility of an inquiry, which we define as the probability that a model succeeds across meaning-preserving formulations. The distribution of accessibility across inquiries reveals a structure that accuracy and existing best-, worst-, or average-case prompt scores collapse: how broadly a model's capabilities extend across inquiries and how dependably those capabilities can be elicited. Two models can have the same average performance while one succeeds across more inquiries unreliably and the other succeeds across fewer inquiries but more consistently, leading to opposite preferences depending on whether the task allows multiple retries until one success or when repeated consistent success is required. Across six benchmarks and more than twenty models, this tradeoff is common: for about half of the model pairs with nearly equal mean accessibility, which model looks better depends on whether one success across five formulations is enough or all five must succeed, quantities we call Access@5 and Reliable@5, respectively. For some pairs the two measures diverge significantly, by up to 10 points in opposite directions. Accessibility also reveals that scaling, reasoning, and prompt optimization can redistribute capability across inquiries in ways aggregate accuracy obscures, sometimes broadening the set of inquiries a model can succeed on while making success on them less repeatable. Accessibility turns formulation sensitivity from a nuisance into a measurable property of capability, distinguishing what a model can sometimes succeed on from what users can actually depend on.

Introduction

Language model benchmarks ask each question in one fixed wording, but people who use a model in reality can phrase the same request in many different ways. A model that answers one wording correctly may fail on another, yet a benchmark score only tells us how the model handled the benchmark’s wording, not how dependably it handles the underlying request.

It is well established that models are sensitive to wording: rephrasing instructions, templates, or questions without changing their meaning can change both scores and model rankings [28, 3, 17, 13]. In response, recent work evaluates models over several wordings and reports the best, worst, or average score across them [28, 3, 35, 26]. These summaries establish that wording matters, but they do not show how success is distributed across the underlying questions. This distinction is important because the same average performance can arise from very different model behaviors. A model may succeed on a relatively narrow set of questions but answer those questions consistently however they are phrased. Another may succeed on a much broader set of questions, but only under specific formulations. The first offers narrower but dependable capability; the second broader but less dependable capability. A single benchmark score, and even an average over multiple wordings, can make these models look similar despite this fundamental difference.

We therefore treat each underlying question, which we call an inquiry, as the object of evaluation, rather than any one wording of it, which we call a formulation of that inquiry. For each inquiry, we measure its accessibility: the share of meaning-preserving formulations that the model answers correctly. We generate the formulations automatically and keep only those that independent judges verify as equivalent to the original. Human annotators agree with this verification on 97–98% of annotated pairs.

On six benchmarks spanning factual question answering, mathematical reasoning, general knowledge, and tool use, and by evaluating more than twenty models, we show that the spread of accessibility across a benchmark’s inquiries, which we call the model’s accessibility profile, answers several practical questions at once: how often the model succeeds on a randomly worded request, which inquiries it answers correctly under every formulation, which under at least one, and how it performs when a user can retry versus when every attempt must succeed.

Contributions and key findings.

  1. Inquiry-level accessibility. We introduce inquiry-level accessibility evaluation, where an accessibility profile simultaneously captures average performance, capability that is reachable under at least some formulations, and capability that remains reliable across formulations. It also determines performance when one success across repeated attempts is enough versus when every attempt must succeed. We further theoretically characterize when these two views necessarily prefer different models (Proposition 1).

  1. RQ1: What does accessibility reveal that accuracy hides? Models with similar average performance can differ sharply in how widely their successes extend and how consistently they hold. For about half of the model pairs with nearly equal mean accessibility, which model looks better depends on whether one success across five formulations is enough or all five must succeed, and for some pairs the gap is significant, up to 10 points in each direction.

  1. RQ2: Is this variation caused by the formulation or by decoding noise? In a controlled study, answers to different equivalent formulations disagree in correctness more often than repeated samples of one formulation in 11 of 12 settings, and varying the formulation raises the chance of at least one success while lowering the chance that all attempts succeed.

  1. RQ3: What does accessibility reveal about model changes? Scaling, reasoning, and prompt optimization redistribute capability across inquiries in ways aggregate scores conceal. For example, enabling native thinking in Qwen3 changes MMLU accuracy by only −3.0 points on average, yet makes 25% of inquiries more accessible and 32% less, and lowers the share answered reliably across formulations by 10.9 points.

Related Work

Prompt sensitivity and multi-prompt evaluation. Language models are sensitive to meaning-preserving changes in instruction templates, formatting, and wording ([37, 5, 42, 17]), and paraphrases of benchmark questions change individual answers even when aggregate accuracy and coarse rankings are stable ([12, 27, 7] (Faghi et al. 2026)). Evaluation over multiple prompts has been proposed in response: use-case-specific metrics over instruction paraphrases ([28]), worst-, best-, and average-case performance over paraphrases of each user query ([3]), efficient estimation of performance quantiles across templates ([35]), and the number of prompt resamplings needed for a stable estimate ([26]); related work constructs user-grounded paraphrases ([4]), removes surface-form artifacts in scoring ([21, 20]), or trains for paraphrase invariance ([7]). These works primarily summarize prompt-level or aggregate variation. We instead take the success probability of each inquiry across its formulations as the primitive object, study the distribution of these probabilities across inquiries, derive operational quantities from that one distribution, and analyze how it moves under model interventions.

Repeated trials and model changes. pass@kpass@k [6] and passkpass^k [41] measure whether at least one or all of kk attempts succeed, usually over decoding samples, and reliable@kk has been defined over kk related prompts [10]; perturbation-based uncertainty estimation varies inputs as well as samples [14, 9]. We derive the at-least-once and all-succeed quantities from the accessibility profile, give a condition under which they rank models in opposite orders, and compare formulation with decoding variability in terms of task success. Model updates are known to break inputs that previously worked [2, 39, 11], and studies of scaling, reasoning, and prompt optimization usually report changes in aggregate accuracy [1, 33]. We analyze such changes through each inquiry’s accessibility before and after, which exposes movement and regressions that leave the canonical label unchanged.

Inquiry-Centric Evaluation

Inquiries, Formulations, and Accessibility

A benchmark item is one textual realization of an underlying question. We call the underlying semantic object an inquiry qq and each meaning-preserving realization of it a formulation ff (e.g., "What is the capital of France?" and "Which city is France’s capital?" are both valid formulations of the same inquiry). We hold the deployment configuration cc fixed (system and task instructions, tools, decoding, and scoring) and vary only the formulation. For a model MM and a distribution DqD_q over reasonable formulations of qq, the accessibility of qq is defined as:

Aq(M;c)=Pr⁡f∼Dq[Y(M,c,f)=1](1)A_q(M;c) = \Pr_{f \sim D_q}\left[Y(M,c,f)=1\right] \tag*{(1)}

where Y(M,c,f)Y(M,c,f) indicates that the desired behavior (a correct answer, a solved problem, a correct tool call) is observed for formulation ff. When MM and cc are fixed, we write AqA_q for brevity.

Estimating Accessibility

DqD_q is generally unknown, so we approximate it with a uniformly weighted set of validated formulations f1,…,fmf_1,\ldots,f_m and estimate

A^q(M;c)=1m∑i=1myi,(2)\widehat{A}_q(M;c)=\frac{1}{m}\sum_{i=1}^{m}y_i, \tag*{(2)}

where yiy_i indicates success on fif_i. Paraphrasing is our procedure for sampling formulations. This estimate requires formulations that preserve the inquiry while covering meaningful variation (§ 4).

Accessibility Profiles

Over a benchmark BB with inquiries q1,…,qnq_1,\ldots,q_n, a model induces an accessibility profile {Aqi(M;c)}i=1n\{A_{q_i}(M;c)\}_{i=1}^{n}, estimated by {A^qi(M;c)}\{\widehat{A}_{q_i}(M;c)\}. For interpretation, we call an inquiry absent if A^=0\widehat{A}=0, brittle if 0<A^<10<\widehat{A}<1, or reliable if A^=1\widehat{A}=1, and latent as long as A^>0\widehat{A}>0. Because accessibility is estimated from finitely many formulations, these labels are empirical: reliable means success on every formulation we tested, and absent means failure on every formulation we tested, not necessarily across the entire formulation space. Three kinds of summaries are derived from the same profile. Mean accessibility μ=Eq∼B[Aq]\mu=\mathbb{E}_{q\sim B}[A_q] (3) is the success probability for a random inquiry and formulation. We take inquiries uniformly over the benchmark and write q∼Bq\sim B for brevity. The reliability curve

R(τ)=Pr⁡q∼B[Aq≥τ](3)R(\tau)=\Pr_{q\sim B}[A_q\ge\tau] \tag*{(3)}

gives the share of inquiries meeting an accessibility requirement τ\tau. The area under the reliability curve equals the mean accessibility μ\mu, since an inquiry with accessibility aa counts toward the curve for every τ≤a\tau\le a (formal proof in Appendix A.1). Its left end (τ→0+\tau\to0^+) is the latent share L=P(A>0)L=P(A>0), and its right end is the reliable share R=P(A=1)R = P(A = 1). Finally, when a system faces kk independently sampled formulations, the probability that at least one formulation succeeds is

Access⁡@k=Eq[1−(1−Aq)k],(4)\operatorname{Access}@k = \mathbb{E}_{q}\left[1 - (1 - A_q)^k\right], \tag*{(4)}

and the probability that all kk formulations succeed would be

Reliable⁡@k=Eq[Aqk].(5)\operatorname{Reliable}@k = \mathbb{E}_{q}\left[A_q^k\right]. \tag*{(5)}

At k=1k = 1, both quantities reduce to mean accessibility μ\mu. As k→∞k \to\infty, Access⁡@k\operatorname{Access}@k approaches the latent share L=P(Aq>0)L = P(A_q > 0), since any inquiry with nonzero accessibility eventually succeeds at least once. In contrast, Reliable⁡@k\operatorname{Reliable}@k approaches the reliable share R=P(Aq=1)R = P(A_q = 1), since only inquiries with accessibility one can succeed on every attempt indefinitely. Thus, the same accessibility profile captures deployments that value repeated opportunity to succeed and those that require repeated consistency.

An important point to note is that equal areas need not mean equal profiles. Two models can have the same mean accessibility μ\mu, and hence the same expected success on a random formulation, while distributing it very differently. A model whose reliability curve starts high but falls quickly makes the desired behavior reachable on many inquiries, but on few of them under every formulation; a model whose curve starts lower but stays flat reaches fewer inquiries, yet what it reaches holds up across formulations. Access⁡@k\operatorname{Access}@k rewards the first shape and Reliable⁡@k\operatorname{Reliable}@k the second, and at k=2k = 2 the trade-off is exact: Access⁡@2+Reliable⁡@2=2μ\operatorname{Access}@2 + \operatorname{Reliable}@2 = 2\mu. The shape of the curve decides which model is preferred:

Proposition 1. If two profiles AA and A′A' have equal mean and their reliability curves cross once, with R(τ)≤R′(τ)R(\tau) \le R'(\tau) below the crossing and R(τ)≥R′(τ)R(\tau) \ge R'(\tau) above it, then Reliable⁡@k(A)≥Reliable⁡@k(A′)\operatorname{Reliable}@k(A) \ge\operatorname{Reliable}@k(A') and Access⁡@k(A)≤Access⁡@k(A′)\operatorname{Access}@k(A) \le\operatorname{Access}@k(A') for every k≥1k \ge1.

The proof is a standard convex-order argument (Appendix § 10.§ 2). We estimate both quantities per inquiry without bias for k≤mk \le m, as (ck)/(mk)\binom{c}{k}/\binom{m}{k} and 1−(m−ck)/(mk)1 - \binom{m-c}{k}/\binom{m}{k} for cc successes among mm formulations.

Experimental Setup

Benchmarks and models. We evaluate six benchmarks spanning factual question answering (SimpleQA (Wei et al., [38]), TruthfulQA (Lin et al., [25])), mathematical reasoning (GSM8K (Cobbe et al., [8]), MATH-500 (Hendrycks et al., [19]; Lightman et al., [24])), knowledge (MMLU (Hendrycks et al., [18])), and tool calling (BFCL (Patil et al., [34])), with 18–21 non-thinking models per benchmark from the Qwen3 (Yang et al., [40]), Qwen2.5 (Qwen Team, 2024), Gemma-3 [15], Llama (Grattafiori et al., 2024), Mistral (Jiang et al., [22]), Phi (Li et al., [23]), and OpenAI [29, 30, 31, 32] families, plus Qwen3 native-thinking variants (Table 2 in Appendix § 12).

ModelSimpleQATruthfulQAGSM8KMATH-500MMLUBFCL
Qwen-3 0.6B / 1.7B / 4B / 8B / 14B / 32B✓✓✓✓✓✓
Qwen-2.5 7B✓✓✓✓✓✓
Gemma-3 1B✓✓✓✓––
Gemma-3 4B / 12B / 27B✓✓✓✓✓✓
LLaMA-3 8B, LLaMA-3.1 8B✓✓✓✓✓✓
Mistral 7B, Phi-1.5✓✓✓✓✓–
GPT-3.5, GPT-4o✓✓✓✓✓✓
GPT-4.1 / Mini / Nano, GPT-5.4-Mini✓✓✓✓✓✓
Models212121212018

Table 2. Models per benchmark in RQ1 (✓ = evaluated). Inquiries (median formulations per inquiry): SimpleQA 618 (7), TruthfulQA 471 (9), GSM8K 902 (7), MATH-500 106 (8), MMLU 634 (7), BFCL 600 (10).

Formulations. We generate candidate formulations of each inquiry with GPT-4.1-mini using controlled structural and stylistic variation, remove near-verbatim candidates, and retain only valid formulations that LLM judges verify as semantically equivalent (Appendix § 11). Human annotation of 1,072 pairs on SimpleQA, TruthfulQA, and GSM8K agrees with the verification pipeline on 96.9–98.0% of pairs (Table 1). For each benchmark, every model is evaluated on the same inquiries and the same formulations, and we retain inquiries with at least five valid formulations: between 106 (MATH-500) and 902 (GSM8K) inquiries per benchmark, with a median of 7–10 formulations each (Table 2).

BenchmarkPairsAgreement
SimpleQA46096.9%
TruthfulQA23298.0%
GSM8K38097.7%

Table 1. Human validation of formulation equivalence. We report the number of annotated formulation pairs and agreement between human judgments and the automatic semantic-verification pipeline.

Scoring. Each model answers every retained formulation under the fixed configuration, with greedy decoding unless stated otherwise. Responses are scored with benchmark-specific binary scorers (symbolic answer verification for math, answer extraction for MMLU, multi-judge grading for open-ended QA, and structured checking of tool calls), and Eq. (2) aggregates them per inquiry.

Results

RQ1: What Does the Accessibility Profile Reveal That Accuracy Hides?

Benchmark accuracy is used to decide whether a model can perform a task and which of two models is better. Both uses assume that one wording of each inquiry is representative of how the model responds to it. We test this assumption by comparing each canonical outcome Yorig(q)Y_{\mathrm{orig}}(q) with the estimated accessibility A^q\hat{A}_q, and then ask whether even the mean of the profile suffices to compare models.

Accuracy is one point inside a wide envelope. Because the canonical formulation is among the evaluated formulations, every reliable inquiry (solved under every formulation, A^q=1\hat{A}_q = 1) is a canonical success and every canonical success is latent (solved under at least one, A^q>0\hat{A}_q > 0), so R≤Acc≤LR \le\mathrm{Acc} \le L. Accuracy is one point inside this envelope. Its width, L−RL - R, is the brittle share (0<A^q<10 < \hat{A}_q < 1): inquiries whose outcome depends on the wording. The envelope is wide (Figure 1): the median brittle share is 55% on TruthfulQA, 37% on MMLU, and 20–24% on GSM8K, MATH-500, and BFCL, with SimpleQA, where most inquiries are absent (A^q=0\hat{A}_q = 0) for every model, the low-capability exception at 6% (Table 3 in Appendix § 13).

Figure 1. Benchmark accuracy is one point inside a capability envelope. Inquiries per model are reliable (A^q=1\hat{A}_q = 1), brittle (0<A^q<10 < \hat{A}_q < 1), or absent (A^q=0\hat{A}_q = 0). Black ticks mark benchmark accuracy and blue diamonds the latent share P(A^q>0)P(\hat{A}_q > 0); models are ordered by accuracy. All non-thinking models on a shared inquiry set per benchmark; SimpleQA, MATH-500, and MMLU in Figure 7.

Figure 7. Capability envelopes for all six benchmarks, including SimpleQA and MATH-500 (as in Figure 1).

Benchmark#M#InqAccRRLLBrittle (L−RL - R)Majority reversal
SimpleQA216182.90.56.56.0 (max 27.5)2.1 (max 9.1)
TruthfulQA2147149.318.576.655.0 (max 64.5)15.3 (max 18.5)
GSM8K2190293.877.897.819.7 (max 71.3)3.4 (max 23.6)
MATH-5002110684.068.993.423.6 (max 59.4)4.7 (max 20.8)
MMLU2063470.646.187.237.0 (max 67.8)11.3 (max 22.6)
BFCL1860074.258.284.123.5 (max 45.0)4.9 (max 9.0)

Table 3. Per-benchmark summary (medians over models, %). R=P(A^=1)R = P(\hat{A} = 1) is the reliable share and L=P(A^>0)L = P(\hat{A} > 0) the latent share; accuracy always lies in [R,L][R, L], and the brittle share L−RL - R is the width of that interval. Majority reversal is the share of inquiries whose canonical outcome disagrees with the majority of formulations. Maxima over models in parentheses.

Even mean accessibility is not enough: the shape of the profile decides. Averaging over formulations does not remove the problem, because models with the same mean accessibility can distribute it very differently. Figure 2 shows reliability curves (Eq. (3)) for four near-tied models (in terms of accuracy) on GSM8K and four on BFCL. On GSM8K, the four models are within 2.8 points of accuracy and have nearly identical latent shares, yet their reliable shares differ by up to 11.1 points (95% CI [8.6, 13.7]), a gap that appears only as the requirement approaches τ=1\tau= 1. On BFCL, the curves cross: GPT-5.4-Mini has a lower latent but a higher reliable share than Qwen-3 1.7B at nearly equal mean accessibility. By Proposition 1, GPT-5.4-Mini must therefore rank higher under every Reliable@k and lower under every Access@k; at k=5k = 5 it leads by 9.6 points ([5.5, 14.0]) and trails by 6.8 points ([$-10.8, -2.8]). Which model is better depends on whether the deployment needs one success or success every time, and neither accuracy nor mean accessibility can tell.

Figure 2. Similar accuracy can conceal different reliability profiles. Reliability curves R(τ)=P(Aq≥τ)R(\tau) = P(A_q \ge\tau) for near-tied models on GSM8K (left; accuracy within 2.8 pp) and BFCL (right; within 2.0 pp). The left end is the latent share, the right end (dot) the reliable share.

Such reversals are common. Among the 118 model pairs whose mean accessibility differs by at most 2 points (TruthfulQA, GSM8K, MATH-500, MMLU, BFCL), Access@5 and Reliable@5 rank the two models in opposite orders for about half (53%). For 11 pairs both gaps exceed 2 points, 8 of them significantly in both directions, and 10 of these 11 have reliability curves that cross exactly once, as Proposition 1 describes. The largest case is LLaMA-3 8B versus LLaMA-3.1 8B on MMLU: with mean accessibility within 0.1 points, the former is 10.0 points higher in Reliable@5 ([6.3, 13.8]) and 9.3 points lower in Access@5 ([−12.8, −5.8]). Accuracy performs no better. Across all models it ranks similarly to the reliable share (Kendall’s τ\tau of 0.83–0.93 per benchmark), but models are compared at a resolution of a few points, and there the two come apart. Consider the 164 model pairs whose accuracies are within 3 points of each other. In 143 of them (87%), the two reliable shares differ by more than the two accuracies. In 40 (24%), the more accurate model has the smaller reliable share, significantly so in 16 (Figure 10 in Appendix § 15).

Figure 10. All model pairs within 3 points of accuracy: accuracy gap (x) versus reliable-share gap (y). Filled markers: paired bootstrap 95% CI of the reliable-share difference excludes zero. The dashed line marks equal gaps.

The canonical label can even point the wrong way. For some inquiries the canonical outcome lies on the minority side of the model’s behavior: the benchmark records a success although most equivalent formulations fail, or a failure although most succeed. Such majority reversals affect a median of 15.3% of TruthfulQA and 11.3% of MMLU inquiries, and up to 23.6% for an individual model, in both directions (Table 3, Figure 8–Figure 9). For these inquiries the label gives the opposite impression of how the model typically behaves, and any conclusion drawn from them individually, such as an error analysis, inherits this error.

Figure 8. Majority reversals as a share of all inquiries; one dot per model, with the median (value above) and interquartile range.

Figure 9. Majority reversals conditional on the canonical outcome, P(A^<0.5∣Yorig=1)P(\hat{A}<0.5 \mid Y_{\mathrm{orig}}=1) and P(A^>0.5∣Yorig=0)P(\hat{A}>0.5 \mid Y_{\mathrm{orig}}=0). A model is omitted from a group with fewer than 20 canonical successes (failures); counts below the axis.

RQ2: Is Formulation Variability More Than Decoding Noise?

The structure in RQ1 is meaningful only if changing the wording changes behavior more than simply sampling again does. In a controlled sub-study, four models (GPT-4.1-Mini, GPT-4o, Qwen-3 8B, LLaMA-3.1 8B) answer 150 SimpleQA, 102 TruthfulQA, and 150 GSM8K inquiries under seven equivalent formulations each, with five samples per formulation at temperature 0.7. This lets us compare, under identical sampling, answers to the same wording with answers to different wordings (Figure 3).

Figure 3. Changing the wording changes behavior more than resampling it. Rows are model–benchmark settings; T=0.7T = 0.7. (a) Probability that two answers differ in correctness, for two samples of the same wording (blue) and for two different equivalent wordings (orange). (b) With five attempts: Access@5 and Reliable@5 over five different wordings minus the same over five samples of one wording. Only paraphrases count as different wordings. Bars: 95% bootstrap CIs over inquiries. ⋆: the 95% CI of the difference excludes zero.

Different wordings disagree more than repeated samples. Two answers to different formulations differ in correctness more often than two samples of the same formulation (both at T=0.7T = 0.7) in 11 of 12 settings (95% CI of the difference excludes zero; Figure 3): on TruthfulQA, for example, Qwen-3 8B disagrees with itself in 6.6% of repeated samples but in 17.2% of answers to different wordings. The exception is LLaMA-3.1 8B on SimpleQA, which answers about 1% of inquiries correctly. The same holds under the greedy decoding used in our main experiments: repeated greedy runs of one wording differ in correctness in 0.2–4.9% of cases, reflecting nondeterminism in the inference backends (the generated text itself changes), whereas different wordings differ in up to 16.4% (Figure 11 and Table 4 in Appendix § 16).

ModelFormulations flippedInquiries moved (up / down)Canonical flips
Qwen-3 8B2.2%11.6% (6.3 / 5.3)2.0%
Qwen-3 8B (thinking)1.7%8.5% (3.3 / 5.2)0.9%
Gemma-3 4B2.3%11.1% (5.3 / 5.8)2.4%
GPT-4.1-Mini1.2%5.5% (3.0 / 2.6)1.0%

Table 4. Greedy rerun of the same model on the full GSM8K inquiry set. “Formulations flipped”: paraphrases whose outcome differs between the two runs. “Inquiries moved”: share whose estimated accessibility differs (up / down).

Figure 11. RQ2 including greedy decoding. (a) Greedy: repeated runs of the same wording (blue; collected for three models on the canonical wording of a separate sample of 200 inquiries per benchmark; for GSM8K with a 256-token limit, which may inflate repeat disagreement) versus different wordings (orange). (b) T=0.7T = 0.7 samples. (c) Budget of five attempts. ⋆: the 95% CI of the difference excludes zero ((b) and (c); the repeats in (a) come from a separate sample, so (a) has no paired CI).

Varying the wording is both a more generous and a stricter test. With a budget of five attempts, spreading them over five wordings rather than five samples of one wording raises Access@5 by 1.1 to 9.7 points and lowers Reliable@5 by 0.5 to 11.3 points, both significantly in 11 of 12 settings (Figure 3b, also other budgets in Figure 12). Repeatedly sampling one prompt, as pass@k and pass{sup}k` usually do, therefore underestimates the capability a model can reach and overestimates how reliably it delivers it to users who rephrase. Accessibility therefore measures structure induced by the choice of formulation, not a repackaging of decoding stochasticity.

Figure 12. Access@kk (top) and Reliable@kk (bottom) for k≤5k \le5: solid lines distribute the kk attempts over kk different wordings, dashed lines draw kk samples of one wording (T=0.7T = 0.7).

RQ3: What Do Accessibility Profiles Reveal About Model Changes?

Whether a model change helps is usually decided by its change in benchmark accuracy. We compare, for every inquiry, the canonical label and the accessibility before and after 119 model changes, each on one benchmark: 48 scaling steps (Qwen3, Gemma-3, GPT-4.1), 30 switches to native thinking (Qwen3), 21 switches to chain-of-thought prompting, and 20 prompt optimizations (GEPA, MIPROv2).2

Model changes redistribute capability rather than simply adding or removing it. Figure 4 compares how often canonical labels change with how often inquiry accessibility changes. Accessibility changes are much more common than label changes, and inquiries can become either more or less accessible. For example, on GSM8K, scaling changes the accessibility of 29% of inquiries on average (24% improve and 5% worsen), and native thinking changes 27% (20% improve and 7% worsen) (Figure 4). The results for native thinking on MMLU show that a small change in overall accuracy can hide much larger changes across individual inquiries (Figure 13 in Appendix § 17). Averaged over the six Qwen3 sizes, accuracy drops by only 3.0 points, but 25% of inquiries become more accessible and 32% become less accessible. The reliable share also drops by 10.9 points.

Figure 4. Canonical labels reveal only part of what changes under an intervention. Per row, the upper bar shows the share of inquiries whose canonical outcome improves, stays the same, or worsens; the lower bar the share whose accessibility AqA_q increases, stays the same, or decreases (means over the model changes in the row, count in parentheses). Rows are the benchmarks on which every intervention was run; all benchmarks in Appendix H. Right: change in accuracy and in mean accessibility, Δμ=mean gain−mean loss\Delta\mu= \text{mean gain} - \text{mean loss}.

Figure 13. All intervention–benchmark rows (as in Figure 4), including smallest-to-largest scaling and GEPA and MIPROv2 separately.

Accuracy does not reveal what kind of capability changed. The redistribution also changes reliability in ways accuracy does not track (Figure 5). On GSM8K, the reliable share grows faster than accuracy for all ten scaling steps and all six native-thinking comparisons, so moderate accuracy gains correspond to larger gains in inquiries answered correctly under every wording. On MMLU, native thinking does the opposite for all six Qwen3 sizes: Qwen3-8B gains 6.2 points of accuracy but loses 4.4 points of reliable share, and Qwen3-32B loses 7.1 points of accuracy but 23.2 of reliable share. More broadly, most changes lie above the diagonal on GSM8K, MATH-500, and BFCL (20 of 26, 11 of 16, and 10 of 15), whereas 10 of 15 lie below it on MMLU. The same accuracy change can therefore mean that a model became more reliable, or only that its performance on the canonical wording shifted.

Figure 5. The same change in accuracy can correspond to very different changes in reliable capability. Each point is one model change; x: change in benchmark accuracy, y: change in reliable share R=P(A^q=1)R = P(\hat{A}_q = 1). Points above the dashed line gain reliable capability faster than accuracy (or lose it more slowly). Counts give the changes on each side of the diagonal and exact ties.

Figure 6 shows such redistributions directly, in the encoding of Figure 1: a profile reshuffled at almost unchanged accuracy, accuracy rising while the reliable share falls, and reliability gains far exceeding accuracy gains. Paired profiles for all 119 model changes are in Appendix I.

Figure 6. Accessibility profiles before and after a model change. Each pair of bars shows the model before (upper) and after (lower) the change, with the same encoding as Figure 1. Right: change in accuracy and in reliable share (points), and the share of inquiries whose accessibility increases or decreases.

Discussion

A formulation is an observation; the inquiry is the object being evaluated; its accessibility AqA_q is the quantity to estimate; and the questions practitioners ask correspond to different summaries of the same accessibility profile, of which single-formulation accuracy is one noisy observation.

What evaluations should report. Our results suggest treating correctness as a distribution over semantically equivalent formulations rather than as a single outcome per item. Reporting the reliable share RR and the latent share LL alongside accuracy exposes the width of the envelope that accuracy collapses, and the reliability curve R(τ)R(\tau) shows where between these endpoints models differ. This need not multiply the evaluation cost: in our benchmarks, evaluating two formulations on half of the inquiries, at the cost of one conventional run, recovered mean accessibility about as precisely as accuracy on every benchmark except the small MATH-500, and the two-formulation latent and reliable shares for every model on the four benchmarks with 600 or more inquiries (Appendix § 19).

Why aggregate rankings look reassuring. Rewording leaves coarse model rankings largely intact (Lunardi et al., [27]), and so do our data: accuracy and mean accessibility order models almost identically (Kendall’s τ\tau of 0.90–0.99), whereas the reliable and latent shares agree only at τ\tau of 0.56–0.85. Averaging preserves the coarse ordering but discards the shape of the profile, which decides whether capability is broad but brittle or narrow but dependable, and whether near-tied models reverse between deployment regimes (RQ1). Ranking stability is not evidence of behavioral equivalence.

Deployment regimes and pipeline design. Accessibility profiles provide a practical diagnosis of how a model’s capability is distributed: whether success is broadly reachable but brittle, or narrower but dependable. This distinction matters because different deployment regimes value these behaviors differently. When failures are cheap to detect and retry (e.g., a verifier, test suite, or human checks the output), the relevant quantity is Access@kk; when failures may go undetected or outputs feed later components automatically, it is Reliable@kk. Proposition 1 formalizes when two models with similar average performance are necessarily preferred in opposite regimes.

The profile also informs the retry budget: an inquiry with accessibility AqA_q requires 1/Aq1/A_q attempts on average to obtain a success. The expected number of attempts to solve an arbitrary query sampled from a benchmark B\mathcal{B} therefore depends on the whole profile, not only on the mean accessibility (Eq. (3)), since Eq∼B[1/Aq]≠1/Eq∼B[Aq]\mathbb{E}_{q\sim\mathcal{B}}[1/A_q]\ne1/\mathbb{E}_{q\sim\mathcal{B}}[A_q]. On BFCL, Qwen-3 1.7B and GPT-5.4-Mini have nearly equal mean accessibility, yet Qwen-3 1.7B requires 1.92 attempts per solvable inquiry versus 1.50 for GPT-5.4-Mini, about 28% more calls, in exchange for reaching more inquiries. Thus, in agentic systems built from many model calls, accessibility profiles can inform not only which model to use, but also where retries or verification are worth the cost and how the available token/compute budget should be allocated across pipeline components.

Evaluating model changes. A change in accuracy is the net result of gains and losses across inquiries (RQ3). For model updates, we recommend reporting the gains and losses behind Δμ\Delta\mu and tracking per-inquiry regressions, extending negative-flip counting (Yan et al., [39]; Echterhoff et al., [11]) from single outputs to accessibility.

Limitations. Accessibility is estimated from a finite set of formulations that approximates, but does not sample, the distribution of real user phrasings; weighting formulations by observed frequency is a natural extension. Endpoint labels (reliable, absent) are therefore empirical: with more formulations, an inquiry can only move from reliable or absent to brittle, so the reported brittle shares are lower bounds with respect to the formulation distribution; conversely, inference-backend nondeterminism adds some variation that is not caused by the wording, which we quantify for GSM8K (RQ2, RQ3) but not for every benchmark. RQ2 and the prompted-CoT and prompt-optimization analyses are smaller sub-studies. Finally, we use binary task scoring and English benchmarks; extending the framework to graded scores, other languages, and other modalities is future work.

Conclusion

Prior work established that language models are sensitive to how a request is phrased. We treat this variation as information about the evaluated capability: each inquiry has an accessibility, and a model has an accessibility profile. The profile reveals what single-formulation accuracy hides, from reliability differences between near-tied models to redistributions of capability under model changes, and reporting it gives a more faithful basis for comparing, selecting, and updating models.

Code and Data Availability

The access-eval toolkit, the validated formulation sets, and the code and data that reproduce every figure, table, and number in this paper are available at https://github.com/kazemf78/access-eval. The full model answers and judge grades are available at https://github.com/kazemf78/access-eval-data.

Acknowledgments

This work was supported in part by NSF CAREER Award 1942230, the ONR PECASE Award N00014-25-1-2378, ARO Early Career Program Award 310902-00001, Army Grant W911NF-21-2-0076, NSF Award CCF-2212458, NSF Award 2229885 (NSF Institute for Trustworthy AI in Law and Society, TRAILS), MURI Grant 14262683, DARPA AIQ Grant HR00112590066, an award from Google Research, and an award from Meta 314593-00001.

Proofs for Section 3.3

Area Under the Reliability Curve

For every inquiry, Aq=∫011{Aq≥τ} dτA_q = \int_0^1 \mathbf{1}\{A_q \ge\tau\}\,d\tau, because the indicator equals 1 for τ∈[0,Aq]\tau\in[0,A_q] and 0 otherwise. Taking the expectation over inquiries and exchanging expectation and integral (the integrand is bounded),

μ=Eq[Aq]=∫01Eq[1{Aq≥τ}] dτ=∫01Pr⁡q[Aq≥τ] dτ=∫01R(τ) dτ.\mu= \mathbb{E}_q[A_q] = \int_0^1 \mathbb{E}_q[\mathbf{1}\{A_q \ge\tau\}]\,d\tau= \int_0^1 \Pr_q[A_q \ge\tau]\,d\tau= \int_0^1 R(\tau)\,d\tau.

The area under the reliability curve is therefore exactly the mean accessibility.

Proof of Proposition 1

Write E[⋅]\mathbb{E}[\cdot] for expectation over inquiries, R(t)=P(A≥t)R(t) = P(A \ge t), and R′(t)=P(A′≥t)R'(t) = P(A' \ge t). For any twice-differentiable φ\varphi on [0,1][0,1], Taylor’s theorem with integral remainder gives, for a∈[0,1]a \in[0,1],

φ(a)=φ(0)+φ′(0)a+∫01φ′′(t)(a−t)+ dt.\varphi(a) = \varphi(0) + \varphi'(0)a + \int_0^1 \varphi''(t)(a-t)_+\,dt.

and E[(A−t)+]=∫t1R(s) ds\mathbb{E}[(A-t)_+] = \int_t^1 R(s)\,ds. Taking expectations and subtracting the same identity for A′A', the linear terms cancel because E[A]=E[A′]\mathbb{E}[A] = \mathbb{E}[A'], so

E[φ(A)]−E[φ(A′)]=∫01φ′′(t)H(t) dt,H(t)=∫t1(R(s)−R′(s)) ds.\mathbb{E}[\varphi(A)] - \mathbb{E}[\varphi(A')] = \int_0^1 \varphi''(t)H(t)\,dt,\qquad H(t) = \int_t^1 \bigl(R(s)-R'(s)\bigr)\,ds.

Here H(1)=0H(1) = 0, and H(0)=E[A]−E[A′]=0H(0) = \mathbb{E}[A] - \mathbb{E}[A'] = 0 because ∫01R(s) ds=E[A]\int_0^1 R(s)\,ds = \mathbb{E}[A]. Since H′(t)=R′(t)−R(t)H'(t) = R'(t) - R(t), the single-crossing assumption makes HH non-decreasing on [0,τ∗][0,\tau^*] and non-increasing on [τ∗,1][\tau^*,1]. Together with H(0)=H(1)=0H(0) = H(1) = 0, this gives H≥0H \ge0 on [0,1][0,1]. (This is the Karlin–Novikoff cut criterion: AA dominates A′A' in the convex order.)

For φ(a)=ak\varphi(a) = a^k, φ′′(t)=k(k−1)tk−2≥0\varphi''(t) = k(k-1)t^{k-2} \ge0, hence Reliable@k(A)−Reliable@k(A′)≥0\mathrm{Reliable}@k(A) - \mathrm{Reliable}@k(A') \ge0. For φ(a)=1−(1−a)k\varphi(a) = 1-(1-a)^k, φ′′(t)=−k(k−1)(1−t)k−2≤0\varphi''(t) = -k(k-1)(1-t)^{k-2} \le0, hence Access@k(A)−Access@k(A′)≤0\mathrm{Access}@k(A) - \mathrm{Access}@k(A') \le0. At k=1k=1 both differences are E[A]−E[A′]=0\mathbb{E}[A] - \mathbb{E}[A'] = 0.

Remarks. (i) The identity Access@2+Reliable@2=2μ\mathrm{Access}@2 + \mathrm{Reliable}@2 = 2\mu follows from 1−(1−a)2+a2=2a1-(1-a)^2+a^2=2a. (ii) The proposition concerns population profiles. With finitely many formulations, we apply it to the empirical profiles and report the unbiased per-inquiry estimators with bootstrap confidence intervals. (iii) The effective-breadth/repeatability decomposition of Appendix K is the k=2k=2 case of this view: D=Reliable@2/μD=\mathrm{Reliable}@2/\mu.

Formulation Construction and Validation

This section provides implementation and validation details for the formulation-construction procedure described in Section 4. Our goal is to construct formulation sets that satisfy two requirements: they should exhibit meaningful linguistic variation while preserving the same underlying inquiry and desired behavior.

Controlled Formulation Generation

For each original inquiry, we generate candidate formulations using GPT-4.1-mini. Rather than relying on unconstrained paraphrasing, generation is guided by lightweight, non-adversarial cues designed to vary surface form while preserving meaning, assumptions, intent, and required information.

We use two complementary types of generation cues.

Structural variation. Structural cues encourage changes in wording and syntax without introducing a specific style or tone. We randomly sample from a pool of transformations including reordering clauses, changing sentence openers, varying syntactic structure, substituting safe synonyms, compressing or expanding phrasing, switching between active and passive voice, and changing between statement and question form where appropriate.

Stylistic variation. We additionally generate formulations conditioned on stylistic attributes including formal, casual, polite, confident, neutral, supportive, and inquisitive. These cues are intended to capture natural variation in how users may express the same request rather than adversarial or performance-targeted perturbations.

Prompt for Structural Paraphrasing

You are a careful, concise rewriter. Paraphrase the user’s question below without changing its meaning, intent, or sentiment.

  • Use different wording and structure.

  • Keep the output natural and fluent.

  • Output only the final paraphrase as a single line.

Diversity tip: [structural tip] Question: [original question]

Prompt for Stylistic Paraphrasing

You are a careful, concise rewriter. Rewrite the user’s question with the requested tone or attribute, but do not change its meaning, intent, assumptions, or sentiment.

  • Use different wording and structure.

  • Keep the output natural and fluent.

  • Output only the final paraphrase as a single line.

Tone/attribute: [tone] — [description] Question: [original question]

Diversity Filtering

Because inquiry accessibility is estimated across alternative formulations, near-verbatim rewrites provide little additional information about the formulation space. We therefore filter candidates based on lexical overlap with the original formulation.

We define lexical overlap as the proportion of shared word tokens between the candidate and original formulation, normalized by the length of the shorter text. Candidates with lexical overlap above 0.8 are discarded. This removes near-duplicate rewrites while retaining formulations with meaningful lexical and structural variation.

Semantic Verification

Diversity alone is insufficient: candidate formulations must also preserve the same underlying inquiry and desired behavior. We therefore apply semantic verification after diversity filtering.

Independent LLM judges. Candidate formulations are evaluated for semantic equivalence using GPT-5 and GPT-5.4-mini. Candidates are retained only when the required judges agree that they express the same inquiry as the original formulation. Because formulation generation uses GPT-4.1-mini, generation and semantic verification are performed by different models.

The semantic-equivalence prompt is:

Prompt for Semantic-Equivalence Verification

You are a precise evaluator of meaning. Determine if the candidate paraphrase expresses the SAME intent and meaning as the original question, without adding or removing constraints or changing sentiment. Could they be used interchangeably in the same context? If they are questions, do they ask the same thing, carry the same assumptions, and admit the same answer? Original: [original] Candidate: [candidate] Respond with EXACTLY one token: YES or NO.

Bidirectional NLI filtering. We additionally require bidirectional entailment using facebook/bart-large-mnli. For an original formulation Q0Q^0 and candidate QiQ^i, we compute entailment in both directions and retain the candidate only when both scores exceed 0.75. This additional constraint helps reject candidates that omit information, introduce new constraints, or otherwise alter the inquiry. The final formulation sets therefore satisfy both diversity and semantic-preservation criteria.

Human Validation of Semantic Equivalence

We independently evaluate whether retained formulations preserve the same underlying inquiry through human annotation. Across 1,072 formulation pairs distributed among five annotators, each pair receives a binary YES/NO judgment indicating whether the two formulations “ask the exact same thing,” i.e., whether they can be used interchangeably without changing the desired answer or behavior.

Agreement between human judgments and the automatic semantic-verification pipeline is consistently high, ranging from 96.9% to 98.0% across benchmarks.

The reported agreement measures human–pipeline agreement, not inter-annotator agreement; specifically, it is the fraction of annotated pairs for which the human judgment matches the final decision of the semantic-verification pipeline.

Summary

Our construction procedure targets two complementary requirements for inquiry-level evaluation. Diversity filtering ensures that sampled formulations contain meaningful variation in expression, while semantic verification ensures that this variation does not change the underlying inquiry or desired behavior. High agreement with human judgments provides additional evidence that the resulting formulation sets preserve semantic equivalence.

Benchmarks and Models

Table 2 lists the non-thinking models evaluated on each benchmark in RQ1 (all models on a benchmark share the same inquiries and formulations). A formulation is valid for a benchmark if every model answered it and every answer could be scored (for open-ended QA, the judges agree); we retain inquiries with at least five valid formulations. Gemma-3 1B is not evaluated on MMLU and BFCL, and Mistral 7B and Phi-1.5 are not evaluated on BFCL. RQ3 additionally uses the native-thinking variants of the six Qwen3 models on TruthfulQA, GSM8K, MATH-500, MMLU, and BFCL. The prompted-CoT sub-study uses the same 21 models on TruthfulQA, the prompt-optimization sub-study uses Qwen-3 8B, Gemma-3 4B, LLaMA-3.1 8B, GPT-4.1-Mini, and GPT-5.4-Mini on GSM8K and TruthfulQA, and the RQ2 sub-study uses GPT-4.1-Mini, GPT-4o, Qwen-3 8B, and LLaMA-3.1 8B.

Additional RQ1 Results

Representativeness of the Canonical Outcome

Figure 8 shows, for every model and benchmark, the share of all inquiries on which the canonical outcome disagrees with the majority of formulations, split by direction: a canonical success although most formulations fail, P(Yorig=1,A^<0.5)P(Y_{\mathrm{orig}}=1,\hat{A}<0.5), and a canonical failure although most formulations succeed, P(Yorig=0,A^>0.5)P(Y_{\mathrm{orig}}=0,\hat{A}>0.5). Both directions occur on every benchmark, so the canonical wording neither systematically favors nor systematically disadvantages models.

Conditioning on the canonical outcome gives the same picture from the other side (Figure 9). On GSM8K, for the median model, 64% of canonical failures are answered correctly under at least one alternative formulation (41–81% across models) and 28% under a majority of formulations (14–44%), while 18% of canonical successes fail under at least one alternative formulation (8–82%). The corresponding medians are 56%, 13%, and 65% on TruthfulQA and 35%, 9%, and 23% on BFCL.

Near-Tied Model Pairs

Across TruthfulQA, GSM8K, MATH-500, MMLU, and BFCL, 164 model pairs lie within 3 points of accuracy (median gap 1.4). For 143 (87%) the reliable shares differ by more than the accuracies; the median reliable-share gap is 3.8 points (95% CI [3.4, 4.5]), 37% of pairs ([34, 45]) differ by at least 5 points, and for 89 pairs (54%) a paired bootstrap interval excludes zero. The per-benchmark median ranges from 3.1 (TruthfulQA, [2.3, 4.9]) to 4.4 points (MMLU, [2.7, 6.0]). In 40 pairs (24%) the more accurate model has the smaller reliable share (16 significantly). In 48% of pairs ([40, 55]) the model with the larger reliable share has the smaller latent share, i.e., the reliability curves cross (56% on BFCL, 82% on MMLU). Among the 118 pairs whose mean accessibility differs by at most 2 points, Access@5 and Reliable@5 rank the two models in opposite orders for 62 (53%); the disagreement is at least 2 points in both measures for 11 pairs, 8 of them significantly in both directions, and 10 of these 11 have reliability curves that cross exactly once. The largest significant case is LLaMA-3 8B versus LLaMA-3.1 8B on MMLU (mean accessibility within 0.1 points): the former is 10.0 points higher in Reliable@5 ([6.3, 13.8]) and 9.3 points lower in Access@5 ([-12.8, -5.8]). SimpleQA is excluded: most inquiries are absent for every model, and near-tied models there share almost no reliable mass (median reliable-share gap 0.3 points).

Additional RQ2 Results

Greedy decoding and run-to-run variation. Figure 11 adds the greedy comparison to the main RQ2 figure. To measure run-to-run variation under greedy decoding on the inquiries used in the paper, we reran four models on the full GSM8K inquiry set (902 inquiries, 6,596 formulations) with identical settings (Table 4). Between 1.2% and 2.3% of formulation outcomes change between the two runs, about evenly in both directions and with no systematic shift in accuracy; the generated text itself differs, reflecting nondeterminism in the inference backends. LLaMA-3.1 8B is not part of this baseline: its original answers were produced with a different version of the inference software, so a rerun would not isolate run-to-run variation.

Additional RQ3 Results

Most changes lie above the diagonal on GSM8K, MATH-500, and BFCL (20 of 26, 11 of 16, and 10 of 15), whereas 10 of 15 lie below it on MMLU. Among the 83 changes that raise accuracy, the loss component of Δμ\Delta\mu amounts to a median of 32% of the gain; among the 19 changes whose accuracy moves by at most one point, a median of 18% of inquiries still change accessibility (up to 74%). Prompt optimization on GSM8K lowers accuracy by 2.6 points on average (GEPA: 3.8) while making 18% of inquiries less accessible (35% for Gemma-3 4B with GEPA).

Paired Profiles for All Model Changes

Table 5 summarizes every model change by intervention family and benchmark, and Figures 14–17 show the paired profiles of every individual change (before: upper bar; after: lower bar; same encoding as Figure 1).

Estimating Profiles Under a Fixed Budget

Evaluating mm formulations for every inquiry costs mm times a conventional run. We ask whether the profile can instead be estimated at the cost of a single run of NN calls, by evaluating KK random formulations on n=N/Kn = N/K random inquiries. For each benchmark and model we draw 2,000 such subsamples from the paper’s data (for each inquiry, the number of successes among KK of its $m formulations is hypergeometric) and record how often each estimate falls within 5 points of its full-data value: mean accessibility, and the latent and reliable shares over KK formulations (Access@KK and Reliable@KK). For comparison, we apply the same criterion to accuracy on the same nn inquiries. Table 6 reports the share of models for which an estimate meets this tolerance in at least 95% of subsamples.

Table 6: Share of models whose estimate is within 5 points of the full-data value in at least 95% of equal-cost subsamples (n=N/Kn = N/K inquiries, KK formulations each). Acc.: accuracy on the same nn inquiries.

BenchmarkKKnnAcc.μ\muAccess@KReliable@K
SimpleQA23091.001.001.001.00
32060.950.950.951.00
TruthfulQA22351.001.000.480.52
31570.000.290.000.29
GSM8K24511.001.001.001.00
33001.001.001.000.90
MATH-5002530.430.290.190.05
MMLU23171.001.001.001.00
32110.551.000.600.30
BFCL23001.001.001.001.00
32000.500.670.610.06

Table 6.

ChangeBenchmarknnAccμ\muRRLLAccess@5Reliable@5Aq↑A_q \uparrowAq↓A_q \downarrow
ScalingTruthfulQA1041.7→50.940.8→48.951.5→22.571.1→76.965.2→72.019.8→26.142%21%
ScalingGSM8K1088.0→94.685.7→92.667.7→81.095.7→97.895.1→97.671.3→83.524%5%
ScalingMATH-5001082.5→90.780.7→89.164.8→77.391.8→95.990.6→95.467.8→79.826%6%
ScalingMMLU967.2→75.367.7→75.947.0→56.185.8→90.483.5→88.750.1→59.535%19%
ScalingBFCL969.7→76.368.3→74.850.5→60.680.6→84.178.2→82.355.9→65.127%12%
ThinkingTruthfulQA639.5→54.039.3→52.213.7→23.969.7→80.063.8→75.417.7→28.847%16%
ThinkingGSM8K689.6→92.987.5→91.670.5→78.396.4→97.495.9→97.173.9→81.420%7%
ThinkingMATH-500684.4→93.782.7→91.465.7→77.292.9→97.692.2→97.369.1→80.727%5%
ThinkingMMLU670.1→67.170.7→67.951.6→40.686.3→89.584.4→87.254.7→44.925%32%
ThinkingBFCL670.6→74.968.9→73.651.4→55.280.9→87.578.5→84.656.7→60.628%14%
CoTTruthfulQA2144.4→45.142.8→43.127.0→27.960.1→60.257.2→57.429.4→30.221%20%
Prompt opt.TruthfulQA1049.0→56.746.2→54.619.2→28.374.7→79.268.8→74.524.4→33.840%18%
Prompt opt.GSM8K1092.2→89.686.0→84.260.0→57.097.2→97.096.4→95.967.6→64.611%18%

Table 5. Model changes by family and benchmark (means over the nn changes, in %; before→after). Aq↑/Aq↓A_q \uparrow/ A_q \downarrow: share of inquiries whose accessibility increases / decreases.

With two formulations on half of the inquiries, every quantity meets the tolerance for every model on SimpleQA, GSM8K, MMLU, and BFCL, exactly as accuracy does, and mean accessibility is as recoverable as accuracy on every benchmark except the small MATH-500. When fewer inquiries remain, because KK is larger or the benchmark is small (TruthfulQA, MATH-500), accuracy is not recovered this precisely either, and the reliable share is the hardest quantity to estimate (e.g., MMLU and BFCL at K=3K = 3). In practice, a benchmark with a few hundred inquiries supports estimating the profile with two formulations per inquiry at the cost of a single conventional run.

Figure 14. Paired profiles for consecutive scaling steps.

Breadth–Repeatability Decomposition of Mean Accessibility

Mean accessibility μ=Eq[Aq]\mu= \mathbb{E}_{q}[A_{q}] does not distinguish whether success is spread moderately across many inquiries or concentrated on a smaller subset of inquiries on which it is highly accessible. This appendix gives a continuous decomposition of μ\mu that separates these two cases and applies it to the model changes studied in Section 5.3.

Definition. For a profile with μ>0\mu> 0, define

D=Eq[Aq2]Eq[Aq],B=Eq[Aq]2Eq[Aq2],so thatμ=BD.(6)D = \frac{\mathbb{E}_{q}[A_{q}^{2}]}{\mathbb{E}_{q}[A_{q}]}, \qquad B = \frac{\mathbb{E}_{q}[A_{q}]^{2}}{\mathbb{E}_{q}[A_{q}^{2}]}, \qquad\text{so that} \qquad\mu= B D. \tag*{(6)}

Figure 15. Paired profiles for enabling native thinking in Qwen3.

Repeatability DD. Sample an inquiry qq uniformly and two formulations f1,f2∼Dqf_1, f_2 \sim D_q independently, and let SiS_i denote success on fif_i. Given qq, S1S_1 and S2S_2 are independent with probability AqA_q each, so

Pr⁡(S2∣S1)=Pr⁡(S1,S2)Pr⁡(S1)=Eq[Aq2]Eq[Aq]=D.\Pr(S_2 \mid S_1) = \frac{\Pr(S_1, S_2)}{\Pr(S_1)} = \frac{\mathbb{E}_q[A_q^2]}{\mathbb{E}_q[A_q]} = D.

DD is therefore the probability that a success is repeated under a fresh formulation of the same inquiry. Equivalently, DD is the mean of AA over inquiries weighted by AA (the accessibility of the inquiry behind a randomly chosen success), and D=μ+Var⁡q(A)/μD = \mu+ \operatorname{Var}_q(A)/\mu.

Figure 16. Paired profiles for prompted chain-of-thought on TruthfulQA.

Effective breadth BB. Let pq=Aq/∑q′Aq′p_q = A_q / \sum_{q'} A_{q'} be the distribution of accessibility mass over the nn inquiries. Then

nB=(∑qAq)2∑qAq2=1∑qpq2,nB = \frac{\left(\sum_q A_q\right)^2}{\sum_q A_q^2} = \frac{1}{\sum_q p_q^2},

the inverse Simpson index (effective number) of this distribution; BB is the corresponding evenness. If Aq=cA_q = c on kk inquiries and 00 elsewhere, then nB=knB = k and D=cD = c exactly.

Properties.

  • μ≤D≤1\mu\le D \le1: A2≤AA^2 \le A on [0,1][0,1] gives D≤1D \le1, and E[A2]≥E[A]2\mathbb{E}[A^2] \ge\mathbb{E}[A]^2 gives D≥μD \ge\mu.

Figure 17. Paired profiles for prompt optimization with GEPA and MIPROv2.

  • μ≤B≤P(A>0)\mu\le B \le P(A > 0): B=μ/D≥μB = \mu/D \ge\mu, and by Cauchy–Schwarz E[A]2=E[A1{A>0}]2≤E[A2]P(A>0)\mathbb{E}[A]^2 = \mathbb{E}[A 1\{A > 0\}]^2 \le\mathbb{E}[A^2] P(A > 0), with equality iff AA is constant on its support. Effective breadth never exceeds the latent share.

  • With CV the coefficient of variation of AA across inquiries, B=1/(1+CV2)B = 1/(1 + \mathrm{CV}^2) and D=μ(1+CV2)D = \mu(1 + \mathrm{CV}^2). At fixed μ\mu, the split between BB and DD is thus governed by a single quantity: how unevenly accessibility is distributed across inquiries. BB is invariant to rescaling AA, whereas DD scales with it, so DD partly tracks μ\mu.

  • Changes decompose additively in logs: Δlog⁡μ=Δlog⁡B+Δlog⁡D\Delta\log\mu= \Delta\log B + \Delta\log D.

Estimation. With kk successes among mm formulations, we estimate Eq[A2]\mathbb{E}_q[A^2] by the plug-in mean of A^q2=(k/m)2\hat{A}_q^2 = (k/m)^2. Because E[A^2∣q]=A2+A(1−A)/m\mathbb{E}[\hat{A}^2 \mid q] = A^2 + A(1 - A)/m, the plug-in estimate slightly inflates DD and deflates BB; it corresponds to drawing the second formulation from the same mm formulations with replacement. The unbiased alternative k(k−1)/(m(m−1))k(k - 1)/(m(m - 1)) corresponds to requiring a different second formulation. On our data the two estimators place 118 of the 119 model changes below in the same quadrant of Figure 18.

Figure 18. Breadth–repeatability decomposition of model changes. Each point is one model change; Δlog⁡B\Delta\log B (xx) and Δlog⁡D\Delta\log D (yy) sum to Δlog⁡μ\Delta\log\mu, so the dotted line marks unchanged mean accessibility. Corner labels give the number of changes in each quadrant.

Results. Figure 18 places each model change from Section 5.3 by Δlog⁡B\Delta\log B and Δlog⁡D\Delta\log D. For most beneficial scaling and reasoning changes, breadth and repeatability increase together: 36 of 46 consecutive scaling steps, and all 18 native-thinking comparisons on TruthfulQA, GSM8K, and MATH-500. The exceptions are informative. On MMLU, native thinking lowers repeatability for all six Qwen3 sizes; for three of them effective breadth increases at the same time, so successful behavior is spread across more inquiries but is less reproducible across formulations. The same broader-but-less-repeatable pattern appears for three of six sizes on BFCL. Prompt optimization on GSM8K tends instead to reduce both quantities (6 of 10 changes across GEPA and MIPROv2), whereas on TruthfulQA it mostly increases both (8 of 10). Prompted CoT on TruthfulQA spans all four regimes (8, 2, 5, and 6 of 21 changes in the broader/more-repeatable, broader/less-repeatable, narrower/more-repeatable, and narrower/less-repeatable quadrants).

References

  1. [1]Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. arXiv preprint arXiv:2507.19457, 2025.arxiv.org/abs/2507.19457
  2. [2]Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S. Weld, Walter S. Lasecki, and Eric Horvitz. Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 2019.DOI
  3. [3]Bowen Cao, Deng Cai, Zhisong Zhang, Yuexian Zou, and Wai Lam. On the Worst Prompt Performance of Large Language Models. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.10248
  4. [4]Cléa Chataigner, Rebecca Ma, Prakhar Ganesh, Yuhao Chen, Afaf Taïk, Elliot Creager, and Golnoosh Farnadi. Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework. arXiv preprint arXiv:2505.03563, 2025.arxiv.org/abs/2505.03563
  5. [5]Anwoy Chatterjee, H S V N S Kowndinya Renduchintala, Sumit Bhatia, and Tanmoy Chakraborty. POSIX: A Prompt Sensitivity Index For Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.arxiv.org/abs/2410.02185
  6. [6]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, et al. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
  7. [7]Minjoon Choi. RoParQ: Paraphrase-Aware Alignment of Large Language Models Towards Robustness to Paraphrased Questions. arXiv preprint arXiv:2511.21568, 2025.arxiv.org/abs/2511.21568
  8. [8]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  9. [9]Kyle Cox, Jiawei Xu, Yikun Han, Rong Xu, Tianhao Li, Chi-Yang Hsu, Tianlong Chen, Walter Gerych, and Ying Ding. Mapping from Meaning: Addressing the Miscalibration of Prompt-Sensitive Language Models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.DOI
  10. [10]Jianshuo Dong, Yutong Zhang, Yan Liu, Zhenyu Zhong, Tao Wei, Chao Zhang, and Han Qiu. Revisiting the Reliability of Language Models in Instruction-Following. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, 2026.
  11. [11]Jessica Echterhoff, Fartash Faghri, Raviteja Vemulapalli, Ting-Yao Hu, Chun-Liang Li, Oncel Tuzel, and Hadi Pouransari. MUSCLE: A Model Update Strategy for Compatible LLM Evolution. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.arxiv.org/abs/2407.09435
  12. [12]Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and Improving Consistency in Pretrained Language Models. Transactions of the Association for Computational Linguistics, 9, 2021.arxiv.org/abs/2102.01017
  13. [13]Kazem Faghih, Yize Cheng, Shoumik Saha, Mobina Pournemat, Armin Gerami, and Soheil Feizi. Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy. arXiv preprint arXiv:2607.22554, 2026.arxiv.org/abs/2607.22554
  14. [14]Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Kamalika Das. SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, 2024.arxiv.org/abs/2403.02509
  15. [15]Gemma Team. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.
  16. [17]Elia Habba, Ofir Arviv, Itay Itzhak, Yotam Perlitz, Elron Bandel, Leshem Choshen, Michal Shmueli-Scheuer, and Gabriel Stanovsky. DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 11744–11763, 2025.DOI
  17. [18]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a.
  18. [19]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2021b.
  19. [20]Andong Hua, Kenan Tang, Chenhe Gu, Jindong Gu, Eric Wong, and Yao Qin. Flaw of Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025.arxiv.org/abs/2509.01790
  20. [21]João Maria Janeiro, Mathurin Videau, Andrea Caciolai, Benjamin Piowarski, Patrick Gallinari, and Loic Barrault. Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEval. arXiv preprint arXiv:2606.10657, 2026.arxiv.org/abs/2606.10657
  21. [22]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.arxiv.org/abs/2310.06825
  22. [23]Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need II: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023.arxiv.org/abs/2309.05463
  23. [24]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024.arxiv.org/abs/2305.20050
  24. [25]Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214–3252, 2022.arxiv.org/abs/2109.07958
  25. [26]Gili Lior, Eliya Habba, Shahar Levy, Avi Caciularu, and Gabriel Stanovsky. ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments. In Findings of the Association for Computational Linguistics: EMNLP 2025, 2025.
  26. [27]Riccardo Lunardi, Vincenzo Della Mea, Stefano Mizzaro, and Kevin Rojtero. On Robustness and Reliability of Benchmark-Based Evaluation of LLMs. arXiv preprint arXiv:2509.04013, 2025.arxiv.org/abs/2509.04013
  27. [28]Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of What Art? A Call for Multi-Prompt LLM Evaluation. Transactions of the Association for Computational Linguistics, 12, 2024.arxiv.org/abs/2401.00595
  28. [29]OpenAI. Introducing ChatGPT. https://openai.com/index/chatgpt/, 2022.
  29. [30]OpenAI. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024.
  30. [31]OpenAI. Introducing GPT-4.1 in the API. https://openai.com/index/gpt-4-1/, 2025.
  31. [32]OpenAI. Introducing GPT-5.4 mini and nano. https://openai.com/index/introducing-gpt-5-4-mini-and-nano/, 2026.
  32. [33]Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
  33. [34]Shishir G. Patil, Huanzhi Mao, Fanija Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Jon Stoica, and Joseph E. Gonzalez. The Berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In International Conference on Machine Learning, 2025.
  34. [35]Felipe Maia Polo, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun, and Mikhail Yurochkin. Efficient multi-prompt evaluation of LLMs. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2405.17202
  35. [37]Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.11324
  36. [38]Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024.arxiv.org/abs/2411.04368
  37. [39]Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Stefano Soatto. Positive-Congruent Training: Towards Regression-Free Model Updates. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.arxiv.org/abs/2011.09161
  38. [40]An Yang et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  39. [41]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045, 2024.arxiv.org/abs/2406.12045
  40. [42]Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.arxiv.org/abs/2410.12405

Paper details

vlnc:W_pyc5bb9f
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents