WorksComputer Science
SOTA or Luck? The Winner's Curse in LLM Leaderboards
Abstract
Public LLM leaderboards guide model evaluation and adoption, yet the reported performance of top-ranked models can be systematically overstated. This leaderboard winner's curse arises because ranking noisy benchmark scores rewards sampling luck alongside genuine ability. The same selection effect can distort scores throughout the leaderboard, a phenomenon we call the rank-induced bias. We develop an -out-of- bootstrap to correct for this bias. It assumes a fixed model pool with independent, identically distributed bounded tasks while allowing arbitrary within-task correlation among models. Under exact population tiers separated by positive gaps, we prove that the bootstrap reduces ranking bias to expected residual bias . Across 14 benchmarks from 5 benchmark families, the rank-1 adjustments range from approximately 0.09 to 3.86 percentage points. In our SWE-bench Verified snapshot, the tied leaders receive an adjustment of 0.90 percentage points. Such overstatement is large enough to make the apparent winner statistically fragile. Thus, we use the same bootstrap to further quantify the intervals of plausible rank positions for each model and group models into tiers. We establish sufficient conditions for consistent recovery of population tiers. On SWE-bench Verified, this procedure places 13 of 173 models in the leading tier. We combine the score adjustment and rank intervals in the Rank Uncertainty and Leaderboard Evaluation Report (RULER). We validate its finite-sample performance using sample splitting and calibrated simulations, and advocate publishing RULER alongside raw rankings so readers can assess both reported performance and the precision of the ordering. RULER also provides guidance for benchmark design that accounts for selection effects and ranking uncertainty.
Introduction
Public leaderboards are central to the evaluation of large language models. Organizations use benchmark rankings to choose models, researchers to claim state-of-the-art performance, and funding bodies to assess progress. Yet benchmark scores are no more than sample averages over a small set of tasks, and sample averages inherit the randomness of the underlying data. Thus, selecting the model with the highest observed score rewards sampling luck as well as genuine ability. As a result, the true performance of the winning model tends to be overestimated. This overestimation is the Winner’s Curse (WC), a phenomenon well understood in auction theory [10], genome-wide association studies [38], and clinical trials [37], but largely unexamined in LLM evaluation. The consequences extend beyond statistical reporting: leaderboard rankings influence model procurement, deployment spending, and research investment, yet these decisions often rely on point rankings without accounting for the uncertainty behind small score differences. As a result, economic resources can be directed toward apparent winners whose performance advantages are not reliably established.
We also stress that LLM leaderboards can inherit the WC dynamics at a much larger scale: the number of competing models has exploded, capability gaps at the top have narrowed, and new State-Of-The-Art (SOTA) claims arrive weekly based on fractional-percentage-point improvements. These are precisely the conditions under which the WC is at its worst [35]. Additionally, the same selection effect extends throughout a leaderboard: among models with the same mean performance, scores near the top tend to be inflated and those near the bottom deflated. We call this rank-induced bias. We develop an -out-of- bootstrap [8, 35] to (1) estimate and correct this bias and (2) quantify ranking uncertainty by resampling tasks and re-ranking models altogether. We package these results in a Rank Uncertainty and Leaderboard Evaluation Report (RULER): score adjustments for selection, a rank probability matrix (RPM) showing how often each model occupies each rank, and rank intervals used to group models into tiers. We advocate publishing RULER alongside raw rankings so readers can assess both the reported performance and the precision of model ranking.
Existing work on leaderboard reliability does not correct rank-induced bias. The literature has studied adaptive test-set reuse [9, 17, 30], benchmark lottery and prompt sensitivity [12, 1], run-to-run variance (Bouthillier et al., 2021), item-level and pairwise uncertainty [28, 36, 11], and data contamination [14]. There is a separate research direction concerning the vendors tuning models to public benchmarks, which has motivated private evaluation suites [23]. Because rank-induced bias comes from sampling noise, it persists on private benchmarks too. Our work also relates to the post-selection inference literature. Empirical Bayes methods shrink estimates toward a common prior, often under parametric assumptions [18, 20]. Selective inference conditions on the selection event and requires a separate derivation for each selection procedure [22, 3]. The -out-of- bootstrap [8] needs neither. It requires only i.i.d. bounded tasks, allows arbitrary correlation among models within a task, and yields the score correction and the RPM from a single set of resamples. Xu et al. [35] use it to correct the score of a single selected winner with independent alternatives in inference-then-optimize settings. Appendix H reviews further related work.
We apply RULER to SWE-Bench Verified, which evaluates 173 models on 500 software engineering tasks at the time of our snapshot, the 2 tied leaders score 79.2%. RULER estimates that the score is inflated by 0.9 percentage points, giving adjusted scores of 78.3%. This inflation exceeds the typical margin by which a new entry takes the lead. In 2025, the top score on SWE-bench Verified rose 13 times. The median increase was 0.8 percentage points, and 7 of the 13 increases were smaller than the estimated inflation. RULER also reports a significant ranking uncertainty, as shown in Figure 1. Panel (a) shows that the 2 tied leaders receive only 30.5% and 29.0% probability of remaining as leaders, while the model at rank 3 receives 26.4%. In addition, panel (b) shows that 80% rank intervals for the 2 leaders both span ranks 1–6, and 13 models have overlapping rank intervals, suggesting an indistinguishable top tier. In addition to SWE-Bench Verified, we also apply RULER to 14 benchmarks from 5 families (SWE-bench, Tau2-Bench, LiveBench, AlpacaEval 2, and WildBench), spanning software engineering, agentic customer service, instruction following, and judged response quality. The same patterns frequently recur: rank-1 adjustments exceed 1 percentage point in 7 benchmarks, and the leading tier contains multiple models in 12 benchmarks out of a total of 14 (Table 4). Because organizations, researchers, and funding bodies act on these
Figure 1. The winner on SWE-bench Verified remains statistically unresolved.
| Benchmark | Number of Tasks | Number of Models | Position 1 Score (Raw) | Position 1 Score (Adjusted) | Number of Tiers | Number of Models in Tier 1 |
| SWE Bash-only | 500 | 41 | 76.80 | 75.71 | 10 | 6 |
| SWE Verified | 500 | 173 | 79.20 | 78.30 | 15 | 13 |
| SWE Multimodal | 510 | 12 | 36.47 | 35.84 | 3 | 5 |
| SWE Multilingual | 300 | 14 | 72.67 | 71.17 | 4 | 9 |
| SWE Pro | 395 | 14 | 44.30 | 43.43 | 6 | 4 |
| Tau2 Airline | 50 | 7 | 69.00 | 65.93 | 2 | 5 |
| Tau2 Retail | 114 | 7 | 84.21 | 82.01 | 2 | 5 |
| Tau2 Telecom | 114 | 7 | 98.25 | 98.07 | 6 | 1 |
| AlpacaEval 2 | 805 | 208 | 77.83 | 77.29 | 21 | 3 |
| WildBench v2 | 988 | 63 | 79.81 | 79.71 | 15 | 2 |
| LiveBench pooled | 318 | 81 | 77.56 | 77.44 | 9 | 1 |
| LiveBench Coding | 128 | 81 | 86.72 | 85.61 | 8 | 2 |
| LiveBench Language | 140 | 81 | 69.81 | 67.91 | 7 | 4 |
| LiveBench Instr. | 50 | 81 | 86.78 | 82.93 | 5 | 19 |
Table 4. Score overstatement and unresolved leaders occur across benchmark families.
Notes. The snapshot contains 173 models and 500 tasks, with and bootstrap resamples. Panel (a) shows the top-10 RPM, whose entries are bootstrap rank probabilities. Panel (b) shows 80% rank-mass intervals, with blue points marking observed ranks. Tiers use the common-intersection rule. Appendix § 12 shows the leading 30 × 30 RPM.
rankings, overstated winning scores and unresolved orderings can misdirect model adoption, research credit, and investment.
We validate RULER using sample splitting and a calibrated simulation. In the split study, one half of each task pool selects the ranking and the other half evaluates the selected models. We average the resulting score errors over 200 splits within each benchmark and then average their magnitudes across benchmarks. At rank 1, adjustment reduces score errors from 2.38 to 0.73 pp. A simulation calibrated to SWE-bench Verified supplies a second check with known model means: the correction reduces mean winner bias from 1.41 to 0.21 pp (§ 3).
Contributions. Our paper puts forth a conceptual shift: While people typically treat the numbers on LLM leaderboards as collections of model means that truthfully reflect the models’ performance, they are in fact post-selection objects because of ranking. A model might be ranked at the top due to noise in the data, inflating the performance of the top model(s). We correct for this rank-induced inflation and quantify rank uncertainty. Specifically, we make the following contributions.
Full-leaderboard bootstrap inference. We apply an -out-of- rank-induced bias correction across the entire leaderboard, allowing arbitrary model correlation within each task. RULER combines position-level score adjustments, RPMs that report how often each model occupies each rank under task resampling, and rank-mass intervals that group models into tiers. Under exact population tiers separated by positive gaps, we derive an all-position bias expansion and prove that the bootstrap reduces rank-induced bias to an expected residual of . We also show that the RPM converges to a block-diagonal matrix of within-tier Gaussian rank probabilities, and we establish sufficient conditions for recovering the population tiers.
Validation and empirical audit. We validate the correction via sample splitting—one half of the tasks selects the rank order while the held-out half rescores the selected models—and a calibrated simulation with known model means. Both exercises show substantial reductions in average rank-1 selection error. Auditing 14 benchmarks from 5 families, we find position-1
adjustments from 0.09 to 3.86 pp and multiple models sharing the leading tier in most benchmarks (§ 3–4 and Appendix § 11).
Benchmark design guidance. RULER also translates the analysis into a reporting standard and an asymptotic task-pool sizing rule. Under the exact-tier approximation, tasks are required to decrease the expected rank-1 inflation below , where is the within-tier Gaussian winning gap. Designers can use this calculation to plan larger task pools and publish RULER’s score adjustments, RPMs, rank intervals, and tiers alongside raw rankings to show the uncertainty remaining at the available sample size (§ 5).
Model, Bootstrap, and Theory
We develop a statistical framework for benchmark outcomes that enables the bootstrap inference for LLM leaderboards. We state the model and theory for bounded task-level scores, with binary pass/fail outcomes as the primary application to SWE-bench. Throughout, an LLM “model” denotes an evaluated leaderboard entry, including its agent or scaffold when applicable. All proofs appear in Appendix C.
Let denote the score for model on task , with binary success or failure as a special case. The observed data form an outcome matrix , the transpose of whose -th row we denote . We assume that these task rows are i.i.d. draws from a common distribution on . We impose no parametric form on and allow arbitrary correlation and dependence among different models’ outcomes on the same task. Our primary quantity of interest is the true mean score for each model , along with the induced true ranking. We collect these into the score vector and write for the within-task covariance across models. The true ranking is a permutation such that with a declared arbitrary tie breaking, where is the index of the model whose true ability places it at rank .
In practice, we observe only the empirical mean scores , which we collect into , and the corresponding empirical ranking , defined analogously by . We assume that there is an arbitrary but fixed tie-breaking rule that is being used throughout in computation. Because leaderboards are read off sorted score vectors, our inferential targets are not the individual but functionals of the sorted . Each is unbiased for , but sorting introduces a selection effect that no single-model standard error can capture. We formalize two targets. The first is the rank-induced bias at position ,
which is the bias induced by sorting. The quantity measures the classical WC: the expected score error of the winning model. The second target is the population RPM , with entry giving the probability that model would be placed at rank on an independent draw of tasks from the same task distribution. A concentrated row signals a stable ranking, while a diffuse row signals that the model’s position is poorly resolved by the available task pool. The bootstrap RPM defined below summarizes rankings across -task resamples. Under the conditions of Proposition 2.4, it shares the population RPM’s limiting rank probabilities. Ranking is nonsmooth at tied mean scores, so standard plug-in approximations can fail where selection effects are most pronounced.
Bootstrap correction and components of RULER
Both the rank-induced bias and the leaderboard uncertainty are functionals of the task distribution: if we knew the population distribution, we could repeatedly draw new leaderboards, rank the models, and directly measure the rank-induced bias. Although we do not observe the population, the leaderboard we already observe, i.e., the empirical distribution of tasks, is a good approximation of the population distribution. Thus, we can bootstrap from the observed tasks in the leaderboard, which preserves the dependence among models evaluated on the same task, and estimate the rank-induced bias from the observed leaderboard itself. The difficulty is that ranking is nonsmooth at ties, so the usual smooth-functional justification for the standard bootstrap fails [19]. The -out-of- bootstrap [8] fixes this by resampling fewer tasks with . We follow the literature standards and use for choosing . To the best of our knowledge, there is no universally applicable way to tune the hyperparameter with theoretical guarantees. Thus, we provide an empirical guidance under Appendix § 7 and show that RULER performs relatively robust under different in Appendix § 11.§ 2.
Score Adjustment. The -out-of- bootstrap works by drawing independent resamples of tasks with replacement. Letting and denote their means and rankings, the adjustment for position is
Intuitively, it measures how much a bootstrap-selected model’s resample score exceeds its original score, then rescales this excess to the full-sample noise level. The adjustment belongs to a rank position, rather than to a particular model.
RPM and rank interval. We can use the same -task resamples to measure the uncertainty in the original leaderboard. The first uncertainty measure is the probability of each model occupying a specific rank position, called the Rank Probability Matrix (RPM):
where is the indicator function.1 The RPM shows how uncertain a leaderboard ranking can be. Thus, we develop a second metric, the rank interval, that measures the possible rank positions of each model. Let denote the probability that model ranks at or above position , and the smallest rank position at which this cumulative probability reaches . Then, the rank interval for model is computed as:
These positions contain at least of model ’s bootstrap rank mass. We default to throughout this paper.
Model tiers. Finally, small benchmark samples often leave the ranking of closely performing models unresolved, with no clear winner. We quantify this uncertainty using tiers of models whose rank intervals have a common intersection. Specifically, we start with the first model on the leaderboard for the first tier, and use its rank interval to represent all possible rank positions for the first tier. Then, we add the next model on the board if its rank interval contains a common rank position as the first tier. If yes, we take an intersection of the rank intervals of all models in the tier. Otherwise, close the tier and start a new one. As a result, the rank intervals of all models in a tier share at least one rank. Models with equal observed scores are always placed in the same tier (Appendix D).
RULER. Together, the score adjustment, RPM, and rank intervals with model tiers form the Rank Uncertainty and Leaderboard Evaluation Report (RULER).
Theoretical properties
We now state the theoretical results that justify the -out-of- bootstrap behind RULER. We show that RULER’s score adjustment reduces rank-induced bias to and that the reported tiers can consistently recover the population model tiers. For a fixed number of models , we make the following assumptions.
Assumption 2.1 (Exact population tiers with positive gaps). The models partition into , with for and . If , the smallest adjacent gap is bounded below by a fixed .
Assumption 2.1 is a deliberate modeling idealization rather than a literal claim that in reality there are always different models with identical population performance. In practice, models may differ in their true mean scores, but often these differences are very small and become statistically indistinguishable with the finite available data. We therefore use exact ties as a tractable approximation for groups of closely performing models whose ordering is difficult to resolve. This idealization captures the selection effects that are particularly pronounced among closely competing models. Under this assumption, we derive theoretical results to provide an informative benchmark for understanding rank-induced bias, and evaluate the practical usefulness of the proposed correction through the finite-sample validation in Section 3. The tiers reported by RULER similarly represent unresolved rankings rather than assertions of exact equality in population performance.
In addition to Assumption 2.1, note that we have assumed that tasks are i.i.d. to facilitate a clean theoretical analysis, as discussed at the beginning of Section 2. However, our sensitivity analysis shows that RULER’s score adjustments are broadly robust across alternative resampling schemes for non-i.i.d. tasks on SWE-bench, as discussed in Section 4.2 and Appendix F.1.
Next, we formalize the magnitude of the rank-induced bias in a leaderboard. Let denote the number of models sitting above tier , a mean-zero Gaussian vector with covariance , and the -th largest coordinate in . Equivalently, is the -th ordered statistics of .
Proposition 2.2 (First-order rank-induced bias at every fixed position). At position ,
Therefore, is the position-induced effect. The winner’s curse bias [35] is a special case of Eq. (4) with . The proposition shows that rank-induced bias can be . Next, we show that the -out-of- bootstrap score adjustment in Eq. (1) reduces this bias to .
Theorem 2.3 (Bootstrap score adjustment). Assume that as , , , and . At every fixed position ,
The second equation in (5) states that, at every fixed position, the adjusted score is unbiased for the true score of the model placed there, up to an error of order . The argument has two steps. First, the bootstrap reproduces the selection effect on the -task scale. In a resample of tasks, each model’s resample score differs from its observed score by noise of order . Within a tier, observed score gaps are of order , which is negligible relative to this noise because . Between tiers, observed gaps remain near the population gaps and dominate the noise. Resample rankings therefore keep the tiers in order and rank models within a tier by noise alone, as in the population. The average in Eq. (1) before rescaling thus equals . Second, the factor converts this to , the leading term of in Proposition 2.2. Subtracting therefore removes the first-order bias.
Finally, we show that the RPM and model tiers estimated using the same -out-of- bootstrap are consistent estimators. For , define and note that . Further, we define the population rank probability matrix (RPM) with entries . These are probabilities over repeated samples of tasks. They depend on , although that dependence is suppressed in the notation.
Proposition 2.4 (Convergence of RPM and consistent tier recovery). Assume the fixed- bounded iid task model and the exact population tiers with positive gaps. Let and be deterministic positive integer sequences with , , and as . Suppose for distinct in each non-singleton tier.
For every tier , model , and ,
For and ,
Consequently, the bootstrap RPM consistently estimates the population RPM:
For every fixed , the common-intersection rule returns exactly , in that order, with probability tending to one.
Proposition 2.4 shows that RULER separates genuine performance differences between tiers while retaining ranking uncertainty within them. As the number of tasks grows, the RPM assigns vanishing probability to ranks outside a model’s population tier. Within a tier, the models have equal true scores, so their relative ranks remain sensitive to sampling noise. The resulting rank intervals share a common position within each tier and separate across tiers with probability tending to one, allowing RULER to recover the population partition. The full proof appears in Appendix § 8.4.
Notes. Each benchmark uses 200 random half-splits, with and resamples on the selection half. Before RULER uses the selected model’s score on selection half minus its score on evaluation half . After RULER subtracts the estimated bias from that difference. Magnitude of mean error first averages these signed differences over splits within each benchmark, then takes the absolute value and averages across the 14 benchmarks. Its reduction is reported as a percentage. Errors are in percentage points (pp)
Validation
Having established RULER’s theoretical guarantees, we now validate its performance. Specifically, we use a held-out test and calibrated simulations to evaluate how much its score adjustment reduces selection error in finite samples. Both studies show substantial reductions in average selection error.
Held-out Test
The held-out test separates rank selection from evaluation by assigning them to non-overlapping subsets of the task pool. For each benchmark, we randomly split the tasks into a selection half with tasks and an evaluation half with the remaining tasks. We treat the selection half as if it were the entire benchmark. Using only its tasks, we rank the models by their scores to obtain the ranking , and apply the bootstrap in Equation (1) with to obtain the adjustment at each position . The selected models are then scored on . Because the evaluation half plays no role in ranking or adjustment, its score provides an unbiased estimate of the selected model’s mean under independent task sampling.2
For each position , we compare the raw and adjusted errors
The expectation of the raw error is the rank-induced bias of a leaderboard with tasks. If the bootstrap removes the bias present in the selection half, the adjusted error should be centered near zero.3 We repeat the test over 200 random partitions per benchmark. Appendix § 10.§ 1 documents the implementation details.
At rank 1, RULER’s score adjustment reduces the magnitude of mean split error from 2.38 to 0.73 percentage points (pp), a reduction of 69.5% across all 14 benchmarks (Table 1). Positions 2–5 in the same table also show similar improvement, ranging from 46.9% to 55.0%. Appendix § 10.§ 1 reports the complete results. Thus, the held-out test broadly demonstrates that the score adjustment can correct the selection effects generally across the leaderboard.
| Rank | Before RULER | After RULER | Reduction (%) |
| 1 | 2.38 | 0.73 | 69.5 |
| 2 | 1.43 | 0.74 | 48.0 |
| 3 | 1.34 | 0.71 | 46.9 |
| 4 | 1.10 | 0.50 | 55.0 |
| 5 | 1.23 | 0.58 | 52.7 |
Table 1. The correction reduces average selection error across 14 benchmarks.
Calibrated Simulation
While the held-out test uses an independently-estimated score as its reference, we provide a second validation using simulated benchmarks where the models’ true scores are known. To mimic realistic settings, we calibrate the simulations to SWE-bench Verified. Specifically, we fit a logistic model combining model-specific abilities with a shared task difficulty, so outcomes for different models on the same task are dependent (see Appendix § 10.§ 2 for more details). From this fitted model, we generate 100 synthetic benchmarks and apply the -out-of- bootstrap with 2,000 resamples to each. Table 2 compares the raw and corrected scores of the models at positions 1–5 with their known means.
| Rank | Mean Signed Error (pp) — No Correction | Mean Signed Error (pp) — -out-of- bootstrap | RMSE (pp) — No Correction | RMSE (pp) — -out-of- bootstrap |
| 1 | +1.41 ± 0.14 | +0.21 ± 0.15 | 1.97 ± 0.13 | 1.55 ± 0.10 |
| 2 | +0.80 ± 0.17 | −0.06 ± 0.17 | 1.85 ± 0.18 | 1.74 ± 0.14 |
| 3 | +0.66 ± 0.18 | −0.12 ± 0.18 | 1.87 ± 0.16 | 1.78 ± 0.14 |
| 4 | +0.98 ± 0.16 | +0.23 ± 0.16 | 1.85 ± 0.11 | 1.64 ± 0.13 |
| 5 | +0.96 ± 0.17 | +0.24 ± 0.18 | 1.97 ± 0.15 | 1.79 ± 0.13 |
Table 2. The -out-of- correction removes most of the winner’s bias and lowers its root mean squared error. Notes. Entries are the mean signed error and root mean squared error (RMSE) of the selected model’s score relative to its known mean, ± standard errors, in pp. The -out-of- bootstrap uses and 2,000 resamples.
The -out-of- correction provides more accurate evaluation to model performance. The raw winner overstates its true mean by 1.41 pp on average, and the -out-of- correction reduces it to 0.21 pp. The correction also brings the mean errors at positions 2–5 closer to zero. It lowers the winner’s RMSE from 1.97 to 1.55 pp and reduces the RMSE at positions 4 and 5 by about 0.2 pp. Appendix § 10.§ 2 also compares the -out-of- correction with the standard bootstrap (), a beta-binomial empirical-Bayes posterior mean, and a correction built from sample splitting. The -out-of- also performs well generally among the 4 corrections.
Empirical Study Across Benchmark Families
The validation shows that RULER can substantially reduce average selection error. We now use it to ask how much public leaderboard scores are overstated and how precisely their task pools resolve the model ordering. Across the 14 benchmarks, rank-1 adjustments exceed 1 pp in 7, and 12 have multiple models in the leading tier. We begin with SWE-bench Verified to show how the score and rank summaries work together, then assess how broadly the findings hold across benchmark families and alternative reporting choices. Appendices D documents the implementation details
Leaderboard Results
SWE-bench Verified evaluates 173 models on 500 tasks extracted from 12 GitHub repositories in our snapshot.4 Its 2 tied leaders score 79.2%. RULER estimates a downward adjustment of 0.90 pp
Notes. The table shows the top 7 models from SWE-bench Verified. RULER uses and . Score Correction is the amount subtracted from the raw score. Rank intervals contain at least 80% of each model’s bootstrap rank mass.
Notes. The raw and adjusted scores are expressed as percentages. Tier counts and Tier 1 sizes use 80% rank mass and the common-intersection rule. All 14 benchmarks use and . Appendix § 9 documents the implementation details.
for each, giving adjusted scores of 78.30% (Table 3). Thus the winning score includes a measurable selection advantage.
| Raw Rank | Model | Raw Score (%) | Score Correction (pp) | 80% Rank Interval | Tier |
| 1–2 | live-SWE + Claude 4.5 Opus med. | 79.20 | 0.90 | [1,6] | 1 |
| 1–2 | Sonar + Claude 4.5 Opus | 79.20 | 0.90 | [1,6] | 1 |
| 3 | TRAE + Doubao-Seed-Code | 78.80 | 0.60 | [1,7] | 1 |
| 4 | live-SWE + Gemini 3 Pro | 77.40 | 0.69 | [2,13] | 1 |
| 5–7 | Atlassian Rovo Dev | 76.80 | 0.63 | [3,15] | 1 |
| 5–7 | EPAM AI/Run v20250719 + Claude 4 Sonnet | 76.80 | 0.63 | [3,15] | 1 |
| 5–7 | mini-SWE-agent + Claude 4.5 Opus (high) | 76.80 | 0.63 | [3,16] | 1 |
Table 3. RULER reveals score overstatement and unresolved ranks among SWE-bench Verified’s leaders.
In addition to score adjustment, RULER also suggests that the ranking is much less decisive than the raw scores suggest. The 2 leaders receive 30.5% and 29.0% of the bootstrap rank-1 mass, while the model at rank 3 receives 26.4%. Both leaders’ 80% rank intervals span positions 1–6, and the rank-3 model’s interval spans 1–7. Uncertainty extends farther down the leaderboard: the models tied at positions 9–10 each have an interval of [5,21] (Figure 1). The common-intersection rule groups 13 models into the leading tier, with rank 6 contained in every member’s interval, and the full leaderboard has 15 tiers in total.
These patterns extend beyond SWE-bench Verified: score overstatement and unresolved leaders appear across benchmark families (Table 4). On Tau2 Airline, the leader’s score falls by 3.07 pp after adjustment and 5 of 7 models share the leading tier. LiveBench instruction following has the largest adjustment, 3.86 pp, with 19 of 81 models in the leading tier. Both benchmarks contain only 50 tasks. Even a small correction can accompany an unresolved leader, as WildBench’s 0.09-pp adjustment and 2-model leading tier demonstrate. Across the audit, RULER therefore reveals both how much the winning score is overstated and how much ambiguity remains in the model ordering. Appendix § 9 explains how we apply the same procedure to each benchmark’s task-level scores.
Robustness Study
The score adjustment for SWE-bench Verified is robust to the i.i.d. task assumptions. Appendix § 11.§ 1 examines alternative sampling schemes for different assumptions on task distributions. Specifically, we compare pooled resampling with repository-stratified resampling, which approximately fixes the mix of repositories, and a cluster stress test, which allows tasks from the same repository to move together. Verified’s position-1 adjustment is 1.12 pp under pooled resampling, 1.20 pp under stratification, and 1.16 pp under clustering. Other benchmarks, such as Telecom, could be more sensitive to alternative assumptions on task distributions, as detailed in Appendix § 11.§ 1. We also examine the RULER’s robustness against different resample size for the -out-of- bootstrap and the retained rank mass used for model tiering. Across resample-size exponents from 0.4 to 0.9, score adjustments are stable and keep their direction across most rank positions. Increasing rank mass broadens the reported uncertainty, expanding Verified’s leading tier from 13 models at 80% to 15 at 90%, 24 at 95%, and 29 at 99%. Appendix § 11 gives the procedures and full results.
Conclusion
Public LLM leaderboards rank noisy estimates of model performance, thus the ranking rewards sampling luck alongside genuine ability. As a result, the winner’s score is overstated (the winner’s curse), scores throughout the leaderboard are distorted (rank-induced bias), and the ordering itself is uncertain. RULER addresses both the score distortion and the rank uncertainty. Its score adjustments correct each reported score for the selection effect of its position, and its RPM, rank intervals, and tiers show how firmly each model holds that position. Theorem 2.3 shows that the adjustment reduces rank-induced bias from to , and Proposition 2.4 shows that RULER’s tiering can recover the population model tiers. Across 14 benchmarks, rank-1 adjustments exceed 1 pp in 7 and the leading tier contains multiple models in 12 of the benchmarks. Score distortion and rank uncertainty are therefore common across the audited benchmark families. Because both arise from ranking noisy scores, they are likely to affect other leaderboards that compare closely matched models on limited task pools.
These findings have implications for model adopters, benchmark designers, and model developers. First, firms choosing a model should account for both the uncertainty in the ranking and the overstatement of top scores. A choice based on raw scores and ranks overstates the selected model’s expected performance and treats an unresolved ordering as settled, which can lead to suboptimal adoption decisions. Second, benchmark designers can reduce both problems by enlarging the task pool. Under the exact-tier approximation, halving expected winner inflation requires approximately 4 times as many tasks when the model pool and task distribution remain stable. Appendix § 12 provides a detailed sizing calculation. Finally, model developers who claim state-of-the-art performance should lead by a margin wider than the winner’s curse. On SWE-bench Verified, 7 of the 13 increases in the top score during 2025 were smaller than the estimated inflation of the current leaders. A confident claim needs a lead larger than the estimated inflation, and RULER’s tiers show whether the new leader is separated from its closest competitors.
Scope and Limitations. RULER describes what a specified benchmark can resolve about the evaluated models. Its score adjustments and rank summaries concern sampling tasks from that benchmark’s target population. Their relevance to deployment depends on the benchmark’s representativeness and the meaning of its score. The audit conditions on released responses and judge annotations, so variation from generating new responses or rerunning judges requires a separate assessment. Contamination, adaptive submissions, and selective reporting can introduce additional biases beyond task-sampling selection. RULER’s finite-sample performance also depends on the data-generating process. When mean differences are comparable to sampling noise, the correction can leave residual bias and may overcorrect.
Guide to the appendices. Appendix A uses a two-model example to build intuition for how the ability gap and resample size affect score correction and ranking uncertainty, and Appendix B uses simulations with known model means to guide the choice of resample size. Appendix C proves the main results. Appendix D documents data and implementation. Appendix E reports supplementary validation results, and Appendix F reports robustness studies. Appendix G provides supplementary RULER reports and task-pool planning, followed by related work in Appendix H.
Intuition from a Two-Model Example
With two models, the relation between the ability gap and sampling noise is easy to see: a near tie produces frequent changes in the winner, while a large gap makes the better model’s position more stable. The same example shows why very small resamples can obscure a real gap.
We use independent Bernoulli outcomes for two models on tasks, with probabilities (0.500, 0.505) or (0.500, 0.600). We generate 3,000 benchmark draws and use , with 2,000 bootstrap replicates per draw and seed 20260502. Multinomial sampling of the four joint-outcome cells is exactly equivalent in distribution to resampling complete binary task rows. Observed score ties are broken uniformly. Bootstrap RPM ties receive fractional mass. Bootstrap selection gaps are averaged over tied winners.
With a 0.5-pp ability gap, selection produces substantial score inflation and leaves the winner uncertain. The raw winning score averages 51.51%, while the selected model’s true ability averages 50.28% (Figure 2). Mean bias falls from 1.23 pp (Monte Carlo SE 0.034) to a residual of 0.22 pp (SE 0.037) after correction. Exact summation over the two independent binomial count distributions gives population raw bias 1.246 pp. The worse model wins in 43.4% of draws, and the better model receives only 54.1% rank-1 mass in the averaged bootstrap RPM. Thus the correction reduces inflation while the RPM continues to show an unresolved ordering.
Figure 2. Score correction reduces inflation while near-tied models remain hard to rank. Notes. Two models have independent Bernoulli outcomes with probabilities (0.500, 0.505) or (0.500, 0.600) on tasks. Each case uses 3,000 benchmark draws, , and 2,000 resamples per draw. Left: raw and adjusted winning scores for the 0.5-pp gap, with the selected model’s mean true ability marked by the dashed line. Middle and right: fractional-tie RPMs averaged over benchmark draws for the 0.5-pp and 10-pp gaps.
A 10-pp ability gap largely resolves the winner: the better model’s averaged rank-1 mass rises to 96.9%, and exact population winner bias falls to 0.0078 pp. The bootstrap nevertheless applies a mean adjustment of 0.176 pp, leaving a simulated mean residual of −0.238 pp (SE 0.043).5 example shows that the finite-sample correction can overshoot even when the winning model is well separated.
Smaller resamples make that separation harder to detect. An additional experiment uses , 3,000 draws, and 2,000 resamples per draw. For the 10-pp gap, the better model’s averaged rank-1 mass falls to 80.8%, compared with 96.9% at , and the mean adjustment rises to approximately 0.822 pp. Raw bias of 0.033 pp then becomes a residual of −0.789 pp. For the 0.5-pp gap, however, raw bias of 1.257 pp becomes a smaller residual of 0.048 pp. The examples expose the tradeoff: smaller resamples suppress apparent differences near a tie but can also obscure genuine ability gaps. The experiments use different random streams, so the comparison across resample sizes is illustrative rather than paired.
Guidance on Choosing the Resample Size for the -out-of- Bootstrap
The theory requires but does not fix the resample size for a given benchmark. We write with exponent , so is the standard bootstrap. The two-model example in Appendix § 6 shows what the exponent trades off: smaller resamples suppress apparent differences near a tie but can also obscure genuine ability gaps. The true means of the audited models are unknown, so the audit cannot show which exponent is most accurate. A simulation with known model means measures that accuracy directly. The most accurate exponent depends mainly on how closely model abilities are spaced and on the number of models. When abilities are closely spaced, every exponent from 0.4 to 0.9 is nearly as accurate as the best, and the default is within 8% of the most accurate exponent unless all models are tied.
Each synthetic benchmark has models and tasks with independent Bernoulli outcomes. The true model means are evenly spaced over an interval of width centered at , where is the standard error of one model’s score. Thus is both the mean and the median ability. In the tied design (), every model has mean . In the close design (), the means span 2 standard errors. In the spread design (), they span 15. The width does not change with , so larger leaderboards pack their models more densely. The grid crosses , , and with 11 exponents . Each design uses 1,000 synthetic benchmarks and resamples. Every exponent is applied to the same benchmarks, so comparisons across exponents are paired.
For each synthetic benchmark, the error at position is the corrected score at that position minus the true mean of the model observed there. We average its square over all positions and over benchmarks and take the square root to obtain the root-mean-square error (RMSE). Table 5–Table 7 report it for every exponent. In each column, the 3 smallest displayed values are shaded from dark to light, and equal displayed values share a shade, so a long shaded block marks exponents with nearly equal accuracy. We call the exponents tied for the smallest displayed RMSE in a column the most accurate.
| \multicolumn{6}{c}{} | \multicolumn{6}{c}{} | |||||||||||
| 2 | 5 | 10 | 20 | 50 | 100 | 2 | 5 | 10 | 20 | 50 | 100 | |
| tasks | ||||||||||||
| 0.1 | 1.87 | 1.35 | 1.05 | 0.87 | 0.81 | 0.83 | 1.64 | 1.23 | 1.05 | 0.94 | 0.91 | 0.94 |
| 0.2 | 1.87 | 1.35 | 1.03 | 0.79 | 0.64 | 0.63 | 1.64 | 1.20 | 0.97 | 0.82 | 0.75 | 0.76 |
| 0.3 | 1.88 | 1.35 | 1.03 | 0.77 | 0.53 | 0.45 | 1.64 | 1.18 | 0.92 | 0.70 | 0.55 | 0.51 |
| 0.4 | 1.88 | 1.35 | 1.03 | 0.77 | 0.51 | 0.39 | 1.65 | 1.18 | 0.91 | 0.68 | 0.47 | 0.39 |
| 0.5 | 1.89 | 1.36 | 1.03 | 0.77 | 0.51 | 0.38 | 1.65 | 1.18 | 0.92 | 0.67 | 0.46 | 0.36 |
| 0.6 | 1.91 | 1.37 | 1.05 | 0.78 | 0.52 | 0.39 | 1.67 | 1.19 | 0.92 | 0.68 | 0.46 | 0.35 |
| 0.7 | 1.93 | 1.40 | 1.07 | 0.80 | 0.55 | 0.42 | 1.69 | 1.22 | 0.94 | 0.70 | 0.48 | 0.37 |
| 0.8 | 1.98 | 1.46 | 1.12 | 0.85 | 0.60 | 0.48 | 1.72 | 1.27 | 0.98 | 0.74 | 0.53 | 0.43 |
| 0.9 | 2.04 | 1.55 | 1.21 | 0.95 | 0.71 | 0.60 | 1.77 | 1.35 | 1.06 | 0.83 | 0.63 | 0.53 |
| 0.95 | 2.08 | 1.62 | 1.27 | 1.02 | 0.79 | 0.69 | 1.80 | 1.41 | 1.11 | 0.89 | 0.70 | 0.61 |
| 1 | 2.12 | 1.69 | 1.35 | 1.11 | 0.89 | 0.79 | 1.83 | 1.47 | 1.18 | 0.96 | 0.78 | 0.70 |
| tasks | ||||||||||||
| 0.1 | 0.90 | 0.67 | 0.53 | 0.44 | 0.41 | 0.41 | 0.80 | 0.59 | 0.52 | 0.47 | 0.46 | 0.47 |
| 0.2 | 0.90 | 0.66 | 0.52 | 0.39 | 0.29 | 0.26 | 0.79 | 0.57 | 0.46 | 0.38 | 0.33 | 0.32 |
| 0.3 | 0.90 | 0.66 | 0.51 | 0.39 | 0.26 | 0.20 | 0.79 | 0.56 | 0.45 | 0.34 | 0.25 | 0.21 |
| 0.4 | 0.90 | 0.66 | 0.52 | 0.39 | 0.26 | 0.19 | 0.79 | 0.56 | 0.45 | 0.34 | 0.23 | 0.18 |
| 0.5 | 0.90 | 0.67 | 0.52 | 0.39 | 0.26 | 0.19 | 0.80 | 0.57 | 0.45 | 0.34 | 0.23 | 0.17 |
| 0.6 | 0.91 | 0.67 | 0.52 | 0.39 | 0.26 | 0.19 | 0.80 | 0.57 | 0.45 | 0.34 | 0.23 | 0.17 |
| 0.7 | 0.92 | 0.68 | 0.53 | 0.40 | 0.27 | 0.20 | 0.81 | 0.58 | 0.46 | 0.35 | 0.23 | 0.18 |
| 0.8 | 0.94 | 0.71 | 0.55 | 0.42 | 0.29 | 0.22 | 0.83 | 0.60 | 0.47 | 0.36 | 0.25 | 0.20 |
| 0.9 | 0.97 | 0.76 | 0.59 | 0.47 | 0.35 | 0.28 | 0.87 | 0.64 | 0.51 | 0.41 | 0.30 | 0.25 |
| 0.95 | 1.00 | 0.80 | 0.63 | 0.51 | 0.39 | 0.33 | 0.89 | 0.67 | 0.54 | 0.44 | 0.34 | 0.29 |
| 1 | 1.02 | 0.84 | 0.68 | 0.56 | 0.45 | 0.39 | 0.91 | 0.71 | 0.58 | 0.49 | 0.39 | 0.34 |
Table 5. When all models are tied, exponents below the default minimize error, and their advantage over the default grows with the number of models.
| \multicolumn{6}{c | }{} | \multicolumn{6}{c | }{} | |||||||||
| 2 | 5 | 10 | 20 | 50 | 100 | 2 | 5 | 10 | 20 | 50 | 100 | |
| tasks | ||||||||||||
| 0.1 | 2.47 | 2.81 | 2.81 | 2.64 | 2.67 | 2.69 | 2.11 | 2.44 | 2.38 | 2.23 | 2.23 | 2.26 |
| 0.2 | 2.43 | 2.80 | 2.80 | 2.61 | 2.61 | 2.62 | 2.08 | 2.44 | 2.39 | 2.22 | 2.22 | 2.24 |
| 0.3 | 2.32 | 2.73 | 2.74 | 2.57 | 2.55 | 2.55 | 1.97 | 2.39 | 2.35 | 2.20 | 2.18 | 2.19 |
| 0.4 | 2.20 | 2.62 | 2.63 | 2.49 | 2.47 | 2.46 | 1.88 | 2.29 | 2.25 | 2.13 | 2.11 | 2.12 |
| 0.5 | 2.13 | 2.54 | 2.52 | 2.41 | 2.38 | 2.36 | 1.82 | 2.23 | 2.15 | 2.06 | 2.04 | 2.04 |
| 0.6 | 2.09 | 2.52 | 2.44 | 2.33 | 2.28 | 2.26 | 1.79 | 2.21 | 2.08 | 2.00 | 1.96 | 1.95 |
| 0.7 | 2.08 | 2.53 | 2.42 | 2.29 | 2.22 | 2.19 | 1.78 | 2.22 | 2.06 | 1.97 | 1.91 | 1.89 |
| 0.8 | 2.08 | 2.53 | 2.46 | 2.29 | 2.19 | 2.15 | 1.78 | 2.22 | 2.09 | 1.97 | 1.89 | 1.86 |
| 0.9 | 2.08 | 2.48 | 2.50 | 2.33 | 2.19 | 2.14 | 1.78 | 2.17 | 2.13 | 2.00 | 1.89 | 1.85 |
| 0.95 | 2.08 | 2.43 | 2.51 | 2.35 | 2.20 | 2.14 | 1.78 | 2.13 | 2.14 | 2.02 | 1.89 | 1.85 |
| 1 | 2.08 | 2.38 | 2.51 | 2.38 | 2.21 | 2.15 | 1.78 | 2.08 | 2.13 | 2.04 | 1.91 | 1.86 |
| tasks | ||||||||||||
| 0.1 | 1.29 | 1.41 | 1.40 | 1.35 | 1.37 | 1.37 | 1.12 | 1.20 | 1.20 | 1.16 | 1.16 | 1.17 |
| 0.2 | 1.29 | 1.42 | 1.41 | 1.35 | 1.34 | 1.34 | 1.13 | 1.22 | 1.21 | 1.17 | 1.16 | 1.16 |
| 0.3 | 1.26 | 1.40 | 1.40 | 1.34 | 1.33 | 1.32 | 1.11 | 1.21 | 1.21 | 1.16 | 1.14 | 1.14 |
| 0.4 | 1.21 | 1.37 | 1.37 | 1.32 | 1.31 | 1.30 | 1.06 | 1.18 | 1.18 | 1.14 | 1.12 | 1.12 |
| 0.5 | 1.16 | 1.32 | 1.32 | 1.27 | 1.26 | 1.25 | 1.01 | 1.14 | 1.14 | 1.10 | 1.08 | 1.08 |
| 0.6 | 1.12 | 1.28 | 1.26 | 1.21 | 1.20 | 1.19 | 0.97 | 1.10 | 1.08 | 1.05 | 1.03 | 1.03 |
| 0.7 | 1.09 | 1.29 | 1.22 | 1.17 | 1.14 | 1.13 | 0.95 | 1.11 | 1.05 | 1.01 | 0.98 | 0.98 |
| 0.8 | 1.09 | 1.30 | 1.23 | 1.16 | 1.11 | 1.09 | 0.95 | 1.12 | 1.06 | 1.00 | 0.96 | 0.94 |
| 0.9 | 1.09 | 1.29 | 1.26 | 1.17 | 1.11 | 1.08 | 0.95 | 1.10 | 1.08 | 1.01 | 0.96 | 0.94 |
| 0.95 | 1.09 | 1.26 | 1.27 | 1.19 | 1.11 | 1.08 | 0.95 | 1.08 | 1.09 | 1.02 | 0.96 | 0.94 |
| 1 | 1.09 | 1.23 | 1.26 | 1.20 | 1.12 | 1.08 | 0.95 | 1.05 | 1.09 | 1.04 | 0.97 | 0.94 |
Table 7. With well-separated models, large exponents minimize error. Notes. As in Table 5, except that the true means are evenly spaced over around , with .
When all models are tied, every observed difference is noise, and exponents well below the default give lower error (Table 5). The most accurate exponents lie between 0.1 and 0.7 and rise with , reaching 0.4–0.6 at . The default has an RMSE 8–18% above that of the most accurate exponent for and 47–57% above it at . The standard bootstrap is 12–110% above it.
With closely spaced models, error changes little across a wide range of exponents (Table 6). In every column, each exponent from 0.4 to 0.9 has an RMSE within 8% of the most accurate exponent.
| \multicolumn{6}{c}{ } | \multicolumn{6}{c}{ } | |||||||||||
| 2 | 5 | 10 | 20 | 50 | 100 | 2 | 5 | 10 | 20 | 50 | 100 | |
| tasks | ||||||||||||
| 0.1 | 2.30 | 1.78 | 1.54 | 1.42 | 1.39 | 1.40 | 1.98 | 1.56 | 1.39 | 1.33 | 1.31 | 1.32 |
| 0.2 | 2.31 | 1.79 | 1.54 | 1.38 | 1.31 | 1.30 | 1.99 | 1.56 | 1.36 | 1.27 | 1.22 | 1.22 |
| 0.3 | 2.32 | 1.80 | 1.54 | 1.38 | 1.27 | 1.23 | 2.01 | 1.56 | 1.34 | 1.22 | 1.13 | 1.11 |
| 0.4 | 2.33 | 1.80 | 1.54 | 1.38 | 1.26 | 1.21 | 2.01 | 1.56 | 1.34 | 1.21 | 1.10 | 1.06 |
| 0.5 | 2.34 | 1.81 | 1.55 | 1.38 | 1.26 | 1.20 | 2.02 | 1.57 | 1.34 | 1.20 | 1.09 | 1.05 |
| 0.6 | 2.35 | 1.81 | 1.55 | 1.38 | 1.25 | 1.20 | 2.03 | 1.57 | 1.34 | 1.20 | 1.08 | 1.04 |
| 0.7 | 2.36 | 1.82 | 1.55 | 1.38 | 1.25 | 1.19 | 2.03 | 1.58 | 1.35 | 1.20 | 1.08 | 1.03 |
| 0.8 | 2.38 | 1.85 | 1.57 | 1.39 | 1.25 | 1.19 | 2.05 | 1.61 | 1.36 | 1.22 | 1.09 | 1.03 |
| 0.9 | 2.40 | 1.91 | 1.62 | 1.43 | 1.28 | 1.22 | 2.06 | 1.67 | 1.40 | 1.25 | 1.12 | 1.06 |
| 0.95 | 2.40 | 1.96 | 1.66 | 1.47 | 1.32 | 1.25 | 2.06 | 1.70 | 1.44 | 1.29 | 1.15 | 1.09 |
| 1 | 2.39 | 2.01 | 1.71 | 1.52 | 1.37 | 1.30 | 2.06 | 1.75 | 1.48 | 1.33 | 1.19 | 1.13 |
| tasks | ||||||||||||
| 0.1 | 1.13 | 0.89 | 0.77 | 0.72 | 0.69 | 0.70 | 0.99 | 0.76 | 0.70 | 0.67 | 0.66 | 0.66 |
| 0.2 | 1.14 | 0.89 | 0.77 | 0.70 | 0.64 | 0.63 | 1.00 | 0.76 | 0.67 | 0.62 | 0.59 | 0.58 |
| 0.3 | 1.14 | 0.89 | 0.78 | 0.70 | 0.63 | 0.61 | 1.01 | 0.76 | 0.67 | 0.60 | 0.55 | 0.54 |
| 0.4 | 1.15 | 0.89 | 0.78 | 0.70 | 0.63 | 0.61 | 1.01 | 0.76 | 0.67 | 0.60 | 0.55 | 0.53 |
| 0.5 | 1.15 | 0.90 | 0.78 | 0.70 | 0.63 | 0.60 | 1.01 | 0.76 | 0.67 | 0.60 | 0.55 | 0.53 |
| 0.6 | 1.15 | 0.90 | 0.78 | 0.70 | 0.63 | 0.60 | 1.02 | 0.77 | 0.67 | 0.60 | 0.55 | 0.52 |
| 0.7 | 1.16 | 0.90 | 0.78 | 0.70 | 0.63 | 0.60 | 1.02 | 0.77 | 0.67 | 0.60 | 0.54 | 0.52 |
| 0.8 | 1.17 | 0.92 | 0.78 | 0.70 | 0.63 | 0.59 | 1.04 | 0.78 | 0.67 | 0.60 | 0.54 | 0.52 |
| 0.9 | 1.18 | 0.95 | 0.81 | 0.72 | 0.64 | 0.60 | 1.05 | 0.81 | 0.69 | 0.62 | 0.55 | 0.53 |
| 0.95 | 1.18 | 0.97 | 0.83 | 0.74 | 0.66 | 0.62 | 1.05 | 0.83 | 0.71 | 0.64 | 0.57 | 0.54 |
| 1 | 1.17 | 1.00 | 0.86 | 0.77 | 0.69 | 0.65 | 1.05 | 0.86 | 0.74 | 0.67 | 0.59 | 0.56 |
Table 6. With closely spaced models, error is nearly flat across a wide range of exponents. Notes. As in Table 5, except that the true means are evenly spaced over around , with .
variation.
Notes. Entries are the RMSE in pp of the corrected scores over all positions and 1,000 synthetic benchmarks with independent Bernoulli outcomes. Every model has true mean . Each correction uses and , and is the standard bootstrap. All exponents use the same benchmarks (seed 20261103). In each column, bold type with the darkest, medium, and lightest shading marks the smallest, second-smallest, and third-smallest displayed values. Monte Carlo standard errors are at most 0.04 pp.
The default is within 5% of the most accurate exponent for and within 8% for every , while the standard bootstrap is 4–13% above it.
When the means span a wide range, larger exponents are more accurate (Table 7). The most accurate exponents lie between 0.7 and 1, and the default is within 5% of the best. Small exponents blur genuine gaps between models, as in the two-model example in Appendix § 6.
The most accurate exponents depend mainly on and the spacing of model abilities. For each and design, the shaded exponents overlap across all 4 combinations of task count and success rate in the grid. The default is within 8% of the most accurate exponent in the close and spread designs. Smaller exponents improve on the default by more than 8% only in the tied design, where the gain grows with . An exponent near the default therefore suits leaderboards whose models range from closely to widely spaced, and smaller exponents pay off mainly when many models are tied.
The spacing of a real leaderboard can be gauged by dividing the range of observed scores by , where is the mean score. Sampling noise widens observed ranges, so this gauge overstates the true spacing. Even 10 tied models have an expected observed range of about 3 standard errors. The designs draw models independently. When outcomes on a task are positively correlated across models, score differences are less noisy than this gauge assumes, so it understates the separation. Measured this way, the 14 audited benchmarks span 2.7–51.4 standard errors (median 14.8). Their 10 leading models, or all models when there are fewer, span 0.6–14.3 standard errors and lie within 3 standard errors in 6 benchmarks. In these 6 benchmarks, the leading groups are as closely spaced as the tied or close design. Appendix F.2 examines the audited corrections over the exponents from 0.4 to 0.9.
Proofs of the Main Results
On a high level, to prove the correctness of the bootstrap procedure for all positions, we use a joint conditional central limit theorem to provide the common approximation for score bias and rank probabilities. Since positive gaps separate the population tiers with probabilities approaching one, the key arguments concern the ordering of Gaussian score errors within each tier. Below we first remark on the notations and then build the proofs step by step in this section.
Let denote the information in the observed sample. Given , the bootstrap draw vectors independently with replacement from the observed rows, and set
Write and for expectation and probability conditional on . For a vector , write for its -th largest coordinate among the indices in . In particular, is this order statistic of the Gaussian vector and (, resp.) is the -th largest component of those coordinates of (, resp.) in . The symbols and denote the Euclidean and maximum norms, respectively.
Joint normal approximation and moment bounds
Lemma C.1 (Joint CLT, bootstrap CLT, and moment bounds). It holds that . For an independent empirical -resample, in probability as . Moreover and .
Proof. The proof of original-sample central limit theorem is standard and follows from Cramér–Wold theorem. We next show that converges in conditional distribution almost surely, which implies conditional convergence in distribution in probability.
1. Almost sure convergence of distribution. Define the empirical covariance matrix by
By the strong law of large numbers, and on an which is of probability one. Fix . Note that the observed rows are fixed for fixed and any . Write and for expectation and covariance under bootstrap sampling. To highlight the dependence on and , below we write , and instead of , and . For , define the centered, normalized bootstrap row
Within each row of this triangular array, the vectors are independent. Since a bootstrap row is uniform on the observed rows,
Consequently,
Both a bootstrap row and the empirical mean belong to . Thus every possible bootstrap outcome satisfies
For any , the assumption implies that for all sufficiently large . For those , the Lindeberg sum is exactly zero:
Equations (11) and (12) verify the hypotheses of the ordinary multivariate Lindeberg–Feller theorem. Applying that theorem under the bootstrap law for this fixed data sequence gives that on :
2. Second moment bounds. Finally, for any random variable , the inequality gives
Independence of the original task rows implies, for each coordinate ,
Conditional independence of the bootstrap rows similarly gives
Upper bounding the maximum by the sum allows us to conclude the proof.
Bias at each leaderboard position
Proof of Proposition 2.2. We first control mistakes in the ordering between different population tiers. We then take expectations of the within-tier order statistic.
Let denote the largest absolute scaled score error. For , define the event
On , if , , and , then
Thus every observed score in a higher tier exceeds every observed score in a lower tier. By Markov’s inequality and Lemma C.1,
When , take to be the whole sample space. No gap or cross-tier probability bound is needed in that case.
Let us write
On , the index belongs to and occupies position within that tier. Every model in has the same population mean. Sorting its empirical means is therefore equivalent to sorting the coordinates of in that tier. Hence on , including when the empirical means tie. Both and are bounded by . Cauchy–Schwarz gives
Taking an order statistic is continuous in the maximum norm. Lemma C.1 and the continuous mapping theorem yield
Finally, we notice that so for all . A uniform bound on the second moment implied uniform integrability, so . Combining with (14), we concludes the proof. □
Bootstrap score correction
The following deterministic fact will control changes in the score of a selected model when the ranking is perturbed.
Lemma C.2 (Ordering fact). Let be a positive integer, and let . Write for the -th largest coordinate of , for . Put , and let be any permutation such that . Then
and
Both conclusions hold for every way of ordering tied coordinates.
Proof. Set . At least coordinates of are at least . The corresponding coordinates of are at least , so . Also, at least coordinates of are at most . Their coordinates in are at most , so . This proves (15), including in the presence of ties.
Since sorts , its selected value satisfies , regardless of which tied index occupies rank . Consequently,
and the triangle inequality gives
This argument compares sorted values directly. Its bound does not depend on how many coordinates change rank. □
Proof of Theorem 2.3. We first replace the average over resamples by its conditional mean. We then show that this mean estimates the same constant as in Proposition 2.2, and finally bound the error from using finitely many resamples. As the first step, let us consider sorting . Define and we first study its properties. Also for convenience we let denote the largest absolute bootstrap scaled score error and similarly .
1. Convergence results for . Lemma C.1 and continuity of the order-statistic map give
by the continuous mapping theorem for weak convergence in probability. For each such sequence, . Therefore, applying the uniform integrability for weak convergence in probability, it holds that
Further, we note that
Dominated convergence therefore gives
2. Cross-tier mistakes in the bootstrap. Let be the event that every bootstrap score in a higher population tier exceeds every bootstrap score in a lower tier. For , define this event to be the whole bootstrap sample space. For , fix an original sample in the event from the proof of Proposition 2.2. Its empirical cross-tier score gaps exceed . If , every bootstrap mean differs from its empirical mean by less than . Each bootstrap cross-tier gap then exceeds . Markov’s inequality gives, for these original samples,
Thus the conditional probability of a cross-tier mistake tends to zero on uniformly.
3. Convergence of the bootstrapped mean. For a single bootstrap resample, define
where is index of the -th largest coordinate of . Define the empirical mean drift, on the bootstrap scale, by
The moment bound in Lemma C.1 gives
In particular, tends to zero in and in probability under original sampling. For , the bootstrap mean decomposes exactly as
Consequently, within , sorting the bootstrap means is equivalent to sorting . Put
On , the globally selected index belongs to and . Apply (16) with and perturbation . It gives
Off , both and are at most . When , for an original sample in , conditional Cauchy–Schwarz and (19) yield
For an original sample outside , the uniform bound
still holds. Taking expectations over the original data, we obtain for
Here , by (20). When , the cross-tier events always hold, and the same conclusion follows from the bound alone. Combining (23) with (18), we see that
Together with Proposition 2.2 and the identity in (25) below, this implies the second identity in (5).
4. Finitely many bootstrap resamples. As the final step, we prove the first identity in (5). (1) can be rewritten exactly as
These summands are conditionally i.i.d. with the conditional law of . Their conditional mean is , and each conditional variance is at most . Consequently,
In particular,
Together with (24), this proves the stronger conclusion
and hence also in probability. We conclude the proof.
Rank probabilities and tier recovery
Proof of Proposition 2.4. We first remark on a few facts. Recall that as argued in previous proofs,
Together, they imply that . For a fixed tier , define its tie set by
For a singleton tier this set is empty. For a nonsingleton tier, each is a centered normal variable with strictly positive variance, so
Suppose that . Let be the indicator that model occupies position when the coordinates in are sorted in decreasing order of with any fixed tie rule. Let be the tie-averaged weight used by the RPM: if ties with other coordinates in , the tied block occupies consecutive positions and at each of them. Both functions take values in . They coincide and are locally constant outside . Thus their discontinuity sets have Gaussian probability zero, and . Write for the within-tier rank of among , so that , and for the model at position when the original empirical means in are sorted with the fixed tie rule. For the -th resample, let be the rank weight of model at global position , with the tie averaging of (2), so that . Its conditional mean is the RPM with infinitely many resamples.
1. Original-sample rank probabilities. Within , all population means equal . Sorting the original empirical means in that tier is therefore exactly the same as sorting the coordinates of in that tier. Hence
Lemma C.1 gives . Boundedness and almost-sure continuity of under the Gaussian law then imply
On , the global and within-tier rankings agree after the position shift . Consequently,
This proves the first convergence in (6). Also, for ,
2. The conditional bootstrap rank probabilities. Define the empirical drift vector and the bootstrap errors centered at the population means by
By the original-sample moment bound,
Thus in probability. Together with the conditional bootstrap limit in Lemma C.1, this gives
This follows from a subsequence argument. From any subsequence, select a further subsequence along which almost surely. On almost every such original data sequence, the conditional CLT holds by Lemma C.1, and ordinary Slutsky convergence applies under that fixed conditional law. This establishes (28) in probability by the subsequence principle.
Within , sorting bootstrap means is equivalent to sorting . The same Gaussian-null discontinuity set in (27) therefore yields
On , the global bootstrap rank weight at equals this within-tier weight. Since all weights lie in ,
It follows that . For and ,
3. A finite number of bootstrap resamples. Write for variance conditional on . The weights are conditionally independent and identically distributed in . Thus
Conditional Chebyshev’s inequality and show that . Combining this with steps 1 and 2 proves all the limits in (6)–(7). For each , the deterministic quantity and the random quantity therefore converge to the same constant. Since is fixed, taking a maximum over the entries proves (8).
Rank intervals stay inside their tier and contain its center
Fix and put and . For , define the limiting cumulative rank probabilities by
The RPM limits and finiteness of imply
Since and , with probability tending to one,
By the definition of , these inequalities give
The Gaussian vector is centrally symmetric: . Since within-tier Gaussian ties have probability zero, negating all coordinates reverses their ranks. Therefore
Define the middle global rank or ranks of the tier by
When is odd, symmetry gives
and . When is even, symmetry gives . Consequently, in both cases,
This includes a singleton tier, for which . Because , convergence in (30) implies, with probability tending to one,
The quantile definition then gives
There are finitely many models and tiers, so the block inclusions in (31) and the center inclusions in (33) hold for all models simultaneously with probability tending to one. The argument allows zero probability at a middle rank. It uses the strict gap between the quantile thresholds and , rather than positivity of every Gaussian ordering probability.
5. The common-intersection rule recovers the partition. Consider the event on which and all the interval inclusions from step 4 hold. Its probability tends to one. On this event, the observed leaderboard visits as consecutive blocks, and no observed score tie contains models from two different population tiers.
Every interval belonging to contains . Hence every observed tie block within is compatible, and the running intersection remains nonempty while all models in that tier are added. The procedure cannot split a population tier. Each such interval also lies inside , while intervals from the next tier lie inside its disjoint rank block. Their intersection is empty, so the procedure closes the current tier at exactly the population boundary. It cannot merge two population tiers. Thus the reported ordered partition is exactly on this event, proving the claim.
Data, Implementation, and Reproducibility
This appendix explains how to construct a RULER report from task-level benchmark scores. It gives the reporting conventions, score definitions, snapshot sources, and computational details needed to reproduce the audit.
Resampling and reporting conventions
The full audit draws resamples of complete task rows with replacement. Every row contains the outcomes of all models on the same task, preserving their within-task dependence. The validation experiments use . The exponent is a predeclared, outcome-independent heuristic. Appendix B gives guidance on choosing it, and Appendix F.2 examines the sensitivity of the audit to it.
Score adjustments and observed ties. For the score correction in Equation (1), each replicate ranks models by decreasing resample score. To give models with equal observed scores a common displayed adjustment, we average the adjustments over the positions occupied by the tie. Verified’s displayed gap between positions 1 and 5 consequently falls from 2.40 to 2.13 pp after tie-averaged adjustments of approximately 0.90 and 0.63 pp.6
Rank probabilities and intervals. For the RPM, we average uniformly over tied ranks. If models tie at positions in a resample, each receives mass at each of those positions. The indicator in Equation (2) is averaged over these tie-breaking permutations before averaging over resamples. Each row and column of the full RPM sums to one. Models with equal observed scores can still have different RPM rows because they may solve different tasks and have different dependence with their competitors.
Rank intervals use the smallest integer rank whose cumulative mass reaches each quantile threshold. The implementation permits roundoff when comparing cumulative masses and identifying score ties. We report 80% central rank mass by default and 90%, 95%, and 99% in the sensitivity analysis. These intervals summarize the -resample distribution. Proposition 2.4 establishes its connection to population rank probabilities under exact tiers.
Tiers and leaderboard order. To form tiers, traverse models in raw-score order while keeping each observed score tie intact. A new block joins the current tier only if all its intervals and the current tier’s intersection share a rank. Requiring a common rank prevents a chain of pairwise overlaps from joining models whose intervals have no overall intersection. If the intervals inside an observed score tie have no common rank, the implementation stops with an error, because that block is incompatible with the tier summary. This did not occur in the audit. The report retains the raw ordering and displays the raw score, tie-averaged adjustment, rank interval, tier, and RPM. Adjusted scores are reported in that original order, even when their values would imply a different ordering.
Complete reports. The interactive RULER report supplies model identities, complete RPMs, all 3 interval levels, and tiers for every benchmark.
Benchmark scores and snapshot sources
For each benchmark, we form a matrix with tasks as rows and models as columns, using the task-level score defined in Table 8. A model’s benchmark score is the mean of its column. RULER resamples whole rows, so all models are evaluated on the same sampled tasks and the procedure preserves their dependence within each task. The resulting score adjustments and rank summaries describe uncertainty from sampling tasks. For judged outcomes, they condition on the released responses and judge annotations.
| Family | Score and sampling unit | Target population and interpretation |
| SWE-bench (5 benchmarks) | Binary issue resolution. Unit: issue. | Software issues under each variant’s task and scaffold rules, evaluated by the retained model-plus-agent systems. The analysis conditions on submitted trajectories. Model-plus-agent systems can differ across entries. |
| Tau2-Bench (3 domains) | Released task reward in . Unit: customer-service scenario. | Airline, retail, or telecom scenarios under the released agents and environment. Frozen outcomes retain the realized agent randomness and environment. |
| AlpacaEval 2 | Raw weighted rate: soft preference minus 1. Unit: instruction. | Instructions compared against a common GPT-4-Turbo reference, conditional on released responses and judge annotations. Instructions are equally weighted. The raw score differs from the length-controlled leaderboard metric. |
| WildBench v2 | Checklist-guided judge score divided by 10, in . Unit: prompt/session. | Real-user-style prompts under the frozen scoring protocol. Uncertainty concerns sampling prompts, conditional on the released judge scores. |
| LiveBench (4 benchmarks) | Released task score in . Unit: task/question. | Tasks in the retained model-coverage subset. We report Coding, Language, Instruction Following, and their task-weighted pooled mean. This subset and pooled score differ from the full live leaderboard. |
Table 8. Each RULER report refers to a specified score and task population.
SWE-bench and Tau2-Bench. For SWE-bench, a task is a software issue and its score is 1 if the evaluated model-plus-agent system resolves it and 0 otherwise. The column mean is therefore the issue-resolution rate. For SWE-bench Pro, we use the 395 tasks for which every system has a released record. For Tau2-Bench, a task is a customer-service scenario and its released reward lies in . RULER applies to these rewards in the same way, separately for Airline, Retail, and Telecom.
AlpacaEval 2. We analyze the raw weighted win rate against the common GPT-4-Turbo reference, rather than the length-controlled leaderboard score [16]. Each instruction has a soft judge preference in , from which we subtract 1 to obtain a score in . The “weighted” rate averages these soft preferences, with all valid instructions receiving equal weight. The official AlpacaEval scorer computes the same average. Each resampled row contains all models’ scores for the same instruction, preserving the shared reference and prompt.
WildBench. The task-level outcome is the released checklist-guided judge score for a prompt or session [24]. Dividing its 1–10 score by 10 puts it in . We average these normalized scores across the 988 prompts and use each prompt’s complete vector of model scores as a resampling row.
LiveBench. We use released task-level accuracy scores in [34]. The analyzed matrix contains 81 models on 318 tasks: 128 Coding, 140 Language, and 50 Instruction Following tasks. RULER reports each category separately and a pooled score that weights all 318 tasks equally. The pooled score therefore weights categories by their task counts and describes this analyzed subset, rather than the full LiveBench leaderboard metric.
Notes. The listed populations are modeling targets. Released tasks need not be random samples from them. Pooled analyses assume independent task rows from the target and resample each row’s complete model vector. Judged scores condition on the released responses and annotations.
Snapshot sources. Table 9 records the date and source of every benchmark snapshot. Table 4 gives the corresponding numbers of tasks and models.
| Benchmark | Snapshot date | Frozen source |
| SWE Verified | 2026-02-27 | Official leaderboard, 7c4289f, and task outcomes, 82be2b5 (2026-01-26). |
| SWE Bash-only | 2026-02-27 | Official leaderboard, 7c4289f, and task outcomes, 2f95aad (2026-02-27). |
| SWE Multimodal | 2026-02-27 | Official leaderboard, 7c4289f, and task outcomes, cff22d0 (2025-11-21). |
| SWE Multilingual | 2026-02-27 | Official leaderboard, 7c4289f, and task outcomes, 6083d31 (2026-02-24). |
| SWE Pro | 2026-03-05 | Docent per-task resolution records, collection 032fb63d-4992-4bfc-911d-3b7dafcb931f. |
| Tau2-Bench | 2026-04-10 | Released leaderboard outcomes from sierra-research/tau2-bench for Airline, Retail, and Telecom. |
| AlpacaEval 2 | 2025-08-09 | tatsu-lab/alpaca_eval, cd543a1, using weighted_alpaca_eval_gpt4_turbo annotations. |
| WildBench v2 | 2024-08-01 | allenai/WildBench, d6b8dca, release v2.0625, with judge gpt-4o-2024-05-13. |
| LiveBench | 2025-04-07 | Hugging Face livebench/model_judgment, 9704e5d, using the released leaderboard split. |
Table 9. Each benchmark audit is tied to an explicitly dated data source.
Computation. RULER reuses released task outcomes, so producing the report requires no language-model training, inference, or judge calls. Computation uses CPU-based Python with NumPy, SciPy, pandas, statsmodels, and Matplotlib. The full audit uses 5,000 resamples per benchmark with seed 42. The sensitivity sweep comprises 154 such runs, the split experiment uses 200 partitions per benchmark with 2,000 resamples each, and the calibrated comparison uses 100 synthetic benchmarks with 2,000 resamples per bootstrap method. Complete runtime and peak-memory measurements are not reported.
Notes. Dates for the 4 classic SWE-bench variants refer to the leaderboard revision, with outcome-revision dates shown separately. Pro and Tau2 dates identify the local snapshot retrievals. AlpacaEval, WildBench, and LiveBench dates identify the pinned upstream revisions. Revision timestamps are interpreted in UTC.
Supplementary Results for Validation
The held-out test and calibrated simulation in § 3 assess the correction using different reference scores. We describe their protocols and report the complete results below.
Sample-splitting protocol and results
Each of the 14 benchmarks uses 200 random partitions into a selection half of tasks and an evaluation half of tasks. SWE-bench splits balance repository representation. AlpacaEval uses 5 source-dataset strata and pooled LiveBench uses 3 categories. WildBench, individual LiveBench categories, and Tau2 use unstratified splits. When strata have odd sizes, the remaining assignments are randomized to reach the exact overall half-size. This approximately fixes the population mix but does not remove dependence among issues from the same repository. Every selection half uses and bootstrap resamples.
For benchmark , the signed mean errors and are the averages of and in Equation (9) over the 200 random partitions. Table 1 reports and . Averaging signed errors first isolates the mean selection error within each benchmark. Taking its magnitude before combining benchmarks prevents overstatement in one benchmark from canceling understatement in another.
On Verified, where each half contains 250 tasks, the mean winner error falls from 1.96 to 0.18 pp (Table 10). Table 11 gives all 70 benchmark–position combinations. Each residual is computed within its split before summarization. The split SE is the standard deviation across partitions divided by and measures Monte Carlo precision conditional on the frozen task pool.
| Rank | Mean Error Before RULER | Mean Score Correction | Mean Error After RULER |
| 1 | 1.96 | 1.79 | +0.18 |
| 2 | 0.66 | 1.38 | -0.72 |
| 3 | 0.72 | 1.24 | -0.52 |
| 4 | 1.41 | 1.16 | +0.26 |
| 5 | 1.52 | 1.08 | +0.44 |
Table 10. The correction substantially reduces SWE-bench Verified’s mean winner error.
| Benchmark | Rank | Error Before RULER | Score Correction | Error After RULER |
| SWE Bash-only | 1 | |||
| SWE Bash-only | 2 | |||
| SWE Bash-only | 3 | |||
| SWE Bash-only | 4 | |||
| SWE Bash-only | 5 | |||
| SWE Verified | 1 | |||
| SWE Verified | 2 | |||
| SWE Verified | 3 | |||
| SWE Verified | 4 | |||
| SWE Verified | 5 | |||
| SWE Multimodal | 1 | |||
| SWE Multimodal | 2 | |||
| SWE Multimodal | 3 | |||
| SWE Multimodal | 4 | |||
| SWE Multimodal | 5 | |||
| SWE Multilingual | 1 | |||
| SWE Multilingual | 2 | |||
| SWE Multilingual | 3 | |||
| SWE Multilingual | 4 | |||
| SWE Multilingual | 5 | |||
| SWE Pro | 1 | |||
| SWE Pro | 2 | |||
| SWE Pro | 3 | |||
| SWE Pro | 4 | |||
| SWE Pro | 5 | |||
| Tau2 Airline | 1 | |||
| Tau2 Airline | 2 | |||
| Tau2 Airline | 3 | |||
| Tau2 Airline | 4 | |||
| Tau2 Airline | 5 | |||
| Tau2 Retail | 1 | |||
| Tau2 Retail | 2 | |||
| Benchmark | Rank | Error Before RULER | Score Correction | Error After RULER |
| Tau2 Retail | 3 | |||
| Tau2 Retail | 4 | |||
| Tau2 Retail | 5 | |||
| Tau2 Telecom | 1 | |||
| Tau2 Telecom | 2 | |||
| Tau2 Telecom | 3 | |||
| Tau2 Telecom | 4 | |||
| Tau2 Telecom | 5 | |||
| AlpacaEval 2 | 1 | |||
| AlpacaEval 2 | 2 | |||
| AlpacaEval 2 | 3 | |||
| AlpacaEval 2 | 4 | |||
| AlpacaEval 2 | 5 | |||
| WildBench v2 | 1 | |||
| WildBench v2 | 2 | |||
| WildBench v2 | 3 | |||
| WildBench v2 | 4 | |||
| WildBench v2 | 5 | |||
| LiveBench pooled | 1 | |||
| LiveBench pooled | 2 | |||
| LiveBench pooled | 3 | |||
| LiveBench pooled | 4 | |||
| LiveBench pooled | 5 | |||
| LiveBench Coding | 1 | |||
| LiveBench Coding | 2 | |||
| LiveBench Coding | 3 | |||
| LiveBench Coding | 4 | |||
| LiveBench Coding | 5 | |||
| LiveBench Instr. | 1 | |||
| LiveBench Instr. | 2 | |||
| LiveBench Instr. | 3 | |||
| LiveBench Instr. | 4 | |||
| LiveBench Instr. | 5 | |||
| LiveBench Language | 1 | |||
| LiveBench Language | 2 | |||
| LiveBench Language | 3 | |||
| LiveBench Language | 4 | |||
| LiveBench Language | 5 |
Table 11. The reduction in selection error varies across benchmarks and ranks.
Notes. Results average 200 random half-splits with , , and . Error Before RULER is the selected model’s selection-half score minus its evaluation-half score. Score Correction is the estimated bias. Error After RULER subtracts that correction within each split. Entries are means across splits in percentage points (pp).
Notes. All 70 benchmark–position combinations use 200 random half-splits. Selection halves contain tasks, with and . Error Before RULER is the selected model’s selection-half score minus its evaluation-half score. Score Correction is the estimated bias. Error After RULER subtracts that correction within each split. Entries are means and standard errors (SEs) across splits. All scores and errors are in pp. SEs measure Monte Carlo precision over partitions of the same frozen task pool, conditional on that pool.
Calibrated simulation and alternative adjustments
Whereas the held-out test uses a second sample to evaluate selected models, the calibrated simulation makes their true marginal means available. We retain dependence across models through a shared task difficulty and generate outcomes as
with conditionally independent model outcomes given the shared task effect. The common effect produces cross-model dependence, while the marginal model means provide known targets for the selected-score errors.
A Bernoulli GLM with model and task effects is fitted to the 472 nonconstant rows of Verified. The 28 tasks with constant outcomes across models are omitted from fitting, while the empirical audit uses all 500 tasks. Task effects are centered and their mean absorbed into the model intercepts. Their standard deviation is . Marginal model means are
computed with 120-node Gauss–Hermite quadrature. Synthetic benchmark uses seed , and its -out-of- and standard bootstraps use seeds and , respectively. Sample splitting, described below, uses seed . Each method uses the same selected model at each raw rank on a given synthetic benchmark. Ties retain the original sort rule.
For the EB baseline, let . Fit positive by maximizing the beta-binomial marginal composite log-likelihood
where binomial coefficients are constant in . Optimize within using analytic gradients and two initializations: and a moment-based concentration split according to , with numerical clipping. Select the successful fit with the lowest negative log-likelihood. A bounded derivative-free fallback is provided if needed. The posterior mean is , so the adjustment for the selected model is minus that mean. This composite likelihood ignores cross-model dependence and imposes a beta population model. The comparison concerns this specified EB baseline.
Sample splitting can also be used to correct scores. The held-out test in § 3 estimates the half-sample bias with the split error in Equation (9). Because Proposition 2.2 shows that the leading term of the bias is proportional to , averaging over many random partitions and multiplying the average by gives a first-order estimate of . Subtracting this estimate from the score at position of the full leaderboard gives the corrected score. We use 200 random partitions of each synthetic benchmark into 2 halves of tasks, so the multiplier is . The correction has 2 limitations. First, a half-sample ranking is noisier than the full ranking and often places different models at a given position. The split error therefore reflects selection among tasks, and rescaling it to tasks corrects only the leading term of the bias. Second, each half contains only half of the tasks, so the estimate is more variable than the -out-of- adjustment. At rank 1, the sample-splitting adjustment has a standard deviation of 0.49 pp across synthetic benchmarks, compared with 0.29 pp for the -out-of- adjustment.
Table 12 reports the mean signed errors of all 4 corrections. The -out-of- correction leaves a mean winner residual of 0.21 pp, compared with 0.46 pp for the standard bootstrap and 1.10 pp for EB. Thus the 3 adjustments remove approximately 85%, 68%, and 22% of mean winner bias, respectively. Resampling tasks therefore removes more bias than resampling all .
| Rank | No Correction | -out-of- bootstrap | Standard bootstrap | EB | Sample splitting |
| 1 | |||||
| 2 | |||||
| 3 | |||||
| 4 | |||||
| 5 |
Table 12. The -out-of- correction and sample splitting leave the smallest mean winner residuals. Mean Signed Error (pp)
Notes. Entries are the mean signed error of the selected model’s score relative to its known mean, Monte Carlo SE, in pp over the 100 synthetic benchmarks of Table 2. No Correction gives the raw selection bias. The other columns give residuals after each correction. The -out-of- and standard bootstraps use and , respectively, with 2,000 resamples each. EB denotes empirical Bayes. It replaces each selected model’s score with its posterior mean under a beta-binomial model fitted to the benchmark. Sample splitting subtracts the split error averaged over 200 random half-splits and rescaled to tasks.
Notes. Entries are the root mean squared error (RMSE) of the selected model’s score relative to its known mean, Monte Carlo SE from the delta method, in pp over the synthetic benchmarks and corrections of Table 12.
Notes. Each entry is a method’s RMSE minus that of the -out-of- bootstrap over the same 100 synthetic benchmarks, delta-method Monte Carlo SE of the paired difference, in pp. Positive entries indicate a higher RMSE than the -out-of- bootstrap. EB denotes empirical Bayes.
Sample splitting leaves a mean residual of 0.06 pp, removing nearly all of the winner bias, but its adjustment is more variable and its winner RMSE is higher than that of the -out-of- correction. At rank 1, the -out-of- correction has the lowest RMSE, 1.55 pp, compared with 1.62 pp for the standard bootstrap, 1.64 pp for sample splitting, and 1.75 pp for EB (Table 13). It also has the lowest RMSE at positions 2–5 and ties EB at rank 2 to the displayed precision.
| Rank | No Correction | -out-of- bootstrap | Standard bootstrap | EB | Sample splitting |
| 1 | |||||
| 2 | |||||
| 3 | |||||
| 4 | |||||
| 5 |
Table 13. The -out-of- correction gives the winner’s score the lowest root mean squared error. RMSE (pp)
Notes. Methods use the same 100 synthetic benchmarks with 472 tasks and 173 models. The -out-of- and standard bootstraps use and , respectively, with 2,000 resamples. EB denotes empirical Bayes and uses the fitted beta-binomial posterior mean. Sample splitting uses 200 random half-splits per benchmark. Mean Score Correction is the mean amount subtracted from the selected model’s score. Mean Residual is its mean signed error after correction, and RMSE is the root mean squared error after correction. All 3 measures are in pp and use the selected model’s known mean as the reference. At rank 2, the -out-of- and EB RMSEs tie to the displayed precision.
Because all methods correct the same selected models on the same draws, their squared errors can be compared benchmark by benchmark. At rank 1, the paired RMSE difference from the -out-of- correction is 0.07 pp (SE 0.02) for the standard bootstrap, 0.09 pp (SE 0.03) for sample splitting, and 0.20 pp (SE 0.07) for EB (Table 14). At rank 2, EB and the -out-of- correction have the same RMSE to the displayed precision (SE of the paired difference 0.07).
| Rank | No Correction | Standard bootstrap | EB | Sample splitting |
| 1 | ||||
| 2 | ||||
| 3 | ||||
| 4 | ||||
| 5 |
Table 14. At rank 1, every alternative has a higher RMSE than the -out-of- correction on the same synthetic benchmarks. RMSE minus -out-of- RMSE (pp)
At rank 1, RMSE differs by at most 0.20 pp across methods, whereas mean residuals differ by up to 1.03 pp (Table 15). These results distinguish removing a systematic selection effect from eliminating sampling variation.
| Rank | Correction Method | Mean Score Correction (pp) | Mean Residual (pp) | RMSE (pp) |
| 1 | -out-of- bootstrap | 1.20 | +0.21 | 1.55 |
| 1 | Standard bootstrap | 0.95 | +0.46 | 1.62 |
| 1 | Beta-binomial EB | 0.32 | +1.10 | 1.75 |
| 1 | Sample splitting | 1.35 | +0.06 | 1.64 |
| 2 | -out-of- bootstrap | 0.86 | -0.06 | 1.74 |
| 2 | Standard bootstrap | 0.60 | +0.20 | 1.77 |
| 2 | Beta-binomial EB | 0.31 | +0.49 | 1.74 |
| 2 | Sample splitting | 0.82 | -0.02 | 1.77 |
| 3 | -out-of- bootstrap | 0.78 | -0.12 | 1.78 |
| 3 | Standard bootstrap | 0.53 | +0.13 | 1.80 |
| 3 | Beta-binomial EB | 0.30 | +0.36 | 1.79 |
| 3 | Sample splitting | 0.73 | -0.07 | 1.84 |
| 4 | -out-of- bootstrap | 0.75 | +0.23 | 1.64 |
| 4 | Standard bootstrap | 0.57 | +0.41 | 1.71 |
| 4 | Beta-binomial EB | 0.29 | +0.69 | 1.72 |
| 4 | Sample splitting | 0.82 | +0.16 | 1.73 |
| 5 | -out-of- bootstrap | 0.72 | +0.24 | 1.79 |
| 5 | Standard bootstrap | 0.56 | +0.40 | 1.87 |
| 5 | Beta-binomial EB | 0.28 | +0.68 | 1.85 |
| 5 | Sample splitting | 0.84 | +0.13 | 1.87 |
Table 15. The -out-of- correction has the lowest RMSE across the first 5 ranks.
Robustness Studies
The robustness studies distinguish three choices that affect a RULER report. First, we examine the i.i.d. tasks assumption. We try alternative bootstrap methods that relaxes the i.i.d. assumption and see whether the estimated selection effect and ranking uncertainty depend on how tasks are grouped. Second, we change the resample size within the range that Appendix § 7 recommends and see whether the estimated inflation is robust to the exponent. Finally, we give different rank mass use for model tiering.
Notes. Pooled resamples task rows, Stratified resamples within repositories, and Cluster applies a shared random multiplier to all tasks in each repository. Each method reports the position-1 score correction and the number of models in Tier 1 at 80% rank mass. All schemes use . Pooled resampling uses . Actual stratified resample sizes are 268 for Bash-only, 268 for Verified, 273 for Multimodal, 174 for Multilingual, and 217 for Pro. Cluster results use shared Webb multipliers and the unscaled selection gap, as specified in Appendix § 11.§ 1.
Beyond i.i.d. tasks
The theory in § 2 and the -out-of- bootstrap treat task rows as i.i.d. draws from a common task distribution. Real benchmarks can depart from this assumption in different ways. A benchmark may be assembled with a fixed mix of sources, such as the repositories of SWE-bench, so that a new draw of tasks would keep roughly the same composition. Tasks from the same source may also share features, such as a codebase, that move all models’ outcomes on those tasks together. Each pattern corresponds to a resampling scheme. Pooled resampling of task rows matches i.i.d. tasks, stratified resampling within repositories matches a fixed repository mix, and cluster resampling matches shared repository effects. If the three schemes give close score adjustments and rank summaries, RULER’s conclusions are robust to the i.i.d. task assumption.
Pooled resampling is the baseline. It draws independent rows from the empirical task distribution and uses the factor in Equation (1). For stratified resampling, repository with tasks receives . We sample independently with replacement within each repository, concatenate the sampled rows, and use the actual size in the rescaling. This approximately fixes repository composition while continuing to treat issues within a repository as independent. On Multilingual, rounding produces rather than the pooled .
For cluster resampling, we use a wild-cluster bootstrap. Define the repository residual vector and the perturbed means
Each cluster’s multiplier is shared across all its tasks and all models, so a draw shifts every task in a repository together. The Webb multipliers take the six values with equal probability. We rank the perturbed means and use the unscaled selection gap, which corresponds to in Equation (1). These pseudo-means can lie outside . The independent-task theorem does not establish the calibration of this procedure, and the small cluster counts—12 repositories for Verified and 3 issue subcategories for Tau2 Telecom—make these comparisons stress tests.
Across the 5 SWE-bench variants, pooled and stratified resampling give leading-tier sizes that differ by at most 1 model and score adjustments that differ by at most 0.27 pp (Table 16). Both summaries are therefore robust to a fixed repository mix. The cluster stress test can change rank summaries more substantially. On Verified, the pooled, stratified, and cluster position-1 adjustments are 1.12, 1.20, and 1.16 pp, respectively, while the leading tier contains 13 models under pooled resampling, 12 under stratified resampling, and 4 under clustering. Table 17 gives the corresponding rank-1 masses. Shared repository effects thus leave Verified’s estimated inflation nearly unchanged but can alter the reported ordering. Tau2 Telecom is more sensitive. Clustering its 3 labeled issue subcategories raises the adjustment from 0.18 to 1.05 pp and lowers the leader’s rank-1 mass from 94.3% to 75.5%. With only 3 clusters, this result serves as a sensitivity diagnostic.
| Benchmark | Number of Repositories | Position 1 Correction (pp) | Number of Models in Tier 1 | ||||
| Pooled | Stratified | Cluster | Pooled | Stratified | Cluster | ||
| SWE Bash-only | 12 | 1.09 | 1.21 | 0.86 | 6 | 6 | 4 |
| SWE Verified | 12 | 1.12 | 1.20 | 1.16 | 13 | 12 | 4 |
| SWE Multimodal | 12 | 0.94 | 0.82 | 0.84 | 5 | 5 | 5 |
| SWE Multilingual | 41 | 1.50 | 1.23 | 1.72 | 9 | 8 | 7 |
| SWE Pro | 11 | 0.87 | 0.87 | 0.54 | 4 | 4 | 3 |
Table 16. SWE-bench score adjustments remain stable across alternative sampling schemes.
| Benchmark | Resampling Scheme | Model 1 | Model 2 | Model 3 | Number of Tiers | Number of Models in Tier 1 |
| SWE Bash-only | Pooled | 41.6 | 16.8 | 18.4 | 10 | 6 |
| SWE Bash-only | Stratified | 43.1 | 15.5 | 18.1 | 10 | 6 |
| SWE Bash-only | Cluster | 57.4 | 18.1 | 15.9 | 12 | 4 |
| SWE Verified | Pooled | 30.5 | 29.0 | 26.4 | 15 | 13 |
| SWE Verified | Stratified | 29.0 | 29.4 | 26.2 | 15 | 12 |
| SWE Verified | Cluster | 30.6 | 29.1 | 33.8 | 24 | 4 |
| SWE Multimodal | Pooled | 34.8 | 34.0 | 22.0 | 3 | 5 |
| SWE Multimodal | Stratified | 34.8 | 34.3 | 22.2 | 3 | 5 |
| SWE Multimodal | Cluster | 31.8 | 37.5 | 27.0 | 3 | 5 |
| SWE Multilingual | Pooled | 44.8 | 33.6 | 9.1 | 4 | 9 |
| SWE Multilingual | Stratified | 45.5 | 35.2 | 9.0 | 4 | 8 |
| SWE Multilingual | Cluster | 51.9 | 34.1 | 7.9 | 4 | 7 |
| SWE Pro | Pooled | 74.9 | 21.1 | 1.9 | 6 | 4 |
| SWE Pro | Stratified | 74.7 | 21.3 | 1.8 | 6 | 4 |
| SWE Pro | Cluster | 87.6 | 10.8 | 0.3 | 6 | 3 |
Table 17. Repository clustering changes leading tiers in some SWE-bench variants. Notes. Models 1–3 are the first 3 models in the original displayed order, including any observed score ties. Each probability column gives the percentage of bootstrap rank mass assigned to rank 1 for that model. Pooled, Stratified, and Cluster use the sampling schemes and resample sizes defined in Table 16, with . The last 2 columns report the total number of tiers and the number of models in Tier 1 under the 80% common-intersection rule.
Choice of resample size
Appendix B shows that, averaged over all leaderboard positions, exponents from 0.4 to 0.9 are nearly equally accurate when models are closely spaced. We therefore recompute the position-1 adjustment of every audited benchmark at , using , , and seed 42. Table 18 reports the 84 adjustments, and Table 19 gives their exact resample sizes. The default was fixed before inspecting outcomes.
| Benchmark | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 |
| SWE Bash-only | 1.61 | 1.59 | 1.53 | 1.38 | 1.26 | 1.09 |
| SWE Verified | 1.91 | 1.90 | 1.77 | 1.56 | 1.34 | 1.12 |
| SWE Multimodal | 1.13 | 1.13 | 1.13 | 1.04 | 1.02 | 0.94 |
| SWE Multilingual | 2.10 | 2.16 | 2.15 | 1.94 | 1.74 | 1.50 |
| SWE Pro | 2.10 | 1.90 | 1.67 | 1.43 | 1.16 | 0.87 |
| Tau2 Airline | 6.13 | 5.75 | 5.40 | 4.80 | 4.00 | 3.07 |
| Tau2 Retail | 3.01 | 2.93 | 2.78 | 2.60 | 2.45 | 2.20 |
| Tau2 Telecom | 1.23 | 1.23 | 0.93 | 0.58 | 0.36 | 0.18 |
| AlpacaEval 2 | 1.33 | 1.13 | 0.93 | 0.77 | 0.64 | 0.53 |
| WildBench v2 | 0.35 | 0.30 | 0.24 | 0.17 | 0.13 | 0.10 |
| LiveBench pooled | 2.58 | 2.19 | 1.65 | 0.96 | 0.39 | 0.12 |
| LiveBench Coding | 4.12 | 3.42 | 2.77 | 2.13 | 1.53 | 1.11 |
| LiveBench Instr. | 4.02 | 4.09 | 4.15 | 4.17 | 4.09 | 3.86 |
| LiveBench Language | 4.07 | 3.62 | 3.03 | 2.53 | 2.09 | 1.90 |
Table 18. Score adjustments stay positive across resample-size exponents from 0.4 to 0.9, and the default gives the smallest. Resample-Size Exponent
| Benchmark | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 |
| SWE Bash-only | 12 | 22 | 41 | 77 | 144 | 268 |
| SWE Verified | 12 | 22 | 41 | 77 | 144 | 268 |
| SWE Multimodal | 12 | 22 | 42 | 78 | 146 | 273 |
| SWE Multilingual | 9 | 17 | 30 | 54 | 95 | 169 |
| SWE Pro | 10 | 19 | 36 | 65 | 119 | 217 |
| Tau2 Airline | 4 | 7 | 10 | 15 | 22 | 33 |
| Tau2 Retail | 6 | 10 | 17 | 27 | 44 | 70 |
| Tau2 Telecom | 6 | 10 | 17 | 27 | 44 | 70 |
| AlpacaEval 2 | 14 | 28 | 55 | 108 | 211 | 412 |
| WildBench v2 | 15 | 31 | 62 | 124 | 248 | 495 |
| LiveBench pooled | 10 | 17 | 31 | 56 | 100 | 178 |
| LiveBench Coding | 6 | 11 | 18 | 29 | 48 | 78 |
| LiveBench Instr. | 4 | 7 | 10 | 15 | 22 | 33 |
| LiveBench Language | 7 | 11 | 19 | 31 | 52 | 85 |
Table 19. Resample sizes range from 4 to 495 tasks across exponents from 0.4 to 0.9. Resample-Size Exponent
Across this range, the correction keeps its direction, and the default is conservative. Every adjustment is positive, and in each of the 14 benchmarks the default gives the smallest adjustment in the range. A smaller exponent in the range would therefore estimate more inflation at the top of each leaderboard than the audit reports. In 12 benchmarks, including the 6 whose 10 leading models lie within 3 standard errors, the adjustment also keeps its order of magnitude across the range. Tau2 Telecom and pooled LiveBench show by far the largest relative changes.
These 2 benchmarks have well-separated leaders, which receive 94.3% and 99.0% of the bootstrap rank-1 mass at the default, and their small default adjustments grow several-fold at smaller exponents. For a well-separated leader, the spread design in Appendix B shows that smaller exponents are less accurate, and the two-model example in Appendix A shows that they overcorrect. The simulations Notes. Entries are position-1 adjustments in pp for 14 benchmarks and 6 exponents. Each run uses , , and seed 42. Score ties follow the deterministic sort convention in Appendix D. The default exponent is .
Notes. Entries are the exact resample sizes for the runs in Table 18. Column headings give , and the default exponent is 0.9.
therefore favor the small default adjustments for these 2 leaders.
Choice of rank-interval level
Retaining more rank mass gives a broader summary of the unresolved ordering. On Verified, increasing central rank mass from 80% to 90%, 95%, and 99% expands the leading tier from 13 models to 15, 24, and 29, respectively. Table 20 reports the corresponding partitions in every Notes. The 2 column groups separately report the total number of tiers and the number of models in Tier 1. Within each group, the percentage headings specify the central rank mass retained in each model’s interval. Partitions are recomputed from the same full-precision RPMs, using , , and the common-intersection rule. Only the retained rank mass changes across columns.
| Benchmark | Number of Tiers | Number of Models in Tier 1 | ||||||
| 80% | 90% | 95% | 99% | 80% | 90% | 95% | 99% | |
| SWE Bash-only | 10 | 9 | 7 | 6 | 6 | 9 | 13 | 16 |
| SWE Verified | 15 | 13 | 11 | 8 | 13 | 15 | 24 | 29 |
| SWE Multimodal | 3 | 3 | 2 | 2 | 5 | 5 | 8 | 10 |
| SWE Multilingual | 4 | 3 | 3 | 3 | 9 | 11 | 12 | 12 |
| SWE Pro | 6 | 6 | 4 | 4 | 4 | 4 | 5 | 5 |
| Tau2 Airline | 2 | 1 | 1 | 1 | 5 | 7 | 7 | 7 |
| Tau2 Retail | 2 | 2 | 2 | 2 | 5 | 6 | 6 | 6 |
| Tau2 Telecom | 6 | 4 | 3 | 2 | 1 | 2 | 3 | 3 |
| AlpacaEval 2 | 21 | 16 | 14 | 11 | 3 | 4 | 4 | 5 |
| WildBench v2 | 15 | 12 | 12 | 10 | 2 | 3 | 3 | 6 |
| LiveBench pooled | 9 | 7 | 7 | 5 | 1 | 1 | 1 | 6 |
| LiveBench Coding | 8 | 6 | 5 | 4 | 2 | 3 | 5 | 9 |
| LiveBench Language | 7 | 6 | 5 | 4 | 4 | 5 | 7 | 15 |
| LiveBench Instr. | 5 | 4 | 4 | 3 | 19 | 26 | 30 | 42 |
Table 20. Leading tiers often expand when the retained rank mass increases.
benchmark. The underlying RPM is unchanged. The difference is how much of each model’s rank distribution the interval includes.
Wider intervals can change where the common-intersection sweep closes a tier, so the resulting partitions need not form a nested hierarchy. We therefore recompute each level from the full-precision RPM using the same reporting conventions (Appendix § 9).
Benchmark Planning and Supplementary Reports
The score adjustment provides a scale for planning larger task pools, and the full RPM reveals how ranking uncertainty extends beyond the models highlighted in the main text. This appendix develops the planning calculation and provides larger rank-distribution displays for Verified and Tau2-Bench.
Task-pool sizing for a target level of score inflation
More tasks reduce the expected selection advantage of the winner. In the exact-tier regime, , so halving expected inflation requires approximately 4 times as many tasks. For a target expected bias , the corresponding planning rule is .
For Verified, the position-1 adjustment gives pp. Table 21 translates this estimate into task counts for different targets. The calculation holds the model pool, task distribution, and covariance structure fixed and uses the exact-tier approximation. Local gaps or changes in these populations can alter the extrapolation, so the table gives approximate planning sizes rather than guaranteed minima.
| Target Expected Winner Bias (pp) | Approximate Number of Tasks |
| 2.00 | 158 |
| 1.00 | 631 |
| 0.50 | 2,522 |
| 0.25 | 10,086 |
| 0.10 | 63,035 |
Table 21. Under the exact-tier approximation, halving expected winner inflation requires about 4 times as many tasks.
Notes. Planning sizes use with from Verified’s position-1 estimate. Counts are rounded upward using full precision. The exact-tier approximation holds the model pool, task distribution, and covariance structure fixed. Values are approximate planning sizes.
Figure 3. Ranking uncertainty extends beyond SWE-bench Verified’s leading models.
Notes. The display is the leading submatrix of the full RPM, based on 500 tasks, , and . Red horizontal lines mark 80% tier boundaries in raw model order. Cropped rows need not sum to one, while full RPM rows do. Axis indices identify raw-order model rows without imposing an order within score ties.
Additional rank probability matrices
The larger RPM places Verified’s leading tier in the context of the first 30 models and rank positions. Uncertainty extends beyond the observed leaders: the models tied at raw positions 9–10 each have an 80% rank interval of [5,21]. The display shows where rank mass is concentrated and how far it spreads across the leaderboard.
The Tau2 domains illustrate different degrees of ranking resolution. Airline and Retail each place 5 of 7 models in the leading tier, and Telecom’s leader forms a tier of its own under pooled resampling. Figure 4 shows the RPMs behind those tier summaries. Red lines mark the 80% tier boundaries. Airline has only tasks, and each resample draws of them. Telecom’s sensitivity to clustering is discussed in § 4.2.
Figure 4. Tau2 domains differ in how sharply they resolve the leading tier.
Notes. The RPMs use pooled task resampling with , , and , with tasks for Airline and for Retail and Telecom. Red lines mark 80% tier boundaries from the common-intersection rule. In panels (a) and (b), 5 of 7 models share the leading tier. In panel (c), the leading tier contains only the leader. § 4.2 reports Telecom’s sensitivity to issue-subcategory clustering.
Extended Related Work
Selection, comparison, and rank uncertainty. Rank-induced bias connects leaderboard interpretation to the broader problem of selection. The WC was documented in oil-lease auctions [10, 33] and formalized as the optimizer’s curse [32]. Selective inference studies inference after selection [22, 3], while empirical Bayes uses information across estimates for shrinkage [18, 20]. RULER builds on classical -out-of- resampling [8, 7] and its use for selected-winner correction [35]. It extends the correction to every leaderboard position while accounting for cross-model dependence within a task. The RPM and rank intervals complement the score correction by showing how much of the ordering the task pool resolves.
Statistical comparison predates LLM leaderboards. Koehn [21] studies significance testing for machine-translation evaluation, and Mathur et al. [25] examines how automatic evaluation metrics, outliers, and system comparisons affect conclusions about translation quality. Our target is the mean estimation error of the model selected by its position after ranking. Rank confidence sets [27, 26] and classifier-comparison procedures [15, 13, 6] address inferential comparisons. RULER complements these approaches by estimating the score error associated with selection and displaying the resampled ordering. Its rank-mass intervals summarize that distribution rather than provide simultaneous confidence coverage.
Validity and reproducibility. Raji et al. [29] questions the scope and meaning of general-purpose AI benchmarks. Belz et al. [5] develops quantified reproducibility assessment, extended across common result types by Belz and Thomson [4]. Schmidtova et al. [31] surveys automatic metric practices in natural-language generation. These concerns extend beyond task sampling: a precise score can measure the wrong capability, and implementations can vary independently of the task pool (Table 22).
| Concern | Question | Relationship to this paper |
| Statistical comparison | Does a prespecified difference exceed evaluation noise? | Selection after sorting changes which error is averaged. |
| Selection bias | How much does a chosen position benefit from noise? | Our bias target. RPMs additionally describe ranking variability. |
| Construct validity | Does the task pool measure the intended capability? | Our target is conditional on the chosen task population. |
| Reproducibility | How much do independently repeated implementations or studies agree? | Frozen outcomes do not capture all reproduction variability. |
| Metric validity | Does the score agree with the quality notion of interest? | Debiasing a score does not validate its underlying metric. |
Table 22. Selection bias is distinct from statistical comparison, metric validity, and reproducibility.
Other leaderboard failure modes. Leaderboards can also become unreliable through changes in the evaluation process or the models submitted to it. Adaptive holdout reuse ([9, 17, 30]) and contamination ([14]) can compromise an evaluation’s independence from model development. Private evaluation suites ([23]) can reduce leakage and adaptation. Benchmark and prompt sensitivity ([12, 1]), run-to-run variance (Bouthillier et al., 2021), and efficient or uncertainty-aware evaluation ([28, 36, 11, 2]) concern further sources of variability and the design of evaluations. Rank-induced bias remains even for a fixed, uncontaminated model pool evaluated once on held-out tasks, because sorting selects favorable noise along with ability.
References
- [1]Norah Alzhahrani, Hisham Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Tw aresh, Areeb Alowisheq, et al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13787–13805, 2024.
- [2]Siavash Ameli, Siyuan Zhuang, Ion Stoica, and Michael W. Mahoney. A statistical framework for ranking LLM-based chatbots. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=rAoEub6Nw2.arxiv.org/abs/2412.18407
- [3]Isaiah Andrews, Toru Kitagawa, and Adam McCloskey. Inference on winners. Quarterly Journal of Economics, 139(1):305–358, 2024.
- [4]Anya Belz and Craig Thomson. Quantified reproducibility assessment for four common types of evaluation results in NLP/ML. Computational Linguistics, 2026. doi: 10.1162/COLI.a.625.DOI
- [5]Anya Belz, Maja Popovic, and Simon Mille. Quantified reproducibility assessment of NLP results. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16–28, 2022. doi: 10.18653/v1/2022.acl-long.2.DOI
- [7]Peter J. Bickel and Anat Sakov. On the choice of m in the m out of n bootstrap and confidence bounds for extrema. Statistica Sinica, 18(3):967–985, 2008.
- [8]Peter J. Bickel, Friedrich Götze, and Willem R. van Zwet. Resampling fewer than n observations: Gains, losses, and remedies for losses. Statistica Sinica, 7(1):1–31, 1997.
- [9]Avrim Blum and Moritz Hardt. The ladder: A reliable leaderboard for machine learning competitions. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1006–1014, 2015. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. Accounting for variance in machine learning benchmarks. Proceedings of machine learning and systems, 3:747–769, 2021.
- [10]E. C. Capen, R. V. Clapp, and W. M. Campbell. Competitive bidding in high-risk situations. Journal of Petroleum Technology, 23(6):641–653, 1971.DOI
- [11]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning, 2024.arxiv.org/abs/2403.04132
- [12]Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. In Proceedings of the 35th Conference on Neural Information Processing Systems: Datasets and Benchmarks Track, 2021.arxiv.org/abs/2107.07002
- [13]Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(1):1–30, 2006.DOI
- [14]Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8706–8719, 2024.arxiv.org/abs/2311.09783
- [15]Thomas G. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895–1923, 1998. doi: 10.1162/089976698300017197.DOI
- [16]Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. URL https://arxiv.org/abs/2404.04475.
- [17]Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Tոնiann Pitassi, Omer Reingold, and Aaron Roth. Generalization in adaptive data analysis and holdout reuse. In Advances in Neural Information Processing Systems, volume 28, pages 2350–2358, 2015.arxiv.org/abs/1506.02629
- [18]Bradley Efron. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106(496):1602–1614, 2011.pubmed.ncbi.nlm.nih.gov/22505788
- [19]Zheng Fang and Andres Santos. Inference on directionally differentiable functions. Review of Economic Studies, 86(1):377–412, 2019.arxiv.org/abs/1404.3763
- [20]Jiaying Gu and Roger Koenker. Invidious comparisons: Ranking and selection as compound decisions. Econometrica, 91(1):1–41, 2023.arxiv.org/abs/2012.12550
- [21]Philipp Koehn. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395. Association for Computational Linguistics, 2004. URL https://aclanthology.org/W04-3250/.
- [22]Jason D. Lee, Dennis L. Sun, Yuekai Sun, and Jonathan E. Taylor. Exact post-selection inference, with application to the lasso. Annals of Statistics, 44(3):907–927, 2016.DOI
- [23]Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasუნaga, Yian Zhang, Deepak Narayanan, Yuhui Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022.arxiv.org/abs/2211.09110
- [24]Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. WildBench: Benchmarking LLMs with challenging tasks from real users in the wild. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=MKEHCx25xp.arxiv.org/abs/2406.04770
- [25]Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, 2020. doi: 10.18653/v1/2020.acl-main.448.DOI
- [26]Magne Mogstad, Joseph P. Romano, Azeem M. Shaikh, and Daniel Wilhelm. Statistical uncertainty in the ranking of journals and universities. AEA Papers and Proceedings, 112:630–634, 2022.
- [27]Magne Mogstad, Joseph P. Romano, Azeem M. Shaikh, and Daniel Wilhelm. Inference for ranks with applications to mobility across neighbourhoods and academic achievement across countries. Review of Economic Studies, 91(1):476–518, 2024.
- [28]Felipe Maia Polo, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun, and Mikhail Yurochkin. Efficient multi-prompt evaluation of llms. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 22483–22512. Curran Associates, Inc., 2024. doi: 10.52202/079017-0707. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/28236482f64a72eec43706b6f3a6c511-Paper-Conference.pdf.DOI
- [29]Iniolụwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.15366, 2021. URL https://arxiv.org/abs/2111.15366.
- [30]Rebecca Roelofs, Vaishaal Shankar, Benjamin Recht, Sara Fridovich-Keil, Moritz Hardt, John Miller, and Ludwig Schmidt. A meta-analysis of overfitting in machine learning. In Advances in Neural Information Processing Systems, volume 32, 2019.
- [31]Patricia Schmidtova, Saad Mahamood, Simone Balloccu, Ondrej Dusek, Albert Gatt, Dimitra Gkatzia, David M. Howcroft, Ondrej Platek, and Adarsa Sivaprasad. Automatic metrics in natural language generation: A survey of current evaluation practices. In Proceedings of the 17th International Natural Language Generation Conference, pages 557–583, 2024. doi: 10.18653/v1/2024.inlg-main.44.DOI
- [32]James E. Smith and Robert L. Winkler. The optimizer’s curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52(3):311–322, 2006.DOI
- [33]Richard H. Thaler. Anomalies: The winner’s curse. Journal of Economic Perspectives, 2(1):191–202, 1988.DOI
- [34]Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddhartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. LiveBench: A challenging, contamination-free LLM benchmark. arXiv preprint arXiv:2406.19314, 2024. URL https://arxiv.org/abs/2406.19314.
- [35]Sikun Xu, Raphael Thomadsen, and Dennis Zhang. The winner’s curse in data-driven decision making: Evidence and solutions. SSRN Working Paper, 2025. Available at SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5930537.DOI
- [36]Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F. Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking LLMs via uncertainty quantification. In Advances in Neural Information Processing Systems: Datasets and Benchmarks Track, volume 37, 2024.
- [37]Hua Zhong and Ross L. Prentice. Bias-reduced estimators and confidence intervals for odds ratios in genome-wide association studies. Biostatistics, 9(4):621–634, 2008.DOI
- [38]Sebastian Zöllner and Jonathan K. Pritchard. Overcoming the winner’s curse: Estimating penetrance parameters from case-control data. American Journal of Human Genetics, 80(4):605–615, 2007.



