betaFeedback

WorksComputer Science

SOTA or Luck? The Winner's Curse in LLM Leaderboards

Abstract

Public LLM leaderboards guide model evaluation and adoption, yet the reported performance of top-ranked models can be systematically overstated. This leaderboard winner's curse arises because ranking noisy benchmark scores rewards sampling luck alongside genuine ability. The same selection effect can distort scores throughout the leaderboard, a phenomenon we call the rank-induced bias. We develop an mm-out-of-NN bootstrap to correct for this bias. It assumes a fixed model pool with independent, identically distributed bounded tasks while allowing arbitrary within-task correlation among models. Under exact population tiers separated by positive gaps, we prove that the bootstrap reduces Θ(N−1/2)Θ(N^{-1/2}) ranking bias to expected residual bias o(N−1/2)o(N^{-1/2}). Across 14 benchmarks from 5 benchmark families, the rank-1 adjustments range from approximately 0.09 to 3.86 percentage points. In our SWE-bench Verified snapshot, the tied leaders receive an adjustment of 0.90 percentage points. Such overstatement is large enough to make the apparent winner statistically fragile. Thus, we use the same bootstrap to further quantify the intervals of plausible rank positions for each model and group models into tiers. We establish sufficient conditions for consistent recovery of population tiers. On SWE-bench Verified, this procedure places 13 of 173 models in the leading tier. We combine the score adjustment and rank intervals in the Rank Uncertainty and Leaderboard Evaluation Report (RULER). We validate its finite-sample performance using sample splitting and calibrated simulations, and advocate publishing RULER alongside raw rankings so readers can assess both reported performance and the precision of the ordering. RULER also provides guidance for benchmark design that accounts for selection effects and ranking uncertainty.

Introduction

Public leaderboards are central to the evaluation of large language models. Organizations use benchmark rankings to choose models, researchers to claim state-of-the-art performance, and funding bodies to assess progress. Yet benchmark scores are no more than sample averages over a small set of tasks, and sample averages inherit the randomness of the underlying data. Thus, selecting the model with the highest observed score rewards sampling luck as well as genuine ability. As a result, the true performance of the winning model tends to be overestimated. This overestimation is the Winner’s Curse (WC), a phenomenon well understood in auction theory [10], genome-wide association studies [38], and clinical trials [37], but largely unexamined in LLM evaluation. The consequences extend beyond statistical reporting: leaderboard rankings influence model procurement, deployment spending, and research investment, yet these decisions often rely on point rankings without accounting for the uncertainty behind small score differences. As a result, economic resources can be directed toward apparent winners whose performance advantages are not reliably established.

We also stress that LLM leaderboards can inherit the WC dynamics at a much larger scale: the number of competing models has exploded, capability gaps at the top have narrowed, and new State-Of-The-Art (SOTA) claims arrive weekly based on fractional-percentage-point improvements. These are precisely the conditions under which the WC is at its worst [35]. Additionally, the same selection effect extends throughout a leaderboard: among models with the same mean performance, scores near the top tend to be inflated and those near the bottom deflated. We call this rank-induced bias. We develop an mm-out-of-NN bootstrap [8, 35] to (1) estimate and correct this bias and (2) quantify ranking uncertainty by resampling tasks and re-ranking models altogether. We package these results in a Rank Uncertainty and Leaderboard Evaluation Report (RULER): score adjustments for selection, a rank probability matrix (RPM) showing how often each model occupies each rank, and rank intervals used to group models into tiers. We advocate publishing RULER alongside raw rankings so readers can assess both the reported performance and the precision of model ranking.

Existing work on leaderboard reliability does not correct rank-induced bias. The literature has studied adaptive test-set reuse [9, 17, 30], benchmark lottery and prompt sensitivity [12, 1], run-to-run variance (Bouthillier et al., 2021), item-level and pairwise uncertainty [28, 36, 11], and data contamination [14]. There is a separate research direction concerning the vendors tuning models to public benchmarks, which has motivated private evaluation suites [23]. Because rank-induced bias comes from sampling noise, it persists on private benchmarks too. Our work also relates to the post-selection inference literature. Empirical Bayes methods shrink estimates toward a common prior, often under parametric assumptions [18, 20]. Selective inference conditions on the selection event and requires a separate derivation for each selection procedure [22, 3]. The mm-out-of-NN bootstrap [8] needs neither. It requires only i.i.d. bounded tasks, allows arbitrary correlation among models within a task, and yields the score correction and the RPM from a single set of resamples. Xu et al. [35] use it to correct the score of a single selected winner with independent alternatives in inference-then-optimize settings. Appendix H reviews further related work.

We apply RULER to SWE-Bench Verified, which evaluates 173 models on 500 software engineering tasks at the time of our snapshot, the 2 tied leaders score 79.2%. RULER estimates that the score is inflated by 0.9 percentage points, giving adjusted scores of 78.3%. This inflation exceeds the typical margin by which a new entry takes the lead. In 2025, the top score on SWE-bench Verified rose 13 times. The median increase was 0.8 percentage points, and 7 of the 13 increases were smaller than the estimated inflation. RULER also reports a significant ranking uncertainty, as shown in Figure 1. Panel (a) shows that the 2 tied leaders receive only 30.5% and 29.0% probability of remaining as leaders, while the model at rank 3 receives 26.4%. In addition, panel (b) shows that 80% rank intervals for the 2 leaders both span ranks 1–6, and 13 models have overlapping rank intervals, suggesting an indistinguishable top tier. In addition to SWE-Bench Verified, we also apply RULER to 14 benchmarks from 5 families (SWE-bench, Tau2-Bench, LiveBench, AlpacaEval 2, and WildBench), spanning software engineering, agentic customer service, instruction following, and judged response quality. The same patterns frequently recur: rank-1 adjustments exceed 1 percentage point in 7 benchmarks, and the leading tier contains multiple models in 12 benchmarks out of a total of 14 (Table 4). Because organizations, researchers, and funding bodies act on these

Figure 1. The winner on SWE-bench Verified remains statistically unresolved.

BenchmarkNumber of TasksNumber of ModelsPosition 1 Score (Raw)Position 1 Score (Adjusted)Number of TiersNumber of Models in Tier 1
SWE Bash-only5004176.8075.71106
SWE Verified50017379.2078.301513
SWE Multimodal5101236.4735.8435
SWE Multilingual3001472.6771.1749
SWE Pro3951444.3043.4364
Tau2 Airline50769.0065.9325
Tau2 Retail114784.2182.0125
Tau2 Telecom114798.2598.0761
AlpacaEval 280520877.8377.29213
WildBench v29886379.8179.71152
LiveBench pooled3188177.5677.4491
LiveBench Coding1288186.7285.6182
LiveBench Language1408169.8167.9174
LiveBench Instr.508186.7882.93519

Table 4. Score overstatement and unresolved leaders occur across benchmark families.

Notes. The snapshot contains 173 models and 500 tasks, with m=⌊N0.9⌋m = \lfloor N^{0.9} \rfloor and B=5,000B = 5{,}000 bootstrap resamples. Panel (a) shows the top-10 RPM, whose entries are bootstrap rank probabilities. Panel (b) shows 80% rank-mass intervals, with blue points marking observed ranks. Tiers use the common-intersection rule. Appendix § 12 shows the leading 30 × 30 RPM.

rankings, overstated winning scores and unresolved orderings can misdirect model adoption, research credit, and investment.

We validate RULER using sample splitting and a calibrated simulation. In the split study, one half of each task pool selects the ranking and the other half evaluates the selected models. We average the resulting score errors over 200 splits within each benchmark and then average their magnitudes across benchmarks. At rank 1, adjustment reduces score errors from 2.38 to 0.73 pp. A simulation calibrated to SWE-bench Verified supplies a second check with known model means: the correction reduces mean winner bias from 1.41 to 0.21 pp (§ 3).

Contributions. Our paper puts forth a conceptual shift: While people typically treat the numbers on LLM leaderboards as collections of model means that truthfully reflect the models’ performance, they are in fact post-selection objects because of ranking. A model might be ranked at the top due to noise in the data, inflating the performance of the top model(s). We correct for this rank-induced inflation and quantify rank uncertainty. Specifically, we make the following contributions.

  • Full-leaderboard bootstrap inference. We apply an mm-out-of-NN rank-induced bias correction across the entire leaderboard, allowing arbitrary model correlation within each task. RULER combines position-level score adjustments, RPMs that report how often each model occupies each rank under task resampling, and rank-mass intervals that group models into tiers. Under exact population tiers separated by positive gaps, we derive an all-position bias expansion and prove that the bootstrap reduces Θ(N−1/2)\Theta(N^{-1/2}) rank-induced bias to an expected residual of o(N−1/2)o(N^{-1/2}). We also show that the RPM converges to a block-diagonal matrix of within-tier Gaussian rank probabilities, and we establish sufficient conditions for recovering the population tiers.

  • Validation and empirical audit. We validate the correction via sample splitting—one half of the tasks selects the rank order while the held-out half rescores the selected models—and a calibrated simulation with known model means. Both exercises show substantial reductions in average rank-1 selection error. Auditing 14 benchmarks from 5 families, we find position-1

adjustments from 0.09 to 3.86 pp and multiple models sharing the leading tier in most benchmarks (§ 3–4 and Appendix § 11).

  • Benchmark design guidance. RULER also translates the analysis into a reporting standard and an asymptotic task-pool sizing rule. Under the exact-tier approximation, N≳(δ/ε)2N \gtrsim(\delta/\varepsilon)^2 tasks are required to decrease the expected rank-1 inflation below ε\varepsilon, where δ\delta is the within-tier Gaussian winning gap. Designers can use this calculation to plan larger task pools and publish RULER’s score adjustments, RPMs, rank intervals, and tiers alongside raw rankings to show the uncertainty remaining at the available sample size (§ 5).

Model, Bootstrap, and Theory

We develop a statistical framework for benchmark outcomes that enables the bootstrap inference for LLM leaderboards. We state the model and theory for bounded task-level scores, with binary pass/fail outcomes as the primary application to SWE-bench. Throughout, an LLM “model” denotes an evaluated leaderboard entry, including its agent or scaffold when applicable. All proofs appear in Appendix C.

Let Yij∈[0,1]Y_{ij} \in[0,1] denote the score for model j∈{1,…,J}j \in\{1,\ldots,J\} on task i∈{1,…,N}i \in\{1,\ldots,N\}, with binary success or failure as a special case. The observed data form an N×JN \times J outcome matrix Y=(Yij)\mathbf{Y} = (Y_{ij}), the transpose of whose ii-th row we denote Yi⋅:=(Yi1,…,YiJ)T\mathbf{Y}_{i\cdot} := (Y_{i1},\ldots,Y_{iJ})^{\mathsf{T}}. We assume that these task rows Y1⋅,…,YN⋅\mathbf{Y}_{1\cdot},\ldots,\mathbf{Y}_{N\cdot} are i.i.d. draws from a common distribution PP on [0,1]J[0,1]^J. We impose no parametric form on PP and allow arbitrary correlation and dependence among different models’ outcomes on the same task. Our primary quantity of interest is the true mean score τj=E[Yij]\tau_j = \mathbb{E}[Y_{ij}] for each model jj, along with the induced true ranking. We collect these into the score vector τ:=(τ1,…,τJ)T\boldsymbol{\tau} := (\tau_1,\ldots,\tau_J)^{\mathsf{T}} and write Σ:=Cov⁡(Yi⋅)\boldsymbol{\Sigma} := \operatorname{Cov}(\mathbf{Y}_{i\cdot}) for the within-task covariance across models. The true ranking is a permutation π∗:{1,…,J}→{1,…,J}\pi^* : \{1,\ldots,J\} \to\{1,\ldots,J\} such that τπ∗(1)≥τπ∗(2)≥⋯≥τπ∗(J)\tau_{\pi^*(1)} \ge\tau_{\pi^*(2)} \ge\cdots\ge\tau_{\pi^*(J)} with a declared arbitrary tie breaking, where π∗(k)\pi^*(k) is the index of the model whose true ability places it at rank kk.

In practice, we observe only the empirical mean scores τ^j=N−1∑iYij\widehat{\tau}_j = N^{-1}\sum_i Y_{ij}, which we collect into τ^:=(τ^1,…,τ^J)T\widehat{\boldsymbol{\tau}} := (\widehat{\tau}_1,\ldots,\widehat{\tau}_J)^{\mathsf{T}}, and the corresponding empirical ranking π^\widehat{\pi}, defined analogously by τ^π^(1)≥⋯≥τ^π^(J)\widehat{\tau}_{\widehat{\pi}(1)} \ge\cdots\ge\widehat{\tau}_{\widehat{\pi}(J)}. We assume that there is an arbitrary but fixed tie-breaking rule that is being used throughout in computation. Because leaderboards are read off sorted score vectors, our inferential targets are not the individual τ^j\widehat{\tau}_j but functionals of the sorted τ^\widehat{\boldsymbol{\tau}}. Each τ^j\widehat{\tau}_j is unbiased for τj\tau_j, but sorting introduces a selection effect that no single-model standard error can capture. We formalize two targets. The first is the rank-induced bias at position kk,

bN(k):=E[τ^π^(k)−τπ^(k)],b_N(k) := \mathbb{E}\left[\widehat{\tau}_{\widehat{\pi}(k)} - \tau_{\widehat{\pi}(k)}\right],

which is the bias induced by sorting. The quantity bN(1)b_N(1) measures the classical WC: the expected score error of the winning model. The second target is the population RPM R∈[0,1]J×J\mathbf{R} \in[0,1]^{J \times J}, with entry Rjk=P(π^(k)=j)R_{jk} = \mathbb{P}(\widehat{\pi}(k) = j) giving the probability that model jj would be placed at rank kk on an independent draw of NN tasks from the same task distribution. A concentrated row signals a stable ranking, while a diffuse row signals that the model’s position is poorly resolved by the available task pool. The bootstrap RPM defined below summarizes rankings across mm-task resamples. Under the conditions of Proposition 2.4, it shares the population RPM’s limiting rank probabilities. Ranking is nonsmooth at tied mean scores, so standard plug-in approximations can fail where selection effects are most pronounced.

Bootstrap correction and components of RULER

Both the rank-induced bias and the leaderboard uncertainty are functionals of the task distribution: if we knew the population distribution, we could repeatedly draw new leaderboards, rank the models, and directly measure the rank-induced bias. Although we do not observe the population, the leaderboard we already observe, i.e., the empirical distribution of tasks, is a good approximation of the population distribution. Thus, we can bootstrap from the observed tasks in the leaderboard, which preserves the dependence among models evaluated on the same task, and estimate the rank-induced bias from the observed leaderboard itself. The difficulty is that ranking is nonsmooth at ties, so the usual smooth-functional justification for the standard bootstrap fails [19]. The mm-out-of-NN bootstrap [8] fixes this by resampling fewer tasks with m=o(N)m = o(N). We follow the literature standards and use m=⌊Nγ⌋m = \lfloor N^\gamma\rfloor for choosing mm. To the best of our knowledge, there is no universally applicable way to tune the hyperparameter γ\gamma with theoretical guarantees. Thus, we provide an empirical guidance under Appendix § 7 and show that RULER performs relatively robust under different γ\gamma in Appendix § 11.§ 2.

Score Adjustment. The mm-out-of-NN bootstrap works by drawing BB independent resamples of m<Nm < N tasks with replacement. Letting τ^j(b)\hat{\tau}_j^{(b)} and π^(b)\hat{\pi}^{(b)} denote their means and rankings, the adjustment for position kk is

b^N(k)=mN1B∑b=1B(τ^π^(b)(k)(b)−τ^π^(b)(k)).(1)\hat{b}_N(k) = \sqrt{\frac{m}{N}} \frac{1}{B} \sum_{b=1}^{B} \left(\hat{\tau}_{\hat{\pi}^{(b)}(k)}^{(b)} - \hat{\tau}_{\hat{\pi}^{(b)}(k)}\right). \tag*{(1)}

Intuitively, it measures how much a bootstrap-selected model’s resample score exceeds its original score, then rescales this excess to the full-sample noise level. The adjustment belongs to a rank position, rather than to a particular model.

RPM and rank interval. We can use the same mm-task resamples to measure the uncertainty in the original leaderboard. The first uncertainty measure is the probability of each model occupying a specific rank position, called the Rank Probability Matrix (RPM):

R^jk=B−1∑b=1B1[model j ranks at position k in the b-th bootstrap sample],(2)\hat{R}_{jk} = B^{-1} \sum_{b=1}^{B} \mathbb{1}[\text{model }j\text{ ranks at position }k\text{ in the }b\text{-th bootstrap sample}], \tag*{(2)}

where 1[⋅]\mathbb{1}[\cdot] is the indicator function.1 The RPM shows how uncertain a leaderboard ranking can be. Thus, we develop a second metric, the rank interval, that measures the possible rank positions of each model. Let Fj(k)=∑s≤kR^jsF_j(k) = \sum_{s \leq k} \hat{R}_{js} denote the probability that model jj ranks at or above position kk, and Qj(u)=min⁡{k:Fj(k)≥u}Q_j(u) = \min\{k : F_j(k) \geq u\} the smallest rank position at which this cumulative probability reaches uu. Then, the rank interval for model jj is computed as:

Ij(α)=[Qj(α/2),Qj(1−α/2)].(3)I_j(\alpha) = [Q_j(\alpha/2), Q_j(1-\alpha/2)]. \tag*{(3)}

These positions contain at least 1−α1-\alpha of model jj’s bootstrap rank mass. We default to α=0.20\alpha= 0.20 throughout this paper.

Model tiers. Finally, small benchmark samples often leave the ranking of closely performing models unresolved, with no clear winner. We quantify this uncertainty using tiers of models whose rank intervals have a common intersection. Specifically, we start with the first model on the leaderboard for the first tier, and use its rank interval to represent all possible rank positions for the first tier. Then, we add the next model on the board if its rank interval contains a common rank position as the first tier. If yes, we take an intersection of the rank intervals of all models in the tier. Otherwise, close the tier and start a new one. As a result, the rank intervals of all models in a tier share at least one rank. Models with equal observed scores are always placed in the same tier (Appendix D).

RULER. Together, the score adjustment, RPM, and rank intervals with model tiers form the Rank Uncertainty and Leaderboard Evaluation Report (RULER).

Theoretical properties

We now state the theoretical results that justify the mm-out-of-NN bootstrap behind RULER. We show that RULER’s score adjustment reduces Θ(N−1/2)\Theta(N^{-1/2}) rank-induced bias to o(N−1/2)o(N^{-1/2}) and that the reported tiers can consistently recover the population model tiers. For a fixed number of models JJ, we make the following assumptions.

Assumption 2.1 (Exact population tiers with positive gaps). The models partition into S1,…,SLS_1,\ldots,S_L, with τj=τ(ℓ)\tau_j=\tau^{(\ell)} for j∈Sℓj\in S_\ell and τ(1)>⋯>τ(L)\tau^{(1)}>\cdots>\tau^{(L)}. If L>1L>1, the smallest adjacent gap is bounded below by a fixed Δ>0\Delta>0.

Assumption 2.1 is a deliberate modeling idealization rather than a literal claim that in reality there are always different models with identical population performance. In practice, models may differ in their true mean scores, but often these differences are very small and become statistically indistinguishable with the finite available data. We therefore use exact ties as a tractable approximation for groups of closely performing models whose ordering is difficult to resolve. This idealization captures the selection effects that are particularly pronounced among closely competing models. Under this assumption, we derive theoretical results to provide an informative benchmark for understanding rank-induced bias, and evaluate the practical usefulness of the proposed correction through the finite-sample validation in Section 3. The tiers reported by RULER similarly represent unresolved rankings rather than assertions of exact equality in population performance.

In addition to Assumption 2.1, note that we have assumed that tasks are i.i.d. to facilitate a clean theoretical analysis, as discussed at the beginning of Section 2. However, our sensitivity analysis shows that RULER’s score adjustments are broadly robust across alternative resampling schemes for non-i.i.d. tasks on SWE-bench, as discussed in Section 4.2 and Appendix F.1.

Next, we formalize the magnitude of the rank-induced bias in a leaderboard. Let Kℓ=∑q<ℓ∣Sq∣K_\ell=\sum_{q<\ell}|S_q| denote the number of models sitting above tier ℓ\ell, Z∼N(0,Σ)Z\sim\mathcal{N}(0,\Sigma) a mean-zero Gaussian vector with covariance Σ\Sigma, and Zℓ,(r)Z_{\ell,(r)} the rr-th largest coordinate in SℓS_\ell. Equivalently, Zℓ,(r)Z_{\ell,(r)} is the ∣Sℓ∣−r+1|S_\ell|-r+1-th ordered statistics of {Zj:j∈Sℓ}\{Z_j:j\in S_\ell\}.

Proposition 2.2 (First-order rank-induced bias at every fixed position). At position k=Kℓ+rk=K_\ell+r,

bN(k)=δℓrN+o(N−1/2),δℓr=E[Zℓ,(r)].(4)b_N(k)=\frac{\delta_{\ell r}}{\sqrt{N}}+o(N^{-1/2}), \qquad\delta_{\ell r}=\mathbb{E}[Z_{\ell,(r)}]. \tag*{(4)}

Therefore, δℓr\delta_{\ell r} is the position-induced effect. The winner’s curse bias [35] is a special case of Eq. (4) with bN(1)b_N(1). The proposition shows that rank-induced bias can be Θ(N−1/2)\Theta(N^{-1/2}). Next, we show that the mm-out-of-NN bootstrap score adjustment in Eq. (1) reduces this bias to o(N−1/2)o(N^{-1/2}).

Theorem 2.3 (Bootstrap score adjustment). Assume that as N→∞N \to\infty, m→∞m \to\infty, m/N→0m/N \to0, and B→∞B \to\infty. At every fixed position k=Kℓ+rk = K_{\ell} + r,

b^N(k)=δℓr/N+oP(N−1/2),E[τ^π^(k)−b^N(k)]=E[τπ^(k)]+o(N−1/2).(5)\widehat{b}_N(k) = \delta_{\ell r}/\sqrt{N} + o_P(N^{-1/2}), \qquad\mathbb{E}[\widehat{\tau}_{\widehat{\pi}}(k) - \widehat{b}_N(k)] = \mathbb{E}[\tau_{\widehat{\pi}}(k)] + o(N^{-1/2}). \tag*{(5)}

The second equation in (5) states that, at every fixed position, the adjusted score is unbiased for the true score of the model placed there, up to an error of order o(N−1/2)o(N^{-1/2}). The argument has two steps. First, the bootstrap reproduces the selection effect on the mm-task scale. In a resample of mm tasks, each model’s resample score differs from its observed score by noise of order m−1/2m^{-1/2}. Within a tier, observed score gaps are of order N−1/2N^{-1/2}, which is negligible relative to this noise because m/N→0m/N \to0. Between tiers, observed gaps remain near the population gaps and dominate the noise. Resample rankings therefore keep the tiers in order and rank models within a tier by noise alone, as in the population. The average in Eq. (1) before rescaling thus equals δℓr/m+oP(m−1/2)\delta_{\ell r}/\sqrt{m} + o_P(m^{-1/2}). Second, the factor m/N\sqrt{m/N} converts this to δℓr/N\delta_{\ell r}/\sqrt{N}, the leading term of bN(k)b_N(k) in Proposition 2.2. Subtracting b^N(k)\widehat{b}_N(k) therefore removes the first-order bias.

Finally, we show that the RPM and model tiers estimated using the same mm-out-of-NN bootstrap are consistent estimators. For j∈Sℓj \in S_{\ell}, define pjr(ℓ):=Pr⁡{Zj is rth largest in Sℓ}p_{jr}^{(\ell)} := \Pr\{Z_j\text{ is rth largest in }S_{\ell}\} and note that ∑r=1∣Sℓ∣pjr(ℓ)=1\sum_{r=1}^{|S_{\ell}|} p_{jr}^{(\ell)} = 1. Further, we define the population rank probability matrix (RPM) with entries Rjk:=Pr⁡{π^(k)=j}R_{jk} := \Pr\{\widehat{\pi}(k) = j\}. These are probabilities over repeated samples of NN tasks. They depend on NN, although that dependence is suppressed in the notation.

Proposition 2.4 (Convergence of RPM and consistent tier recovery). Assume the fixed-JJ bounded iid task model and the exact population tiers with positive gaps. Let m=mNm = m_N and B=BNB = B_N be deterministic positive integer sequences with m→∞m \to\infty, m/N→0m/N \to0, and B→∞B \to\infty as N→∞N \to\infty. Suppose Var⁡(Zj−Zh)>0\operatorname{Var}(Z_j - Z_h) > 0 for distinct j,hj,h in each non-singleton tier.

For every tier SℓS_{\ell}, model j∈Sℓj \in S_{\ell}, and r∈{1,…,qℓ}r \in\{1,\ldots,q_{\ell}\},

Rj,Kℓ+r→pjr(ℓ),R^j,Kℓ+r→Ppjr(ℓ).(6)R_{j,K_{\ell}+r} \xrightarrow{} p_{jr}^{(\ell)}, \qquad\widehat{R}_{j,K_{\ell}+r} \xrightarrow{P} p_{jr}^{(\ell)}. \tag*{(6)}

For j∈Sℓj \in S_{\ell} and k∉{Kℓ+1,…,Kℓ+qℓ}k \notin\{K_{\ell}+1,\ldots,K_{\ell}+q_{\ell}\},

Rjk→0,R^jk→P0.(7)R_{jk} \xrightarrow{} 0, \qquad\widehat{R}_{jk} \xrightarrow{P} 0. \tag*{(7)}

Consequently, the bootstrap RPM consistently estimates the population RPM:

max⁡1≤j,k≤J∣R^jk−Rjk∣→P0.(8)\max_{1\leq j,k\leq J} \left|\widehat{R}_{jk} - R_{jk}\right| \xrightarrow{P} 0. \tag*{(8)}

For every fixed α∈(0,1)\alpha\in(0,1), the common-intersection rule returns exactly (S1,…,SL)(S_1,\ldots,S_L), in that order, with probability tending to one.

Proposition 2.4 shows that RULER separates genuine performance differences between tiers while retaining ranking uncertainty within them. As the number of tasks grows, the RPM assigns vanishing probability to ranks outside a model’s population tier. Within a tier, the models have equal true scores, so their relative ranks remain sensitive to sampling noise. The resulting rank intervals share a common position within each tier and separate across tiers with probability tending to one, allowing RULER to recover the population partition. The full proof appears in Appendix § 8.4.

Notes. Each benchmark uses 200 random half-splits, with m=⌊NS0.9⌋m = \lfloor N_S^{0.9} \rfloor and B=2,000B = 2{,}000 resamples on the selection half. Before RULER uses the selected model’s score on selection half SS minus its score on evaluation half EE. After RULER subtracts the estimated bias from that difference. Magnitude of mean error first averages these signed differences over splits within each benchmark, then takes the absolute value and averages across the 14 benchmarks. Its reduction is reported as a percentage. Errors are in percentage points (pp)

Validation

Having established RULER’s theoretical guarantees, we now validate its performance. Specifically, we use a held-out test and calibrated simulations to evaluate how much its score adjustment reduces selection error in finite samples. Both studies show substantial reductions in average selection error.

Held-out Test

The held-out test separates rank selection from evaluation by assigning them to non-overlapping subsets of the task pool. For each benchmark, we randomly split the tasks into a selection half SS with NS=⌊N/2⌋N_S = \lfloor N/2 \rfloor tasks and an evaluation half EE with the remaining NE=N−NSN_E = N - N_S tasks. We treat the selection half as if it were the entire benchmark. Using only its NSN_S tasks, we rank the models by their scores τ^jS\hat{\tau}^{S}_{j} to obtain the ranking π^S\hat{\pi}_S, and apply the bootstrap in Equation (1) with m=⌊NS0.9⌋m = \lfloor N_S^{0.9} \rfloor to obtain the adjustment b^S(k)\hat{b}^{S}(k) at each position kk. The selected models are then scored on EE. Because the evaluation half plays no role in ranking or adjustment, its score τ^jE\hat{\tau}^{E}_{j} provides an unbiased estimate of the selected model’s mean under independent task sampling.2

For each position kk, we compare the raw and adjusted errors

dk=τ^π^S(k)S−τ^π^S(k)E,ek=dk−b^S(k).(9)d_k = \hat{\tau}^{S}_{\hat{\pi}_S(k)} - \hat{\tau}^{E}_{\hat{\pi}_S(k)}, \qquad e_k = d_k - \hat{b}^{S}(k). \tag*{(9)}

The expectation of the raw error dkd_k is the rank-induced bias bNS(k)b_{N_S}(k) of a leaderboard with NSN_S tasks. If the bootstrap removes the bias present in the selection half, the adjusted error eke_k should be centered near zero.3 We repeat the test over 200 random partitions per benchmark. Appendix § 10.§ 1 documents the implementation details.

At rank 1, RULER’s score adjustment reduces the magnitude of mean split error from 2.38 to 0.73 percentage points (pp), a reduction of 69.5% across all 14 benchmarks (Table 1). Positions 2–5 in the same table also show similar improvement, ranging from 46.9% to 55.0%. Appendix § 10.§ 1 reports the complete results. Thus, the held-out test broadly demonstrates that the score adjustment can correct the selection effects generally across the leaderboard.

RankBefore RULERAfter RULERReduction (%)
12.380.7369.5
21.430.7448.0
31.340.7146.9
41.100.5055.0
51.230.5852.7

Table 1. The correction reduces average selection error across 14 benchmarks.

Calibrated Simulation

While the held-out test uses an independently-estimated score as its reference, we provide a second validation using simulated benchmarks where the models’ true scores are known. To mimic realistic settings, we calibrate the simulations to SWE-bench Verified. Specifically, we fit a logistic model combining model-specific abilities with a shared task difficulty, so outcomes for different models on the same task are dependent (see Appendix § 10.§ 2 for more details). From this fitted model, we generate 100 synthetic benchmarks and apply the mm-out-of-NN bootstrap with 2,000 resamples to each. Table 2 compares the raw and corrected scores of the models at positions 1–5 with their known means.

RankMean Signed Error (pp) — No CorrectionMean Signed Error (pp) — mm-out-of-NN bootstrapRMSE (pp) — No CorrectionRMSE (pp) — mm-out-of-NN bootstrap
1+1.41 ± 0.14+0.21 ± 0.151.97 ± 0.131.55 ± 0.10
2+0.80 ± 0.17−0.06 ± 0.171.85 ± 0.181.74 ± 0.14
3+0.66 ± 0.18−0.12 ± 0.181.87 ± 0.161.78 ± 0.14
4+0.98 ± 0.16+0.23 ± 0.161.85 ± 0.111.64 ± 0.13
5+0.96 ± 0.17+0.24 ± 0.181.97 ± 0.151.79 ± 0.13

Table 2. The mm-out-of-NN correction removes most of the winner’s bias and lowers its root mean squared error. Notes. Entries are the mean signed error and root mean squared error (RMSE) of the selected model’s score relative to its known mean, ± standard errors, in pp. The mm-out-of-NN bootstrap uses m=⌊N0.9⌋m = \lfloor N^{0.9}\rfloor and 2,000 resamples.

The mm-out-of-NN correction provides more accurate evaluation to model performance. The raw winner overstates its true mean by 1.41 pp on average, and the mm-out-of-NN correction reduces it to 0.21 pp. The correction also brings the mean errors at positions 2–5 closer to zero. It lowers the winner’s RMSE from 1.97 to 1.55 pp and reduces the RMSE at positions 4 and 5 by about 0.2 pp. Appendix § 10.§ 2 also compares the mm-out-of-NN correction with the standard bootstrap (m=Nm = N), a beta-binomial empirical-Bayes posterior mean, and a correction built from sample splitting. The mm-out-of-NN also performs well generally among the 4 corrections.

Empirical Study Across Benchmark Families

The validation shows that RULER can substantially reduce average selection error. We now use it to ask how much public leaderboard scores are overstated and how precisely their task pools resolve the model ordering. Across the 14 benchmarks, rank-1 adjustments exceed 1 pp in 7, and 12 have multiple models in the leading tier. We begin with SWE-bench Verified to show how the score and rank summaries work together, then assess how broadly the findings hold across benchmark families and alternative reporting choices. Appendices D documents the implementation details

Leaderboard Results

SWE-bench Verified evaluates 173 models on 500 tasks extracted from 12 GitHub repositories in our snapshot.4 Its 2 tied leaders score 79.2%. RULER estimates a downward adjustment of 0.90 pp

Notes. The table shows the top 7 models from SWE-bench Verified. RULER uses m=⌊N0.9⌋m = \lfloor N^{0.9}\rfloor and B=5,000B = 5{,}000. Score Correction is the amount subtracted from the raw score. Rank intervals contain at least 80% of each model’s bootstrap rank mass.

Notes. The raw and adjusted scores are expressed as percentages. Tier counts and Tier 1 sizes use 80% rank mass and the common-intersection rule. All 14 benchmarks use m=⌊N0.9⌋m = \lfloor N^{0.9}\rfloor and B=5,000B = 5{,}000. Appendix § 9 documents the implementation details.

for each, giving adjusted scores of 78.30% (Table 3). Thus the winning score includes a measurable selection advantage.

Raw RankModelRaw Score (%)Score Correction (pp)80% Rank IntervalTier
1–2live-SWE + Claude 4.5 Opus med.79.200.90[1,6]1
1–2Sonar + Claude 4.5 Opus79.200.90[1,6]1
3TRAE + Doubao-Seed-Code78.800.60[1,7]1
4live-SWE + Gemini 3 Pro77.400.69[2,13]1
5–7Atlassian Rovo Dev76.800.63[3,15]1
5–7EPAM AI/Run v20250719 + Claude 4 Sonnet76.800.63[3,15]1
5–7mini-SWE-agent + Claude 4.5 Opus (high)76.800.63[3,16]1

Table 3. RULER reveals score overstatement and unresolved ranks among SWE-bench Verified’s leaders.

In addition to score adjustment, RULER also suggests that the ranking is much less decisive than the raw scores suggest. The 2 leaders receive 30.5% and 29.0% of the bootstrap rank-1 mass, while the model at rank 3 receives 26.4%. Both leaders’ 80% rank intervals span positions 1–6, and the rank-3 model’s interval spans 1–7. Uncertainty extends farther down the leaderboard: the models tied at positions 9–10 each have an interval of [5,21] (Figure 1). The common-intersection rule groups 13 models into the leading tier, with rank 6 contained in every member’s interval, and the full leaderboard has 15 tiers in total.

These patterns extend beyond SWE-bench Verified: score overstatement and unresolved leaders appear across benchmark families (Table 4). On Tau2 Airline, the leader’s score falls by 3.07 pp after adjustment and 5 of 7 models share the leading tier. LiveBench instruction following has the largest adjustment, 3.86 pp, with 19 of 81 models in the leading tier. Both benchmarks contain only 50 tasks. Even a small correction can accompany an unresolved leader, as WildBench’s 0.09-pp adjustment and 2-model leading tier demonstrate. Across the audit, RULER therefore reveals both how much the winning score is overstated and how much ambiguity remains in the model ordering. Appendix § 9 explains how we apply the same procedure to each benchmark’s task-level scores.

Robustness Study

The score adjustment for SWE-bench Verified is robust to the i.i.d. task assumptions. Appendix § 11.§ 1 examines alternative sampling schemes for different assumptions on task distributions. Specifically, we compare pooled resampling with repository-stratified resampling, which approximately fixes the mix of repositories, and a cluster stress test, which allows tasks from the same repository to move together. Verified’s position-1 adjustment is 1.12 pp under pooled resampling, 1.20 pp under stratification, and 1.16 pp under clustering. Other benchmarks, such as Telecom, could be more sensitive to alternative assumptions on task distributions, as detailed in Appendix § 11.§ 1. We also examine the RULER’s robustness against different resample size for the mm-out-of-NN bootstrap and the retained rank mass used for model tiering. Across resample-size exponents from 0.4 to 0.9, score adjustments are stable and keep their direction across most rank positions. Increasing rank mass broadens the reported uncertainty, expanding Verified’s leading tier from 13 models at 80% to 15 at 90%, 24 at 95%, and 29 at 99%. Appendix § 11 gives the procedures and full results.

Conclusion

Public LLM leaderboards rank noisy estimates of model performance, thus the ranking rewards sampling luck alongside genuine ability. As a result, the winner’s score is overstated (the winner’s curse), scores throughout the leaderboard are distorted (rank-induced bias), and the ordering itself is uncertain. RULER addresses both the score distortion and the rank uncertainty. Its score adjustments correct each reported score for the selection effect of its position, and its RPM, rank intervals, and tiers show how firmly each model holds that position. Theorem 2.3 shows that the adjustment reduces rank-induced bias from Θ(N−1/2)\Theta(N^{-1/2}) to O(N−1/2)O(N^{-1/2}), and Proposition 2.4 shows that RULER’s tiering can recover the population model tiers. Across 14 benchmarks, rank-1 adjustments exceed 1 pp in 7 and the leading tier contains multiple models in 12 of the benchmarks. Score distortion and rank uncertainty are therefore common across the audited benchmark families. Because both arise from ranking noisy scores, they are likely to affect other leaderboards that compare closely matched models on limited task pools.

These findings have implications for model adopters, benchmark designers, and model developers. First, firms choosing a model should account for both the uncertainty in the ranking and the overstatement of top scores. A choice based on raw scores and ranks overstates the selected model’s expected performance and treats an unresolved ordering as settled, which can lead to suboptimal adoption decisions. Second, benchmark designers can reduce both problems by enlarging the task pool. Under the exact-tier approximation, halving expected winner inflation requires approximately 4 times as many tasks when the model pool and task distribution remain stable. Appendix § 12 provides a detailed sizing calculation. Finally, model developers who claim state-of-the-art performance should lead by a margin wider than the winner’s curse. On SWE-bench Verified, 7 of the 13 increases in the top score during 2025 were smaller than the estimated inflation of the current leaders. A confident claim needs a lead larger than the estimated inflation, and RULER’s tiers show whether the new leader is separated from its closest competitors.

Scope and Limitations. RULER describes what a specified benchmark can resolve about the evaluated models. Its score adjustments and rank summaries concern sampling tasks from that benchmark’s target population. Their relevance to deployment depends on the benchmark’s representativeness and the meaning of its score. The audit conditions on released responses and judge annotations, so variation from generating new responses or rerunning judges requires a separate assessment. Contamination, adaptive submissions, and selective reporting can introduce additional biases beyond task-sampling selection. RULER’s finite-sample performance also depends on the data-generating process. When mean differences are comparable to sampling noise, the correction can leave residual bias and may overcorrect.

Guide to the appendices. Appendix A uses a two-model example to build intuition for how the ability gap and resample size affect score correction and ranking uncertainty, and Appendix B uses simulations with known model means to guide the choice of resample size. Appendix C proves the main results. Appendix D documents data and implementation. Appendix E reports supplementary validation results, and Appendix F reports robustness studies. Appendix G provides supplementary RULER reports and task-pool planning, followed by related work in Appendix H.

Intuition from a Two-Model Example

With two models, the relation between the ability gap and sampling noise is easy to see: a near tie produces frequent changes in the winner, while a large gap makes the better model’s position more stable. The same example shows why very small resamples can obscure a real gap.

We use independent Bernoulli outcomes for two models on N=500N = 500 tasks, with probabilities (0.500, 0.505) or (0.500, 0.600). We generate 3,000 benchmark draws and use m=⌊N0.9⌋m = \lfloor N^{0.9}\rfloor, with 2,000 bootstrap replicates per draw and seed 20260502. Multinomial sampling of the four joint-outcome cells is exactly equivalent in distribution to resampling complete binary task rows. Observed score ties are broken uniformly. Bootstrap RPM ties receive fractional mass. Bootstrap selection gaps are averaged over tied winners.

With a 0.5-pp ability gap, selection produces substantial score inflation and leaves the winner uncertain. The raw winning score averages 51.51%, while the selected model’s true ability averages 50.28% (Figure 2). Mean bias falls from 1.23 pp (Monte Carlo SE 0.034) to a residual of 0.22 pp (SE 0.037) after correction. Exact summation over the two independent binomial count distributions gives population raw bias 1.246 pp. The worse model wins in 43.4% of draws, and the better model receives only 54.1% rank-1 mass in the averaged bootstrap RPM. Thus the correction reduces inflation while the RPM continues to show an unresolved ordering.

Figure 2. Score correction reduces inflation while near-tied models remain hard to rank. Notes. Two models have independent Bernoulli outcomes with probabilities (0.500, 0.505) or (0.500, 0.600) on N=500N = 500 tasks. Each case uses 3,000 benchmark draws, m=⌊N0.9⌋m = \lfloor N^{0.9}\rfloor, and 2,000 resamples per draw. Left: raw and adjusted winning scores for the 0.5-pp gap, with the selected model’s mean true ability marked by the dashed line. Middle and right: fractional-tie RPMs averaged over benchmark draws for the 0.5-pp and 10-pp gaps.

A 10-pp ability gap largely resolves the winner: the better model’s averaged rank-1 mass rises to 96.9%, and exact population winner bias falls to 0.0078 pp. The bootstrap nevertheless applies a mean adjustment of 0.176 pp, leaving a simulated mean residual of −0.238 pp (SE 0.043).5 example shows that the finite-sample correction can overshoot even when the winning model is well separated.

Smaller resamples make that separation harder to detect. An additional experiment uses m=⌊N0.6⌋m = \lfloor N^{0.6} \rfloor, 3,000 draws, and 2,000 resamples per draw. For the 10-pp gap, the better model’s averaged rank-1 mass falls to 80.8%, compared with 96.9% at m=⌊N0.9⌋m = \lfloor N^{0.9} \rfloor, and the mean adjustment rises to approximately 0.822 pp. Raw bias of 0.033 pp then becomes a residual of −0.789 pp. For the 0.5-pp gap, however, raw bias of 1.257 pp becomes a smaller residual of 0.048 pp. The examples expose the tradeoff: smaller resamples suppress apparent differences near a tie but can also obscure genuine ability gaps. The experiments use different random streams, so the comparison across resample sizes is illustrative rather than paired.

Guidance on Choosing the Resample Size for the mm-out-of-NN Bootstrap

The theory requires m=o(N)m = o(N) but does not fix the resample size for a given benchmark. We write m=max⁡{2,⌊Nγ⌋}m = \max\{2, \lfloor N^\gamma\rfloor\} with exponent γ∈(0,1]\gamma\in(0, 1], so γ=1\gamma= 1 is the standard bootstrap. The two-model example in Appendix § 6 shows what the exponent trades off: smaller resamples suppress apparent differences near a tie but can also obscure genuine ability gaps. The true means of the audited models are unknown, so the audit cannot show which exponent is most accurate. A simulation with known model means measures that accuracy directly. The most accurate exponent depends mainly on how closely model abilities are spaced and on the number of models. When abilities are closely spaced, every exponent from 0.4 to 0.9 is nearly as accurate as the best, and the default γ=0.9\gamma= 0.9 is within 8% of the most accurate exponent unless all models are tied.

Each synthetic benchmark has JJ models and NN tasks with independent Bernoulli outcomes. The true model means are evenly spaced over an interval of width wσw\sigma centered at pp, where σ=p(1−p)/N\sigma= \sqrt{p(1-p)/N} is the standard error of one model’s score. Thus pp is both the mean and the median ability. In the tied design (w=0w = 0), every model has mean pp. In the close design (w=2w = 2), the means span 2 standard errors. In the spread design (w=15w = 15), they span 15. The width does not change with JJ, so larger leaderboards pack their models more densely. The grid crosses N∈{500,2,000}N \in\{500, 2{,}000\}, p∈{0.5,0.75}p \in\{0.5, 0.75\}, and J∈{2,5,10,20,50,100}J \in\{2, 5, 10, 20, 50, 100\} with 11 exponents γ∈{0.1,0.2,…,0.9,0.95,1}\gamma\in\{0.1, 0.2, \ldots, 0.9, 0.95, 1\}. Each design uses 1,000 synthetic benchmarks and B=2,000B = 2{,}000 resamples. Every exponent is applied to the same benchmarks, so comparisons across exponents are paired.

For each synthetic benchmark, the error at position kk is the corrected score at that position minus the true mean of the model observed there. We average its square over all JJ positions and over benchmarks and take the square root to obtain the root-mean-square error (RMSE). Table 5–Table 7 report it for every exponent. In each column, the 3 smallest displayed values are shaded from dark to light, and equal displayed values share a shade, so a long shaded block marks exponents with nearly equal accuracy. We call the exponents tied for the smallest displayed RMSE in a column the most accurate.

γ/J\gamma/ J\multicolumn{6}{c}{p=0.5p = 0.5}\multicolumn{6}{c}{p=0.75p = 0.75}
2510205010025102050100
N=500N = 500 tasks
0.11.871.351.050.870.810.831.641.231.050.940.910.94
0.21.871.351.030.790.640.631.641.200.970.820.750.76
0.31.881.351.030.770.530.451.641.180.920.700.550.51
0.41.881.351.030.770.510.391.651.180.910.680.470.39
0.51.891.361.030.770.510.381.651.180.920.670.460.36
0.61.911.371.050.780.520.391.671.190.920.680.460.35
0.71.931.401.070.800.550.421.691.220.940.700.480.37
0.81.981.461.120.850.600.481.721.270.980.740.530.43
0.92.041.551.210.950.710.601.771.351.060.830.630.53
0.952.081.621.271.020.790.691.801.411.110.890.700.61
12.121.691.351.110.890.791.831.471.180.960.780.70
N=2,000N = 2{,}000 tasks
0.10.900.670.530.440.410.410.800.590.520.470.460.47
0.20.900.660.520.390.290.260.790.570.460.380.330.32
0.30.900.660.510.390.260.200.790.560.450.340.250.21
0.40.900.660.520.390.260.190.790.560.450.340.230.18
0.50.900.670.520.390.260.190.800.570.450.340.230.17
0.60.910.670.520.390.260.190.800.570.450.340.230.17
0.70.920.680.530.400.270.200.810.580.460.350.230.18
0.80.940.710.550.420.290.220.830.600.470.360.250.20
0.90.970.760.590.470.350.280.870.640.510.410.300.25
0.951.000.800.630.510.390.330.890.670.540.440.340.29
11.020.840.680.560.450.390.910.710.580.490.390.34

Table 5. When all models are tied, exponents below the default minimize error, and their advantage over the default grows with the number of models.

γ/J\gamma/ J\multicolumn{6}{c}{p=0.5p = 0.5}\multicolumn{6}{c}{p=0.75p = 0.75}
2510205010025102050100
N=500N = 500 tasks
0.12.472.812.812.642.672.692.112.442.382.232.232.26
0.22.432.802.802.612.612.622.082.442.392.222.222.24
0.32.322.732.742.572.552.551.972.392.352.202.182.19
0.42.202.622.632.492.472.461.882.292.252.132.112.12
0.52.132.542.522.412.382.361.822.232.152.062.042.04
0.62.092.522.442.332.282.261.792.212.082.001.961.95
0.72.082.532.422.292.222.191.782.222.061.971.911.89
0.82.082.532.462.292.192.151.782.222.091.971.891.86
0.92.082.482.502.332.192.141.782.172.132.001.891.85
0.952.082.432.512.352.202.141.782.132.142.021.891.85
12.082.382.512.382.212.151.782.082.132.041.911.86
N=2,000N = 2{,}000 tasks
0.11.291.411.401.351.371.371.121.201.201.161.161.17
0.21.291.421.411.351.341.341.131.221.211.171.161.16
0.31.261.401.401.341.331.321.111.211.211.161.141.14
0.41.211.371.371.321.311.301.061.181.181.141.121.12
0.51.161.321.321.271.261.251.011.141.141.101.081.08
0.61.121.281.261.211.201.190.971.101.081.051.031.03
0.71.091.291.221.171.141.130.951.111.051.010.980.98
0.81.091.301.231.161.111.090.951.121.061.000.960.94
0.91.091.291.261.171.111.080.951.101.081.010.960.94
0.951.091.261.271.191.111.080.951.081.091.020.960.94
11.091.231.261.201.121.080.951.051.091.040.970.94

Table 7. With well-separated models, large exponents minimize error. Notes. As in Table 5, except that the true means are evenly spaced over 15σ15\sigma around pp, with σ=p(1−p)/N\sigma= \sqrt{p(1-p)/N}.

When all models are tied, every observed difference is noise, and exponents well below the default give lower error (Table 5). The most accurate exponents lie between 0.1 and 0.7 and rise with JJ, reaching 0.4–0.6 at J=100J = 100. The default γ=0.9\gamma= 0.9 has an RMSE 8–18% above that of the most accurate exponent for J≤10J \le10 and 47–57% above it at J=100J = 100. The standard bootstrap is 12–110% above it.

With closely spaced models, error changes little across a wide range of exponents (Table 6). In every column, each exponent from 0.4 to 0.9 has an RMSE within 8% of the most accurate exponent.

γ/J\gamma/ J\multicolumn{6}{c}{ p=0.5p = 0.5 }\multicolumn{6}{c}{ p=0.75p = 0.75 }
2510205010025102050100
N=500N = 500 tasks
0.12.301.781.541.421.391.401.981.561.391.331.311.32
0.22.311.791.541.381.311.301.991.561.361.271.221.22
0.32.321.801.541.381.271.232.011.561.341.221.131.11
0.42.331.801.541.381.261.212.011.561.341.211.101.06
0.52.341.811.551.381.261.202.021.571.341.201.091.05
0.62.351.811.551.381.251.202.031.571.341.201.081.04
0.72.361.821.551.381.251.192.031.581.351.201.081.03
0.82.381.851.571.391.251.192.051.611.361.221.091.03
0.92.401.911.621.431.281.222.061.671.401.251.121.06
0.952.401.961.661.471.321.252.061.701.441.291.151.09
12.392.011.711.521.371.302.061.751.481.331.191.13
N=2,000N = 2{,}000 tasks
0.11.130.890.770.720.690.700.990.760.700.670.660.66
0.21.140.890.770.700.640.631.000.760.670.620.590.58
0.31.140.890.780.700.630.611.010.760.670.600.550.54
0.41.150.890.780.700.630.611.010.760.670.600.550.53
0.51.150.900.780.700.630.601.010.760.670.600.550.53
0.61.150.900.780.700.630.601.020.770.670.600.550.52
0.71.160.900.780.700.630.601.020.770.670.600.540.52
0.81.170.920.780.700.630.591.040.780.670.600.540.52
0.91.180.950.810.720.640.601.050.810.690.620.550.53
0.951.180.970.830.740.660.621.050.830.710.640.570.54
11.171.000.860.770.690.651.050.860.740.670.590.56

Table 6. With closely spaced models, error is nearly flat across a wide range of exponents. Notes. As in Table 5, except that the true means are evenly spaced over 2σ2\sigma around pp, with σ=p(1−p)/N\sigma= \sqrt{p(1-p)/N}.

variation.

Notes. Entries are the RMSE in pp of the corrected scores over all JJ positions and 1,000 synthetic benchmarks with independent Bernoulli outcomes. Every model has true mean pp. Each correction uses m=max⁡{2,⌊Nγ⌋}m = \max\{2, \lfloor N^\gamma\rfloor\} and B=2,000B = 2{,}000, and γ=1\gamma= 1 is the standard bootstrap. All exponents use the same benchmarks (seed 20261103). In each column, bold type with the darkest, medium, and lightest shading marks the smallest, second-smallest, and third-smallest displayed values. Monte Carlo standard errors are at most 0.04 pp.

The default is within 5% of the most accurate exponent for J≥20J \ge20 and within 8% for every JJ, while the standard bootstrap is 4–13% above it.

When the means span a wide range, larger exponents are more accurate (Table 7). The most accurate exponents lie between 0.7 and 1, and the default is within 5% of the best. Small exponents blur genuine gaps between models, as in the two-model example in Appendix § 6.

The most accurate exponents depend mainly on JJ and the spacing of model abilities. For each JJ and design, the shaded exponents overlap across all 4 combinations of task count and success rate in the grid. The default γ=0.9\gamma= 0.9 is within 8% of the most accurate exponent in the close and spread designs. Smaller exponents improve on the default by more than 8% only in the tied design, where the gain grows with JJ. An exponent near the default therefore suits leaderboards whose models range from closely to widely spaced, and smaller exponents pay off mainly when many models are tied.

The spacing of a real leaderboard can be gauged by dividing the range of observed scores by pˉ(1−pˉ)/N\sqrt{\bar{p}(1-\bar{p})/N}, where pˉ\bar{p} is the mean score. Sampling noise widens observed ranges, so this gauge overstates the true spacing. Even 10 tied models have an expected observed range of about 3 standard errors. The designs draw models independently. When outcomes on a task are positively correlated across models, score differences are less noisy than this gauge assumes, so it understates the separation. Measured this way, the 14 audited benchmarks span 2.7–51.4 standard errors (median 14.8). Their 10 leading models, or all models when there are fewer, span 0.6–14.3 standard errors and lie within 3 standard errors in 6 benchmarks. In these 6 benchmarks, the leading groups are as closely spaced as the tied or close design. Appendix F.2 examines the audited corrections over the exponents from 0.4 to 0.9.

Proofs of the Main Results

On a high level, to prove the correctness of the bootstrap procedure for all positions, we use a joint conditional central limit theorem to provide the common approximation for score bias and rank probabilities. Since positive gaps separate the population tiers with probabilities approaching one, the key arguments concern the ordering of Gaussian score errors within each tier. Below we first remark on the notations and then build the proofs step by step in this section.

Let FN=σ(Y1,…,YN)\mathcal{F}_N = \sigma(Y_1,\ldots,Y_N) denote the information in the observed sample. Given FN\mathcal{F}_N, the bootstrap draw mm vectors Y1∗,…,Ym∗Y_1^*,\ldots,Y_m^* independently with replacement from the NN observed rows, and set

τ^∗=1m∑r=1mYr∗,WN=N(τ^−τ),Wm∗=m(τ^∗−τ^).\hat{\tau}^{*} = \frac{1}{m}\sum_{r=1}^{m}Y_{r}^{*}, \qquad W_N = \sqrt{N}(\hat{\tau}-\tau), \qquad W_m^{*} = \sqrt{m}(\hat{\tau}^{*}-\hat{\tau}).

Write E∗\mathbb{E}_{*} and Pr⁡∗\Pr_{*} for expectation and probability conditional on FN\mathcal{F}_N. For a vector x∈RJ\mathbf{x} \in\mathbb{R}^{J}, write xℓ,(r)x_{\ell,(r)} for its rr-th largest coordinate among the indices in SℓS_{\ell}. In particular, Zℓ,(r)Z_{\ell,(r)} is this order statistic of the Gaussian vector Z∼N(0,Σ)\mathbf{Z} \sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma}) and WN,ℓ,(r)W_{N,\ell,(r)} (Wm,ℓ,(r)∗W_{m,\ell,(r)}^{*}, resp.) is the rr-th largest component of those coordinates of WN\mathbf{W}_N (Wm∗\mathbf{W}_m^{*}, resp.) in SℓS_{\ell}. The symbols ∥⋅∥2\lVert\cdot\rVert_{2} and ∥⋅∥∞\lVert\cdot\rVert_{\infty} denote the Euclidean and maximum norms, respectively.

Joint normal approximation and moment bounds

Lemma C.1 (Joint CLT, bootstrap CLT, and moment bounds). It holds that WN=N(τ^−τ)⇒Z∼N(0,Σ)\mathbf{W}_N = \sqrt{N}(\hat{\tau}-\tau) \Rightarrow\mathbf{Z} \sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma}). For an independent empirical mm-resample, Wm∗=m(τ^∗−τ^)⇒Z\mathbf{W}_m^{*} = \sqrt{m}(\hat{\tau}^{*}-\hat{\tau}) \Rightarrow\mathbf{Z} in probability as m,N→∞m,N \to\infty. Moreover E∥WN∥∞2≤J/4\mathbb{E}\lVert\mathbf{W}_N\rVert_{\infty}^{2} \le J/4 and E∗∥Wm∗∥∞2≤J/4\mathbb{E}_{*}\lVert\mathbf{W}_m^{*}\rVert_{\infty}^{2} \le J/4.

Proof. The proof of original-sample central limit theorem is standard and follows from Cramér–Wold theorem. We next show that Wm∗\mathbf{W}_m^{*} converges in conditional distribution almost surely, which implies conditional convergence in distribution in probability.

1. Almost sure convergence of distribution. Define the empirical covariance matrix by

Σ^:=1N∑i=1N(Yi⋅−τ^)(Yi⋅−τ^)T=1N∑i=1NYi⋅Yi⋅T−τ^τ^T.(10)\widehat{\Sigma} := \frac{1}{N}\sum_{i=1}^{N}(\mathbf{Y}_{i\cdot}-\widehat{\tau})(\mathbf{Y}_{i\cdot}-\widehat{\tau})^{\mathsf{T}} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{Y}_{i\cdot}\mathbf{Y}_{i\cdot}^{\mathsf{T}}-\widehat{\tau}\widehat{\tau}^{\mathsf{T}}. \tag*{(10)}

By the strong law of large numbers, τ^→τ\widehat{\tau} \to\tau and Σ^→Σ\widehat{\Sigma} \to\Sigma on an Ω0\Omega_0 which is of probability one. Fix ω∈Ω0\omega\in\Omega_0. Note that the observed rows are fixed for fixed NN and any ω\omega. Write Eω∗\mathbb{E}^{*}_{\omega} and Cov⁡ω∗\operatorname{Cov}^{*}_{\omega} for expectation and covariance under bootstrap sampling. To highlight the dependence on NN and ω\omega, below we write mNm_N, τ^N(ω)\widehat{\tau}_N(\omega) and Σ^N(ω)\widehat{\Sigma}_N(\omega) instead of mm, τ^\widehat{\tau} and Σ^\widehat{\Sigma}. For 1≤r≤mN1 \le r \le m_N, define the centered, normalized bootstrap row

UN,r∗:=Yr⋅∗−τ^N(ω)mN.U^{*}_{N,r} := \frac{Y^{*}_{r\cdot}-\widehat{\tau}_N(\omega)}{\sqrt{m_N}}.

Within each row of this triangular array, the vectors UN,r∗U^{*}_{N,r} are independent. Since a bootstrap row is uniform on the observed rows,

Eω∗[Yr⋅∗]=τ^(ω),Cov⁡ω∗(Yr⋅∗)=Σ^N(ω).\mathbb{E}^{*}_{\omega}[Y^{*}_{r\cdot}] = \widehat{\tau}(\omega), \qquad\operatorname{Cov}^{*}_{\omega}(Y^{*}_{r\cdot}) = \widehat{\Sigma}_N(\omega).

Consequently,

Eω∗[UN,r∗]=0,∑r=1mNCov⁡ω∗(UN,r∗)=Σ^N(ω)⟶Σ.(11)\mathbb{E}^{*}_{\omega}[U^{*}_{N,r}] = 0, \qquad\sum_{r=1}^{m_N}\operatorname{Cov}^{*}_{\omega}(U^{*}_{N,r}) = \widehat{\Sigma}_N(\omega) \longrightarrow\Sigma. \tag*{(11)}

Both a bootstrap row and the empirical mean belong to [0,1]J[0,1]^J. Thus every possible bootstrap outcome satisfies

∥UN,r∗∥2≤JmN.\lVert U^{*}_{N,r}\rVert_2 \le\frac{\sqrt{J}}{\sqrt{m_N}}.

For any ε>0\varepsilon> 0, the assumption mN→∞m_N \to\infty implies that mN>J/ε2m_N > J/\varepsilon^2 for all sufficiently large NN. For those NN, the Lindeberg sum is exactly zero:

∑r=1mNEω∗[∥UN,r∗∥221{∥UN,r∗∥2>ε}]=0.(12)\sum_{r=1}^{m_N}\mathbb{E}^{*}_{\omega}\left[\lVert U^{*}_{N,r}\rVert_2^2\mathbf{1}\{\lVert U^{*}_{N,r}\rVert_2 > \varepsilon\}\right] = 0. \tag*{(12)}

Equations (11) and (12) verify the hypotheses of the ordinary multivariate Lindeberg–Feller theorem. Applying that theorem under the bootstrap law for this fixed data sequence gives that on Ω0\Omega_0:

∑r=1mNUN,r∗=WmN∗⇒N(0,Σ).\sum_{r=1}^{m_N}U^{*}_{N,r} = W^{*}_{m_N} \Rightarrow\mathcal{N}(0,\Sigma).

2. Second moment bounds. Finally, for any random variable X∈[0,1]X \in[0,1], the inequality X2≤XX^2 \le X gives

Var⁡(X)=E[X2]−(EX)2≤(EX)(1−EX)≤14.\operatorname{Var}(X) = \mathbb{E}[X^2]-(\mathbb{E}X)^2 \le(\mathbb{E}X)(1-\mathbb{E}X) \le\frac{1}{4}.

Independence of the original task rows implies, for each coordinate jj,

E[WN,j2]=Var⁡(Y1j)≤14.\mathbb{E}[W_{N,j}^2] = \operatorname{Var}(Y_{1j}) \le\frac{1}{4}.

Conditional independence of the bootstrap rows similarly gives

E∗[(Wm,j∗)2]=1N∑i=1N(Yij−τ^j)2=1N∑i=1NYij2−τ^j2≤τ^j(1−τ^j)≤14.\mathbb{E}_{*}[(W^{*}_{m,j})^2] = \frac{1}{N}\sum_{i=1}^{N}(Y_{ij}-\widehat{\tau}_j)^2 = \frac{1}{N}\sum_{i=1}^{N}Y_{ij}^2-\widehat{\tau}_j^2 \le\widehat{\tau}_j(1-\widehat{\tau}_j) \le\frac{1}{4}.

Upper bounding the maximum by the sum allows us to conclude the proof.

Bias at each leaderboard position

Proof of Proposition 2.2. We first control mistakes in the ordering between different population tiers. We then take expectations of the within-tier order statistic.

Let MN=∥WN∥∞M_N=\lVert\mathbf{W}_N\rVert_\infty denote the largest absolute scaled score error. For L>1L>1, define the event

AN={max⁡1≤j≤J∣τ^j−τj∣<Δ/4}.A_N=\left\{\max_{1\le j\le J}\left|\widehat{\tau}_j-\tau_j\right|<\Delta/4\right\}.

On ANA_N, if j∈Shj\in S_h, j′∈Sh′j'\in S_{h'}, and h<h′h<h', then

τ^j−τ^j′>τj−τj′−Δ/2≥Δ/2>0.\widehat{\tau}_j-\widehat{\tau}_{j'}>\tau_j-\tau_{j'}-\Delta/2\ge\Delta/2>0.

Thus every observed score in a higher tier exceeds every observed score in a lower tier. By Markov’s inequality and Lemma C.1,

Pr⁡(ANc)=Pr⁡{MN≥N Δ/4}≤16EMN2NΔ2≤4JNΔ2⟶0.(13)\Pr(A_N^c)=\Pr\{M_N\ge\sqrt{N}\,\Delta/4\}\le\frac{16\mathbb{E}M_N^2}{N\Delta^2}\le\frac{4J}{N\Delta^2}\longrightarrow0. \tag*{(13)}

When L=1L=1, take ANA_N to be the whole sample space. No gap or cross-tier probability bound is needed in that case.

Let us write

TN,k:=N(τ^π^(k)−τπ^(k))=WN,π^(k),VN:=WN,ℓ,(r).T_{N,k}:=\sqrt{N}\left(\widehat{\tau}_{\widehat{\pi}(k)}-\tau_{\widehat{\pi}(k)}\right)=W_{N,\widehat{\pi}(k)},\qquad V_N:=W_{N,\ell,(r)}.

On ANA_N, the index π^(k)\widehat{\pi}(k) belongs to SℓS_\ell and occupies position rr within that tier. Every model in SℓS_\ell has the same population mean. Sorting its empirical means is therefore equivalent to sorting the coordinates of WNW_N in that tier. Hence TN,k=VNT_{N,k}=V_N on ANA_N, including when the empirical means tie. Both ∣TN,k∣\lvert T_{N,k}\rvert and ∣VN∣\lvert V_N\rvert are bounded by MNM_N. Cauchy–Schwarz gives

E∣TN,k−VN∣≤2E[MN1{ANc}]≤2EMN2Pr⁡(ANc)≤JPr⁡(ANc)⟶0.(14)\mathbb{E}\lvert T_{N,k}-V_N\rvert\le2\mathbb{E}\left[M_N1\{A_N^c\}\right]\le2\sqrt{\mathbb{E}M_N^2\Pr(A_N^c)}\le\sqrt{J}\sqrt{\Pr(A_N^c)}\longrightarrow0. \tag*{(14)}

Taking an order statistic is continuous in the maximum norm. Lemma C.1 and the continuous mapping theorem yield

VN⇒Zℓ,(r).V_N\Rightarrow Z_{\ell,(r)}.

Finally, we notice that VN2≤∥WN∥∞2V_N^2\le\lVert W_N\rVert_\infty^2 so E[VN2]≤J/4\mathbb{E}[V_N^2]\le J/4 for all NN. A uniform bound on the second moment implied uniform integrability, so E[VN]→E[Zℓ,(r)]=δℓr\mathbb{E}[V_N]\to\mathbb{E}[Z_{\ell,(r)}]=\delta_{\ell r}. Combining with (14), we concludes the proof. □

Bootstrap score correction

The following deterministic fact will control changes in the score of a selected model when the ranking is perturbed.

Lemma C.2 (Ordering fact). Let qq be a positive integer, and let x,d∈Rqx,d\in\mathbb{R}^q. Write x(r)x_{(r)} for the rr-th largest coordinate of xx, for 1≤r≤q1\le r\le q. Put y=x+dy=x+d, and let ρ\rho be any permutation such that yρ(1)≥⋯≥yρ(q)y_{\rho(1)}\ge\cdots\ge y_{\rho(q)}. Then

∣y(r)−x(r)∣≤∥d∥∞,(15)\lvert y_{(r)}-x_{(r)}\rvert\le\lVert d\rVert_\infty, \tag*{(15)}

and

∣xρ(r)−x(r)∣≤2∥d∥∞.(16)\lvert x_{\rho(r)}-x_{(r)}\rvert\le2\lVert d\rVert_\infty. \tag*{(16)}

Both conclusions hold for every way of ordering tied coordinates.

Proof. Set ϵ=∥d∥∞\epsilon= \lVert d\rVert_{\infty}. At least rr coordinates of xx are at least x(r)x_{(r)}. The corresponding coordinates of yy are at least x(r)−ϵx_{(r)}-\epsilon, so y(r)≥x(r)−ϵy_{(r)} \ge x_{(r)}-\epsilon. Also, at least q−r+1q-r+1 coordinates of xx are at most x(r)x_{(r)}. Their coordinates in yy are at most x(r)+ϵx_{(r)}+\epsilon, so y(r)≤x(r)+ϵy_{(r)} \le x_{(r)}+\epsilon. This proves (15), including in the presence of ties.

Since ρ\rho sorts yy, its selected value satisfies yρ(r)=y(r)y_{\rho(r)} = y_{(r)}, regardless of which tied index occupies rank rr. Consequently,

xρ(r)−x(r)=y(r)−x(r)−dρ(r),x_{\rho(r)}-x_{(r)}=y_{(r)}-x_{(r)}-d_{\rho(r)},

and the triangle inequality gives

∣xρ(r)−x(r)∣≤∣y(r)−x(r)∣+∣dρ(r)∣≤2ϵ.\lvert x_{\rho(r)}-x_{(r)}\rvert\le\lvert y_{(r)}-x_{(r)}\rvert+\lvert d_{\rho(r)}\rvert\le2\epsilon.

This argument compares sorted values directly. Its bound does not depend on how many coordinates change rank. □

Proof of Theorem 2.3. We first replace the average over BB resamples by its conditional mean. We then show that this mean estimates the same constant as in Proposition 2.2, and finally bound the error from using finitely many resamples. As the first step, let us consider sorting Wm,j∗W^{*}_{m,j}. Define qN(ℓ,r):=E∗Wm,ℓ,(r)∗q_N(\ell,r):=\mathbb{E}_{*}W^{*}_{m,\ell,(r)} and we first study its properties. Also for convenience we let Mm∗:=∥Wm∗∥∞M^{*}_{m}:=\lVert W^{*}_{m}\rVert_{\infty} denote the largest absolute bootstrap scaled score error and similarly MN:=∥WN∥∞M_N:=\lVert W_N\rVert_{\infty}.

1. Convergence results for qN(ℓ,r)q_N(\ell,r). Lemma C.1 and continuity of the order-statistic map give

Wm,ℓ,(r)∗⇒Zℓ,(r)in probabilityW^{*}_{m,\ell,(r)} \Rightarrow Z_{\ell,(r)} \qquad\text{in probability}

by the continuous mapping theorem for weak convergence in probability. For each such sequence, E∗[(Wm,ℓ,(r)∗)2]≤J/4\mathbb{E}_{*}[(W^{*}_{m,\ell,(r)})^2]\le J/4. Therefore, applying the uniform integrability for weak convergence in probability, it holds that

qN(ℓ,r)⟶Pδℓr.(17)q_N(\ell,r)\longrightarrow_{P}\delta_{\ell r}. \tag*{(17)}

Further, we note that

∣qN(ℓ,r)∣≤(E∗[(Mm∗)2])1/2≤J2.\lvert q_N(\ell,r)\rvert\le\left(\mathbb{E}_{*}\left[(M^{*}_{m})^2\right]\right)^{1/2}\le\frac{\sqrt{J}}{2}.

Dominated convergence therefore gives

E∣qN(ℓ,r)−δℓr∣⟶0.(18)\mathbb{E}\lvert q_N(\ell,r)-\delta_{\ell r}\rvert\longrightarrow0. \tag*{(18)}

2. Cross-tier mistakes in the bootstrap. Let CN,m∗C^{*}_{N,m} be the event that every bootstrap score in a higher population tier exceeds every bootstrap score in a lower tier. For L=1L=1, define this event to be the whole bootstrap sample space. For L>1L>1, fix an original sample in the event ANA_N from the proof of Proposition 2.2. Its empirical cross-tier score gaps exceed Δ/2\Delta/2. If Mm∗<m Δ/8M^{*}_{m}<\sqrt{m}\,\Delta/8, every bootstrap mean differs from its empirical mean by less than Δ/8\Delta/8. Each bootstrap cross-tier gap then exceeds Δ/2−2(Δ/8)=Δ/4>0\Delta/2-2(\Delta/8)=\Delta/4>0. Markov’s inequality gives, for these original samples,

Pr⁡∗{(CN,m∗)c}≤Pr⁡∗{Mm∗≥m Δ/8}≤64E∗[(Mm∗)2]mΔ2≤16JmΔ2.(19)\Pr_{*}\{(C^{*}_{N,m})^{c}\}\le\Pr_{*}\{M^{*}_{m}\ge\sqrt{m}\,\Delta/8\}\le\frac{64\mathbb{E}_{*}[(M^{*}_{m})^2]}{m\Delta^2}\le\frac{16J}{m\Delta^2}. \tag*{(19)}

Thus the conditional probability of a cross-tier mistake tends to zero on ANA_N uniformly.

3. Convergence of the bootstrapped mean. For a single bootstrap resample, define

μN(k):=E∗Wm,π^∗(k)∗,\mu_N(k):=\mathbb{E}_{*}W^{*}_{m,\widehat{\pi}^{*}(k)},

where π^∗(k)\hat{\pi}^{*}(k) is index of the kk-th largest coordinate of τ^∗\hat{\tau}^{*}. Define the empirical mean drift, on the bootstrap scale, by

dN,j:=m(τ^j−τj)=mNWN,j,aN:=max⁡1≤j≤J∣dN,j∣=mNMN.d_{N,j}:=\sqrt{m}(\hat{\tau}_{j}-\tau_{j})=\sqrt{\frac{m}{N}}W_{N,j},\qquad a_{N}:=\max_{1\leq j\leq J}|d_{N,j}|=\sqrt{\frac{m}{N}}M_{N}.

The moment bound in Lemma C.1 gives

EaN2≤Jm4N⟶0.(20)\mathbb{E}a_{N}^{2}\leq\frac{Jm}{4N}\longrightarrow0. \tag*{(20)}

In particular, aNa_{N} tends to zero in L2L^{2} and in probability under original sampling. For j∈Sℓj\in S_{\ell}, the bootstrap mean decomposes exactly as

τ^j∗=τ(ℓ)+Wm,j∗+dN,jm.\hat{\tau}_{j}^{*}=\tau^{(\ell)}+\frac{W_{m,j}^{*}+d_{N,j}}{\sqrt{m}}.

Consequently, within SℓS_{\ell}, sorting the bootstrap means is equivalent to sorting Wm,j∗+dN,jW_{m,j}^{*}+d_{N,j}. Put

qN(ℓ,r):=E∗Wm,ℓ,(r)∗.q_{N}(\ell,r):=\mathbb{E}_{*}W_{m,\ell,(r)}^{*}.

On CN,m∗C_{N,m}^{*}, the globally selected index π^∗(k)\hat{\pi}^{*}(k) belongs to SℓS_{\ell} and k=Kℓ+rk=K_{\ell}+r. Apply (16) with x=(Wm,j∗:j∈Sℓ)\mathbf{x}=(W_{m,j}^{*}:j\in S_{\ell}) and perturbation d=(dN,j:j∈Sℓ)\mathbf{d}=(d_{N,j}:j\in S_{\ell}). It gives

∣Wm,π^∗(k)∗−Wm,ℓ,(r)∗∣≤2max⁡j∈Sℓ∣dN,j∣≤2aNon CN,m∗.(21)\left|W_{m,\hat{\pi}^{*}(k)}^{*}-W_{m,\ell,(r)}^{*}\right|\leq2\max_{j\in S_{\ell}}|d_{N,j}|\leq2a_{N}\quad\text{on }C_{N,m}^{*}. \tag*{(21)}

Off CN,m∗C_{N,m}^{*}, both ∣Wm,π^∗(k)∗∣\left|W_{m,\hat{\pi}^{*}(k)}^{*}\right| and ∣Wm,ℓ,(r)∗∣\left|W_{m,\ell,(r)}^{*}\right| are at most Mm∗M_{m}^{*}. When L>1L>1, for an original sample in ANA_{N}, conditional Cauchy–Schwarz and (19) yield

∣μN(k)−qN(ℓ,r)∣≤E∗∣Wm,π^∗(k)∗−Wm,ℓ,(r)∗∣≤2aN+2E∗[(Mm∗)2]Pr⁡∗{(CN,m∗)c}≤2aN+4Jm Δ.(22)\begin{aligned} |\mu_{N}(k)-q_{N}(\ell,r)| &\leq\mathbb{E}_{*}|W_{m,\hat{\pi}^{*}(k)}^{*}-W_{m,\ell,(r)}^{*}|\\ &\leq2a_{N}+2\sqrt{\mathbb{E}_{*}[(M_{m}^{*})^{2}]\Pr_{*}\{(C_{N,m}^{*})^{c}\}}\\ &\leq2a_{N}+\frac{4J}{\sqrt{m}\,\Delta}. \tag*{(22)} \end{aligned}

For an original sample outside ANA_{N}, the uniform bound

∣μN(k)−qN(ℓ,r)∣≤E∗∣Wm,π^∗(k)∗∣+E∗∣Wm,ℓ,(r)∗∣≤J|\mu_{N}(k)-q_{N}(\ell,r)|\leq\mathbb{E}_{*}|W_{m,\hat{\pi}^{*}(k)}^{*}|+\mathbb{E}_{*}|W_{m,\ell,(r)}^{*}|\leq\sqrt{J}

still holds. Taking expectations over the original data, we obtain for L>1L>1

E∣μN(k)−qN(ℓ,r)∣≤2EaN+4Jm Δ+JPr⁡(ANc)⟶0.(23)\mathbb{E}|\mu_{N}(k)-q_{N}(\ell,r)|\leq2\mathbb{E}a_{N}+\frac{4J}{\sqrt{m}\,\Delta}+\sqrt{J}\Pr(A_{N}^{c})\longrightarrow0. \tag*{(23)}

Here EaN≤Jm/(4N)→0\mathbb{E}a_{N}\leq\sqrt{Jm/(4N)}\to0, by (20). When L=1L=1, the cross-tier events always hold, and the same conclusion follows from the bound 2EaN2\mathbb{E}a_{N} alone. Combining (23) with (18), we see that

μN(k)⟶δℓrin L1.(24)\mu_{N}(k)\longrightarrow\delta_{\ell r}\quad\text{in }L^{1}. \tag*{(24)}

Together with Proposition 2.2 and the identity E∗[Nb^N(k)]=μN(k)\mathbb{E}_{*}[\sqrt{N}\hat{b}_{N}(k)]=\mu_{N}(k) in (25) below, this implies the second identity in (5).

4. Finitely many bootstrap resamples. As the final step, we prove the first identity in (5). (1) can be rewritten exactly as

N b^N(k)=1B∑b=1Bm(τ^π^(b)(k)(b)−τ^π^(b)(k)).\sqrt{N}\,\hat b_N(k)=\frac{1}{B}\sum_{b=1}^{B}\sqrt{m}\left(\hat{\tau}^{(b)}_{\hat{\pi}^{(b)}(k)}-\hat{\tau}_{\hat{\pi}^{(b)}(k)}\right).

These summands are conditionally i.i.d. with the conditional law of Wm,π^∗(k)∗W^*_{m,\hat{\pi}^*(k)}. Their conditional mean is μN(k)\mu_N(k), and each conditional variance is at most J/4J/4. Consequently,

E∗[N b^N(k)]=μN(k),E∗[(N b^N(k)−μN(k))2]≤J4B.(25)\mathbb{E}_*\left[\sqrt{N}\,\hat b_N(k)\right]=\mu_N(k),\qquad \mathbb{E}_*\left[\left(\sqrt{N}\,\hat b_N(k)-\mu_N(k)\right)^2\right]\leq\frac{J}{4B}. \tag*{(25)}

In particular,

E∣N b^N(k)−μN(k)∣≤J4B⟶0.\mathbb{E}\left|\sqrt{N}\,\hat b_N(k)-\mu_N(k)\right|\leq\sqrt{\frac{J}{4B}}\longrightarrow0.

Together with (24), this proves the stronger conclusion

N b^N(k)⟶δℓrin L1,\sqrt{N}\,\hat b_N(k)\longrightarrow\delta_{\ell r}\quad\text{in }L^1,

and hence also in probability. We conclude the proof.

Rank probabilities and tier recovery

Proof of Proposition 2.4. We first remark on a few facts. Recall that as argued in previous proofs,

Pr⁡(ANc)≤4JNΔ2⟶0,Pr⁡∗{(CN,m∗)c}≤16JmΔ2on AN.(26)\Pr(A_N^c)\leq\frac{4J}{N\Delta^2}\longrightarrow0,\qquad \Pr_*\{(C^*_{N,m})^c\}\leq\frac{16J}{m\Delta^2}\quad\text{on }A_N. \tag*{(26)}

Together, they imply that Pr⁡∗{(CN,m∗)c}→P0\Pr_*\{(C^*_{N,m})^c\}\to_P 0. For a fixed tier ℓ\ell, define its tie set by

Hℓ:={x∈RJ:xj=xh for some distinct j,h∈Sℓ}.\mathcal{H}_\ell:=\{x\in\mathbb{R}^J:x_j=x_h\text{ for some distinct }j,h\in S_\ell\}.

For a singleton tier this set is empty. For a nonsingleton tier, each Zj−ZhZ_j-Z_h is a centered normal variable with strictly positive variance, so

Pr⁡{Z∈Hℓ}≤∑j,h∈Sℓj<hPr⁡{Zj=Zh}=0.(27)\Pr\{Z\in\mathcal{H}_\ell\}\leq\sum_{\substack{j,h\in S_\ell\\j<h}}\Pr\{Z_j=Z_h\}=0. \tag*{(27)}

Suppose that j∈Sℓj\in S_\ell. Let gjr(ℓ)(x)g_{jr}^{(\ell)}(x) be the indicator that model jj occupies position rr when the coordinates in SℓS_\ell are sorted in decreasing order of xx with any fixed tie rule. Let g~jr(ℓ)(x)\widetilde g_{jr}^{(\ell)}(x) be the tie-averaged weight used by the RPM: if xjx_j ties with q−1q-1 other coordinates in SℓS_\ell, the tied block occupies qq consecutive positions and g~jr(ℓ)(x)=1/q\widetilde g_{jr}^{(\ell)}(x)=1/q at each of them. Both functions take values in [0,1][0,1]. They coincide and are locally constant outside Hℓ\mathcal{H}_\ell. Thus their discontinuity sets have Gaussian probability zero, and Egjr(ℓ)(Z)=Eg~jr(ℓ)(Z)=pjr(ℓ)\mathbb{E}g_{jr}^{(\ell)}(Z)=\mathbb{E}\widetilde g_{jr}^{(\ell)}(Z)=p_{jr}^{(\ell)}. Write Rj(ℓ)R_j^{(\ell)} for the within-tier rank of ZjZ_j among {Zh:h∈Sℓ}\{Z_h:h\in S_\ell\}, so that pjr(ℓ)=Pr⁡{Rj(ℓ)=r}p_{jr}^{(\ell)}=\Pr\{R_j^{(\ell)}=r\}, and π^ℓ(r)\hat\pi_\ell(r) for the model at position rr when the original empirical means in SℓS_\ell are sorted with the fixed tie rule. For the bb-th resample, let wjk(b)∈[0,1]w_{jk}^{(b)}\in[0,1] be the rank weight of model jj at global position kk, with the tie averaging of (2), so that R^jk=B−1∑b=1Bwjk(b)\hat R_{jk}=B^{-1}\sum_{b=1}^{B}w_{jk}^{(b)}. Its conditional mean R^jk∞:=E∗wjk(1)\hat R_{jk}^{\infty}:=\mathbb{E}_*w_{jk}^{(1)} is the RPM with infinitely many resamples.

1. Original-sample rank probabilities. Within SℓS_\ell, all population means equal τ(ℓ)\tau^{(\ell)}. Sorting the original empirical means in that tier is therefore exactly the same as sorting the coordinates of WNW_N in that tier. Hence

Pr⁡{π^ℓ(r)=j}=Egjr(ℓ)(WN).\Pr\{\widehat{\pi}_\ell(r)=j\}=\mathbb{E}g_{jr}^{(\ell)}(W_N).

Lemma C.1 gives WN⇒ZW_N\Rightarrow Z. Boundedness and almost-sure continuity of gjr(ℓ)g_{jr}^{(\ell)} under the Gaussian law then imply

Pr⁡{π^ℓ(r)=j}⟶pjr(ℓ).\Pr\{\widehat{\pi}_\ell(r)=j\}\longrightarrow p_{jr}^{(\ell)}.

On ANA_N, the global and within-tier rankings agree after the position shift KℓK_\ell. Consequently,

∣Rj,Kℓ+r−Pr⁡{π^ℓ(r)=j}∣≤Pr⁡(ANc)⟶0.\left|R_{j,K_\ell+r}-\Pr\{\widehat{\pi}_\ell(r)=j\}\right|\leq\Pr(A_N^c)\longrightarrow0.

This proves the first convergence in (6). Also, for k∉Bℓ:={Kℓ+1,…,Kℓ+∣Sℓ∣}k\notin B_\ell:=\{K_\ell+1,\ldots,K_\ell+|S_\ell|\},

Rjk≤Pr⁡(ANc)⟶0.R_{jk}\leq\Pr(A_N^c)\longrightarrow0.

2. The conditional bootstrap rank probabilities. Define the empirical drift vector and the bootstrap errors centered at the population means by

dN:=m(τ^−τ)=m/N WN,VN,m∗:=m(τ^∗−τ)=Wm∗+dN.d_N:=\sqrt{m}(\widehat{\tau}-\tau)=\sqrt{m/N}\,W_N,\qquad V_{N,m}^*:=\sqrt{m}(\widehat{\tau}^*-\tau)=W_m^*+d_N.

By the original-sample moment bound,

E∥dN∥2≤Jm4N⟶0.\mathbb{E}\lVert d_N\rVert^2\leq\frac{Jm}{4N}\longrightarrow0.

Thus dN→0d_N\to0 in probability. Together with the conditional bootstrap limit in Lemma C.1, this gives

VN,m∗⇒Zin probability.(28)V_{N,m}^*\Rightarrow Z\qquad\text{in probability}. \tag*{(28)}

This follows from a subsequence argument. From any subsequence, select a further subsequence along which dN→0d_N\to0 almost surely. On almost every such original data sequence, the conditional CLT holds by Lemma C.1, and ordinary Slutsky convergence applies under that fixed conditional law. This establishes (28) in probability by the subsequence principle.

Within SℓS_\ell, sorting bootstrap means is equivalent to sorting VN,m∗V_{N,m}^*. The same Gaussian-null discontinuity set in (27) therefore yields

E∗g~jr(ℓ)(VN,m∗)→Ppjr(ℓ).\mathbb{E}_*\widetilde{g}_{jr}^{(\ell)}(V_{N,m}^*)\xrightarrow{P}p_{jr}^{(\ell)}.

On CN,m∗C_{N,m}^*, the global bootstrap rank weight at Kℓ+rK_\ell+r equals this within-tier weight. Since all weights lie in [0,1][0,1],

∣R^j,Kℓ+r∞−E∗g~jr(ℓ)(VN,m∗)∣≤Pr⁡∗{(CN,m∗)c}→P0.\left|\widehat{R}_{j,K_\ell+r}^{\infty}-\mathbb{E}_*\widetilde{g}_{jr}^{(\ell)}(V_{N,m}^*)\right|\leq\Pr_*\{(C_{N,m}^*)^c\}\xrightarrow{P}0.

It follows that R^j,Kℓ+r∞→Ppjr(ℓ)\widehat{R}_{j,K_\ell+r}^{\infty}\xrightarrow{P}p_{jr}^{(\ell)}. For j∈Sℓj\in S_\ell and k∉Bℓk\notin B_\ell,

0≤R^jk∞≤Pr⁡∗{(CN,m∗)c}→P0.0\leq\widehat{R}_{jk}^{\infty}\leq\Pr_*\{(C_{N,m}^*)^c\}\xrightarrow{P}0.

3. A finite number of bootstrap resamples. Write Var⁡∗\operatorname{Var}_* for variance conditional on FN\mathcal{F}_N. The weights wjk(1),…,wjk(B)w_{jk}^{(1)},\ldots,w_{jk}^{(B)} are conditionally independent and identically distributed in [0,1][0,1]. Thus

E∗R^jk=R^jk∞,E∗[(R^jk−R^jk∞)2]=Var⁡∗(wjk(1))B≤14B.(29)\mathbb{E}_*\widehat{R}_{jk}=\widehat{R}_{jk}^{\infty},\qquad \mathbb{E}_*\left[(\widehat{R}_{jk}-\widehat{R}_{jk}^{\infty})^2\right] =\frac{\operatorname{Var}_*(w_{jk}^{(1)})}{B}\leq\frac{1}{4B}. \tag*{(29)}

Conditional Chebyshev’s inequality and B→∞B \to\infty show that R^jk−R^jk∞→P0\widehat{R}_{jk} - \widehat{R}_{jk}^{\infty} \xrightarrow{P} 0. Combining this with steps 1 and 2 proves all the limits in (6)–(7). For each (j,k)(j,k), the deterministic quantity RjkR_{jk} and the random quantity R^jk\widehat{R}_{jk} therefore converge to the same constant. Since JJ is fixed, taking a maximum over the J2J^2 entries proves (8).

Rank intervals stay inside their tier and contain its center

Fix α∈(0,1)\alpha\in(0,1) and put u:=α/2u := \alpha/2 and v:=1−α/2v := 1-\alpha/2. For j∈Sℓj \in S_\ell, define the limiting cumulative rank probabilities by

Fj∞(s):={0,s≤Kℓ,∑r=1s−Kℓpjr(ℓ),Kℓ<s<Kℓ+qℓ,1,s≥Kℓ+qℓ.F_j^\infty(s) := \begin{cases} 0, & s \le K_\ell,\\ \sum_{r=1}^{s-K_\ell} p_{jr}^{(\ell)}, & K_\ell< s < K_\ell+ q_\ell,\\ 1, & s \ge K_\ell+ q_\ell. \end{cases}

The RPM limits and finiteness of JJ imply

max⁡1≤j≤Jmax⁡0≤s≤J∣Fj(s)−Fj∞(s)∣→P0.(30)\max_{1\le j\le J}\max_{0\le s\le J} \left|F_j(s)-F_j^\infty(s)\right| \xrightarrow{P} 0. \tag*{(30)}

Since Fj∞(Kℓ)=0F_j^\infty(K_\ell)=0 and Fj∞(Kℓ+qℓ)=1F_j^\infty(K_\ell+q_\ell)=1, with probability tending to one,

Fj(Kℓ)<u,Fj(Kℓ+qℓ)>v.F_j(K_\ell)<u,\qquad F_j(K_\ell+q_\ell)>v.

By the definition of QjQ_j, these inequalities give

Ij(α)⊆[Kℓ+1,Kℓ+qℓ].(31)I_j(\alpha)\subseteq[K_\ell+1,K_\ell+q_\ell]. \tag*{(31)}

The Gaussian vector is centrally symmetric: Z=d−Z\mathbf{Z}\overset{d}{=}-\mathbf{Z}. Since within-tier Gaussian ties have probability zero, negating all coordinates reverses their ranks. Therefore

Rj(ℓ)=dqℓ+1−Rj(ℓ).(32)\mathcal{R}_j^{(\ell)}\overset{d}{=}q_\ell+1-\mathcal{R}_j^{(\ell)}. \tag*{(32)}

Define the middle global rank or ranks of the tier by

cℓ−:=Kℓ+⌊qℓ+12⌋,cℓ+:=Kℓ+⌈qℓ+12⌉.c_\ell^-:=K_\ell+\left\lfloor\frac{q_\ell+1}{2}\right\rfloor,\qquad c_\ell^+:=K_\ell+\left\lceil\frac{q_\ell+1}{2}\right\rceil.

When qℓ=2h−1q_\ell=2h-1 is odd, symmetry gives

Pr⁡{Rj(ℓ)≤h−1}=1−Pr⁡{Rj(ℓ)=h}2≤12,\Pr\{\mathcal{R}_j^{(\ell)}\le h-1\}=\frac{1-\Pr\{\mathcal{R}_j^{(\ell)}=h\}}{2}\le\frac{1}{2},

and Pr⁡{Rj(ℓ)≤h}≥1/2\Pr\{\mathcal{R}_j^{(\ell)}\le h\}\ge1/2. When qℓ=2hq_\ell=2h is even, symmetry gives Pr⁡{Rj(ℓ)≤h}=1/2\Pr\{\mathcal{R}_j^{(\ell)}\le h\}=1/2. Consequently, in both cases,

Fj∞(cℓ−)≥12,Fj∞(cℓ+−1)≤12.F_j^\infty(c_\ell^-)\ge\frac{1}{2},\qquad F_j^\infty(c_\ell^+-1)\le\frac{1}{2}.

This includes a singleton tier, for which cℓ−=cℓ+=Kℓ+1c_\ell^-=c_\ell^+=K_\ell+1. Because u<1/2<vu<1/2<v, convergence in (30) implies, with probability tending to one,

Fj(cℓ−)>u,Fj(cℓ+−1)<v.F_j(c_\ell^-)>u,\qquad F_j(c_\ell^+-1)<v.

The quantile definition then gives

Qj(u)≤cℓ−≤cℓ+≤Qj(v),[cℓ−,cℓ+]⊆Ij(α).(33)Q_j(u)\le c_\ell^-\le c_\ell^+\le Q_j(v),\qquad[c_\ell^-,c_\ell^+]\subseteq I_j(\alpha). \tag*{(33)}

There are finitely many models and tiers, so the block inclusions in (31) and the center inclusions in (33) hold for all models simultaneously with probability tending to one. The argument allows zero probability at a middle rank. It uses the strict gap between the quantile thresholds and 1/21/2, rather than positivity of every Gaussian ordering probability.

5. The common-intersection rule recovers the partition. Consider the event on which ANA_N and all the interval inclusions from step 4 hold. Its probability tends to one. On this event, the observed leaderboard visits S1,…,SLS_1,\ldots,S_L as consecutive blocks, and no observed score tie contains models from two different population tiers.

Every interval belonging to SℓS_\ell contains [cℓ−,cℓ+][c_\ell^-,c_\ell^+]. Hence every observed tie block within SℓS_\ell is compatible, and the running intersection remains nonempty while all models in that tier are added. The procedure cannot split a population tier. Each such interval also lies inside [Kℓ+1,Kℓ+qℓ][K_\ell+1,K_\ell+q_\ell], while intervals from the next tier lie inside its disjoint rank block. Their intersection is empty, so the procedure closes the current tier at exactly the population boundary. It cannot merge two population tiers. Thus the reported ordered partition is exactly (S1,…,SL)(S_1,\ldots,S_L) on this event, proving the claim.

Data, Implementation, and Reproducibility

This appendix explains how to construct a RULER report from task-level benchmark scores. It gives the reporting conventions, score definitions, snapshot sources, and computational details needed to reproduce the audit.

Resampling and reporting conventions

The full audit draws B=5,000B=5{,}000 resamples of m=⌊N0.9⌋m=\lfloor N^{0.9}\rfloor complete task rows with replacement. Every row contains the outcomes of all models on the same task, preserving their within-task dependence. The validation experiments use B=2,000B=2{,}000. The exponent is a predeclared, outcome-independent heuristic. Appendix B gives guidance on choosing it, and Appendix F.2 examines the sensitivity of the audit to it.

Score adjustments and observed ties. For the score correction in Equation (1), each replicate ranks models by decreasing resample score. To give models with equal observed scores a common displayed adjustment, we average the adjustments over the positions occupied by the tie. Verified’s displayed gap between positions 1 and 5 consequently falls from 2.40 to 2.13 pp after tie-averaged adjustments of approximately 0.90 and 0.63 pp.6

Rank probabilities and intervals. For the RPM, we average uniformly over tied ranks. If qq models tie at positions a,…,a+q−1a,\ldots,a+q-1 in a resample, each receives mass 1/q1/q at each of those positions. The indicator in Equation (2) is averaged over these tie-breaking permutations before averaging over resamples. Each row and column of the full RPM sums to one. Models with equal observed scores can still have different RPM rows because they may solve different tasks and have different dependence with their competitors.

Rank intervals use the smallest integer rank whose cumulative mass reaches each quantile threshold. The implementation permits 10−1210^{-12} roundoff when comparing cumulative masses and identifying score ties. We report 80% central rank mass by default and 90%, 95%, and 99% in the sensitivity analysis. These intervals summarize the mm-resample distribution. Proposition 2.4 establishes its connection to population rank probabilities under exact tiers.

Tiers and leaderboard order. To form tiers, traverse models in raw-score order while keeping each observed score tie intact. A new block joins the current tier only if all its intervals and the current tier’s intersection share a rank. Requiring a common rank prevents a chain of pairwise overlaps from joining models whose intervals have no overall intersection. If the intervals inside an observed score tie have no common rank, the implementation stops with an error, because that block is incompatible with the tier summary. This did not occur in the audit. The report retains the raw ordering and displays the raw score, tie-averaged adjustment, rank interval, tier, and RPM. Adjusted scores are reported in that original order, even when their values would imply a different ordering.

Complete reports. The interactive RULER report supplies model identities, complete RPMs, all 3 interval levels, and tiers for every benchmark.

Benchmark scores and snapshot sources

For each benchmark, we form a matrix with tasks as rows and models as columns, using the task-level score defined in Table 8. A model’s benchmark score is the mean of its column. RULER resamples whole rows, so all models are evaluated on the same sampled tasks and the procedure preserves their dependence within each task. The resulting score adjustments and rank summaries describe uncertainty from sampling tasks. For judged outcomes, they condition on the released responses and judge annotations.

FamilyScore and sampling unitTarget population and interpretation
SWE-bench (5 benchmarks)Binary issue resolution. Unit: issue.Software issues under each variant’s task and scaffold rules, evaluated by the retained model-plus-agent systems. The analysis conditions on submitted trajectories. Model-plus-agent systems can differ across entries.
Tau2-Bench (3 domains)Released task reward in [0,1][0,1]. Unit: customer-service scenario.Airline, retail, or telecom scenarios under the released agents and environment. Frozen outcomes retain the realized agent randomness and environment.
AlpacaEval 2Raw weighted rate: soft preference minus 1. Unit: instruction.Instructions compared against a common GPT-4-Turbo reference, conditional on released responses and judge annotations. Instructions are equally weighted. The raw score differs from the length-controlled leaderboard metric.
WildBench v2Checklist-guided judge score divided by 10, in [0.1,1][0.1,1]. Unit: prompt/session.Real-user-style prompts under the frozen scoring protocol. Uncertainty concerns sampling prompts, conditional on the released judge scores.
LiveBench (4 benchmarks)Released task score in [0,1][0,1]. Unit: task/question.Tasks in the retained model-coverage subset. We report Coding, Language, Instruction Following, and their task-weighted pooled mean. This subset and pooled score differ from the full live leaderboard.

Table 8. Each RULER report refers to a specified score and task population.

SWE-bench and Tau2-Bench. For SWE-bench, a task is a software issue and its score is 1 if the evaluated model-plus-agent system resolves it and 0 otherwise. The column mean is therefore the issue-resolution rate. For SWE-bench Pro, we use the 395 tasks for which every system has a released record. For Tau2-Bench, a task is a customer-service scenario and its released reward lies in [0,1][0,1]. RULER applies to these rewards in the same way, separately for Airline, Retail, and Telecom.

AlpacaEval 2. We analyze the raw weighted win rate against the common GPT-4-Turbo reference, rather than the length-controlled leaderboard score [16]. Each instruction has a soft judge preference in [1,2][1,2], from which we subtract 1 to obtain a score in [0,1][0,1]. The “weighted” rate averages these soft preferences, with all valid instructions receiving equal weight. The official AlpacaEval scorer computes the same average. Each resampled row contains all models’ scores for the same instruction, preserving the shared reference and prompt.

WildBench. The task-level outcome is the released checklist-guided judge score for a prompt or session [24]. Dividing its 1–10 score by 10 puts it in [0.1,1][0.1,1]. We average these normalized scores across the 988 prompts and use each prompt’s complete vector of model scores as a resampling row.

LiveBench. We use released task-level accuracy scores in [0,1][0,1] [34]. The analyzed matrix contains 81 models on 318 tasks: 128 Coding, 140 Language, and 50 Instruction Following tasks. RULER reports each category separately and a pooled score that weights all 318 tasks equally. The pooled score therefore weights categories by their task counts and describes this analyzed subset, rather than the full LiveBench leaderboard metric.

Notes. The listed populations are modeling targets. Released tasks need not be random samples from them. Pooled analyses assume independent task rows from the target and resample each row’s complete model vector. Judged scores condition on the released responses and annotations.

Snapshot sources. Table 9 records the date and source of every benchmark snapshot. Table 4 gives the corresponding numbers of tasks and models.

BenchmarkSnapshot dateFrozen source
SWE Verified2026-02-27Official leaderboard, 7c4289f, and task outcomes, 82be2b5 (2026-01-26).
SWE Bash-only2026-02-27Official leaderboard, 7c4289f, and task outcomes, 2f95aad (2026-02-27).
SWE Multimodal2026-02-27Official leaderboard, 7c4289f, and task outcomes, cff22d0 (2025-11-21).
SWE Multilingual2026-02-27Official leaderboard, 7c4289f, and task outcomes, 6083d31 (2026-02-24).
SWE Pro2026-03-05Docent per-task resolution records, collection 032fb63d-4992-4bfc-911d-3b7dafcb931f.
Tau2-Bench2026-04-10Released leaderboard outcomes from sierra-research/tau2-bench for Airline, Retail, and Telecom.
AlpacaEval 22025-08-09tatsu-lab/alpaca_eval, cd543a1, using weighted_alpaca_eval_gpt4_turbo annotations.
WildBench v22024-08-01allenai/WildBench, d6b8dca, release v2.0625, with judge gpt-4o-2024-05-13.
LiveBench2025-04-07Hugging Face livebench/model_judgment, 9704e5d, using the released leaderboard split.

Table 9. Each benchmark audit is tied to an explicitly dated data source.

Computation. RULER reuses released task outcomes, so producing the report requires no language-model training, inference, or judge calls. Computation uses CPU-based Python with NumPy, SciPy, pandas, statsmodels, and Matplotlib. The full audit uses 5,000 resamples per benchmark with seed 42. The sensitivity sweep comprises 154 such runs, the split experiment uses 200 partitions per benchmark with 2,000 resamples each, and the calibrated comparison uses 100 synthetic benchmarks with 2,000 resamples per bootstrap method. Complete runtime and peak-memory measurements are not reported.

Notes. Dates for the 4 classic SWE-bench variants refer to the leaderboard revision, with outcome-revision dates shown separately. Pro and Tau2 dates identify the local snapshot retrievals. AlpacaEval, WildBench, and LiveBench dates identify the pinned upstream revisions. Revision timestamps are interpreted in UTC.

Supplementary Results for Validation

The held-out test and calibrated simulation in § 3 assess the correction using different reference scores. We describe their protocols and report the complete results below.

Sample-splitting protocol and results

Each of the 14 benchmarks uses 200 random partitions into a selection half of NS=⌊N/2⌋N_S = \lfloor N/2 \rfloor tasks and an evaluation half of NE=N−NSN_E = N - N_S tasks. SWE-bench splits balance repository representation. AlpacaEval uses 5 source-dataset strata and pooled LiveBench uses 3 categories. WildBench, individual LiveBench categories, and Tau2 use unstratified splits. When strata have odd sizes, the remaining assignments are randomized to reach the exact overall half-size. This approximately fixes the population mix but does not remove dependence among issues from the same repository. Every selection half uses m=⌊NS0.9⌋m = \lfloor N_S^{0.9} \rfloor and B=2,000B = 2{,}000 bootstrap resamples.

For benchmark hh, the signed mean errors dˉhk\bar{d}_{hk} and eˉhk\bar{e}_{hk} are the averages of dkd_k and eke_k in Equation (9) over the 200 random partitions. Table 1 reports 14−1∑h∣dˉhk∣14^{-1}\sum_h|\bar{d}_{hk}| and 14−1∑h∣eˉhk∣14^{-1}\sum_h|\bar{e}_{hk}|. Averaging signed errors first isolates the mean selection error within each benchmark. Taking its magnitude before combining benchmarks prevents overstatement in one benchmark from canceling understatement in another.

On Verified, where each half contains 250 tasks, the mean winner error falls from 1.96 to 0.18 pp (Table 10). Table 11 gives all 70 benchmark–position combinations. Each residual is computed within its split before summarization. The split SE is the standard deviation across partitions divided by 200\sqrt{200} and measures Monte Carlo precision conditional on the frozen task pool.

RankMean Error Before RULERMean Score CorrectionMean Error After RULER
11.961.79+0.18
20.661.38-0.72
30.721.24-0.52
41.411.16+0.26
51.521.08+0.44

Table 10. The correction substantially reduces SWE-bench Verified’s mean winner error.

BenchmarkRankError Before RULERScore CorrectionError After RULER
SWE Bash-only1+2.20±0.25+2.20 \pm0.25+1.60±0.02+1.60 \pm0.02+0.60±0.26+0.60 \pm0.26
SWE Bash-only2+1.24±0.26+1.24 \pm0.26+1.07±0.01+1.07 \pm0.01+0.17±0.26+0.17 \pm0.26
SWE Bash-only3+0.62±0.25+0.62 \pm0.25+0.79±0.01+0.79 \pm0.01−0.18±0.25-0.18 \pm0.25
SWE Bash-only4+0.20±0.27+0.20 \pm0.27+0.63±0.01+0.63 \pm0.01−0.43±0.27-0.43 \pm0.27
SWE Bash-only5+0.56±0.26+0.56 \pm0.26+0.52±0.01+0.52 \pm0.01+0.04±0.27+0.04 \pm0.27
SWE Verified1+1.96±0.24+1.96 \pm0.24+1.79±0.02+1.79 \pm0.02+0.18±0.25+0.18 \pm0.25
SWE Verified2+0.66±0.25+0.66 \pm0.25+1.38±0.02+1.38 \pm0.02−0.72±0.26-0.72 \pm0.26
SWE Verified3+0.72±0.27+0.72 \pm0.27+1.24±0.01+1.24 \pm0.01−0.52±0.27-0.52 \pm0.27
SWE Verified4+1.41±0.28+1.41 \pm0.28+1.16±0.01+1.16 \pm0.01+0.26±0.29+0.26 \pm0.29
SWE Verified5+1.52±0.25+1.52 \pm0.25+1.08±0.01+1.08 \pm0.01+0.44±0.25+0.44 \pm0.25
SWE Multimodal1+1.68±0.22+1.68 \pm0.22+1.29±0.01+1.29 \pm0.01+0.39±0.23+0.39 \pm0.23
SWE Multimodal2+0.55±0.23+0.55 \pm0.23+0.57±0.01+0.57 \pm0.01−0.01±0.23-0.01 \pm0.23
SWE Multimodal3−0.03±0.24-0.03 \pm0.24+0.14±0.01+0.14 \pm0.01−0.17±0.24-0.17 \pm0.24
SWE Multimodal4−0.13±0.25-0.13 \pm0.25−0.11±0.01-0.11 \pm0.01−0.02±0.25-0.02 \pm0.25
SWE Multimodal5−0.77±0.26-0.77 \pm0.26−0.09±0.02-0.09 \pm0.02−0.69±0.26-0.69 \pm0.26
SWE Multilingual1+2.62±0.37+2.62 \pm0.37+2.39±0.03+2.39 \pm0.03+0.23±0.39+0.23 \pm0.39
SWE Multilingual2+0.88±0.38+0.88 \pm0.38+1.70±0.02+1.70 \pm0.02−0.82±0.39-0.82 \pm0.39
SWE Multilingual3+0.92±0.35+0.92 \pm0.35+1.26±0.02+1.26 \pm0.02−0.34±0.35-0.34 \pm0.35
SWE Multilingual4+0.92±0.38+0.92 \pm0.38+0.96±0.02+0.96 \pm0.02−0.04±0.38-0.04 \pm0.38
SWE Multilingual5+1.65±0.41+1.65 \pm0.41+0.68±0.02+0.68 \pm0.02+0.96±0.41+0.96 \pm0.41
SWE Pro1+1.88±0.30+1.88 \pm0.30+1.59±0.03+1.59 \pm0.03+0.28±0.32+0.28 \pm0.32
SWE Pro2+0.45±0.32+0.45 \pm0.32+0.63±0.03+0.63 \pm0.03−0.18±0.33-0.18 \pm0.33
SWE Pro3+1.83±0.30+1.83 \pm0.30+0.23±0.03+0.23 \pm0.03+1.59±0.31+1.59 \pm0.31
SWE Pro4+0.31±0.31+0.31 \pm0.31−0.27±0.03-0.27 \pm0.03+0.58±0.32+0.58 \pm0.32
SWE Pro5−1.07±0.31-1.07 \pm0.31−0.51±0.05-0.51 \pm0.05−0.56±0.34-0.56 \pm0.34
Tau2 Airline1+4.35±0.85+4.35 \pm0.85+5.94±0.11+5.94 \pm0.11−1.59±0.91-1.59 \pm0.91
Tau2 Airline2+4.31±0.90+4.31 \pm0.90+3.34±0.07+3.34 \pm0.07+0.97±0.92+0.97 \pm0.92
Tau2 Airline3+2.07±0.87+2.07 \pm0.87+1.76±0.06+1.76 \pm0.06+0.31±0.88+0.31 \pm0.88
Tau2 Airline4−0.00±0.83-0.00 \pm0.83+0.25±0.05+0.25 \pm0.05−0.25±0.83-0.25 \pm0.83
Tau2 Airline5−2.02±0.77-2.02 \pm0.77−1.21±0.05-1.21 \pm0.05−0.81±0.77-0.81 \pm0.77
Tau2 Retail1+4.58±0.40+4.58 \pm0.40+3.05±0.06+3.05 \pm0.06+1.53±0.44+1.53 \pm0.44
Tau2 Retail2+1.32±0.37+1.32 \pm0.37+1.34±0.03+1.34 \pm0.03−0.02±0.38-0.02 \pm0.38
BenchmarkRankError Before RULERScore CorrectionError After RULER
Tau2 Retail3+0.48±0.39+0.48 \pm0.39+0.37±0.03+0.37 \pm0.03+0.11±0.40+0.11 \pm0.40
Tau2 Retail4−1.73±0.42-1.73 \pm0.42−0.44±0.03-0.44 \pm0.03−1.30±0.43-1.30 \pm0.43
Tau2 Retail5−1.09±0.45-1.09 \pm0.45−0.81±0.04-0.81 \pm0.04−0.28±0.46-0.28 \pm0.46
Tau2 Telecom1+0.22±0.14+0.22 \pm0.14+0.63±0.02+0.63 \pm0.02−0.41±0.15-0.41 \pm0.15
Tau2 Telecom2+0.05±0.14+0.05 \pm0.14+0.07±0.02+0.07 \pm0.02−0.02±0.15-0.02 \pm0.15
Tau2 Telecom3+0.11±0.38+0.11 \pm0.38−0.70±0.03-0.70 \pm0.03+0.82±0.41+0.82 \pm0.41
Tau2 Telecom4+0.79±0.49+0.79 \pm0.49+1.06±0.04+1.06 \pm0.04−0.27±0.50-0.27 \pm0.50
Tau2 Telecom5+1.64±0.54+1.64 \pm0.54+0.64±0.04+0.64 \pm0.04+0.99±0.55+0.99 \pm0.55
AlpacaEval 21+0.96±0.17+0.96 \pm0.17+0.85±0.01+0.85 \pm0.01+0.11±0.18+0.11 \pm0.18
AlpacaEval 22−0.32±0.17-0.32 \pm0.17+0.17±0.01+0.17 \pm0.01−0.49±0.18-0.49 \pm0.18
AlpacaEval 23−0.06±0.18-0.06 \pm0.18−0.01±0.02-0.01 \pm0.02−0.05±0.19-0.05 \pm0.19
AlpacaEval 24+0.72±0.17+0.72 \pm0.17+0.25±0.02+0.25 \pm0.02+0.47±0.18+0.47 \pm0.18
AlpacaEval 25+0.10±0.19+0.10 \pm0.19+0.41±0.02+0.41 \pm0.02−0.31±0.19-0.31 \pm0.19
WildBench v21+0.14±0.05+0.14 \pm0.05+0.17±0.00+0.17 \pm0.00−0.03±0.05-0.03 \pm0.05
WildBench v22−0.08±0.05-0.08 \pm0.05+0.03±0.01+0.03 \pm0.01−0.10±0.06-0.10 \pm0.06
WildBench v23+0.04±0.05+0.04 \pm0.05+0.11±0.01+0.11 \pm0.01−0.08±0.06-0.08 \pm0.06
WildBench v24+0.33±0.05+0.33 \pm0.05+0.19±0.01+0.19 \pm0.01+0.14±0.06+0.14 \pm0.06
WildBench v25+0.10±0.06+0.10 \pm0.06+0.16±0.00+0.16 \pm0.00−0.06±0.06-0.06 \pm0.06
LiveBench pooled1+0.24±0.28+0.24 \pm0.28+0.85±0.03+0.85 \pm0.03−0.61±0.30-0.61 \pm0.30
LiveBench pooled2+3.98±0.28+3.98 \pm0.28+1.96±0.03+1.96 \pm0.03+2.02±0.29+2.02 \pm0.29
LiveBench pooled3+1.47±0.27+1.47 \pm0.27+1.48±0.02+1.48 \pm0.02−0.01±0.28-0.01 \pm0.28
LiveBench pooled4−0.25±0.28-0.25 \pm0.28+1.10±0.02+1.10 \pm0.02−1.35±0.28-1.35 \pm0.28
LiveBench pooled5−0.68±0.32-0.68 \pm0.32+0.90±0.02+0.90 \pm0.02−1.58±0.33-1.58 \pm0.33
LiveBench Coding1+1.54±0.42+1.54 \pm0.42+2.33±0.06+2.33 \pm0.06−0.79±0.45-0.79 \pm0.45
LiveBench Coding2+0.39±0.44+0.39 \pm0.44+1.97±0.07+1.97 \pm0.07−1.58±0.47-1.58 \pm0.47
LiveBench Coding3+4.75±0.52+4.75 \pm0.52+2.55±0.06+2.55 \pm0.06+2.20±0.54+2.20 \pm0.54
LiveBench Coding4+3.27±0.52+3.27 \pm0.52+2.46±0.04+2.46 \pm0.04+0.82±0.54+0.82 \pm0.54
LiveBench Coding5+2.34±0.53+2.34 \pm0.53+2.22±0.04+2.22 \pm0.04+0.12±0.54+0.12 \pm0.54
LiveBench Instr.1+7.83±0.49+7.83 \pm0.49+4.61±0.07+4.61 \pm0.07+3.22±0.53+3.22 \pm0.53
LiveBench Instr.2+5.49±0.53+5.49 \pm0.53+3.84±0.05+3.84 \pm0.05+1.65±0.55+1.65 \pm0.55
LiveBench Instr.3+4.08±0.52+4.08 \pm0.52+3.51±0.03+3.51 \pm0.03+0.57±0.54+0.57 \pm0.54
LiveBench Instr.4+3.92±0.55+3.92 \pm0.55+3.21±0.03+3.21 \pm0.03+0.71±0.57+0.71 \pm0.57
LiveBench Instr.5+2.94±0.57+2.94 \pm0.57+2.90±0.03+2.90 \pm0.03+0.04±0.58+0.04 \pm0.58
LiveBench Language1+3.16±0.45+3.16 \pm0.45+2.95±0.05+2.95 \pm0.05+0.21±0.48+0.21 \pm0.48
LiveBench Language2+0.26±0.41+0.26 \pm0.41+1.91±0.05+1.91 \pm0.05−1.65±0.44-1.65 \pm0.44
LiveBench Language3−1.59±0.47-1.59 \pm0.47+1.43±0.06+1.43 \pm0.06−3.02±0.49-3.02 \pm0.49
LiveBench Language4+1.44±0.50+1.44 \pm0.50+1.75±0.06+1.75 \pm0.06−0.30±0.52-0.30 \pm0.52
LiveBench Language5+0.77±0.49+0.77 \pm0.49+2.06±0.05+2.06 \pm0.05−1.29±0.50-1.29 \pm0.50

Table 11. The reduction in selection error varies across benchmarks and ranks.

Notes. Results average 200 random half-splits with NS=NE=250N_S = N_E = 250, m=⌊NS0.9⌋m = \lfloor N_S^{0.9} \rfloor, and B=2,000B = 2{,}000. Error Before RULER is the selected model’s selection-half score minus its evaluation-half score. Score Correction is the estimated bias. Error After RULER subtracts that correction within each split. Entries are means across splits in percentage points (pp).

Notes. All 70 benchmark–position combinations use 200 random half-splits. Selection halves contain NS=⌊N/2⌋N_S = \lfloor N/2 \rfloor tasks, with m=⌊NS0.9⌋m = \lfloor N_S^{0.9} \rfloor and B=2,000B = 2{,}000. Error Before RULER is the selected model’s selection-half score minus its evaluation-half score. Score Correction is the estimated bias. Error After RULER subtracts that correction within each split. Entries are means and standard errors (SEs) across splits. All scores and errors are in pp. SEs measure Monte Carlo precision over partitions of the same frozen task pool, conditional on that pool.

Calibrated simulation and alternative adjustments

Whereas the held-out test uses a second sample to evaluate selected models, the calibrated simulation makes their true marginal means available. We retain dependence across models through a shared task difficulty and generate outcomes as

Yij∣Ti∼Bernoulli⁡{σ(αj+Ti)},Ti∼N(0,σT2),σ(x)=(1+e−x)−1,(34)Y_{ij} \mid T_i \sim\operatorname{Bernoulli}\{\sigma(\alpha_j + T_i)\}, \qquad T_i \sim N(0,\sigma_T^2), \qquad\sigma(x) = (1 + e^{-x})^{-1}, \tag*{(34)}

with conditionally independent model outcomes given the shared task effect. The common effect produces cross-model dependence, while the marginal model means provide known targets for the selected-score errors.

A Bernoulli GLM with model and task effects is fitted to the 472 nonconstant rows of Verified. The 28 tasks with constant outcomes across models are omitted from fitting, while the empirical audit uses all 500 tasks. Task effects are centered and their mean absorbed into the model intercepts. Their standard deviation is σ^T=2.6481\hat{\sigma}_T = 2.6481. Marginal model means are

τj=∫σ(α^j+σ^Tz)ϕ(z) dz,(35)\tau_j = \int\sigma(\hat{\alpha}_j + \hat{\sigma}_T z)\phi(z)\,dz, \tag*{(35)}

computed with 120-node Gauss–Hermite quadrature. Synthetic benchmark r=0,…,99r = 0,\ldots,99 uses seed 42+r42+r, and its mm-out-of-NN and standard bootstraps use seeds 42+r+5,000,00042+r+5{,}000{,}000 and 42+r+7,000,00042+r+7{,}000{,}000, respectively. Sample splitting, described below, uses seed 42+r+9,000,00042+r+9{,}000{,}000. Each method uses the same selected model at each raw rank on a given synthetic benchmark. Ties retain the original sort rule.

For the EB baseline, let xj=∑iYijx_j = \sum_i Y_{ij}. Fit positive (a,b)(a,b) by maximizing the beta-binomial marginal composite log-likelihood

ℓ(a,b)=∑j=1J[log⁡B(xj+a,N−xj+b)−log⁡B(a,b)],(36)\ell(a,b) = \sum_{j=1}^{J} [\log B(x_j+a,N-x_j+b)-\log B(a,b)], \tag*{(36)}

where binomial coefficients are constant in (a,b)(a,b). Optimize (log⁡a,log⁡b)(\log a,\log b) within [−10,14]2[-10,14]^2 using analytic gradients and two initializations: (1,1)(1,1) and a moment-based concentration max⁡{pˉ(1−pˉ)/sp2−1,0.1}\max\{\bar{p}(1-\bar{p})/s_p^2-1,0.1\} split according to pˉ\bar{p}, with numerical clipping. Select the successful fit with the lowest negative log-likelihood. A bounded derivative-free fallback is provided if needed. The posterior mean is (xj+a)/(N+a+b)(x_j+a)/(N+a+b), so the adjustment for the selected model is xj/Nx_j/N minus that mean. This composite likelihood ignores cross-model dependence and imposes a beta population model. The comparison concerns this specified EB baseline.

Sample splitting can also be used to correct scores. The held-out test in § 3 estimates the half-sample bias bNS(k)b_{N_S}(k) with the split error dkd_k in Equation (9). Because Proposition 2.2 shows that the leading term of the bias is proportional to N−1/2N^{-1/2}, averaging dkd_k over many random partitions and multiplying the average by NS/N\sqrt{N_S/N} gives a first-order estimate of bN(k)b_N(k). Subtracting this estimate from the score at position kk of the full leaderboard gives the corrected score. We use 200 random partitions of each synthetic benchmark into 2 halves of NS=236N_S = 236 tasks, so the multiplier is 1/2\sqrt{1/2}. The correction has 2 limitations. First, a half-sample ranking is noisier than the full ranking and often places different models at a given position. The split error therefore reflects selection among NSN_S tasks, and rescaling it to NN tasks corrects only the leading term of the bias. Second, each half contains only half of the tasks, so the estimate is more variable than the mm-out-of-NN adjustment. At rank 1, the sample-splitting adjustment has a standard deviation of 0.49 pp across synthetic benchmarks, compared with 0.29 pp for the mm-out-of-NN adjustment.

Table 12 reports the mean signed errors of all 4 corrections. The mm-out-of-NN correction leaves a mean winner residual of 0.21 pp, compared with 0.46 pp for the standard bootstrap and 1.10 pp for EB. Thus the 3 adjustments remove approximately 85%, 68%, and 22% of mean winner bias, respectively. Resampling m<Nm<N tasks therefore removes more bias than resampling all NN.

RankNo Correctionmm-out-of-NN bootstrapStandard bootstrapEBSample splitting
1+1.41±0.14+1.41 \pm0.14+0.21±0.15+0.21 \pm0.15+0.46±0.16+0.46 \pm0.16+1.10±0.14+1.10 \pm0.14+0.06±0.16+0.06 \pm0.16
2+0.80±0.17+0.80 \pm0.17−0.06±0.17-0.06 \pm0.17+0.20±0.18+0.20 \pm0.18+0.49±0.17+0.49 \pm0.17−0.02±0.18-0.02 \pm0.18
3+0.66±0.18+0.66 \pm0.18−0.12±0.18-0.12 \pm0.18+0.13±0.18+0.13 \pm0.18+0.36±0.18+0.36 \pm0.18−0.07±0.18-0.07 \pm0.18
4+0.98±0.16+0.98 \pm0.16+0.23±0.16+0.23 \pm0.16+0.41±0.17+0.41 \pm0.17+0.69±0.16+0.69 \pm0.16+0.16±0.17+0.16 \pm0.17
5+0.96±0.17+0.96 \pm0.17+0.24±0.18+0.24 \pm0.18+0.40±0.18+0.40 \pm0.18+0.68±0.17+0.68 \pm0.17+0.13±0.19+0.13 \pm0.19

Table 12. The mm-out-of-NN correction and sample splitting leave the smallest mean winner residuals. Mean Signed Error (pp)

Notes. Entries are the mean signed error of the selected model’s score relative to its known mean, ±\pm Monte Carlo SE, in pp over the 100 synthetic benchmarks of Table 2. No Correction gives the raw selection bias. The other columns give residuals after each correction. The mm-out-of-NN and standard bootstraps use m=⌊N0.9⌋m = \lfloor N^{0.9}\rfloor and m=Nm = N, respectively, with 2,000 resamples each. EB denotes empirical Bayes. It replaces each selected model’s score with its posterior mean under a beta-binomial model fitted to the benchmark. Sample splitting subtracts the split error averaged over 200 random half-splits and rescaled to NN tasks.

Notes. Entries are the root mean squared error (RMSE) of the selected model’s score relative to its known mean, ±\pm Monte Carlo SE from the delta method, in pp over the synthetic benchmarks and corrections of Table 12.

Notes. Each entry is a method’s RMSE minus that of the mm-out-of-NN bootstrap over the same 100 synthetic benchmarks, ±\pm delta-method Monte Carlo SE of the paired difference, in pp. Positive entries indicate a higher RMSE than the mm-out-of-NN bootstrap. EB denotes empirical Bayes.

Sample splitting leaves a mean residual of 0.06 pp, removing nearly all of the winner bias, but its adjustment is more variable and its winner RMSE is higher than that of the mm-out-of-NN correction. At rank 1, the mm-out-of-NN correction has the lowest RMSE, 1.55 pp, compared with 1.62 pp for the standard bootstrap, 1.64 pp for sample splitting, and 1.75 pp for EB (Table 13). It also has the lowest RMSE at positions 2–5 and ties EB at rank 2 to the displayed precision.

RankNo Correctionmm-out-of-NN bootstrapStandard bootstrapEBSample splitting
11.97±0.131.97 \pm0.131.55±0.101.55 \pm0.101.62±0.101.62 \pm0.101.75±0.121.75 \pm0.121.64±0.091.64 \pm0.09
21.85±0.181.85 \pm0.181.74±0.141.74 \pm0.141.77±0.151.77 \pm0.151.74±0.171.74 \pm0.171.77±0.141.77 \pm0.14
31.87±0.161.87 \pm0.161.78±0.141.78 \pm0.141.80±0.151.80 \pm0.151.79±0.151.79 \pm0.151.84±0.151.84 \pm0.15
41.85±0.111.85 \pm0.111.64±0.131.64 \pm0.131.71±0.121.71 \pm0.121.72±0.111.72 \pm0.111.73±0.141.73 \pm0.14
51.97±0.151.97 \pm0.151.79±0.131.79 \pm0.131.87±0.141.87 \pm0.141.85±0.141.85 \pm0.141.87±0.141.87 \pm0.14

Table 13. The mm-out-of-NN correction gives the winner’s score the lowest root mean squared error. RMSE (pp)

Notes. Methods use the same 100 synthetic benchmarks with 472 tasks and 173 models. The mm-out-of-NN and standard bootstraps use m=⌊N0.9⌋m = \lfloor N^{0.9} \rfloor and m=Nm = N, respectively, with 2,000 resamples. EB denotes empirical Bayes and uses the fitted beta-binomial posterior mean. Sample splitting uses 200 random half-splits per benchmark. Mean Score Correction is the mean amount subtracted from the selected model’s score. Mean Residual is its mean signed error after correction, and RMSE is the root mean squared error after correction. All 3 measures are in pp and use the selected model’s known mean as the reference. At rank 2, the mm-out-of-NN and EB RMSEs tie to the displayed precision.

Because all methods correct the same selected models on the same draws, their squared errors can be compared benchmark by benchmark. At rank 1, the paired RMSE difference from the mm-out-of-NN correction is 0.07 pp (SE 0.02) for the standard bootstrap, 0.09 pp (SE 0.03) for sample splitting, and 0.20 pp (SE 0.07) for EB (Table 14). At rank 2, EB and the mm-out-of-NN correction have the same RMSE to the displayed precision (SE of the paired difference 0.07).

RankNo CorrectionStandard bootstrapEBSample splitting
1+0.42±0.09+0.42 \pm0.09+0.07±0.02+0.07 \pm0.02+0.20±0.07+0.20 \pm0.07+0.09±0.03+0.09 \pm0.03
2+0.11±0.09+0.11 \pm0.09+0.03±0.02+0.03 \pm0.02+0.00±0.07+0.00 \pm0.07+0.03±0.03+0.03 \pm0.03
3+0.08±0.07+0.08 \pm0.07+0.01±0.03+0.01 \pm0.03+0.00±0.05+0.00 \pm0.05+0.05±0.02+0.05 \pm0.02
4+0.21±0.08+0.21 \pm0.08+0.07±0.02+0.07 \pm0.02+0.07±0.06+0.07 \pm0.06+0.09±0.03+0.09 \pm0.03
5+0.18±0.06+0.18 \pm0.06+0.07±0.02+0.07 \pm0.02+0.06±0.04+0.06 \pm0.04+0.08±0.03+0.08 \pm0.03

Table 14. At rank 1, every alternative has a higher RMSE than the mm-out-of-NN correction on the same synthetic benchmarks. RMSE minus mm-out-of-NN RMSE (pp)

At rank 1, RMSE differs by at most 0.20 pp across methods, whereas mean residuals differ by up to 1.03 pp (Table 15). These results distinguish removing a systematic selection effect from eliminating sampling variation.

RankCorrection MethodMean Score Correction (pp)Mean Residual (pp)RMSE (pp)
1mm-out-of-NN bootstrap1.20+0.211.55
1Standard bootstrap0.95+0.461.62
1Beta-binomial EB0.32+1.101.75
1Sample splitting1.35+0.061.64
2mm-out-of-NN bootstrap0.86-0.061.74
2Standard bootstrap0.60+0.201.77
2Beta-binomial EB0.31+0.491.74
2Sample splitting0.82-0.021.77
3mm-out-of-NN bootstrap0.78-0.121.78
3Standard bootstrap0.53+0.131.80
3Beta-binomial EB0.30+0.361.79
3Sample splitting0.73-0.071.84
4mm-out-of-NN bootstrap0.75+0.231.64
4Standard bootstrap0.57+0.411.71
4Beta-binomial EB0.29+0.691.72
4Sample splitting0.82+0.161.73
5mm-out-of-NN bootstrap0.72+0.241.79
5Standard bootstrap0.56+0.401.87
5Beta-binomial EB0.28+0.681.85
5Sample splitting0.84+0.131.87

Table 15. The mm-out-of-NN correction has the lowest RMSE across the first 5 ranks.

Robustness Studies

The robustness studies distinguish three choices that affect a RULER report. First, we examine the i.i.d. tasks assumption. We try alternative bootstrap methods that relaxes the i.i.d. assumption and see whether the estimated selection effect and ranking uncertainty depend on how tasks are grouped. Second, we change the resample size within the range that Appendix § 7 recommends and see whether the estimated inflation is robust to the exponent. Finally, we give different rank mass use for model tiering.

Notes. Pooled resamples task rows, Stratified resamples within repositories, and Cluster applies a shared random multiplier to all tasks in each repository. Each method reports the position-1 score correction and the number of models in Tier 1 at 80% rank mass. All schemes use B=5,000B = 5{,}000. Pooled resampling uses m=⌊N0.9⌋m = \lfloor N^{0.9}\rfloor. Actual stratified resample sizes are 268 for Bash-only, 268 for Verified, 273 for Multimodal, 174 for Multilingual, and 217 for Pro. Cluster results use shared Webb multipliers and the unscaled selection gap, as specified in Appendix § 11.§ 1.

Beyond i.i.d. tasks

The theory in § 2 and the mm-out-of-NN bootstrap treat task rows as i.i.d. draws from a common task distribution. Real benchmarks can depart from this assumption in different ways. A benchmark may be assembled with a fixed mix of sources, such as the repositories of SWE-bench, so that a new draw of tasks would keep roughly the same composition. Tasks from the same source may also share features, such as a codebase, that move all models’ outcomes on those tasks together. Each pattern corresponds to a resampling scheme. Pooled resampling of task rows matches i.i.d. tasks, stratified resampling within repositories matches a fixed repository mix, and cluster resampling matches shared repository effects. If the three schemes give close score adjustments and rank summaries, RULER’s conclusions are robust to the i.i.d. task assumption.

Pooled resampling is the baseline. It draws mm independent rows from the empirical task distribution and uses the factor m/N\sqrt{m/N} in Equation (1). For stratified resampling, repository gg with ngn_g tasks receives mg=max⁡{1,round⁡(ngm/N)}m_g = \max\{1,\operatorname{round}(n_gm/N)\}. We sample independently with replacement within each repository, concatenate the sampled rows, and use the actual size mstrat=∑gmgm_{\mathrm{strat}} = \sum_g m_g in the rescaling. This approximately fixes repository composition while continuing to treat issues within a repository as independent. On Multilingual, rounding produces mstrat=174m_{\mathrm{strat}} = 174 rather than the pooled m=⌊N0.9⌋=169m = \lfloor N^{0.9}\rfloor= 169.

For cluster resampling, we use a wild-cluster bootstrap. Define the repository residual vector and the perturbed means

ηg=∑i∈g(Yi−τ^),τ^∗(b)=τ^+N−1∑gWg(b)ηg.(37)\eta_g = \sum_{i\in g}(\mathbf{Y}_i-\hat{\tau}), \qquad\hat{\tau}^{*(b)} = \hat{\tau} + N^{-1}\sum_g W_g^{(b)}\eta_g. \tag*{(37)}

Each cluster’s multiplier is shared across all its tasks and all models, so a draw shifts every task in a repository together. The Webb multipliers take the six values {−3/2,−1,−1/2,1/2,1,3/2}\left\{-\sqrt{3/2},-1,-\sqrt{1/2},\sqrt{1/2},1,\sqrt{3/2}\right\} with equal probability. We rank the perturbed means and use the unscaled selection gap, which corresponds to m=Nm=N in Equation (1). These pseudo-means can lie outside [0,1][0,1]. The independent-task theorem does not establish the calibration of this procedure, and the small cluster counts—12 repositories for Verified and 3 issue subcategories for Tau2 Telecom—make these comparisons stress tests.

Across the 5 SWE-bench variants, pooled and stratified resampling give leading-tier sizes that differ by at most 1 model and score adjustments that differ by at most 0.27 pp (Table 16). Both summaries are therefore robust to a fixed repository mix. The cluster stress test can change rank summaries more substantially. On Verified, the pooled, stratified, and cluster position-1 adjustments are 1.12, 1.20, and 1.16 pp, respectively, while the leading tier contains 13 models under pooled resampling, 12 under stratified resampling, and 4 under clustering. Table 17 gives the corresponding rank-1 masses. Shared repository effects thus leave Verified’s estimated inflation nearly unchanged but can alter the reported ordering. Tau2 Telecom is more sensitive. Clustering its 3 labeled issue subcategories raises the adjustment from 0.18 to 1.05 pp and lowers the leader’s rank-1 mass from 94.3% to 75.5%. With only 3 clusters, this result serves as a sensitivity diagnostic.

BenchmarkNumber of RepositoriesPosition 1 Correction (pp)Number of Models in Tier 1
PooledStratifiedClusterPooledStratifiedCluster
SWE Bash-only121.091.210.86664
SWE Verified121.121.201.1613124
SWE Multimodal120.940.820.84555
SWE Multilingual411.501.231.72987
SWE Pro110.870.870.54443

Table 16. SWE-bench score adjustments remain stable across alternative sampling schemes.

BenchmarkResampling SchemeModel 1Model 2Model 3Number of TiersNumber of Models in Tier 1
SWE Bash-onlyPooled41.616.818.4106
SWE Bash-onlyStratified43.115.518.1106
SWE Bash-onlyCluster57.418.115.9124
SWE VerifiedPooled30.529.026.41513
SWE VerifiedStratified29.029.426.21512
SWE VerifiedCluster30.629.133.8244
SWE MultimodalPooled34.834.022.035
SWE MultimodalStratified34.834.322.235
SWE MultimodalCluster31.837.527.035
SWE MultilingualPooled44.833.69.149
SWE MultilingualStratified45.535.29.048
SWE MultilingualCluster51.934.17.947
SWE ProPooled74.921.11.964
SWE ProStratified74.721.31.864
SWE ProCluster87.610.80.363

Table 17. Repository clustering changes leading tiers in some SWE-bench variants. Notes. Models 1–3 are the first 3 models in the original displayed order, including any observed score ties. Each probability column gives the percentage of bootstrap rank mass assigned to rank 1 for that model. Pooled, Stratified, and Cluster use the sampling schemes and resample sizes defined in Table 16, with B=5,000B = 5{,}000. The last 2 columns report the total number of tiers and the number of models in Tier 1 under the 80% common-intersection rule.

Choice of resample size

Appendix B shows that, averaged over all leaderboard positions, exponents from 0.4 to 0.9 are nearly equally accurate when models are closely spaced. We therefore recompute the position-1 adjustment of every audited benchmark at γ∈{0.4,0.5,…,0.9}\gamma\in\{0.4, 0.5, \ldots, 0.9\}, using m=max⁡{2,⌊Nγ⌋}m = \max\{2, \lfloor N^\gamma\rfloor\}, B=5,000B = 5{,}000, and seed 42. Table 18 reports the 84 adjustments, and Table 19 gives their exact resample sizes. The default γ=0.9\gamma= 0.9 was fixed before inspecting outcomes.

Benchmark0.40.50.60.70.80.9
SWE Bash-only1.611.591.531.381.261.09
SWE Verified1.911.901.771.561.341.12
SWE Multimodal1.131.131.131.041.020.94
SWE Multilingual2.102.162.151.941.741.50
SWE Pro2.101.901.671.431.160.87
Tau2 Airline6.135.755.404.804.003.07
Tau2 Retail3.012.932.782.602.452.20
Tau2 Telecom1.231.230.930.580.360.18
AlpacaEval 21.331.130.930.770.640.53
WildBench v20.350.300.240.170.130.10
LiveBench pooled2.582.191.650.960.390.12
LiveBench Coding4.123.422.772.131.531.11
LiveBench Instr.4.024.094.154.174.093.86
LiveBench Language4.073.623.032.532.091.90

Table 18. Score adjustments stay positive across resample-size exponents from 0.4 to 0.9, and the default gives the smallest. Resample-Size Exponent γ\gamma

Benchmark0.40.50.60.70.80.9
SWE Bash-only12224177144268
SWE Verified12224177144268
SWE Multimodal12224278146273
SWE Multilingual917305495169
SWE Pro10193665119217
Tau2 Airline4710152233
Tau2 Retail61017274470
Tau2 Telecom61017274470
AlpacaEval 2142855108211412
WildBench v2153162124248495
LiveBench pooled10173156100178
LiveBench Coding61118294878
LiveBench Instr.4710152233
LiveBench Language71119315285

Table 19. Resample sizes range from 4 to 495 tasks across exponents from 0.4 to 0.9. Resample-Size Exponent γ\gamma

Across this range, the correction keeps its direction, and the default is conservative. Every adjustment is positive, and in each of the 14 benchmarks the default gives the smallest adjustment in the range. A smaller exponent in the range would therefore estimate more inflation at the top of each leaderboard than the audit reports. In 12 benchmarks, including the 6 whose 10 leading models lie within 3 standard errors, the adjustment also keeps its order of magnitude across the range. Tau2 Telecom and pooled LiveBench show by far the largest relative changes.

These 2 benchmarks have well-separated leaders, which receive 94.3% and 99.0% of the bootstrap rank-1 mass at the default, and their small default adjustments grow several-fold at smaller exponents. For a well-separated leader, the spread design in Appendix B shows that smaller exponents are less accurate, and the two-model example in Appendix A shows that they overcorrect. The simulations Notes. Entries are position-1 adjustments in pp for 14 benchmarks and 6 exponents. Each run uses m=max⁡{2,⌊Nγ⌋}m = \max\{2, \lfloor N^\gamma\rfloor\}, B=5,000B = 5{,}000, and seed 42. Score ties follow the deterministic sort convention in Appendix D. The default exponent is γ=0.9\gamma= 0.9.

Notes. Entries are the exact resample sizes m=max⁡{2,⌊Nγ⌋}m = \max\{2, \lfloor N^\gamma\rfloor\} for the runs in Table 18. Column headings give γ\gamma, and the default exponent is 0.9.

therefore favor the small default adjustments for these 2 leaders.

Choice of rank-interval level

Retaining more rank mass gives a broader summary of the unresolved ordering. On Verified, increasing central rank mass from 80% to 90%, 95%, and 99% expands the leading tier from 13 models to 15, 24, and 29, respectively. Table 20 reports the corresponding partitions in every Notes. The 2 column groups separately report the total number of tiers and the number of models in Tier 1. Within each group, the percentage headings specify the central rank mass retained in each model’s interval. Partitions are recomputed from the same full-precision RPMs, using m=⌊N0.9⌋m = \lfloor N^{0.9} \rfloor, B=5,000B = 5{,}000, and the common-intersection rule. Only the retained rank mass changes across columns.

BenchmarkNumber of TiersNumber of Models in Tier 1
80%90%95%99%80%90%95%99%
SWE Bash-only10976691316
SWE Verified151311813152429
SWE Multimodal332255810
SWE Multilingual43339111212
SWE Pro66444455
Tau2 Airline21115777
Tau2 Retail22225666
Tau2 Telecom64321233
AlpacaEval 2211614113445
WildBench v2151212102336
LiveBench pooled97751116
LiveBench Coding86542359
LiveBench Language765445715
LiveBench Instr.544319263042

Table 20. Leading tiers often expand when the retained rank mass increases.

benchmark. The underlying RPM is unchanged. The difference is how much of each model’s rank distribution the interval includes.

Wider intervals can change where the common-intersection sweep closes a tier, so the resulting partitions need not form a nested hierarchy. We therefore recompute each level from the full-precision RPM using the same reporting conventions (Appendix § 9).

Benchmark Planning and Supplementary Reports

The score adjustment provides a scale for planning larger task pools, and the full RPM reveals how ranking uncertainty extends beyond the models highlighted in the main text. This appendix develops the planning calculation and provides larger rank-distribution displays for Verified and Tau2-Bench.

Task-pool sizing for a target level of score inflation

More tasks reduce the expected selection advantage of the winner. In the exact-tier regime, bN(1)≈δ/Nb_N(1) \approx\delta/\sqrt{N}, so halving expected inflation requires approximately 4 times as many tasks. For a target expected bias ε\varepsilon, the corresponding planning rule is N≳(δ/ε)2N \gtrsim(\delta/\varepsilon)^2.

For Verified, the position-1 adjustment gives δ^=500 b^500(1)≈25.1\hat{\delta} = \sqrt{500}\,\hat{b}_{500}(1) \approx25.1 pptasks\sqrt{\text{tasks}}. Table 21 translates this estimate into task counts for different targets. The calculation holds the model pool, task distribution, and covariance structure fixed and uses the exact-tier approximation. Local gaps or changes in these populations can alter the extrapolation, so the table gives approximate planning sizes rather than guaranteed minima.

Target Expected Winner Bias (pp)Approximate Number of Tasks
2.00158
1.00631
0.502,522
0.2510,086
0.1063,035

Table 21. Under the exact-tier approximation, halving expected winner inflation requires about 4 times as many tasks.

Notes. Planning sizes use N≳(δ^/ε)2N \gtrsim(\hat{\delta}/\varepsilon)^2 with δ^=500 b^500(1)≈25.1 pptasks\hat{\delta} = \sqrt{500}\,\hat{b}_{500}(1) \approx25.1\ \text{pp}\sqrt{\text{tasks}} from Verified’s position-1 estimate. Counts are rounded upward using full precision. The exact-tier approximation holds the model pool, task distribution, and covariance structure fixed. Values are approximate planning sizes.

Figure 3. Ranking uncertainty extends beyond SWE-bench Verified’s leading models.

Notes. The display is the leading 30×3030 \times30 submatrix of the full 173×173173 \times173 RPM, based on 500 tasks, m=⌊N0.9⌋m = \lfloor N^{0.9} \rfloor, and B=5,000B = 5{,}000. Red horizontal lines mark 80% tier boundaries in raw model order. Cropped rows need not sum to one, while full RPM rows do. Axis indices identify raw-order model rows without imposing an order within score ties.

Additional rank probability matrices

The larger RPM places Verified’s leading tier in the context of the first 30 models and rank positions. Uncertainty extends beyond the observed leaders: the models tied at raw positions 9–10 each have an 80% rank interval of [5,21]. The display shows where rank mass is concentrated and how far it spreads across the leaderboard.

The Tau2 domains illustrate different degrees of ranking resolution. Airline and Retail each place 5 of 7 models in the leading tier, and Telecom’s leader forms a tier of its own under pooled resampling. Figure 4 shows the RPMs behind those tier summaries. Red lines mark the 80% tier boundaries. Airline has only N=50N = 50 tasks, and each resample draws m=⌊500.9⌋=33m = \lfloor50^{0.9} \rfloor= 33 of them. Telecom’s sensitivity to clustering is discussed in § 4.2.

Figure 4. Tau2 domains differ in how sharply they resolve the leading tier.

Notes. The RPMs use pooled task resampling with J=7J = 7, m=⌊N0.9⌋m = \lfloor N^{0.9} \rfloor, and B=5,000B = 5{,}000, with N=50N = 50 tasks for Airline and N=114N = 114 for Retail and Telecom. Red lines mark 80% tier boundaries from the common-intersection rule. In panels (a) and (b), 5 of 7 models share the leading tier. In panel (c), the leading tier contains only the leader. § 4.2 reports Telecom’s sensitivity to issue-subcategory clustering.

Extended Related Work

Selection, comparison, and rank uncertainty. Rank-induced bias connects leaderboard interpretation to the broader problem of selection. The WC was documented in oil-lease auctions [10, 33] and formalized as the optimizer’s curse [32]. Selective inference studies inference after selection [22, 3], while empirical Bayes uses information across estimates for shrinkage [18, 20]. RULER builds on classical mm-out-of-NN resampling [8, 7] and its use for selected-winner correction [35]. It extends the correction to every leaderboard position while accounting for cross-model dependence within a task. The RPM and rank intervals complement the score correction by showing how much of the ordering the task pool resolves.

Statistical comparison predates LLM leaderboards. Koehn [21] studies significance testing for machine-translation evaluation, and Mathur et al. [25] examines how automatic evaluation metrics, outliers, and system comparisons affect conclusions about translation quality. Our target is the mean estimation error of the model selected by its position after ranking. Rank confidence sets [27, 26] and classifier-comparison procedures [15, 13, 6] address inferential comparisons. RULER complements these approaches by estimating the score error associated with selection and displaying the resampled ordering. Its rank-mass intervals summarize that distribution rather than provide simultaneous confidence coverage.

Validity and reproducibility. Raji et al. [29] questions the scope and meaning of general-purpose AI benchmarks. Belz et al. [5] develops quantified reproducibility assessment, extended across common result types by Belz and Thomson [4]. Schmidtova et al. [31] surveys automatic metric practices in natural-language generation. These concerns extend beyond task sampling: a precise score can measure the wrong capability, and implementations can vary independently of the task pool (Table 22).

ConcernQuestionRelationship to this paper
Statistical comparisonDoes a prespecified difference exceed evaluation noise?Selection after sorting changes which error is averaged.
Selection biasHow much does a chosen position benefit from noise?Our bias target. RPMs additionally describe ranking variability.
Construct validityDoes the task pool measure the intended capability?Our target is conditional on the chosen task population.
ReproducibilityHow much do independently repeated implementations or studies agree?Frozen outcomes do not capture all reproduction variability.
Metric validityDoes the score agree with the quality notion of interest?Debiasing a score does not validate its underlying metric.

Table 22. Selection bias is distinct from statistical comparison, metric validity, and reproducibility.

Other leaderboard failure modes. Leaderboards can also become unreliable through changes in the evaluation process or the models submitted to it. Adaptive holdout reuse ([9, 17, 30]) and contamination ([14]) can compromise an evaluation’s independence from model development. Private evaluation suites ([23]) can reduce leakage and adaptation. Benchmark and prompt sensitivity ([12, 1]), run-to-run variance (Bouthillier et al., 2021), and efficient or uncertainty-aware evaluation ([28, 36, 11, 2]) concern further sources of variability and the design of evaluations. Rank-induced bias remains even for a fixed, uncontaminated model pool evaluated once on held-out tasks, because sorting selects favorable noise along with ability.

References

  1. [1]Norah Alzhahrani, Hisham Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Tw aresh, Areeb Alowisheq, et al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13787–13805, 2024.
  2. [2]Siavash Ameli, Siyuan Zhuang, Ion Stoica, and Michael W. Mahoney. A statistical framework for ranking LLM-based chatbots. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=rAoEub6Nw2.arxiv.org/abs/2412.18407
  3. [3]Isaiah Andrews, Toru Kitagawa, and Adam McCloskey. Inference on winners. Quarterly Journal of Economics, 139(1):305–358, 2024.
  4. [4]Anya Belz and Craig Thomson. Quantified reproducibility assessment for four common types of evaluation results in NLP/ML. Computational Linguistics, 2026. doi: 10.1162/COLI.a.625.DOI
  5. [5]Anya Belz, Maja Popovic, and Simon Mille. Quantified reproducibility assessment of NLP results. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16–28, 2022. doi: 10.18653/v1/2022.acl-long.2.DOI
  6. [6]Alessio Benavoli, Giorgio Corani, Janez Demšar, and Marco Zaffalon. Time for a change: A tutorial for comparing multiple classifiers through bayesian analysis. Journal of Machine Learning Research, 18(77):1–36, 2017.arxiv.org/abs/1606.04316
  7. [7]Peter J. Bickel and Anat Sakov. On the choice of m in the m out of n bootstrap and confidence bounds for extrema. Statistica Sinica, 18(3):967–985, 2008.
  8. [8]Peter J. Bickel, Friedrich Götze, and Willem R. van Zwet. Resampling fewer than n observations: Gains, losses, and remedies for losses. Statistica Sinica, 7(1):1–31, 1997.
  9. [9]Avrim Blum and Moritz Hardt. The ladder: A reliable leaderboard for machine learning competitions. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1006–1014, 2015. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. Accounting for variance in machine learning benchmarks. Proceedings of machine learning and systems, 3:747–769, 2021.
  10. [10]E. C. Capen, R. V. Clapp, and W. M. Campbell. Competitive bidding in high-risk situations. Journal of Petroleum Technology, 23(6):641–653, 1971.DOI
  11. [11]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning, 2024.arxiv.org/abs/2403.04132
  12. [12]Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. In Proceedings of the 35th Conference on Neural Information Processing Systems: Datasets and Benchmarks Track, 2021.arxiv.org/abs/2107.07002
  13. [13]Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(1):1–30, 2006.DOI
  14. [14]Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8706–8719, 2024.arxiv.org/abs/2311.09783
  15. [15]Thomas G. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895–1923, 1998. doi: 10.1162/089976698300017197.DOI
  16. [16]Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. URL https://arxiv.org/abs/2404.04475.
  17. [17]Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Tոնiann Pitassi, Omer Reingold, and Aaron Roth. Generalization in adaptive data analysis and holdout reuse. In Advances in Neural Information Processing Systems, volume 28, pages 2350–2358, 2015.arxiv.org/abs/1506.02629
  18. [18]Bradley Efron. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106(496):1602–1614, 2011.pubmed.ncbi.nlm.nih.gov/22505788
  19. [19]Zheng Fang and Andres Santos. Inference on directionally differentiable functions. Review of Economic Studies, 86(1):377–412, 2019.arxiv.org/abs/1404.3763
  20. [20]Jiaying Gu and Roger Koenker. Invidious comparisons: Ranking and selection as compound decisions. Econometrica, 91(1):1–41, 2023.arxiv.org/abs/2012.12550
  21. [21]Philipp Koehn. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395. Association for Computational Linguistics, 2004. URL https://aclanthology.org/W04-3250/.
  22. [22]Jason D. Lee, Dennis L. Sun, Yuekai Sun, and Jonathan E. Taylor. Exact post-selection inference, with application to the lasso. Annals of Statistics, 44(3):907–927, 2016.DOI
  23. [23]Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasუნaga, Yian Zhang, Deepak Narayanan, Yuhui Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022.arxiv.org/abs/2211.09110
  24. [24]Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. WildBench: Benchmarking LLMs with challenging tasks from real users in the wild. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=MKEHCx25xp.arxiv.org/abs/2406.04770
  25. [25]Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, 2020. doi: 10.18653/v1/2020.acl-main.448.DOI
  26. [26]Magne Mogstad, Joseph P. Romano, Azeem M. Shaikh, and Daniel Wilhelm. Statistical uncertainty in the ranking of journals and universities. AEA Papers and Proceedings, 112:630–634, 2022.
  27. [27]Magne Mogstad, Joseph P. Romano, Azeem M. Shaikh, and Daniel Wilhelm. Inference for ranks with applications to mobility across neighbourhoods and academic achievement across countries. Review of Economic Studies, 91(1):476–518, 2024.
  28. [28]Felipe Maia Polo, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun, and Mikhail Yurochkin. Efficient multi-prompt evaluation of llms. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 22483–22512. Curran Associates, Inc., 2024. doi: 10.52202/079017-0707. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/28236482f64a72eec43706b6f3a6c511-Paper-Conference.pdf.DOI
  29. [29]Iniolụwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.15366, 2021. URL https://arxiv.org/abs/2111.15366.
  30. [30]Rebecca Roelofs, Vaishaal Shankar, Benjamin Recht, Sara Fridovich-Keil, Moritz Hardt, John Miller, and Ludwig Schmidt. A meta-analysis of overfitting in machine learning. In Advances in Neural Information Processing Systems, volume 32, 2019.
  31. [31]Patricia Schmidtova, Saad Mahamood, Simone Balloccu, Ondrej Dusek, Albert Gatt, Dimitra Gkatzia, David M. Howcroft, Ondrej Platek, and Adarsa Sivaprasad. Automatic metrics in natural language generation: A survey of current evaluation practices. In Proceedings of the 17th International Natural Language Generation Conference, pages 557–583, 2024. doi: 10.18653/v1/2024.inlg-main.44.DOI
  32. [32]James E. Smith and Robert L. Winkler. The optimizer’s curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52(3):311–322, 2006.DOI
  33. [33]Richard H. Thaler. Anomalies: The winner’s curse. Journal of Economic Perspectives, 2(1):191–202, 1988.DOI
  34. [34]Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddhartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. LiveBench: A challenging, contamination-free LLM benchmark. arXiv preprint arXiv:2406.19314, 2024. URL https://arxiv.org/abs/2406.19314.
  35. [35]Sikun Xu, Raphael Thomadsen, and Dennis Zhang. The winner’s curse in data-driven decision making: Evidence and solutions. SSRN Working Paper, 2025. Available at SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5930537.DOI
  36. [36]Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F. Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking LLMs via uncertainty quantification. In Advances in Neural Information Processing Systems: Datasets and Benchmarks Track, volume 37, 2024.
  37. [37]Hua Zhong and Ross L. Prentice. Bias-reduced estimators and confidence intervals for odds ratios in genome-wide association studies. Biostatistics, 9(4):621–634, 2008.DOI
  38. [38]Sebastian Zöllner and Jonathan K. Pritchard. Overcoming the winner’s curse: Estimating penetrance parameters from case-control data. American Journal of Human Genetics, 80(4):605–615, 2007.

Paper details

vlnc:W_8kzkqkmw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents