betaFeedback

WorksComputer Science

A Hierarchical Statistical Framework for Evaluating Agent Reliability

Abstract

Agents operating in complex scenarios often face uncertainty due to incomplete information about the environment they operate in, ambiguous user queries, complex business rules and the inherently stochastic dynamics of the governing policy. Several new benchmarks have been proposed to evaluate LLM-based agents on complex tasks in multi-tool environments, including settings with policy constraints and varying levels of intent ambiguity. These conditions make it important to evaluate agentic systems systematically and quantify their performance under different conditions to assess their reliability in production. This need is evident in the recently introduced Thinkingbox benchmark, where even the strongest models struggle to perform consistently across diverse business scenarios. This paper presents a statistical framework for evaluating uncertainty and reliability in agentic systems. We define a hierarchy of scenarios, instances, and samples to assess model performance and variability systematically. We apply this framework to common metrics found in the literature: mean pass rate, pass@kpass@k, and passhatkpasshat^k, with uncertainty estimates for each metric. Aggregating metrics at different levels of the hierarchy reveals different structures and properties of the policy model; we connect this hierarchical aggregation with the known process of coarse-graining as intended in the physics and statistical literature. We independently derive exact finite-sample expressions for the sampling variances of the unbiased pass@kpass@k and passhatkpasshatk estimators; related results also appear in recent submissions. Our central contribution is a constructive derivation of hierarchical uncertainty propagation, making explicit its correspondence with coarse-graining and separating execution noise from task heterogeneity.

Introduction

Large language models (LLMs) have enabled increasingly capable AI agents that combine a policy model, tools for interacting with an external environment, and a harness that orchestrates their execution. Given a task, an agent generates a solution trajectory comprising reasoning steps and actions, potentially modifying the environment along the way. Stochastic decision-making, task ambiguity, and complex interactions with the environment can produce different trajectories and outcomes across repeated executions of the same task. Evaluating these systems therefore requires more than a single aggregate performance score: we must establish how consistently they succeed, where failures concentrate, and how precisely performance has been measured.

Yet benchmark reporting often emphasizes a single state-of-the-art (SOTA) result without quantifying its statistical uncertainty [9]. When reported differences between models amount to only a few percentage points, point estimates alone do not establish whether those differences reflect genuine improvements or sampling noise. This limitation is particularly consequential for agentic systems, whose performance can vary both across tasks and across repeated attempts at the same task. Bjarnason et al. [10] quantify this variability on SWE-Bench-Verified and show that small reported improvements can be comparable to evaluation noise. The recently introduced Think-ingbox benchmark [5] likewise highlights substantial variability among strong models, both within and across domains, with uncertainty estimates complicating claims of a clear leader even within a single domain. These observations motivate a statistical treatment that distinguishes variability in agent behavior from uncertainty in benchmark estimates.

Reliability also extends beyond success frequency. Rabanser et al. [11] organize agent reliability into consistency, robustness, predictability, and safety, and find that capability gains need not yield comparable improvements in reliability. Their perspective is complementary to ours: rather than proposing a comprehensive taxonomy of reliability, we focus on the statistical interpretation of success-based metrics and the propagation of uncertainty across repeated attempts, tasks, and scenarios. Such analysis supports, but does not replace, evaluation of robustness or failure severity.

From a user’s perspective, an agent should solve a task reliably without requiring repeated requests. Single-attempt success, summarized by pass@1pass@1 (mean pass rate), is therefore a central measure of practical performance. However, the average alone neither captures how consistently individual tasks are solved nor indicates how precisely performance has been estimated. Repeated evaluation of each task helps characterize execution variability, while confidence intervals quantify uncertainty in the resulting estimates. These measurements provide evidence relevant to production reliability, although their applicability depends on how closely the benchmark reflects deployment conditions. Reliability must also be considered alongside the cost of obtaining a successful outcome. An agent may eventually find a solution through retries, but this behavior can be undesirable for long-horizon tasks that consume substantial time and computational resources. Secondary metrics such as number of tool calls, retries, and token usage therefore complement success-based measures by characterizing the efficiency of solution trajectories. Together, these measurements can guide reinforcement fine-tuning (RFT) toward more consistent and efficient task completion, rather than improvements in aggregate success alone.

Prior work provides complementary approaches to statistical evaluation. Chen et al. [6] introduce the unbiased pass@kpass@k estimator, while Yao et al. [1] introduce passkpass^{k} and its unbiased estimator to measure agent reliability across repeated trials. Hariri et al. [12] develop a Bayesian alternative to pass@kpass@k-based model ranking, with analytical posterior uncertainty and credible intervals for underlying success probabilities. Dewasurendra [13] studies a different problem: finite-sample confidence intervals and identification limits for population pass@kpass@k, particularly when extrapolating beyond the observed sampling budget. These questions are distinct from the exact frequentist sampling variances of the unbiased pass@kpass@k and passkpass^{k} estimators studied here; bootstrap uncertainty estimates likewise do not provide analytical expressions for those variances. Recent manuscripts also analyze finite-sample uncertainty for these metrics [7, 8], but do not address hierarchical uncertainty propagation.

In this work, we present a unified statistical framework for evaluating performance and uncertainty in LLM-based agentic systems. Our contributions are threefold. First, we organize evaluation into scenarios, task instances, and repeated samples, distinguishing execution variability, differences across tasks, and uncertainty in estimated performance. We interpret aggregation through this hierarchy as coarse-graining in physics and statistics [14], showing how averages reveal collective behavior while obscuring task-level differences. Second, we analyze uncertainty in mean pass rate and derive exact finite-sample sampling variances for the unbiased pass@kpass@k and passkpass^{k} estimators, connecting these per-task results to uncertainty in population-level averages. Third, we illustrate the framework on agentic benchmarks to show what these metrics and their uncertainty estimates reveal about reliability. The central contribution is not an isolated metric or variance formula, but an account of how task heterogeneity and finite evaluation budgets jointly shape the conclusions drawn from benchmark results. The framework draws on established statistical results; the coarse-graining connection and analytical derivations are developed in Appendices § 7 and § 8.

Hierarchical data structure showing scenarios, instances, and samples

Figure 1. Hierarchical data structure: scenarios contain multiple instances, each instance is sampled multiple times to account for variability. Aggregating metrics at different levels of this hierarchy corresponds to coarse graining.

Modeling Assumptions and Definitions

Throughout the paper, the policy model governs the agent’s decisions, while the environment includes the tools and infrastructure with which it interacts. We evaluate a fixed agent configuration, including its policy, harness, and environment setup. Following the terminology used in [5], we define the hierarchical structure of our framework as follows:

  1. Scenario: A scenario is defined by a specific set of tools (e.g., a set of MCP servers) and the environment’s initial state. It is grounded in the real-world context it aims to simulate, such as retail, healthcare, finance, etc.

  1. Instance: A specific task that the agent must complete using the tools available in a given scenario. An instance is a data point in our dataset comprising the user’s intent, the system prompt, and other relevant context. A dataset is a collection of instances across one or more scenarios.

  1. Sample: A single attempt by the agent to resolve a given instance, also referred to as a “sample trajectory.” To assess the agent’s performance and variability, we need multiple samples for each instance.

For the analysis, we write πθ(y∣x)\pi_\theta(y \mid x) for the trajectory distribution induced by the policy with parameters θ\theta in this fixed evaluation setup. Here, xx specifies the task and its context, and yy is the resulting trajectory; the distribution includes any modeled environment or user-interaction randomness. Conceptually, population evaluation involves two levels of sampling: a task xx is drawn from an unknown population distribution PtaskP_{\mathrm{task}}, and a trajectory yy is then generated conditionally on that task:

x∼Ptask,y∣x∼πθ(⋅∣x).(1)x \sim P_{\mathrm{task}}, \qquad y \mid x \sim\pi_{\theta}(\cdot\mid x). \tag*{(1)}

In practice, we evaluate a finite dataset D={x1,…,x∣D∣}\mathcal{D} = \{x_1, \ldots, x_{|\mathcal{D}|}\} and repeatedly execute each task. § 3.1 connects this dataset to the task population and the agent’s task-specific success probabilities.

In the agentic context, the final output yy comprises a sequence of actions generated by the policy model and executed by the orchestrator in the environment, and may also include multiturn interactions with the user (see Figure 2). We define a sample trajectory as follows:

Figure 2. Sample trajectory: (left) a single attempt of the agent to resolve a given instance, comprising actions, states, and user interactions. (right) Repeatedly sampling the same instance typically leads to different solution paths and possibly different outcomes.

Y^={A,S,U},(2)\hat{Y} = \{A, S, U\}, \tag*{(2)}

where AA is the sequence of actions taken by the model, SS is the sequence of states observed in the environment, and UU is the sequence of user interactions (if any)1. An evaluation experiment can be understood as a rollout of the policy model πθ\pi_{\theta} on a set of instances, each associated with its own evaluation criterion. As in any experiment, stochasticity arises from various factors. We account for it by running multiple samples per instance and statistically analyzing the results to assess the model’s average performance and variability.

Throughout the paper, we adopt the following modeling assumption: each sample outcome is a random variable ξs,i,r\xi_{s,i,r}, where ss is the scenario index, ii is the instance index, and rr is the sample index. For this analysis, we consider binary outcomes: graders report each experiment as a success or a failure according to whether the task meets a specified set of evaluation criteria. The hierarchical variance decomposition extends to other rewards with finite variance, but the Bernoulli and success-metric formulas below rely on binary outcomes. Under this model, each sample is drawn from a Bernoulli distribution conditional on the true success probability ρi\rho_i of instance ii:

ξs,i,r∣ρi∼Bernoulli⁡(ρi)(3)\xi_{s,i,r} \mid\rho_i \sim\operatorname{Bernoulli}(\rho_i) \tag*{(3)}

Here, ρi\rho_i is the true success probability for instance ii. In practice, we estimate it by running the model NN times on that instance and computing the empirical pass rate pi=Ci/Np_i = C_i/N, where CiC_i is

MetricMain intuition
Pass@1Typical single-attempt performance
Pass@kCan retries rescue the task?
Passk^{k}Is success consistently reproducible?

the number of successful samples. Instances differ in their probability of success, and variation among the resulting estimates reflects both differences between instances and finite-sample noise. The statistical framework below separates these contributions.

Intuitively, evaluation proceeds in two steps: repeated attempts estimate each instance’s pass rate, then averaging these estimates gives a scenario-level mean pass rate. Section 3.2 explains how uncertainty propagates through these averages when estimating performance on a broader task population. From a practical perspective, a task’s estimated success probability indicates how likely an attempt is to succeed, while its uncertainty estimate indicates how precisely that probability is known. Other metrics considered here include pass@k and passk^{k}. For a given task, the former measures the probability that at least one of kk independent attempts succeeds; the latter measures the probability that all kk attempts succeed.

Statistical Analysis Framework

The dataset’s hierarchical structure (scenarios, instances, and samples) requires global metrics that capture model performance at different levels of granularity. Moving up the hierarchy corresponds to a coarse-graining process in which we aggregate performance metrics by averaging. However, this process can obscure important information about model performance in specific domains and on individual instances. For example, Ref. [5] presents a detailed benchmark of several models at the scenario and dataset levels and shows that no model consistently outperforms the others across all scenarios, even when some models lead in the across-scenario rankings. This finding highlights the importance of considering both fine- and coarse-grained metrics. Although pass rates can be aggregated throughout the hierarchy by simple averaging, propagating uncertainty across levels requires additional care, particularly for evaluations with limited numbers of instances [9, 15]. We address this issue in Section 3.2 within the hierarchical uncertainty-propagation framework, and provide an explicit derivation in Appendix § 7, which accounts for variability at multiple levels and their interactions; the derivation borrows the formalism of coarse-graining [14] from the physics and statistical literature. Informally, coarse-graining is the process of zooming out the description of a system, moving from a microscopic to a macroscopic description, e.g. describing a fluid in terms of its individual constituents versus a continuous field. Mathematically, zooming out consists in averaging the microscopic variables, which inevitably leads to a loss of resolution while highlighting collective, population-level aspects.

From task distributions to success probabilities

A benchmark contains tasks, not success probabilities. The connection is that each task has a probability of success for the particular agent being evaluated. Let r(x,y)∈{0,1}r(x,y) \in\{0,1\} indicate whether trajectory yy successfully completes task xx according to its evaluation criterion. For a fixed agent configuration and grader, define

ρθ(x)=Ey∼πθ(⋅∣x)[r(x,y)],ρi≡ρθ(xi).(4)\rho_\theta(x) = \mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}[r(x,y)], \qquad\rho_i \equiv\rho_\theta(x_i). \tag*{(4)}

For a fixed task, ρθ(x)\rho_\theta(x) is a fixed, generally unknown number. Repeated attempts randomize the outcome, not this underlying probability. A benchmark author may anticipate difficulty from task characteristics such as the required reasoning or tool dependencies. However, success probability measures difficulty relative to the evaluated agent and grader: another agent may succeed more or less often on the same task. Like a school assignment, a task can have an intended level of complexity while posing different challenges to different students.

Now imagine selecting tasks from PtaskP_{\mathrm{task}} and recording each selected task’s true success probability. The resulting distribution of probabilities is gg. It describes differences across tasks, not uncertainty about the probability of one fixed task. For example, suppose a population contains two tasks, selected with probabilities 0.75 and 0.25, on which the agent succeeds with probabilities 0.2 and 0.8, respectively. Then gg assigns weight 0.75 to 0.2 and weight 0.25 to 0.8. The weights come from how often tasks are selected; the success probabilities come from the agent’s performance on those tasks. Changing either the task mix or the agent can therefore change gg. Formally, this transformation of the task distribution is called its pushforward through ρθ\rho_\theta [16, 17, 18]:

g=(ρθ)#Ptask,Fg(t)=Pr⁡x∼Ptask(ρθ(x)≤t).(5)g = (\rho_\theta)_{\#}P_{\mathrm{task}}, \qquad F_g(t) = \Pr_{x\sim P_{\mathrm{task}}}\left(\rho_\theta(x) \le t\right). \tag*{(5)}

Here, Fg(t)F_g(t) is the fraction of the task population, weighted by its selection probabilities, on which the agent’s success probability is at most tt. Writing ρ∼g\rho\sim g is shorthand for selecting a task and taking its associated success probability, not an additional independent sampling step:

x∼Ptask⟶ρ=ρθ(x)⟶ξ∣ρ∼Bernoulli⁡(ρ).(6)x \sim P_{\mathrm{task}} \quad\longrightarrow\quad\rho= \rho_\theta(x) \quad\longrightarrow\quad\xi\mid\rho\sim\operatorname{Bernoulli}(\rho). \tag*{(6)}

The population’s single-attempt success probability is therefore the mean of gg: Pr⁡(ξ=1)=Ex∼Ptask[ρθ(x)]=Eρ∼g[ρ]\Pr(\xi= 1) = \mathbb{E}_{x\sim P_{\mathrm{task}}}[\rho_\theta(x)] = \mathbb{E}_{\rho\sim g}[\rho]. In the two-task example, it is 0.75 ×\times 0.2 + 0.25 ×\times 0.8 = 0.35. For the finite dataset D\mathcal{D}, selecting an equally weighted task means using the empirical task distribution

P^D=1∣D∣∑i=1∣D∣δxi,(7)\widehat{P}_{\mathcal{D}} = \frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}\delta_{x_i}, \tag*{(7)}

where δxi\delta_{x_i} is a unit point mass at task xix_i. This describes selection from the benchmark; it does not make the population distribution known. Replacing each task by its true success probability gives

gD=(ρθ)#P^D=1∣D∣∑i=1∣D∣δρθ(xi).(8)g_{\mathcal{D}} = (\rho_\theta)_{\#}\widehat{P}_{\mathcal{D}} = \frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}\delta_{\rho_\theta(x_i)}. \tag*{(8)}

Thus, P^D\widehat{P}_{\mathcal{D}} puts equal weight on the observed tasks, while gDg_{\mathcal{D}} puts the same weight on their true success probabilities. The mean of gDg_{\mathcal{D}} is the true dataset-level pass rate, ∣D∣−1∑iρi|\mathcal{D}|^{-1}\sum_i \rho_i. In practice, the ρi\rho_i are unknown. Replacing them by the observed pass rates pi=Ci/Np_i = C_i/N defined in § 2 gives

g^D=1∣D∣∑i=1∣D∣δpi.(9)\widehat{g}_{\mathcal{D}} = \frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}\delta_{p_i}. \tag*{(9)}

Unlike gDg_{\mathcal{D}}, g^D\widehat{g}_{\mathcal{D}} contains finite-run estimation noise as well as differences in true success probabilities. Even tasks with identical true probabilities can have different observed pass rates after a limited number of attempts. For a single binary outcome, the true success probability determines its distribution, but an estimated pass rate alone does not tell us how precisely that probability is known. Likewise, the dataset’s mean pass rate alone does not reveal how success probabilities vary across tasks.

Finally, estimating performance on the fixed benchmark and estimating performance on a broader task population are different goals. For the benchmark’s own mean, more independent attempts can make estimation uncertainty arbitrarily small. For the population mean, even perfectly known probabilities on those same tasks leave uncertainty about unobserved tasks. The analysis below models benchmark tasks as independent draws from a common task population to quantify this second contribution. This is a modeling assumption, not a guarantee of benchmark construction: deliberately selected or related tasks may require a different sampling model, and more attempts cannot correct an unrepresentative task mix.

Pass rate and hierarchical uncertainty propagation

In this section we estimate the scenario’s population mean success probability using the law of total variance. We first compute per-instance pass rates and associated uncertainties, then aggregate them at the scenario level by averaging across independently sampled instances.

Uncertainty propagation is not trivial, however, because it encompasses both within-instance variability (due to stochasticity in model behavior) and between-instance variability (due to differences in instance difficulty). Consider the first level of coarse-graining, from instances to a scenario:

  1. Within a fixed instance, the NN outcomes are i.i.d. Bernoulli conditional on its true success probability ρi\rho_i. The number of successes is CiC_i, and the empirical pass rate is pip_i:

Ci∣ρi∼Binomial⁡(N,ρi),pi=CiN,σpi2:=Var⁡(pi∣ρi)=ρi(1−ρi)N.(10)C_i \mid\rho_i \sim\operatorname{Binomial}(N,\rho_i), \qquad p_i = \frac{C_i}{N}, \qquad\sigma_{p_i}^{2} := \operatorname{Var}(p_i \mid\rho_i) = \frac{\rho_i(1-\rho_i)}{N}. \tag*{(10)}
  1. Across instances, the success probabilities vary with task difficulty. We model them as independent and identically distributed draws from a common, unknown distribution gg:

ρi∼g,Eg[ρ]=μ,Var⁡g(ρ)=τ2.(11)\rho_i \sim g, \qquad\mathbb{E}_g[\rho] = \mu, \qquad\operatorname{Var}_g(\rho) = \tau^2. \tag*{(11)}

Here, gg is induced by the task distribution within the scenario under consideration through the map ρθ\rho_\theta defined above. We only require that the first two moments of gg exist and are finite.

  1. Conditional on their success probabilities, the runs are independent within and across instances. The instance probabilities ρi\rho_i are also independent across instances.

Now consider M=∣S∣M = |S| instances in scenario SS. The scenario-level empirical pass rate is the average of the instance-level pass rates:

pˉ=1M∑i=1Mpi.(12)\bar{p} = \frac{1}{M}\sum_{i=1}^{M}p_i. \tag*{(12)}

Its uncertainty follows from the law of total variance (see, e.g., Ref. [19] and the derivation in Appendix § 7):

Var⁡(pˉ)=E[Var⁡(pˉ∣ρ1:M)]+Var⁡(E[pˉ∣ρ1:M]).(13)\operatorname{Var}(\bar{p}) = \mathbb{E}[\operatorname{Var}(\bar{p} \mid\rho_{1:M})] + \operatorname{Var}(\mathbb{E}[\bar{p} \mid\rho_{1:M}]). \tag*{(13)}

Conditional on ρ1:M\rho_{1:M}, the instance-level pass rates are independent, so

Var⁡(pˉ∣ρ1:M)=1M2∑i=1MVar⁡(pi∣ρi)=1M2∑i=1Mσpi2.(14)\operatorname{Var}(\bar{p} \mid\rho_{1:M}) = \frac{1}{M^2}\sum_{i=1}^{M}\operatorname{Var}(p_i \mid\rho_i) = \frac{1}{M^2}\sum_{i=1}^{M}\sigma_{p_i}^{2}. \tag*{(14)}

Averaging over the instance distribution gives the expected within-instance contribution:

E[Var⁡(pˉ∣ρ1:M)]=1M2∑i=1MEg[σpi2]=Eg[σpi2]M.(15)\mathbb{E}[\operatorname{Var}(\bar{p} \mid\rho_{1:M})] = \frac{1}{M^2}\sum_{i=1}^{M}\mathbb{E}_g[\sigma_{p_i}^{2}] = \frac{\mathbb{E}_g[\sigma_{p_i}^{2}]}{M}. \tag*{(15)}

The conditional mean is

E[pˉ∣ρ1:M]=1M∑i=1Mρi.(16)\mathbb{E}[\bar{p}\mid\rho_{1:M}] = \frac{1}{M}\sum_{i=1}^{M}\rho_i. \tag*{(16)}

Since the ρi\rho_i are i.i.d. with variance τ2\tau^2, the between-instance contribution is

Var⁡(E[pˉ∣ρ1:M])=Var⁡(1M∑i=1Mρi)=τ2M.(17)\operatorname{Var}(\mathbb{E}[\bar{p}\mid\rho_{1:M}]) = \operatorname{Var}\left(\frac{1}{M}\sum_{i=1}^{M}\rho_i\right) = \frac{\tau^2}{M}. \tag*{(17)}

Combining the two terms gives the total variance of the scenario-level pass rate [9]:

Var⁡(pˉ)=τ2+Eg[σρi2]M.(18)\operatorname{Var}(\bar{p}) = \frac{\tau^2 + \mathbb{E}_g[\sigma_{\rho_i}^2]}{M}. \tag*{(18)}

The first term is between-instance variability and is unaffected by increasing the number of runs NN; this term would still exist when N=1N = 1. The second term is within-instance sampling variability and decreases as NN increases. Both terms decrease as the number of independently sampled instances MM increases. Note that if the different tasks had the same difficulty, meaning the same pass-rate probability, the extra term τ2=0\tau^2 = 0 and only sampling noise would be relevant.

Although run-level variability is averaged as we move up the hierarchy, it remains important at the instance level, where it can identify areas for model improvement and is closely tied to user experience. The same decomposition applies at the next level of the hierarchy, from scenarios to the global level, by treating scenarios as the units being averaged. Applying it recursively yields global performance metrics and uncertainty estimates that account for variability at multiple levels.

To summarize, each instance is sampled NN times to capture within-instance sampling variability; for each instance we can estimate its pass rate and the standard-error on it. The second step, we take an average across instances, that is the average of the pass rates. This mean pass rate now has an error propagated over from each individual instance, and one that captures the variability across instances, that is a population level variability, not a single-instance variability. The two errors scale differently with NN and MM and reveal different aspects of the agent’s performance on the eval set. In the next section we use a toy model as a proxy to show this difference explicitly.

A "random matrix" toy model

To make the two averaging steps explicit, consider a toy model with an N×MN \times M data matrix X^\hat{X}. Columns represent tasks and rows represent repeated attempts. We retain i=1,…,Mi = 1,\ldots,M as the instance index and r=1,…,Nr = 1,\ldots,N as the run index, writing the matrix entry as XriX_{ri}; for l=1l = 1, this corresponds to ξir\xi_{ir} above.

In a real benchmark, each column is a task xix_i whose agent-specific success probability ρθ(xi)\rho_\theta(x_i) is unknown. Here, drawing a probability from gg stands in for sampling a task and taking its associated probability, preserving the binary-outcome model under the assumptions of § 3.1 without generating task content. The model is

ρ1,…,ρM∼iidg,Xri∣ρ1,…,ρM∼Binomial⁡(l,ρi),(19)\rho_1,\ldots,\rho_M \overset{\mathrm{iid}}{\sim} g,\qquad X_{ri}\mid\rho_1,\ldots,\rho_M \sim\operatorname{Binomial}(l,\rho_i), \tag*{(19)}

where entries are conditionally independent given the task probabilities. Draw the MM probabilities once, then generate NN entries in each column using that column’s probability. The positive integer ll counts Bernoulli trials per entry, with l=1l = 1 recovering binary outcomes. The overall average is

Xˉ=1NM∑i=1M∑r=1NXri.(20)\bar{X} = \frac{1}{NM}\sum_{i=1}^{M}\sum_{r=1}^{N}X_{ri}. \tag*{(20)}

For ℓ=1\ell= 1, this is equivalent to Eq. (12), with the two-level average making the hierarchy explicit: repeated entries within each column estimate one task’s mean performance, while differences across column means reflect both task heterogeneity and finite-run noise. Averaging the column means gives the same point estimate as averaging all entries, but uncertainty propagation must preserve the task structure. As in the preceding subsection, the target is population performance: the unconditional variance considers variation in both the selected tasks and their execution outcomes. The conditional moments are

E[Xri∣ρi]=ℓρi,E[Xri2∣ρi]=ℓρi(1−ρi)+ℓ2ρi2.(21)\mathbb{E}[X_{ri}\mid\rho_i] = \ell\rho_i,\qquad\mathbb{E}[X_{ri}^{2}\mid\rho_i] = \ell\rho_i(1-\rho_i) + \ell^{2}\rho_i^{2}. \tag*{(21)}

From here, the unconditional moments can be defined using the shorthand notations:

q=Eρ∼g[ρ(1−ρ)],v=Var⁡ρ∼g(ρ).(22)q = \mathbb{E}_{\rho\sim g}[\rho(1-\rho)],\qquad v = \operatorname{Var}_{\rho\sim g}(\rho). \tag*{(22)}

In the notation of the preceding subsection, v=τ2v=\tau^{2} and q/N=Eg[σρi2]q/N=\mathbb{E}_{g}[\sigma_{\rho_i}^{2}], where σρi2\sigma_{\rho_i}^{2} is the conditional sampling variance of the Bernoulli pass-rate estimator. Thus, for ℓ=1\ell=1, vv and q/Nq/N we recover the between-instance and expected within-instance variance contributions, respectively. Using this in the definition of the unconditional variance of XriX_{ri}:

Var⁡(Xri)=E[Xri2]−E[Xri]2=ℓq+ℓ2E[ρ2]−ℓ2E[ρ]2=ℓq+ℓ2v.(23)\operatorname{Var}(X_{ri}) = \mathbb{E}[X_{ri}^{2}] - \mathbb{E}[X_{ri}]^{2} = \ell q + \ell^{2}\mathbb{E}[\rho^{2}] - \ell^{2}\mathbb{E}[\rho]^{2} = \ell q + \ell^{2}v. \tag*{(23)}

With this, we can compute the unconditional variance of Xˉ\bar{X} using Eq. (20):

Var⁡(Xˉ)=1N2M2Var⁡(∑i=1M∑r=1NXri)=1N2M2∑i=1M∑r=1N∑r′=1NCov⁡(Xri,Xr′i).(24)\operatorname{Var}(\bar{X}) = \frac{1}{N^{2}M^{2}}\operatorname{Var}\left(\sum_{i=1}^{M}\sum_{r=1}^{N}X_{ri}\right) = \frac{1}{N^{2}M^{2}}\sum_{i=1}^{M}\sum_{r=1}^{N}\sum_{r'=1}^{N}\operatorname{Cov}(X_{ri},X_{r'i}). \tag*{(24)}

Figure 3 distinguishes the data matrix from the covariance matrix used in this sum. For a fixed column ii, let xi=(X1i,…,XNi)⊤\mathbf{x}_i=(X_{1i},\ldots,X_{Ni})^{\top}; its unconditional covariance matrix has entries

Data layout and unconditional covariance structure

Figure 3. Data layout and unconditional covariance structure, panels (a)–(b). Each data column shares one latent probability ρi\rho_i. Diagonal covariance entries contain both sampling noise and instance variability; off-diagonal entries within a block contain only the shared instance contribution.

(Σi)rr′=Cov⁡(Xri,Xr′i)=ℓq δrr′+ℓ2v.(25)(\Sigma_i)_{rr'} = \operatorname{Cov}(X_{ri},X_{r'i}) = \ell q\,\delta_{rr'} + \ell^{2}v. \tag*{(25)}

For r≠r′r \ne r', conditional independence gives E[XriXr′i∣ρi]=l2ρi2\mathbb{E}[X_{ri}X_{r'i}\mid\rho_i] = l^2\rho_i^2, hence the off-diagonal covariance is l2vl^2v. This unconditional covariance arises because both entries share the same randomly selected task probability; the executions remain independent conditional on that probability. Different columns are independent and have zero cross-covariance. The diagonal here therefore refers to pairing an entry with itself, not to the diagonal of the rectangular data matrix X^\hat{X}. Figure 4 shows the covariance structure across all task columns. Each of the MM covariance blocks has NN diagonal elements with lq+l2vlq+l^2v and N(N−1)N(N-1) ordered off-diagonal elements with l2vl^2v. Using this in Eq. (24) we obtain the general expression for the unconditional variance:

Unconditional covariance across all task columns

Figure 4. Unconditional covariance across all task columns. Blocks for different columns have zero cross-covariance. The variance of the overall average is the sum of all covariance entries divided by (NM)2(NM)^2.

Var⁡(Xˉ)=MN2M2[N(lq+l2v)+N(N−1)l2v]=lqNM+l2vM.(26)\operatorname{Var}(\bar{X}) = \frac{M}{N^2M^2}\left[N(lq+l^2v)+N(N-1)l^2v\right] = \frac{lq}{NM}+\frac{l^2v}{M}. \tag*{(26)}

Choosing l=1l=1, we now recover Eq. (18). More rows improve the precision of each task mean, reducing the within-task contribution by 1/N1/N. The between-task contribution reflects variation in true success probabilities and does not receive this additional reduction. Both contributions are reduced by averaging over MM tasks. If every task has the same true success probability, v=0v=0; sampling different probabilities from a common gg does not impose that condition.

An explicit numerical example. To make things concrete, we compare two ways of estimating uncertainty in population mean performance from the same observations: preserving the task structure, or incorrectly treating every execution as an independent task. Both produce the same mean, but different standard errors. Figure 5 follows the analysis of a single observed matrix with M=20M=20 tasks and binary outcomes (l=1l=1). The task probabilities ρi\rho_i are drawn once from g=Uniform⁡(0,1)g=\operatorname{Uniform}(0,1), then for each column ii, we sample N=100N=100 entries independently with success probability ρi\rho_i to obtain a 100×20100 \times20 matrix. Eq. (20) can be thought of as a two step process: first take task level means and their errors, then take the mean of these single task means and combine their standard errors hierarchically using Eq. (26). For comparison, flattening the same matrix and treating all entries as independent task-level evidence gives:

Two-step averaging of one observed matrix with four panels: observed outcomes, task-level estimates, overall uncertainty, and more runs for the same tasks

Figure 5. Two-step averaging of one observed matrix with M=20M = 20 tasks. (a) The outcome of the matrix generation process for the MM tasks, showing only the first N=20N = 20 rows of binary outcomes for convenience. (b) Task-level means with estimated SE. (c) The within-task and noise-corrected between-task contributions to the hierarchical variance estimate, compared with the pooled estimate; both methods give the same mean. (d) SE estimates from nested row prefixes of the same 100×20100 \times20 matrix. Repeated attempts are not additional independent tasks.

SE⁡^pooled=sX2NM,sX2=1NM−1∑r=1N∑i=1M(Xri−Xˉ)2.(27)\widehat{\operatorname{SE}}_{\mathrm{pooled}}=\sqrt{\frac{s_X^2}{NM}}, \qquad s_X^2=\frac{1}{NM-1}\sum_{r=1}^{N}\sum_{i=1}^{M}(X_{ri}-\bar{X})^2. \tag*{(27)}

Because averaging is a linear operation and each task has the same number of runs, averaging the task-level means is equivalent to averaging all matrix entries. The pooled SE, however, ignores the hierarchy: additional runs improve estimates of existing tasks rather than providing additional independent tasks. Likewise, simply averaging the individual task SEs does not propagate uncertainty through the second averaging step. At N=M=20N = M = 20, both procedures give Xˉ=0.4825\bar{X} = 0.4825, but the hierarchical SE is 0.07329 and the pooled SE is 0.02502, a factor of approximately 2.93 smaller. The propagated within-task contribution alone has SE 0.01971. The plot also compares estimates from nested row prefixes, N∈{2,5,10,20,50,100}N \in\{2, 5, 10, 20, 50, 100\}, of this same matrix. Thus, the discrepancy is not caused by different observations or a different mean estimator, but by treating repeated executions as additional independent tasks. Panel (d) makes the consequence visible: the pooled SE continues to shrink with more runs, while the hierarchical SE retains uncertainty associated with having evaluated only 20 tasks.

pass@k and its Uncertainty

We now consider another important metric in agent evaluation [6]: pass@kpass@k, the probability that at least one of kk independent attempts succeeds on the same task. The gap between pass@1pass@1 and pass@kpass@k reveals solutions that the current policy can produce but does not produce consistently. This can motivate reinforcement fine-tuning (RFT) aimed at increasing single-attempt success by making successful behaviors more likely [20, 21]. However, pass@kpass@k describes repeated sampling from the current policy; it neither predicts nor bounds the performance achievable after training. Given CiC_i successes in NN independent attempts on task ii, the unbiased estimator of pass@kpass@k is

hk(Ci)=1−(N−Cik)(Nk),1≤k≤N.(28)h_k(C_i) = 1 - \frac{\binom{N-C_i}{k}}{\binom{N}{k}}, \qquad1 \le k \le N. \tag*{(28)}

Here, (N−Cik)=0\binom{N-C_i}{k}=0 when N−Ci<kN-C_i<k. The estimator averages the at-least-one-success indicator over all size-kk subsets drawn without replacement from the observed runs. This subset construction is distinct from the kk future independent attempts whose success probability it estimates. Over repeated evaluations, E[hk(Ci)∣ρi]=1−(1−ρi)k\mathbb{E}[h_k(C_i)\mid\rho_i]=1-(1-\rho_i)^k. Before evaluating its standard error, we summarize a few properties of the estimator; a constructive derivation is provided in Appendix Appendix § 8:

  • If Ci=0C_i=0, then hk(Ci)=0h_k(C_i)=0 for every admissible kk: no observed subset contains a success. This does not imply that the true success probability is zero.

  • If k=1k=1, the estimator reduces to the empirical pass rate, h1(Ci)=Ci/Nh_1(C_i)=C_i/N.

  • For large NN with kk fixed, the estimator can be approximated as (see also Appendix Appendix § 8.§ 3):

hk(Ci)≃1−(1−pi)k,(29)h_k(C_i) \simeq1-(1-p_i)^k, \tag*{(29)}

where pi=Ci/Np_i=C_i/N is the empirical pass rate for instance ii. The expression on the right is a plug-in point estimate, not the exact unbiased estimator at finite NN.

Under Ci∣ρi∼Binomial⁡(N,ρi)C_i\mid\rho_i\sim\operatorname{Binomial}(N,\rho_i), the estimator’s standard error is the square root of its exact conditional sampling variance:

Var⁡(hk(Ci)∣ρi)=∑c=0Nhk(c)2(Nc)ρic(1−ρi)N−c−(1−(1−ρi)k)2.(30)\operatorname{Var}(h_k(C_i)\mid\rho_i)=\sum_{c=0}^{N}h_k(c)^2\binom{N}{c}\rho_i^c(1-\rho_i)^{N-c}-\left(1-(1-\rho_i)^k\right)^2. \tag*{(30)}

Related finite-sample analysis of the pass@kpass@k estimator is presented in the recent manuscript [7]. Appendix Appendix § 8.§ 2 gives our derivation of these moments using the logarithm of the estimator’s characteristic function. The first two cumulants give the mean and variance exactly, without a truncation approximation; further derivatives give higher-order cumulants.

The variance formula is exact at the true probability ρi\rho_i. In applications, substituting pi=Ci/Np_i=C_i/N and taking the square root gives a plug-in estimate of the standard error. This distinction applies to both metrics considered here. In particular, observing Ci=0C_i=0 or Ci=NC_i=N makes this estimated SE zero, but does not establish that the true probability is exactly zero or one.

passkpass^k and its Uncertainty

Unlike pass@kpass@k, pass^k\widehat{pass}_k, introduced in τ\tau-bench [1], is the probability that all kk independent attempts succeed on the same task. For task ii with true success probability ρi\rho_i,

pass^k(ρi)=Pr⁡(X1=1,…,Xk=1∣ρi)=ρik,(31)\widehat{pass}_k(\rho_i)=\Pr(X_1=1,\ldots,X_k=1\mid\rho_i)=\rho_i^k, \tag*{(31)}

where X1,…,XkX_1,\ldots,X_k are conditionally independent Bernoulli outcomes. This is the success–failure counterpart of pass@kpass@k: at least one success is the complement of all kk attempts failing. Given CiC_i successes in NN independent attempts, the unbiased estimator [1] is

Q^k(Ci)=(Cik)(Nk),1≤k≤N.(32)\widehat{Q}_k(C_i)=\frac{\binom{C_i}{k}}{\binom{N}{k}}, \qquad1\le k\le N. \tag*{(32)}
Estimated standard error for $k=1,5,10$ at fixed $N=20$

Figure 6. Estimated standard error for k=1,5,10k=1,5,10 at fixed N=20N=20. Solid lines use the exact variance in Eq. (30) evaluated at pi=Ci/Np_i=C_i/N; open circles show Monte Carlo estimates under the same fitted binomial model. Dashed lines show the delta-method approximation.

with Q^k(Ci)=0\widehat{Q}_k(C_i)=0 if Ci<kC_i<k. This estimator averages the all-success indicator over subsets of kk distinct observed runs, selected without replacement, and satisfies E[Q^k(Ci)∣ρi]=ρik\mathbb{E}[\widehat{Q}_k(C_i)\mid\rho_i]=\rho_i^k. Its exact conditional variance is

Var⁡(Q^k(Ci)∣ρi)=∑r=1k(kr)(N−kk−r)(Nk)(ρi2k−r−ρi2k).(33)\operatorname{Var}(\widehat{Q}_k(C_i)\mid\rho_i)=\sum_{r=1}^{k}\frac{\binom{k}{r}\binom{N-k}{k-r}}{\binom{N}{k}}\left(\rho_i^{2k-r}-\rho_i^{2k}\right). \tag*{(33)}

The coefficient in the summand is the probability that two independently and uniformly chosen size-kk subsets of the NN samples overlap in exactly rr attempts; impossible overlaps have coefficient zero. Their shared samples induce the covariance. Both unbiased estimators are U-statistics, and this subset-overlap argument specializes classical U-statistic variance theory [22, 23]. Recent manuscripts also analyze finite-sample estimation and uncertainty for pass^@k\widehat{\mathrm{pass}}@k [7, 8]; here, these per-task results are incorporated into the hierarchical uncertainty framework. Replacing ρi\rho_i by 1−ρi1-\rho_i in Eq. (33) gives the variance of the unbiased pass@k\mathrm{pass}@k estimator in Eq. (30): taking one minus the all-failure estimator leaves its variance unchanged. A simpler alternative estimates pass^@k\widehat{\mathrm{pass}}@k by substituting the empirical pass rate pi=Ci/Np_i=C_i/N into ρik\rho_i^k:

Q~k(Ci)=pik=(CiN)k.(34)\widetilde{Q}_k(C_i)=p_i^k=\left(\frac{C_i}{N}\right)^k. \tag*{(34)}

Unlike Eq. (32), this plug-in estimator averages the all-success indicator over ordered kk-tuples drawn with replacement from the observed runs. Reusing an observed run is not equivalent to obtaining a fresh independent outcome. For k>1k>1 and 0<ρi<10<\rho_i<1, the estimator is upward biased: because raising a probability to the kkth power is convex, fluctuations in pip_i increase the average of pikp_i^k above ρik\rho_i^k. At k=1k=1, both estimators reduce to pip_i.

Appendix § 9 gives the exact conditional mean and variance of the plug-in estimator. Its standard error measures variation around its own expectation, not its bias relative to ρik\rho_i^k; a smaller SE therefore does not by itself imply a more accurate estimator. For either estimator, we estimate its standard error by substituting pip_i for ρi\rho_i in its exact variance formula and taking the square root. Plugging into a variance formula is a separate choice from using a plug-in point estimator: it can be done for the unbiased estimator as well. Figure 7 compares the point estimators and their estimated standard errors.

Comparison of the unbiased and plug-in estimators for $\mathrm{pass}^{k}$

Figure 7. Comparison of the unbiased and plug-in estimators for passk\mathrm{pass}^{k} at fixed N=20N = 20 and k=5k = 5. (Left) Point estimates as functions of the observed success count CiC_i. (Right) Estimated standard errors from the exact variance formulas evaluated at pi=Ci/Np_i = C_i/N (lines), compared with Monte Carlo estimates under the same fitted binomial model (open markers).

The uncertainty formulas in these two subsections concern individual tasks. Benchmark metrics are obtained by computing the per-task estimates first and then averaging with the chosen task weights; for k>1k > 1, applying the nonlinear metric to a pooled mean pass rate generally gives a different quantity. For inference about a task population, the law of total variance from § 3.2 must also account for differences in the underlying per-task metric, 1−(1−ρi)k1 - (1 - \rho_i)^k or ρik\rho_i^k, across independently sampled tasks. Conditional on a fixed benchmark, only the execution-sampling contribution remains.

Empirical Analysis

In this section we apply the statistical framework developed above to results obtained on the think-ingbox dataset [5]; in particular we apply a the framework introduced in the previous sections to the Qwen-3.8-27b model [24], via a side-by-side (SBS) analysis of the base and reinforcement fine-tuned model, see Ref. [5] for training details. We provide few additional insights into the comparative performance of the two models, beyond what is reported in the original study. Although we choose to presents details for this specific dataset, the formalism and general analysis principle, are independent of the specific benchmark. In the following we refer to the base and reinforcement fine-tuned models’ results simply as base and RLFT, respectively.

The benchmark consists of five different business scenarios, each comprising ∼100\sim100 evaluation instances. Overall metrics aggregate across attempts, tasks, and scenarios using the equal averaging defined above. We first consider mean pass-rate at different level of aggregation, as explained in the previous sections. Figure 8 shows the comparison between the base and RLFT models, both overall and at the scenario level. In particular we focus on the performance gap and report it alongside the corresponding standard errors, propagated according to the statistical framework introduced above and again on the gap itself. Perhaps not surprisingly, the magnitude of the gap varies across scenarios, highlighting differering learning dynamics and the varying impact of reinforcement fine-tuning across different business contexts. The disparity itself depends on the base model (the prior), but can be further increased or mitigated by the fine-tuning process. We can also zoom in at the instance level to better understand the gap distribution and the error characteristics induced by the training dynamics. Figure 9 provides an example of such an instance-level analysis, highlighting how individual evaluation instances contribute to the overall performance gap and its variability. Comparing to the scenario-level analysis in Figure 8, the instance-level view reveals the heterogeneity in model performance across individual evaluation instances, and also how individual errors are washed out in the aggregated metrics, that is the essence of coarse-graining, see § 3. Figure 9 shows only the 104 instances in the banking support scenario, comparing the same instance across the two models. Table 1 reports counts separately for each scenario, including instances with increased, decreased, or unchanged observed pass rates. Within each scenario, we count changes whose per-instance approximate 95% Wald intervals exclude zero: the interval is the observed RLFT-minus-base pass-rate difference plus or minus 1.96 times its estimated standard error, obtained by adding the two binomial plug-in variances under conditional independence of the models’ executions.

Mean pass-rate comparison between the base and RLFT models

Figure 8. Mean pass-rate comparison between the base and RLFT (candidate) models. Left: overall mean pass rates and their paired gap. Right: scenario-level performance gaps (RLFT minus base), with instance counts nn. Error bars indicate standard errors; the blue line and shaded band indicate the overall mean gap and one standard error. The scenario aggregated metrics do not reveal the full picture of model performance across different business scenarios.

Distribution of the performance gap across individual instances in the 104-instance banking support scenario

Figure 9. Distribution of the performance gap across individual instances in the 104-instance banking support scenario, highlighting variability and error characteristics. Green indicates improvements while red indicates regressions. No change is shown in gray.

Metricbookingauto insuranceconsulting servicesretailbanking support
number of instances10410010198104
baseline mps ± ste0.534 ± 0.0330.479 ± 0.0320.457 ± 0.0350.640 ± 0.0410.479 ± 0.030
candidate mps ± ste0.627 ± 0.0350.565 ± 0.0400.545 ± 0.0370.679 ± 0.0420.624 ± 0.034
mean gap ± ste0.093 ± 0.0110.086 ± 0.0110.088 ± 0.0110.039 ± 0.0080.145 ± 0.012
tests improved6258574569
tests degraded2428301628
tests no change181414377
significant changes (95% CI)3532411046
no change total181414377
no change both zero6108101
no change both same nonzero1246276

Table 1. Separate comparisons of the baseline and candidate models within each scenario. Each column reports scenario-level mean pass rates (mps), their standard errors and candidate-minus-baseline gaps, together with counts of instance-level changes. No aggregation across scenarios is performed in this table.

Task Coverage, Reliability and Distributional Shifts

While Pass@1Pass@1 (pass-rate) measures average success probability, larger kk increasingly measures coverage: whether a task has any appreciable chance of being solved. Figure 10 shows the mean pass@5pass@5 and pass@20pass@20 across scenarios for the baseline and candidate models. The two plots tell a different story: while a significant improvement is observed at pass@1pass@1, already at k=5k = 5 the difference between the two models is not statistically significant, and at k=20k = 20 the performance is almost indistinguishable. For the overall task-population comparison, the approximate 95% confidence intervals for the mean differences at k=5k = 5 and k=20k = 20 both include zero. This pattern is consistent with the interpretation that the RL fine-tuned model mainly improves the probability of success for tasks that are already solvable by the baseline, without substantially increasing coverage for harder tasks; in the next section we show the same effect by looking at the distributional shift across the two models. Although the task-coverage point estimates change little in our candidate model, reliability shows a more noticeable improvement, suggesting that the candidate model is more consistent in solving tasks it can handle, even if it does not cover substantially more tasks overall. Although the two metrics are related, they capture different aspects of model performance, and improvements in reliability can be valuable even when task coverage remains similar.

Bar charts comparing base and candidate model coverage and reliability

Figure 10. Blue: Base model, Orange: Candidate model. Top row: mean pass@kpass@k (task coverage). Bottom row: mean passkpass^{k} (reliability). Left column: k=5k = 5; right column: k=20k = 20. Error bars indicate propagated standard errors.

Finally, we examine the effect of model training within a distributional shift point of view. Although this analysis is not part of our discussion on uncertaining quantification, it nevertheless provides a more holistic representation of the effect of model training on the task performance. Figure 11 illustrates the pass rate distribution before and after fine tuning. Two clear trends can be observed: first, there is a noticeable shift towards higher pass rates for the candidate model, particularly in the high pass-rate region; second, the distribution indicates that the model is becoming more reliable in the Goldilocks Zone, where task difficulty is optimal for learning. Finally, some of the tasks that had low pass rate, and were previously outside of the Goldilocks Zone, moved into the Goldilocks Zone and experienced improved success rates, while few regressed. Understanding regression behaviour is paramount for evaluating model improvements and to develop novel training strategies to mitigate negative impacts on previously well-performing tasks.

Pass Rate Distribution Comparison

Figure 11. Distributional shift of model performance across different scenarios of mean pass rate for the baseline and candidate models across different bins of task difficulty. Goldilocks Zone indicates the range of task difficulty where the model can learn more efficiently. The figures identifies a push towards higher pass rates for the candidate model, particularly in the high pass-rate region.

Conclusion

Reliable evaluation of agentic systems requires more than an aggregate success score: it requires identifying what a metric measures, how precisely it has been estimated, and which differences are hidden by aggregation. We have presented a statistical framework organized around scenarios, task instances, and repeated samples, distinguishing stochastic execution variability from finite-sample estimation uncertainty and heterogeneity in task difficulty. Connecting this hierarchy to coarse-graining makes explicit both the value and the limitations of benchmark averages: they summarize collective behavior, but cannot by themselves characterize instance-level reliability or reveal localized regressions.

Within this framework, we have analyzed mean pass rate, pass@kpass@k, and pass^k\widehat{pass}^{k}, including exact finite-sample sampling variances for the unbiased estimators of the latter two metrics and hierarchical uncertainty propagation. These results give a principled basis for attaching uncertainty estimates to commonly reported scores and interpreting model comparisons. They also clarify that single-attempt performance, task coverage under retries, and repeated-success reliability answer different questions; no one of these metrics is a substitute for the others.

The Thinkingbox case study [5] illustrates the practical importance of this distinction. For the base and reinforcement fine-tuned Qwen-3.8-27b models considered here, improved mean pass rate coexists with similar coverage at larger kk and higher point estimates of repeated-success reliability. Together with the instance-level gaps and the shift toward higher pass rates, this pattern shows that the fine-tuned model is making already-solvable tasks more consistently successful, rather than substantially expanding task coverage, although some previously unsolved tasks may still benefit from fine-tuning, as shown by the instance-level and distributional analyses. Scenario averages nevertheless conceal individual improvements and regressions, emphasizing why aggregate gains should be examined alongside paired task-level comparisons and their uncertainty.

The resulting recommendation is to report mean pass rate, coverage, and reliability jointly, with uncertainty estimates at the relevant aggregation levels and sufficient instance-level detail to expose regressions. Extending this analysis to dependent executions and evaluation-budget allocation is a natural direction for future work. Ultimately, progress in agentic systems should be judged not only by whether average performance increases, but by whether that improvement is statistically supported, broadly distributed across tasks, and aligned with the reliability users actually need.

Use of Large Language Models (LLMs)

We used large language models (LLMs) to assist with manuscript drafting and editing, as well as code and math development. The authors developed the research ideas and methodology, derived main mathematical expressions, conducted the experiments, analyzed the results, and reviewed the LLM-assisted content and code. The authors take full responsibility for the final content of this work.

Law of total variance: a coarse-graining derivation

In this section, we provide an explicit evaluation of the hierarchical model described in § 3 using coarse graining methods. We use coarse-graining in the sense of replacing a detailed description by distributions of aggregate observables, a standard viewpoint in statistical mechanics. Here, the corresponding probabilistic operations are aggregation and marginalization, and the resulting variance decomposition is the law of total variance [25]. We evaluate these operations explicitly using the methods of Refs. [26, 14].

We consider one scenario with MM instances and NN runs per instance. As in § 3, we suppress the scenario index ss and write ξir\xi_{ir} for the binary outcome of run rr on instance ii. For simplicity, each instance is sampled the same number of times; the derivation extends directly to variable sample counts. For each instance ii, define the per-instance average over runs:

ξˉi=1N∑r=1Nξir=CiN,(35)\bar{\xi}_i = \frac{1}{N}\sum_{r=1}^{N}\xi_{ir} = \frac{C_i}{N}, \tag*{(35)}

where CiC_i counts the successful runs for instance ii. Conditional on its true success probability ρi\rho_i, each outcome is Bernoulli:

P(ξir∣ρi)=ρi δ(ξir−1)+(1−ρi) δ(ξir).(36)\mathbb{P}(\xi_{ir}\mid\rho_i)=\rho_i\,\delta(\xi_{ir}-1)+(1-\rho_i)\,\delta(\xi_{ir}). \tag*{(36)}

We assume the ρi\rho_i are i.i.d. draws from a distribution g(ρ)g(\rho), representing the variability in instance difficulty. The scenario-level average is

ξ^=1M∑i=1Mξˉi.(37)\hat{\xi}=\frac{1}{M}\sum_{i=1}^{M}\bar{\xi}_i. \tag*{(37)}

For binary outcomes, ξˉi\bar{\xi}_i is the instance pass rate pip_i from § 3, and ξ^\hat{\xi} is the scenario pass rate pˉ\bar{p}. The calculation proceeds in two steps: first, we evaluate the conditional probability distribution of ξˉi\bar{\xi}_i given ρi\rho_i; then, we marginalize over the instance probabilities to obtain the unconditional distribution of ξ^\hat{\xi}. Following Ref. [26], the conditional distribution can be evaluated as follows:

PN(ξˉi∣ρi)=⟨δ(ξˉi−1N∑r=1Nξir)⟩ξir∣ρi=∫∏r=1Ndξir P(ξir∣ρi)δ(ξˉi−1N∑r=1Nξir),=∫dλ2πeiλξˉi∏r=1N∫dξir P(ξir∣ρi)e−iλNξir=∫dλ2πeiλξˉi[ρie−iλN+(1−ρi)]N,=∫dλ2πeiλξˉiχiN(λ/N∣ρi).(38)\begin{aligned} \mathbb{P}_{N}(\bar{\xi}_{i}\mid\rho_{i}) &= \left\langle\delta\left(\bar{\xi}_{i}-\frac{1}{N}\sum_{r=1}^{N}\xi_{ir}\right)\right\rangle_{\xi_{ir}\mid\rho_{i}} = \int\prod_{r=1}^{N}\mathrm{d}\xi_{ir}\,\mathbb{P}(\xi_{ir}\mid\rho_{i})\delta\left(\bar{\xi}_{i}-\frac{1}{N}\sum_{r=1}^{N}\xi_{ir}\right), \\ &= \int\frac{\mathrm{d}\lambda}{2\pi}e^{i\lambda\bar{\xi}_{i}}\prod_{r=1}^{N}\int\mathrm{d}\xi_{ir}\,\mathbb{P}(\xi_{ir}\mid\rho_{i})e^{-i\frac{\lambda}{N}\xi_{ir}} = \int\frac{\mathrm{d}\lambda}{2\pi}e^{i\lambda\bar{\xi}_{i}}\left[\rho_{i}e^{-i\frac{\lambda}{N}}+(1-\rho_{i})\right]^{N}, \\ &= \int\frac{\mathrm{d}\lambda}{2\pi}e^{i\lambda\bar{\xi}_{i}}\chi_{i}^{N}(\lambda/N\mid\rho_{i}). \tag*{(38)} \end{aligned}

where we have used the Fourier representation of the Dirac delta function

δ(x)=∫dλ2πeiλx,(39)\delta(x)=\int\frac{\mathrm{d}\lambda}{2\pi}e^{i\lambda x}, \tag*{(39)}

with λ\lambda a Fourier variable enforcing the Dirac delta constraint. The one-run characteristic function is χi(t∣ρi)=E[e−itξir∣ρi]=(1−ρi)+ρie−it\chi_{i}(t\mid\rho_{i})=\mathbb{E}[e^{-it\xi_{ir}}\mid\rho_{i}]=(1-\rho_{i})+\rho_{i}e^{-it}; independence of the NN runs gives the NNth power in Eq. (A.4). From here, we could evaluate the conditional moments of ξˉi\bar{\xi}_{i} given ρi\rho_{i} by taking derivatives of its characteristic function. We move instead to the second level of the hierarchy and first evaluate the conditional distribution of ξ^\hat{\xi} given the set ρ1:M\rho_{1:M}:

PM,N(ξ^∣ρ1:M)=∫∏i=1Mdξˉi PN(ξˉi∣ρi)δ(ξ^−1M∑i=1Mξˉi)=∫dλ2πeiλξ^∏i=1MχiN(λMN∣ρi)=∫dλ2πexp⁡(iλξ^+∑i=1MNΦi(λMN∣ρi)).(40)\begin{aligned} \mathbb{P}_{M,N}(\hat{\xi}\mid\rho_{1:M}) &= \int\prod_{i=1}^{M}\mathrm{d}\bar{\xi}_{i}\,\mathbb{P}_{N}(\bar{\xi}_{i}\mid\rho_{i})\delta\left(\hat{\xi}-\frac{1}{M}\sum_{i=1}^{M}\bar{\xi}_{i}\right) \\ &= \int\frac{\mathrm{d}\lambda}{2\pi}e^{i\lambda\hat{\xi}}\prod_{i=1}^{M}\chi_{i}^{N}\left(\frac{\lambda}{MN}\mid\rho_{i}\right) \\ &= \int\frac{\mathrm{d}\lambda}{2\pi}\exp\left(i\lambda\hat{\xi}+\sum_{i=1}^{M}N\Phi_{i}\left(\frac{\lambda}{MN}\mid\rho_{i}\right)\right). \tag*{(40)} \end{aligned}

The conditional characteristic function appearing in the inverse Fourier transform in Eq. (A.6) is χξ^(λ∣ρ1:M)=∏i=1MχiN(λ/(MN)∣ρi)\chi_{\hat{\xi}}(\lambda\mid\rho_{1:M})=\prod_{i=1}^{M}\chi_{i}^{N}(\lambda/(MN)\mid\rho_{i}). We define the cumulant-generating function for the one-run characteristic function by Φi(t∣ρi)=log⁡χi(t∣ρi)\Phi_{i}(t\mid\rho_{i})=\log\chi_{i}(t\mid\rho_{i}), where t=λ/(MN)t=\lambda/(MN). For large M,NM,N we expand around t0=0t_{0}=0, writing t=t0+δtt=t_{0}+\delta t, and retain terms through second order in δt\delta t:

Φi(t∣ρi)=−iρiδt−12ρi(1−ρi)δt2+O(δt3).(41)\Phi_{i}(t\mid\rho_{i})=-i\rho_{i}\delta t-\frac{1}{2}\rho_{i}(1-\rho_{i})\delta t^{2}+O(\delta t^{3}). \tag*{(41)}

Substituting this expansion into the logarithm of the conditional characteristic function in Eq. (A.6) gives, through second order in λ\lambda,

log⁡χξ^(λ∣ρ1:M)=−iλM∑i=1Mρi−λ22M2N∑i=1Mρi(1−ρi)+O(λ3).(42)\log\chi_{\hat{\xi}}(\lambda\mid\rho_{1:M})=-\frac{i\lambda}{M}\sum_{i=1}^{M}\rho_{i}-\frac{\lambda^{2}}{2M^{2}N}\sum_{i=1}^{M}\rho_{i}(1-\rho_{i})+O(\lambda^{3}). \tag*{(42)}

This expansion identifies the first two conditional cumulants exactly; no normal approximation is needed to obtain the moments. Thus, conditional on ρ1:M\rho_{1:M},

E[ξ^∣ρ1:M]=1M∑i=1Mρi,(43)\mathbb{E}[\hat{\xi}\mid\rho_{1:M}]=\frac{1}{M}\sum_{i=1}^{M}\rho_{i}, \tag*{(43)}
Var⁡(ξ^∣ρ1:M)=1M2N∑i=1Mρi(1−ρi).(44)\operatorname{Var}(\hat{\xi}\mid\rho_{1:M})=\frac{1}{M^{2}N}\sum_{i=1}^{M}\rho_{i}(1-\rho_{i}). \tag*{(44)}

Marginal Characteristic function

We can now marginalize the conditional characteristic function over the instance probabilities to obtain the unconditional moments of ξ^\hat{\xi}. Under the i.i.d. assumption, this gives:

χξ^(λ)=∫01[∏i=1Mg(ρi) dρi]χξ^(λ∣ρ1:M)(45)\chi_{\hat{\xi}}(\lambda)=\int_{0}^{1}\left[\prod_{i=1}^{M}g(\rho_i)\,d\rho_i\right]\chi_{\hat{\xi}}(\lambda\mid\rho_{1:M}) \tag*{(45)}

Assuming that instances are i.i.d. draws from g(ρ)g(\rho), we can use a moment expansion to second order similar to what done in Eq. A.7. This only assumes that the first two moments of g(ρ)g(\rho) are finite, but otherwise makes no assumption on the form of g(ρ)g(\rho) itself. It is convenient to reintroduce the parametrization δλ^=MN δt\delta\hat{\lambda}=MN\,\delta t and expand to second order in δt\delta t the exponential:

eu+v=1+(u+v)+u22+O(δt3)(46)e^{u+v}=1+(u+v)+\frac{u^2}{2}+O(\delta t^3) \tag*{(46)}
u=−iNδt ρiu=-iN\delta t\,\rho_i
v=−N2δt2 ρi(1−ρi).v=-\frac{N}{2}\delta t^2\,\rho_i(1-\rho_i).

Since the instances are i.i.d., the marginal characteristic function is the MMth power of this expression with t=λ/(MN)t=\lambda/(MN). Taking its logarithm and retaining terms through second order gives

log⁡χξ^(δλ^)=−i δλ^ μ−δλ^22(m2−μ2M+Eg[ρ(1−ρ)]MN)+O(δλ^3),(47)\log\chi_{\hat{\xi}}(\delta\hat{\lambda})=-i\,\delta\hat{\lambda}\,\mu-\frac{\delta\hat{\lambda}^{2}}{2}\left(\frac{m_2-\mu^2}{M}+\frac{\mathbb{E}_g[\rho(1-\rho)]}{MN}\right)+O(\delta\hat{\lambda}^3), \tag*{(47)}

where we have defined:

μ≡Eg[ρ]=∫01g(ρ)ρ dρ,(48)\mu\equiv\mathbb{E}_g[\rho]=\int_{0}^{1}g(\rho)\rho\,d\rho, \tag*{(48)}
m2≡Eg[ρ2]=∫01g(ρ)ρ2 dρ.(49)m_2\equiv\mathbb{E}_g[\rho^2]=\int_{0}^{1}g(\rho)\rho^2\,d\rho. \tag*{(49)}

Plugging this expression back into Eq. A.8 we can read off the unconditional mean and variance of ξ^\hat{\xi}:

E[ξ^]=μ,(50)\mathbb{E}[\hat{\xi}]=\mu, \tag*{(50)}
Var⁡(ξ^)=m2−μ2M+Eg[ρ(1−ρ)]MN=τ2M+Eg[σρi2]M.(51)\begin{aligned} \operatorname{Var}(\hat{\xi})&=\frac{m_2-\mu^2}{M}+\frac{\mathbb{E}_g[\rho(1-\rho)]}{MN}\\ &=\frac{\tau^2}{M}+\frac{\mathbb{E}_g[\sigma_{\rho_i}^2]}{M}. \tag*{(51)} \end{aligned}

The first term arises from variation in success probabilities across independently sampled instances; the second arises from the finite number of Bernoulli runs per instance. Thus the two components emerge constructively by marginalizing and expanding the characteristic function. Their sum matches Eq. (18) in § 3, where the same result is presented through the law of total variance.

Finite-Sample Analysis of pass@kpass@k

We derive the finite-sample distribution and variance of the pass@kpass@k estimator, complementing the main-text definition in Eq. (28). The derivation tracks both the total number of successes among NN runs and the outcome on a selected subset of kk runs.

Fix an instance ii and condition on its true success probability ρi\rho_i. The run outcomes are exchangeable under the i.i.d. Bernoulli model, so without loss of generality we take the selected subset to be the first kk runs. For 1≤k≤N1 \le k \le N, define

Ci=∑r=1Nξir,(52)C_i = \sum_{r=1}^{N} \xi_{ir}, \tag*{(52)}
Di=∏r=1k(1−ξir),Bi=1−Di.(53)D_i = \prod_{r=1}^{k}(1-\xi_{ir}), \qquad B_i = 1-D_i. \tag*{(53)}

Here, CiC_i is the total number of successes, DiD_i indicates that all selected runs fail, and BiB_i indicates that at least one selected run succeeds. The pass@kpass@k estimate conditional on the observed count Ci=ciC_i=c_i is the probability of Bi=1B_i=1 when a size-kk subset is selected uniformly at random. The joint probability mass function of CiC_i and BiB_i can be written using the Kronecker delta representation δK(n)=∫−ππdλ2πeiλn\delta_K(n)=\int_{-\pi}^{\pi}\frac{d\lambda}{2\pi}e^{i\lambda n} for integer nn. Define their joint characteristic function by

PN,k(ci,bi∣ρi)=⟨δK(ci−Ci)δK(bi−Bi)⟩ξir∣ρi(54)\mathbb{P}_{N,k}(c_i,b_i\mid\rho_i)=\left\langle\delta_K(c_i-C_i)\delta_K(b_i-B_i)\right\rangle_{\xi_{ir}\mid\rho_i} \tag*{(54)}
=∫−ππdλ2π∫−ππdt2πeiλci+itbiχiN,k(λ,t∣ρi),(55)=\int_{-\pi}^{\pi}\frac{d\lambda}{2\pi}\int_{-\pi}^{\pi}\frac{dt}{2\pi}e^{i\lambda c_i+itb_i}\chi_i^{N,k}(\lambda,t\mid\rho_i), \tag*{(55)}
χiN,k(λ,t∣ρi):=⟨e−iλCi−itBi⟩ξir∣ρi.(56)\chi_i^{N,k}(\lambda,t\mid\rho_i):=\left\langle e^{-i\lambda C_i-itB_i}\right\rangle_{\xi_{ir}\mid\rho_i}. \tag*{(56)}

Both variables are discrete, which is why the Fourier integrals run over [−π,π][-\pi,\pi]. Setting t=0t=0 marginalizes over BiB_i and recovers the characteristic function of CiC_i, namely ai(λ)Na_i(\lambda)^N, where

ai(λ)=(1−ρi)+ρie−iλ.(57)a_i(\lambda)=(1-\rho_i)+\rho_i e^{-i\lambda}. \tag*{(57)}

This is the count version of the per-instance characteristic function in Eq. (A.4); the Fourier argument there is scaled by 1/N1/N because it describes the sample average Ci/NC_i/N. Since Bi=1−DiB_i=1-D_i and DiD_i is binary, we can easily see that the following identity holds:

e−itBi=e−it(1−Di)=e−it+(1−e−it)Di.(58)e^{-itB_i}=e^{-it(1-D_i)}=e^{-it}+(1-e^{-it})D_i. \tag*{(58)}

The first term contributes the characteristic function of CiC_i. For the second term, independence of the runs gives

⟨e−iλCiDi⟩ξir∣ρi=∏r=1k(1−ρi)∏r=k+1N[(1−ρi)+ρie−iλ](59)\left\langle e^{-i\lambda C_i}D_i\right\rangle_{\xi_{ir}\mid\rho_i} =\prod_{r=1}^{k}(1-\rho_i)\prod_{r=k+1}^{N}\left[(1-\rho_i)+\rho_i e^{-i\lambda}\right] \tag*{(59)}
=(1−ρi)kai(λ)N−k.(60)=(1-\rho_i)^k a_i(\lambda)^{N-k}. \tag*{(60)}

Consequently, the joint characteristic function is

χiN,k(λ,t∣ρi)=e−itai(λ)N+(1−e−it)(1−ρi)kai(λ)N−k.(61)\chi_i^{N,k}(\lambda,t\mid\rho_i)=e^{-it}a_i(\lambda)^N+(1-e^{-it})(1-\rho_i)^k a_i(\lambda)^{N-k}. \tag*{(61)}

Setting λ=0\lambda=0 we obtain the characteristic function of BiB_i alone.

Conditional characteristic function of BiB_i

We now condition on the observed total count Ci=ciC_i=c_i. Define

Im(ci)=∫−ππdλ2πeiλciai(λ)m,(62)I_m(c_i) = \int_{-\pi}^{\pi} \frac{\mathrm{d}\lambda}{2\pi} e^{\mathrm{i}\lambda c_i}a_i(\lambda)^m, \tag*{(62)}
JN−k(ci)=(1−ρi)kIN−k(ci).(63)J_{N-k}(c_i) = (1-\rho_i)^k I_{N-k}(c_i). \tag*{(63)}

The first quantity is the probability mass of Ci=ciC_i=c_i; the second is the joint probability mass of Ci=ciC_i=c_i and Di=1D_i=1. Expanding with the binomial theorem gives

IN(ci)=∑j=0N(Nj)(1−ρi)N−jρij∫−ππdλ2πeiλ(ci−j)=(Nci)(1−ρi)N−ciρici.(64)I_N(c_i) = \sum_{j=0}^{N} \binom{N}{j}(1-\rho_i)^{N-j}\rho_i^j \int_{-\pi}^{\pi} \frac{\mathrm{d}\lambda}{2\pi}e^{\mathrm{i}\lambda(c_i-j)} = \binom{N}{c_i}(1-\rho_i)^{N-c_i}\rho_i^{c_i}. \tag*{(64)}

Similarly,

JN−k(ci)={(N−kci)(1−ρi)N−ciρici,ci≤N−k,0,ci>N−k.(65)J_{N-k}(c_i) = \begin{cases} \binom{N-k}{c_i}(1-\rho_i)^{N-c_i}\rho_i^{c_i}, & c_i \le N-k,\\ 0, & c_i > N-k. \end{cases} \tag*{(65)}

The conditional characteristic function of BiB_i given Ci=ciC_i=c_i is obtained by normalizing the joint Fourier coefficient by IN(ci)I_N(c_i):

Hi,ci(t):=E[e−itBi∣Ci=ci,ρi]=1IN(ci)∫−ππdλ2πeiλciχiN,k(λ,t∣ρi).(66)H_{i,c_i}(t) := \mathbb{E}[e^{-\mathrm{i}tB_i}\mid C_i=c_i,\rho_i] = \frac{1}{I_N(c_i)}\int_{-\pi}^{\pi}\frac{\mathrm{d}\lambda}{2\pi}e^{\mathrm{i}\lambda c_i}\chi_i^{N,k}(\lambda,t\mid\rho_i). \tag*{(66)}

Substituting the expressions for INI_N and JN−kJ_{N-k} yields

Hi,ci(t)=e−it+(1−e−it)JN−k(ci)IN(ci).(67)H_{i,c_i}(t) = e^{-\mathrm{i}t} + (1-e^{-\mathrm{i}t})\frac{J_{N-k}(c_i)}{I_N(c_i)}. \tag*{(67)}

The ratio JN−k(ci)/IN(ci)J_{N-k}(c_i)/I_N(c_i) is the conditional probability that all kk selected runs fail. Therefore, the conditional mean of BiB_i—and hence the pass@k estimate—is

hk(ci)=i∂∂tlog⁡Hi,ci(t)∣t=0=1−JN−k(ci)IN(ci)=1−(N−cik)(Nk),(68)h_k(c_i) = \left.\mathrm{i}\frac{\partial}{\partial t}\log H_{i,c_i}(t)\right|_{t=0} = 1-\frac{J_{N-k}(c_i)}{I_N(c_i)} = 1-\frac{\binom{N-c_i}{k}}{\binom{N}{k}}, \tag*{(68)}

where (N−cik)=0\binom{N-c_i}{k}=0 when N−ci<kN-c_i<k. Thus the estimator for instance ii is Qi,k=hk(Ci)Q_{i,k}=h_k(C_i).

Sampling variance

At fixed ρi\rho_i, the count CiC_i has probability mass function IN(ci)I_N(c_i). The characteristic function of the estimator Qi,kQ_{i,k} is therefore

Ψi,k(u∣ρi)=E[e−iuQi,k∣ρi]=∑ci=0NIN(ci)e−iuhk(ci),(69)\Psi_{i,k}(u\mid\rho_i) = \mathbb{E}[e^{-\mathrm{i}uQ_{i,k}}\mid\rho_i] = \sum_{c_i=0}^{N} I_N(c_i)e^{-\mathrm{i}u h_k(c_i)}, \tag*{(69)}
Φi,k(u∣ρi)=log⁡Ψi,k(u∣ρi).(70)\Phi_{i,k}(u\mid\rho_i) = \log\Psi_{i,k}(u\mid\rho_i). \tag*{(70)}

With the characteristic-function convention used here, the first derivative of Φi,k\Phi_{i,k} at zero is −iE[Qi,k∣ρi]-\mathrm{i}\mathbb{E}[Q_{i,k}\mid\rho_i], and the second is −Var⁡(Qi,k∣ρi)-\operatorname{Var}(Q_{i,k}\mid\rho_i):

Φi,k′(0∣ρi)=−i∑ci=0NIN(ci)hk(ci),(71)\Phi'_{i,k}(0\mid\rho_i) = -\mathrm{i}\sum_{c_i=0}^{N} I_N(c_i)h_k(c_i), \tag*{(71)}
Φi,k′′(0∣ρi)=−∑ci=0NIN(ci)hk(ci)2+[∑ci=0NIN(ci)hk(ci)]2.(72)\Phi''_{i,k}(0\mid\rho_i) = -\sum_{c_i=0}^{N} I_N(c_i)h_k(c_i)^2 + \left[\sum_{c_i=0}^{N} I_N(c_i)h_k(c_i)\right]^2. \tag*{(72)}

The conditional mean simplifies to

μi,k:=E[Qi,k∣ρi]=1−∑ci=0N−k(N−kci)(1−ρi)N−ciρici(73)\mu_{i,k} := \mathbb{E}[Q_{i,k}\mid\rho_i] = 1-\sum_{c_i=0}^{N-k}\binom{N-k}{c_i}(1-\rho_i)^{N-c_i}\rho_i^{c_i} \tag*{(73)}
=1−(1−ρi)k.(74)=1-(1-\rho_i)^k. \tag*{(74)}

The conditional sampling variance is

vi,k:=Var⁡(Qi,k∣ρi)=∑ci=0NIN(ci)hk(ci)2−[1−(1−ρi)k]2.(75)v_{i,k} := \operatorname{Var}(Q_{i,k}\mid\rho_i)=\sum_{c_i=0}^{N}I_N(c_i)h_k(c_i)^2-\left[1-(1-\rho_i)^k\right]^2. \tag*{(75)}

This is the variance formula reported in the main text. Averaging these conditional moments over the distribution of ρi\rho_i gives the corresponding unconditional moments.

Simple identities

For k=1k=1, the finite-sample formula reduces to the empirical pass rate. For Ci<NC_i<N, expanding the binomial coefficients gives

h1(Ci)=1−(N−Ci1)(N1)=1−(N−Ci)!(N−Ci−1)!(N−1)!N!=1−N−CiN=CiN=pi.(76)\begin{aligned} h_1(C_i)&=1-\frac{\binom{N-C_i}{1}}{\binom{N}{1}}=1-\frac{(N-C_i)!}{(N-C_i-1)!}\frac{(N-1)!}{N!}\\ &=1-\frac{N-C_i}{N}=\frac{C_i}{N}=p_i. \tag*{(76)} \end{aligned}

For Ci=NC_i=N, the numerator binomial coefficient is zero, so the same identity gives h1(N)=1=pih_1(N)=1=p_i. For the large-NN approximation, write pi=Ci/Np_i=C_i/N and use the product representation of the binomial-coefficient ratio:

hk(Ci)=1−∏j=0k−1N−Ci−jN−j=1−∏j=0k−11−pi−j/N1−j/N≃1−(1−pi)k,(77)h_k(C_i)=1-\prod_{j=0}^{k-1}\frac{N-C_i-j}{N-j} =1-\prod_{j=0}^{k-1}\frac{1-p_i-j/N}{1-j/N}\simeq1-(1-p_i)^k, \tag*{(77)}

where the approximation holds as NN grows with kk fixed (equivalently, when N≫kN\gg k).

Moments of the Plug-in pass^k Estimator

For a fixed task, let Ci∣ρi∼Binomial⁡(N,ρi)C_i\mid\rho_i\sim\operatorname{Binomial}(N,\rho_i) and Q~k(Ci)=(Ci/N)k\widetilde{Q}_k(C_i)=(C_i/N)^k. To express its exact moments, let (N)j=N(N−1)⋯(N−j+1)(N)_j=N(N-1)\cdots(N-j+1) be the falling factorial and let {mj}\left\{\begin{array}{c}m\\j\end{array}\right\} be a Stirling number of the second kind. Using the identities

Cim=∑j=1m{mj}(Ci)j,E[(Ci)j∣ρi]=(N)jρij,(78)C_i^m=\sum_{j=1}^{m}\left\{\begin{array}{c}m\\j\end{array}\right\}(C_i)_j,\qquad \mathbb{E}[(C_i)_j\mid\rho_i]=(N)_j\rho_i^j, \tag*{(78)}

with (N)j=0(N)_j=0 for j>Nj>N, the exact conditional mean is

E[Q~k(Ci)∣ρi]=1Nk∑j=1k{kj}(N)jρij.(79)\mathbb{E}[\widetilde{Q}_k(C_i)\mid\rho_i] =\frac{1}{N^k}\sum_{j=1}^{k}\left\{\begin{array}{c}k\\j\end{array}\right\}(N)_j\rho_i^j. \tag*{(79)}
Finite-size scaling plot of mean pass rate versus per-sample size

Figure 12. Scenario-level pass-rate estimate as a function of the number nn of runs per instance. The curve shows the mean across subsampling replicas, and the shaded band shows approximate 95% intervals based on the hierarchical standard error from Eq. (18).

For k>1k > 1 and 0<ρi<10 < \rho_i < 1, this exceeds ρik\rho_i^k, whereas the unbiased estimator in Eq. (32) has conditional mean exactly ρik\rho_i^k. Applying the same identity to the second moment gives the exact conditional variance

Var⁡(Q~k(Ci)∣ρi)=1N2k∑j=12k(2kj)(N)jρij−[1Nk∑j=1k(kj)(N)jρij]2.(80)\operatorname{Var}(\widetilde{Q}_k(C_i) \mid\rho_i) = \frac{1}{N^{2k}}\sum_{j=1}^{2k}\binom{2k}{j}(N)_j\rho_i^j-\left[\frac{1}{N^k}\sum_{j=1}^{k}\binom{k}{j}(N)_j\rho_i^j\right]^2. \tag*{(80)}

Its square root is the conditional standard error of the plug-in point estimator. Replacing ρi\rho_i by pi=Ci/Np_i = C_i/N gives a plug-in estimate of this standard error, as discussed in the main text.

Choosing the sample size

In this section, we assess how the scenario-level pass-rate estimate and its uncertainty depend on the number of runs available per instance. For each instance, we have up to Nmax⁡N_{\max} runs (here, Nmax⁡=40N_{\max} = 40). Fix a scenario containing MM instances, and let n≤Nmax⁡n \le N_{\max} be the target number of runs per instance; each included instance must have at least nn observed runs. For subsampling replica bb, draw a set Ri,b(n)\mathcal{R}_{i,b}(n) of nn distinct run indices for each instance ii, and compute

pi,b(n)=1n∑r∈Ri,b(n)ξir,pˉb(n)=1M∑i=1Mpi,b(n).(81)p_{i,b}(n) = \frac{1}{n}\sum_{r\in\mathcal{R}_{i,b}(n)}\xi_{ir}, \qquad\bar{p}_b(n) = \frac{1}{M}\sum_{i=1}^{M}p_{i,b}(n). \tag*{(81)}

The draws are without replacement within each replica, so no observed run is counted twice in that replica; different replicas may overlap. Since the original runs are i.i.d. conditional on each instance’s success probability, each such subset has the same marginal sampling distribution as nn runs from that instance. We repeat the subsampling with RR random seeds and report the replica-averaged scenario pass rate

p^(n)=1R∑b=1Rpˉb(n).(82)\widehat{p}(n) = \frac{1}{R}\sum_{b=1}^{R}\bar{p}_b(n). \tag*{(82)}

Figure 12 shows this mean and approximate 95% intervals, p^(n)±1.96 SE^hier(n)\hat{p}(n) \pm1.96\,\widehat{\mathrm{SE}}_{\mathrm{hier}}(n), using the hierarchical variance propagation in Eq. (18) with N=nN=n. At the full sample size, n=Nmax⁡n=N_{\max}, no subsampling is performed. As nn increases, the within-instance binomial-noise contribution decreases, while between-instance variability remains; accordingly, the aggregate uncertainty does not in general vanish as nn grows. The binomial-noise contribution scales as 1/(Mn)1/(Mn), so increasing the number of runs has diminishing returns, particularly when MM is large. The normal-approximation intervals should be interpreted cautiously for small evaluation sets, where they can understate uncertainty [15]. This analysis illustrates the trade-off between computational cost and the statistical precision of scenario-level performance estimates.

References

  1. [1]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024.arxiv.org/abs/2406.12045
  2. [5]Zhuochun Li, Youngmin Ko, Ali Keramati, Tuhin Kundu, Liang-Chun Tsai, Nicola Ferri, Mirco Milletari, Jiaxiang Liu, Susana Palmaz Lopez Pelaez, Yuepeng Wang, Vadim Smolyakov, Xiang Jiang, Kjartan Ólafsson, and Tommy Guy. One success isn’t reliability: Thinkingbox, a sandbox and benchmark for agents in stateful business workflows. arXiv preprint arXiv:2608.19741, 2026.arxiv.org/abs/2608.19741
  3. [6]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chanzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
  4. [7]Anonymous. Unbiased pass^k and pass^k evaluation with fewer rollouts, 2026. Submitted to ICLR 2027. OpenReview: https://openreview.net/forum?id=SWs3hMG9IC.
  5. [8]Anonymous. Are agent reliability rankings real? a statistical treatment of the pass^k metric, 2026. Submitted to ICLR 2027. OpenReview: https://openreview.net/forum?id=NH98Pcbbwd.
  6. [9]Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024.arxiv.org/abs/2411.00640
  7. [10]Bjarni Haukur Bjarnason, André Silva, and Martin Monperrus. On randomness in agentic evals. arXiv preprint arXiv:2602.07150, 2026.arxiv.org/abs/2602.07150
  8. [11]Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, and Arvind Narayanan. Towards a science of AI agent reliability. arXiv preprint arXiv:2602.16666, 2026.arxiv.org/abs/2602.16666
  9. [12]Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, and Vipin Chaudhary. Don’t Pass@k: A Bayesian framework for large language model evaluation. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2510.04265
  10. [13]Pahan Dewasurendra. Extrapolation is an assumption: Sharp limits for Pass@k forecasting. https://doi.org/10.5281/zenodo.21698609, 2026.
  11. [14]Mehran Kardar. Statistical Physics of Particles. Cambridge University Press, 2007.
  12. [15]Sam Bowyer, Laurence Aitchison, and Desi R. Ivanova. Position: Don’t use the clt in llm evals with fewer than a few hundred datapoints. In Proceedings of the 42nd International Conference on Machine Learning, Vancouver, Canada, PMLR 267, 2025.arxiv.org/abs/2503.01747
  13. [16]Terence Tao. An Introduction to Measure Theory, volume 126 of Graduate Studies in Mathematics. American Mathematical Society, 2011.DOI
  14. [17]Cédric Villani. Topics in Optimal Transportation, volume 58 of Graduate Studies in Mathematics. American Mathematical Society, 2003.DOI
  15. [18]Filippo Santambrogio. Optimal Transport for Applied Mathematicians: Calculus of Variations, PDEs, and Modeling, volume 87 of Progress in Nonlinear Differential Equations and Their Applications. Birkhäuser, 2015.DOI
  16. [19]Wikipedia contributors. Law of total variance. https://en.wikipedia.org/wiki/Law_of_total_variance. Wikipedia.
  17. [20]Kohsei Matsutani, Shota Takashiro, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, and Yutaka Matsuo. Rl squeezes, sft expands: A comparative study of reasoning llms. arXiv preprint arXiv:2509.21128, 2025.arxiv.org/abs/2509.21128
  18. [21]N. Lambert. Reinforcement learning from human feedback. https://rlhfbook.com/, 2025.arxiv.org/abs/2504.12501
  19. [22]Wassily Hoeffding. A class of statistics with asymptotically normal distribution. The Annals of Mathematical Statistics, 19(3):293–325, 1948.
  20. [23]Robert J. Serfling. Approximation Theorems of Mathematical Statistics. Wiley, 1980.
  21. [24]Qwen Team. Qwen3.8-27B model card. https://huggingface.co/Qwen/Qwen3.8-27B, 2026.
  22. [25]Joseph K. Blitzstein and Jessica Hwang. Introduction to Probability. Chapman and Hall/CRC, 2 edition, 2019.
  23. [26]Luca Leuzzi, Enzo Marinari, and Giorgio Parisi. Probability Theory for quantitative scientists. Cambridge University Press, 2025.DOI

Paper details

vlnc:W_y82stzkn
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

large language models (LLMs)

View statement
Looking for related papers…
Contents