WorksComputer Science
A Hierarchical Statistical Framework for Evaluating Agent Reliability
Abstract
Agents operating in complex scenarios often face uncertainty due to incomplete information about the environment they operate in, ambiguous user queries, complex business rules and the inherently stochastic dynamics of the governing policy. Several new benchmarks have been proposed to evaluate LLM-based agents on complex tasks in multi-tool environments, including settings with policy constraints and varying levels of intent ambiguity. These conditions make it important to evaluate agentic systems systematically and quantify their performance under different conditions to assess their reliability in production. This need is evident in the recently introduced Thinkingbox benchmark, where even the strongest models struggle to perform consistently across diverse business scenarios. This paper presents a statistical framework for evaluating uncertainty and reliability in agentic systems. We define a hierarchy of scenarios, instances, and samples to assess model performance and variability systematically. We apply this framework to common metrics found in the literature: mean pass rate, , and , with uncertainty estimates for each metric. Aggregating metrics at different levels of the hierarchy reveals different structures and properties of the policy model; we connect this hierarchical aggregation with the known process of coarse-graining as intended in the physics and statistical literature. We independently derive exact finite-sample expressions for the sampling variances of the unbiased and estimators; related results also appear in recent submissions. Our central contribution is a constructive derivation of hierarchical uncertainty propagation, making explicit its correspondence with coarse-graining and separating execution noise from task heterogeneity.
Introduction
Large language models (LLMs) have enabled increasingly capable AI agents that combine a policy model, tools for interacting with an external environment, and a harness that orchestrates their execution. Given a task, an agent generates a solution trajectory comprising reasoning steps and actions, potentially modifying the environment along the way. Stochastic decision-making, task ambiguity, and complex interactions with the environment can produce different trajectories and outcomes across repeated executions of the same task. Evaluating these systems therefore requires more than a single aggregate performance score: we must establish how consistently they succeed, where failures concentrate, and how precisely performance has been measured.
Yet benchmark reporting often emphasizes a single state-of-the-art (SOTA) result without quantifying its statistical uncertainty [9]. When reported differences between models amount to only a few percentage points, point estimates alone do not establish whether those differences reflect genuine improvements or sampling noise. This limitation is particularly consequential for agentic systems, whose performance can vary both across tasks and across repeated attempts at the same task. Bjarnason et al. [10] quantify this variability on SWE-Bench-Verified and show that small reported improvements can be comparable to evaluation noise. The recently introduced Think-ingbox benchmark [5] likewise highlights substantial variability among strong models, both within and across domains, with uncertainty estimates complicating claims of a clear leader even within a single domain. These observations motivate a statistical treatment that distinguishes variability in agent behavior from uncertainty in benchmark estimates.
Reliability also extends beyond success frequency. Rabanser et al. [11] organize agent reliability into consistency, robustness, predictability, and safety, and find that capability gains need not yield comparable improvements in reliability. Their perspective is complementary to ours: rather than proposing a comprehensive taxonomy of reliability, we focus on the statistical interpretation of success-based metrics and the propagation of uncertainty across repeated attempts, tasks, and scenarios. Such analysis supports, but does not replace, evaluation of robustness or failure severity.
From a user’s perspective, an agent should solve a task reliably without requiring repeated requests. Single-attempt success, summarized by (mean pass rate), is therefore a central measure of practical performance. However, the average alone neither captures how consistently individual tasks are solved nor indicates how precisely performance has been estimated. Repeated evaluation of each task helps characterize execution variability, while confidence intervals quantify uncertainty in the resulting estimates. These measurements provide evidence relevant to production reliability, although their applicability depends on how closely the benchmark reflects deployment conditions. Reliability must also be considered alongside the cost of obtaining a successful outcome. An agent may eventually find a solution through retries, but this behavior can be undesirable for long-horizon tasks that consume substantial time and computational resources. Secondary metrics such as number of tool calls, retries, and token usage therefore complement success-based measures by characterizing the efficiency of solution trajectories. Together, these measurements can guide reinforcement fine-tuning (RFT) toward more consistent and efficient task completion, rather than improvements in aggregate success alone.
Prior work provides complementary approaches to statistical evaluation. Chen et al. [6] introduce the unbiased estimator, while Yao et al. [1] introduce and its unbiased estimator to measure agent reliability across repeated trials. Hariri et al. [12] develop a Bayesian alternative to -based model ranking, with analytical posterior uncertainty and credible intervals for underlying success probabilities. Dewasurendra [13] studies a different problem: finite-sample confidence intervals and identification limits for population , particularly when extrapolating beyond the observed sampling budget. These questions are distinct from the exact frequentist sampling variances of the unbiased and estimators studied here; bootstrap uncertainty estimates likewise do not provide analytical expressions for those variances. Recent manuscripts also analyze finite-sample uncertainty for these metrics [7, 8], but do not address hierarchical uncertainty propagation.
In this work, we present a unified statistical framework for evaluating performance and uncertainty in LLM-based agentic systems. Our contributions are threefold. First, we organize evaluation into scenarios, task instances, and repeated samples, distinguishing execution variability, differences across tasks, and uncertainty in estimated performance. We interpret aggregation through this hierarchy as coarse-graining in physics and statistics [14], showing how averages reveal collective behavior while obscuring task-level differences. Second, we analyze uncertainty in mean pass rate and derive exact finite-sample sampling variances for the unbiased and estimators, connecting these per-task results to uncertainty in population-level averages. Third, we illustrate the framework on agentic benchmarks to show what these metrics and their uncertainty estimates reveal about reliability. The central contribution is not an isolated metric or variance formula, but an account of how task heterogeneity and finite evaluation budgets jointly shape the conclusions drawn from benchmark results. The framework draws on established statistical results; the coarse-graining connection and analytical derivations are developed in Appendices § 7 and § 8.

Figure 1. Hierarchical data structure: scenarios contain multiple instances, each instance is sampled multiple times to account for variability. Aggregating metrics at different levels of this hierarchy corresponds to coarse graining.
Modeling Assumptions and Definitions
Throughout the paper, the policy model governs the agent’s decisions, while the environment includes the tools and infrastructure with which it interacts. We evaluate a fixed agent configuration, including its policy, harness, and environment setup. Following the terminology used in [5], we define the hierarchical structure of our framework as follows:
Scenario: A scenario is defined by a specific set of tools (e.g., a set of MCP servers) and the environment’s initial state. It is grounded in the real-world context it aims to simulate, such as retail, healthcare, finance, etc.
Instance: A specific task that the agent must complete using the tools available in a given scenario. An instance is a data point in our dataset comprising the user’s intent, the system prompt, and other relevant context. A dataset is a collection of instances across one or more scenarios.
Sample: A single attempt by the agent to resolve a given instance, also referred to as a “sample trajectory.” To assess the agent’s performance and variability, we need multiple samples for each instance.
For the analysis, we write for the trajectory distribution induced by the policy with parameters in this fixed evaluation setup. Here, specifies the task and its context, and is the resulting trajectory; the distribution includes any modeled environment or user-interaction randomness. Conceptually, population evaluation involves two levels of sampling: a task is drawn from an unknown population distribution , and a trajectory is then generated conditionally on that task:
In practice, we evaluate a finite dataset and repeatedly execute each task. § 3.1 connects this dataset to the task population and the agent’s task-specific success probabilities.
In the agentic context, the final output comprises a sequence of actions generated by the policy model and executed by the orchestrator in the environment, and may also include multiturn interactions with the user (see Figure 2). We define a sample trajectory as follows:
!Sample trajectory showing a single attempt and repeated sampling of the same instance bbox=0.11,0.08,0.73,0. thirty-five
Figure 2. Sample trajectory: (left) a single attempt of the agent to resolve a given instance, comprising actions, states, and user interactions. (right) Repeatedly sampling the same instance typically leads to different solution paths and possibly different outcomes.
where is the sequence of actions taken by the model, is the sequence of states observed in the environment, and is the sequence of user interactions (if any)1. An evaluation experiment can be understood as a rollout of the policy model on a set of instances, each associated with its own evaluation criterion. As in any experiment, stochasticity arises from various factors. We account for it by running multiple samples per instance and statistically analyzing the results to assess the model’s average performance and variability.
Throughout the paper, we adopt the following modeling assumption: each sample outcome is a random variable , where is the scenario index, is the instance index, and is the sample index. For this analysis, we consider binary outcomes: graders report each experiment as a success or a failure according to whether the task meets a specified set of evaluation criteria. The hierarchical variance decomposition extends to other rewards with finite variance, but the Bernoulli and success-metric formulas below rely on binary outcomes. Under this model, each sample is drawn from a Bernoulli distribution conditional on the true success probability of instance :
Here, is the true success probability for instance . In practice, we estimate it by running the model times on that instance and computing the empirical pass rate , where is
| Metric | Main intuition |
| Pass@1 | Typical single-attempt performance |
| Pass@k | Can retries rescue the task? |
| Pass | Is success consistently reproducible? |
the number of successful samples. Instances differ in their probability of success, and variation among the resulting estimates reflects both differences between instances and finite-sample noise. The statistical framework below separates these contributions.
Intuitively, evaluation proceeds in two steps: repeated attempts estimate each instance’s pass rate, then averaging these estimates gives a scenario-level mean pass rate. Section 3.2 explains how uncertainty propagates through these averages when estimating performance on a broader task population. From a practical perspective, a task’s estimated success probability indicates how likely an attempt is to succeed, while its uncertainty estimate indicates how precisely that probability is known. Other metrics considered here include pass@k and pass. For a given task, the former measures the probability that at least one of independent attempts succeeds; the latter measures the probability that all attempts succeed.
Statistical Analysis Framework
The dataset’s hierarchical structure (scenarios, instances, and samples) requires global metrics that capture model performance at different levels of granularity. Moving up the hierarchy corresponds to a coarse-graining process in which we aggregate performance metrics by averaging. However, this process can obscure important information about model performance in specific domains and on individual instances. For example, Ref. [5] presents a detailed benchmark of several models at the scenario and dataset levels and shows that no model consistently outperforms the others across all scenarios, even when some models lead in the across-scenario rankings. This finding highlights the importance of considering both fine- and coarse-grained metrics. Although pass rates can be aggregated throughout the hierarchy by simple averaging, propagating uncertainty across levels requires additional care, particularly for evaluations with limited numbers of instances [9, 15]. We address this issue in Section 3.2 within the hierarchical uncertainty-propagation framework, and provide an explicit derivation in Appendix § 7, which accounts for variability at multiple levels and their interactions; the derivation borrows the formalism of coarse-graining [14] from the physics and statistical literature. Informally, coarse-graining is the process of zooming out the description of a system, moving from a microscopic to a macroscopic description, e.g. describing a fluid in terms of its individual constituents versus a continuous field. Mathematically, zooming out consists in averaging the microscopic variables, which inevitably leads to a loss of resolution while highlighting collective, population-level aspects.
From task distributions to success probabilities
A benchmark contains tasks, not success probabilities. The connection is that each task has a probability of success for the particular agent being evaluated. Let indicate whether trajectory successfully completes task according to its evaluation criterion. For a fixed agent configuration and grader, define
For a fixed task, is a fixed, generally unknown number. Repeated attempts randomize the outcome, not this underlying probability. A benchmark author may anticipate difficulty from task characteristics such as the required reasoning or tool dependencies. However, success probability measures difficulty relative to the evaluated agent and grader: another agent may succeed more or less often on the same task. Like a school assignment, a task can have an intended level of complexity while posing different challenges to different students.
Now imagine selecting tasks from and recording each selected task’s true success probability. The resulting distribution of probabilities is . It describes differences across tasks, not uncertainty about the probability of one fixed task. For example, suppose a population contains two tasks, selected with probabilities 0.75 and 0.25, on which the agent succeeds with probabilities 0.2 and 0.8, respectively. Then assigns weight 0.75 to 0.2 and weight 0.25 to 0.8. The weights come from how often tasks are selected; the success probabilities come from the agent’s performance on those tasks. Changing either the task mix or the agent can therefore change . Formally, this transformation of the task distribution is called its pushforward through [16, 17, 18]:
Here, is the fraction of the task population, weighted by its selection probabilities, on which the agent’s success probability is at most . Writing is shorthand for selecting a task and taking its associated success probability, not an additional independent sampling step:
The population’s single-attempt success probability is therefore the mean of : . In the two-task example, it is 0.75 0.2 + 0.25 0.8 = 0.35. For the finite dataset , selecting an equally weighted task means using the empirical task distribution
where is a unit point mass at task . This describes selection from the benchmark; it does not make the population distribution known. Replacing each task by its true success probability gives
Thus, puts equal weight on the observed tasks, while puts the same weight on their true success probabilities. The mean of is the true dataset-level pass rate, . In practice, the are unknown. Replacing them by the observed pass rates defined in § 2 gives
Unlike , contains finite-run estimation noise as well as differences in true success probabilities. Even tasks with identical true probabilities can have different observed pass rates after a limited number of attempts. For a single binary outcome, the true success probability determines its distribution, but an estimated pass rate alone does not tell us how precisely that probability is known. Likewise, the dataset’s mean pass rate alone does not reveal how success probabilities vary across tasks.
Finally, estimating performance on the fixed benchmark and estimating performance on a broader task population are different goals. For the benchmark’s own mean, more independent attempts can make estimation uncertainty arbitrarily small. For the population mean, even perfectly known probabilities on those same tasks leave uncertainty about unobserved tasks. The analysis below models benchmark tasks as independent draws from a common task population to quantify this second contribution. This is a modeling assumption, not a guarantee of benchmark construction: deliberately selected or related tasks may require a different sampling model, and more attempts cannot correct an unrepresentative task mix.
Pass rate and hierarchical uncertainty propagation
In this section we estimate the scenario’s population mean success probability using the law of total variance. We first compute per-instance pass rates and associated uncertainties, then aggregate them at the scenario level by averaging across independently sampled instances.
Uncertainty propagation is not trivial, however, because it encompasses both within-instance variability (due to stochasticity in model behavior) and between-instance variability (due to differences in instance difficulty). Consider the first level of coarse-graining, from instances to a scenario:
Within a fixed instance, the outcomes are i.i.d. Bernoulli conditional on its true success probability . The number of successes is , and the empirical pass rate is :
Across instances, the success probabilities vary with task difficulty. We model them as independent and identically distributed draws from a common, unknown distribution :
Here, is induced by the task distribution within the scenario under consideration through the map defined above. We only require that the first two moments of exist and are finite.
Conditional on their success probabilities, the runs are independent within and across instances. The instance probabilities are also independent across instances.
Now consider instances in scenario . The scenario-level empirical pass rate is the average of the instance-level pass rates:
Its uncertainty follows from the law of total variance (see, e.g., Ref. [19] and the derivation in Appendix § 7):
Conditional on , the instance-level pass rates are independent, so
Averaging over the instance distribution gives the expected within-instance contribution:
The conditional mean is
Since the are i.i.d. with variance , the between-instance contribution is
Combining the two terms gives the total variance of the scenario-level pass rate [9]:
The first term is between-instance variability and is unaffected by increasing the number of runs ; this term would still exist when . The second term is within-instance sampling variability and decreases as increases. Both terms decrease as the number of independently sampled instances increases. Note that if the different tasks had the same difficulty, meaning the same pass-rate probability, the extra term and only sampling noise would be relevant.
Although run-level variability is averaged as we move up the hierarchy, it remains important at the instance level, where it can identify areas for model improvement and is closely tied to user experience. The same decomposition applies at the next level of the hierarchy, from scenarios to the global level, by treating scenarios as the units being averaged. Applying it recursively yields global performance metrics and uncertainty estimates that account for variability at multiple levels.
To summarize, each instance is sampled times to capture within-instance sampling variability; for each instance we can estimate its pass rate and the standard-error on it. The second step, we take an average across instances, that is the average of the pass rates. This mean pass rate now has an error propagated over from each individual instance, and one that captures the variability across instances, that is a population level variability, not a single-instance variability. The two errors scale differently with and and reveal different aspects of the agent’s performance on the eval set. In the next section we use a toy model as a proxy to show this difference explicitly.
A "random matrix" toy model
To make the two averaging steps explicit, consider a toy model with an data matrix . Columns represent tasks and rows represent repeated attempts. We retain as the instance index and as the run index, writing the matrix entry as ; for , this corresponds to above.
In a real benchmark, each column is a task whose agent-specific success probability is unknown. Here, drawing a probability from stands in for sampling a task and taking its associated probability, preserving the binary-outcome model under the assumptions of § 3.1 without generating task content. The model is
where entries are conditionally independent given the task probabilities. Draw the probabilities once, then generate entries in each column using that column’s probability. The positive integer counts Bernoulli trials per entry, with recovering binary outcomes. The overall average is
For , this is equivalent to Eq. (12), with the two-level average making the hierarchy explicit: repeated entries within each column estimate one task’s mean performance, while differences across column means reflect both task heterogeneity and finite-run noise. Averaging the column means gives the same point estimate as averaging all entries, but uncertainty propagation must preserve the task structure. As in the preceding subsection, the target is population performance: the unconditional variance considers variation in both the selected tasks and their execution outcomes. The conditional moments are
From here, the unconditional moments can be defined using the shorthand notations:
In the notation of the preceding subsection, and , where is the conditional sampling variance of the Bernoulli pass-rate estimator. Thus, for , and we recover the between-instance and expected within-instance variance contributions, respectively. Using this in the definition of the unconditional variance of :
With this, we can compute the unconditional variance of using Eq. (20):
Figure 3 distinguishes the data matrix from the covariance matrix used in this sum. For a fixed column , let ; its unconditional covariance matrix has entries

Figure 3. Data layout and unconditional covariance structure, panels (a)–(b). Each data column shares one latent probability . Diagonal covariance entries contain both sampling noise and instance variability; off-diagonal entries within a block contain only the shared instance contribution.
For , conditional independence gives , hence the off-diagonal covariance is . This unconditional covariance arises because both entries share the same randomly selected task probability; the executions remain independent conditional on that probability. Different columns are independent and have zero cross-covariance. The diagonal here therefore refers to pairing an entry with itself, not to the diagonal of the rectangular data matrix . Figure 4 shows the covariance structure across all task columns. Each of the covariance blocks has diagonal elements with and ordered off-diagonal elements with . Using this in Eq. (24) we obtain the general expression for the unconditional variance:

Figure 4. Unconditional covariance across all task columns. Blocks for different columns have zero cross-covariance. The variance of the overall average is the sum of all covariance entries divided by .
Choosing , we now recover Eq. (18). More rows improve the precision of each task mean, reducing the within-task contribution by . The between-task contribution reflects variation in true success probabilities and does not receive this additional reduction. Both contributions are reduced by averaging over tasks. If every task has the same true success probability, ; sampling different probabilities from a common does not impose that condition.
An explicit numerical example. To make things concrete, we compare two ways of estimating uncertainty in population mean performance from the same observations: preserving the task structure, or incorrectly treating every execution as an independent task. Both produce the same mean, but different standard errors. Figure 5 follows the analysis of a single observed matrix with tasks and binary outcomes (). The task probabilities are drawn once from , then for each column , we sample entries independently with success probability to obtain a matrix. Eq. (20) can be thought of as a two step process: first take task level means and their errors, then take the mean of these single task means and combine their standard errors hierarchically using Eq. (26). For comparison, flattening the same matrix and treating all entries as independent task-level evidence gives:

Figure 5. Two-step averaging of one observed matrix with tasks. (a) The outcome of the matrix generation process for the tasks, showing only the first rows of binary outcomes for convenience. (b) Task-level means with estimated SE. (c) The within-task and noise-corrected between-task contributions to the hierarchical variance estimate, compared with the pooled estimate; both methods give the same mean. (d) SE estimates from nested row prefixes of the same matrix. Repeated attempts are not additional independent tasks.
Because averaging is a linear operation and each task has the same number of runs, averaging the task-level means is equivalent to averaging all matrix entries. The pooled SE, however, ignores the hierarchy: additional runs improve estimates of existing tasks rather than providing additional independent tasks. Likewise, simply averaging the individual task SEs does not propagate uncertainty through the second averaging step. At , both procedures give , but the hierarchical SE is 0.07329 and the pooled SE is 0.02502, a factor of approximately 2.93 smaller. The propagated within-task contribution alone has SE 0.01971. The plot also compares estimates from nested row prefixes, , of this same matrix. Thus, the discrepancy is not caused by different observations or a different mean estimator, but by treating repeated executions as additional independent tasks. Panel (d) makes the consequence visible: the pooled SE continues to shrink with more runs, while the hierarchical SE retains uncertainty associated with having evaluated only 20 tasks.
pass@k and its Uncertainty
We now consider another important metric in agent evaluation [6]: , the probability that at least one of independent attempts succeeds on the same task. The gap between and reveals solutions that the current policy can produce but does not produce consistently. This can motivate reinforcement fine-tuning (RFT) aimed at increasing single-attempt success by making successful behaviors more likely [20, 21]. However, describes repeated sampling from the current policy; it neither predicts nor bounds the performance achievable after training. Given successes in independent attempts on task , the unbiased estimator of is
Here, when . The estimator averages the at-least-one-success indicator over all size- subsets drawn without replacement from the observed runs. This subset construction is distinct from the future independent attempts whose success probability it estimates. Over repeated evaluations, . Before evaluating its standard error, we summarize a few properties of the estimator; a constructive derivation is provided in Appendix Appendix § 8:
If , then for every admissible : no observed subset contains a success. This does not imply that the true success probability is zero.
If , the estimator reduces to the empirical pass rate, .
For large with fixed, the estimator can be approximated as (see also Appendix Appendix § 8.§ 3):
where is the empirical pass rate for instance . The expression on the right is a plug-in point estimate, not the exact unbiased estimator at finite .
Under , the estimator’s standard error is the square root of its exact conditional sampling variance:
Related finite-sample analysis of the estimator is presented in the recent manuscript [7]. Appendix Appendix § 8.§ 2 gives our derivation of these moments using the logarithm of the estimator’s characteristic function. The first two cumulants give the mean and variance exactly, without a truncation approximation; further derivatives give higher-order cumulants.
The variance formula is exact at the true probability . In applications, substituting and taking the square root gives a plug-in estimate of the standard error. This distinction applies to both metrics considered here. In particular, observing or makes this estimated SE zero, but does not establish that the true probability is exactly zero or one.
and its Uncertainty
Unlike , , introduced in -bench [1], is the probability that all independent attempts succeed on the same task. For task with true success probability ,
where are conditionally independent Bernoulli outcomes. This is the success–failure counterpart of : at least one success is the complement of all attempts failing. Given successes in independent attempts, the unbiased estimator [1] is

Figure 6. Estimated standard error for at fixed . Solid lines use the exact variance in Eq. (30) evaluated at ; open circles show Monte Carlo estimates under the same fitted binomial model. Dashed lines show the delta-method approximation.
with if . This estimator averages the all-success indicator over subsets of distinct observed runs, selected without replacement, and satisfies . Its exact conditional variance is
The coefficient in the summand is the probability that two independently and uniformly chosen size- subsets of the samples overlap in exactly attempts; impossible overlaps have coefficient zero. Their shared samples induce the covariance. Both unbiased estimators are U-statistics, and this subset-overlap argument specializes classical U-statistic variance theory [22, 23]. Recent manuscripts also analyze finite-sample estimation and uncertainty for [7, 8]; here, these per-task results are incorporated into the hierarchical uncertainty framework. Replacing by in Eq. (33) gives the variance of the unbiased estimator in Eq. (30): taking one minus the all-failure estimator leaves its variance unchanged. A simpler alternative estimates by substituting the empirical pass rate into :
Unlike Eq. (32), this plug-in estimator averages the all-success indicator over ordered -tuples drawn with replacement from the observed runs. Reusing an observed run is not equivalent to obtaining a fresh independent outcome. For and , the estimator is upward biased: because raising a probability to the th power is convex, fluctuations in increase the average of above . At , both estimators reduce to .
Appendix § 9 gives the exact conditional mean and variance of the plug-in estimator. Its standard error measures variation around its own expectation, not its bias relative to ; a smaller SE therefore does not by itself imply a more accurate estimator. For either estimator, we estimate its standard error by substituting for in its exact variance formula and taking the square root. Plugging into a variance formula is a separate choice from using a plug-in point estimator: it can be done for the unbiased estimator as well. Figure 7 compares the point estimators and their estimated standard errors.

Figure 7. Comparison of the unbiased and plug-in estimators for at fixed and . (Left) Point estimates as functions of the observed success count . (Right) Estimated standard errors from the exact variance formulas evaluated at (lines), compared with Monte Carlo estimates under the same fitted binomial model (open markers).
The uncertainty formulas in these two subsections concern individual tasks. Benchmark metrics are obtained by computing the per-task estimates first and then averaging with the chosen task weights; for , applying the nonlinear metric to a pooled mean pass rate generally gives a different quantity. For inference about a task population, the law of total variance from § 3.2 must also account for differences in the underlying per-task metric, or , across independently sampled tasks. Conditional on a fixed benchmark, only the execution-sampling contribution remains.
Empirical Analysis
In this section we apply the statistical framework developed above to results obtained on the think-ingbox dataset [5]; in particular we apply a the framework introduced in the previous sections to the Qwen-3.8-27b model [24], via a side-by-side (SBS) analysis of the base and reinforcement fine-tuned model, see Ref. [5] for training details. We provide few additional insights into the comparative performance of the two models, beyond what is reported in the original study. Although we choose to presents details for this specific dataset, the formalism and general analysis principle, are independent of the specific benchmark. In the following we refer to the base and reinforcement fine-tuned models’ results simply as base and RLFT, respectively.
The benchmark consists of five different business scenarios, each comprising evaluation instances. Overall metrics aggregate across attempts, tasks, and scenarios using the equal averaging defined above. We first consider mean pass-rate at different level of aggregation, as explained in the previous sections. Figure 8 shows the comparison between the base and RLFT models, both overall and at the scenario level. In particular we focus on the performance gap and report it alongside the corresponding standard errors, propagated according to the statistical framework introduced above and again on the gap itself. Perhaps not surprisingly, the magnitude of the gap varies across scenarios, highlighting differering learning dynamics and the varying impact of reinforcement fine-tuning across different business contexts. The disparity itself depends on the base model (the prior), but can be further increased or mitigated by the fine-tuning process. We can also zoom in at the instance level to better understand the gap distribution and the error characteristics induced by the training dynamics. Figure 9 provides an example of such an instance-level analysis, highlighting how individual evaluation instances contribute to the overall performance gap and its variability. Comparing to the scenario-level analysis in Figure 8, the instance-level view reveals the heterogeneity in model performance across individual evaluation instances, and also how individual errors are washed out in the aggregated metrics, that is the essence of coarse-graining, see § 3. Figure 9 shows only the 104 instances in the banking support scenario, comparing the same instance across the two models. Table 1 reports counts separately for each scenario, including instances with increased, decreased, or unchanged observed pass rates. Within each scenario, we count changes whose per-instance approximate 95% Wald intervals exclude zero: the interval is the observed RLFT-minus-base pass-rate difference plus or minus 1.96 times its estimated standard error, obtained by adding the two binomial plug-in variances under conditional independence of the models’ executions.

Figure 8. Mean pass-rate comparison between the base and RLFT (candidate) models. Left: overall mean pass rates and their paired gap. Right: scenario-level performance gaps (RLFT minus base), with instance counts . Error bars indicate standard errors; the blue line and shaded band indicate the overall mean gap and one standard error. The scenario aggregated metrics do not reveal the full picture of model performance across different business scenarios.

Figure 9. Distribution of the performance gap across individual instances in the 104-instance banking support scenario, highlighting variability and error characteristics. Green indicates improvements while red indicates regressions. No change is shown in gray.
| Metric | booking | auto insurance | consulting services | retail | banking support |
| number of instances | 104 | 100 | 101 | 98 | 104 |
| baseline mps ± ste | 0.534 ± 0.033 | 0.479 ± 0.032 | 0.457 ± 0.035 | 0.640 ± 0.041 | 0.479 ± 0.030 |
| candidate mps ± ste | 0.627 ± 0.035 | 0.565 ± 0.040 | 0.545 ± 0.037 | 0.679 ± 0.042 | 0.624 ± 0.034 |
| mean gap ± ste | 0.093 ± 0.011 | 0.086 ± 0.011 | 0.088 ± 0.011 | 0.039 ± 0.008 | 0.145 ± 0.012 |
| tests improved | 62 | 58 | 57 | 45 | 69 |
| tests degraded | 24 | 28 | 30 | 16 | 28 |
| tests no change | 18 | 14 | 14 | 37 | 7 |
| significant changes (95% CI) | 35 | 32 | 41 | 10 | 46 |
| no change total | 18 | 14 | 14 | 37 | 7 |
| no change both zero | 6 | 10 | 8 | 10 | 1 |
| no change both same nonzero | 12 | 4 | 6 | 27 | 6 |
Table 1. Separate comparisons of the baseline and candidate models within each scenario. Each column reports scenario-level mean pass rates (mps), their standard errors and candidate-minus-baseline gaps, together with counts of instance-level changes. No aggregation across scenarios is performed in this table.
Task Coverage, Reliability and Distributional Shifts
While (pass-rate) measures average success probability, larger increasingly measures coverage: whether a task has any appreciable chance of being solved. Figure 10 shows the mean and across scenarios for the baseline and candidate models. The two plots tell a different story: while a significant improvement is observed at , already at the difference between the two models is not statistically significant, and at the performance is almost indistinguishable. For the overall task-population comparison, the approximate 95% confidence intervals for the mean differences at and both include zero. This pattern is consistent with the interpretation that the RL fine-tuned model mainly improves the probability of success for tasks that are already solvable by the baseline, without substantially increasing coverage for harder tasks; in the next section we show the same effect by looking at the distributional shift across the two models. Although the task-coverage point estimates change little in our candidate model, reliability shows a more noticeable improvement, suggesting that the candidate model is more consistent in solving tasks it can handle, even if it does not cover substantially more tasks overall. Although the two metrics are related, they capture different aspects of model performance, and improvements in reliability can be valuable even when task coverage remains similar.

Figure 10. Blue: Base model, Orange: Candidate model. Top row: mean (task coverage). Bottom row: mean (reliability). Left column: ; right column: . Error bars indicate propagated standard errors.
Finally, we examine the effect of model training within a distributional shift point of view. Although this analysis is not part of our discussion on uncertaining quantification, it nevertheless provides a more holistic representation of the effect of model training on the task performance. Figure 11 illustrates the pass rate distribution before and after fine tuning. Two clear trends can be observed: first, there is a noticeable shift towards higher pass rates for the candidate model, particularly in the high pass-rate region; second, the distribution indicates that the model is becoming more reliable in the Goldilocks Zone, where task difficulty is optimal for learning. Finally, some of the tasks that had low pass rate, and were previously outside of the Goldilocks Zone, moved into the Goldilocks Zone and experienced improved success rates, while few regressed. Understanding regression behaviour is paramount for evaluating model improvements and to develop novel training strategies to mitigate negative impacts on previously well-performing tasks.

Figure 11. Distributional shift of model performance across different scenarios of mean pass rate for the baseline and candidate models across different bins of task difficulty. Goldilocks Zone indicates the range of task difficulty where the model can learn more efficiently. The figures identifies a push towards higher pass rates for the candidate model, particularly in the high pass-rate region.
Conclusion
Reliable evaluation of agentic systems requires more than an aggregate success score: it requires identifying what a metric measures, how precisely it has been estimated, and which differences are hidden by aggregation. We have presented a statistical framework organized around scenarios, task instances, and repeated samples, distinguishing stochastic execution variability from finite-sample estimation uncertainty and heterogeneity in task difficulty. Connecting this hierarchy to coarse-graining makes explicit both the value and the limitations of benchmark averages: they summarize collective behavior, but cannot by themselves characterize instance-level reliability or reveal localized regressions.
Within this framework, we have analyzed mean pass rate, , and , including exact finite-sample sampling variances for the unbiased estimators of the latter two metrics and hierarchical uncertainty propagation. These results give a principled basis for attaching uncertainty estimates to commonly reported scores and interpreting model comparisons. They also clarify that single-attempt performance, task coverage under retries, and repeated-success reliability answer different questions; no one of these metrics is a substitute for the others.
The Thinkingbox case study [5] illustrates the practical importance of this distinction. For the base and reinforcement fine-tuned Qwen-3.8-27b models considered here, improved mean pass rate coexists with similar coverage at larger and higher point estimates of repeated-success reliability. Together with the instance-level gaps and the shift toward higher pass rates, this pattern shows that the fine-tuned model is making already-solvable tasks more consistently successful, rather than substantially expanding task coverage, although some previously unsolved tasks may still benefit from fine-tuning, as shown by the instance-level and distributional analyses. Scenario averages nevertheless conceal individual improvements and regressions, emphasizing why aggregate gains should be examined alongside paired task-level comparisons and their uncertainty.
The resulting recommendation is to report mean pass rate, coverage, and reliability jointly, with uncertainty estimates at the relevant aggregation levels and sufficient instance-level detail to expose regressions. Extending this analysis to dependent executions and evaluation-budget allocation is a natural direction for future work. Ultimately, progress in agentic systems should be judged not only by whether average performance increases, but by whether that improvement is statistically supported, broadly distributed across tasks, and aligned with the reliability users actually need.
Use of Large Language Models (LLMs)
We used large language models (LLMs) to assist with manuscript drafting and editing, as well as code and math development. The authors developed the research ideas and methodology, derived main mathematical expressions, conducted the experiments, analyzed the results, and reviewed the LLM-assisted content and code. The authors take full responsibility for the final content of this work.
Law of total variance: a coarse-graining derivation
In this section, we provide an explicit evaluation of the hierarchical model described in § 3 using coarse graining methods. We use coarse-graining in the sense of replacing a detailed description by distributions of aggregate observables, a standard viewpoint in statistical mechanics. Here, the corresponding probabilistic operations are aggregation and marginalization, and the resulting variance decomposition is the law of total variance [25]. We evaluate these operations explicitly using the methods of Refs. [26, 14].
We consider one scenario with instances and runs per instance. As in § 3, we suppress the scenario index and write for the binary outcome of run on instance . For simplicity, each instance is sampled the same number of times; the derivation extends directly to variable sample counts. For each instance , define the per-instance average over runs:
where counts the successful runs for instance . Conditional on its true success probability , each outcome is Bernoulli:
We assume the are i.i.d. draws from a distribution , representing the variability in instance difficulty. The scenario-level average is
For binary outcomes, is the instance pass rate from § 3, and is the scenario pass rate . The calculation proceeds in two steps: first, we evaluate the conditional probability distribution of given ; then, we marginalize over the instance probabilities to obtain the unconditional distribution of . Following Ref. [26], the conditional distribution can be evaluated as follows:
where we have used the Fourier representation of the Dirac delta function
with a Fourier variable enforcing the Dirac delta constraint. The one-run characteristic function is ; independence of the runs gives the th power in Eq. (A.4). From here, we could evaluate the conditional moments of given by taking derivatives of its characteristic function. We move instead to the second level of the hierarchy and first evaluate the conditional distribution of given the set :
The conditional characteristic function appearing in the inverse Fourier transform in Eq. (A.6) is . We define the cumulant-generating function for the one-run characteristic function by , where . For large we expand around , writing , and retain terms through second order in :
Substituting this expansion into the logarithm of the conditional characteristic function in Eq. (A.6) gives, through second order in ,
This expansion identifies the first two conditional cumulants exactly; no normal approximation is needed to obtain the moments. Thus, conditional on ,
Marginal Characteristic function
We can now marginalize the conditional characteristic function over the instance probabilities to obtain the unconditional moments of . Under the i.i.d. assumption, this gives:
Assuming that instances are i.i.d. draws from , we can use a moment expansion to second order similar to what done in Eq. A.7. This only assumes that the first two moments of are finite, but otherwise makes no assumption on the form of itself. It is convenient to reintroduce the parametrization and expand to second order in the exponential:
Since the instances are i.i.d., the marginal characteristic function is the th power of this expression with . Taking its logarithm and retaining terms through second order gives
where we have defined:
Plugging this expression back into Eq. A.8 we can read off the unconditional mean and variance of :
The first term arises from variation in success probabilities across independently sampled instances; the second arises from the finite number of Bernoulli runs per instance. Thus the two components emerge constructively by marginalizing and expanding the characteristic function. Their sum matches Eq. (18) in § 3, where the same result is presented through the law of total variance.
Finite-Sample Analysis of
We derive the finite-sample distribution and variance of the estimator, complementing the main-text definition in Eq. (28). The derivation tracks both the total number of successes among runs and the outcome on a selected subset of runs.
Fix an instance and condition on its true success probability . The run outcomes are exchangeable under the i.i.d. Bernoulli model, so without loss of generality we take the selected subset to be the first runs. For , define
Here, is the total number of successes, indicates that all selected runs fail, and indicates that at least one selected run succeeds. The estimate conditional on the observed count is the probability of when a size- subset is selected uniformly at random. The joint probability mass function of and can be written using the Kronecker delta representation for integer . Define their joint characteristic function by
Both variables are discrete, which is why the Fourier integrals run over . Setting marginalizes over and recovers the characteristic function of , namely , where
This is the count version of the per-instance characteristic function in Eq. (A.4); the Fourier argument there is scaled by because it describes the sample average . Since and is binary, we can easily see that the following identity holds:
The first term contributes the characteristic function of . For the second term, independence of the runs gives
Consequently, the joint characteristic function is
Setting we obtain the characteristic function of alone.
Conditional characteristic function of
We now condition on the observed total count . Define
The first quantity is the probability mass of ; the second is the joint probability mass of and . Expanding with the binomial theorem gives
Similarly,
The conditional characteristic function of given is obtained by normalizing the joint Fourier coefficient by :
Substituting the expressions for and yields
The ratio is the conditional probability that all selected runs fail. Therefore, the conditional mean of —and hence the pass@k estimate—is
where when . Thus the estimator for instance is .
Sampling variance
At fixed , the count has probability mass function . The characteristic function of the estimator is therefore
With the characteristic-function convention used here, the first derivative of at zero is , and the second is :
The conditional mean simplifies to
The conditional sampling variance is
This is the variance formula reported in the main text. Averaging these conditional moments over the distribution of gives the corresponding unconditional moments.
Simple identities
For , the finite-sample formula reduces to the empirical pass rate. For , expanding the binomial coefficients gives
For , the numerator binomial coefficient is zero, so the same identity gives . For the large- approximation, write and use the product representation of the binomial-coefficient ratio:
where the approximation holds as grows with fixed (equivalently, when ).
Moments of the Plug-in pass^k Estimator
For a fixed task, let and . To express its exact moments, let be the falling factorial and let be a Stirling number of the second kind. Using the identities
with for , the exact conditional mean is

Figure 12. Scenario-level pass-rate estimate as a function of the number of runs per instance. The curve shows the mean across subsampling replicas, and the shaded band shows approximate 95% intervals based on the hierarchical standard error from Eq. (18).
For and , this exceeds , whereas the unbiased estimator in Eq. (32) has conditional mean exactly . Applying the same identity to the second moment gives the exact conditional variance
Its square root is the conditional standard error of the plug-in point estimator. Replacing by gives a plug-in estimate of this standard error, as discussed in the main text.
Choosing the sample size
In this section, we assess how the scenario-level pass-rate estimate and its uncertainty depend on the number of runs available per instance. For each instance, we have up to runs (here, ). Fix a scenario containing instances, and let be the target number of runs per instance; each included instance must have at least observed runs. For subsampling replica , draw a set of distinct run indices for each instance , and compute
The draws are without replacement within each replica, so no observed run is counted twice in that replica; different replicas may overlap. Since the original runs are i.i.d. conditional on each instance’s success probability, each such subset has the same marginal sampling distribution as runs from that instance. We repeat the subsampling with random seeds and report the replica-averaged scenario pass rate
Figure 12 shows this mean and approximate 95% intervals, , using the hierarchical variance propagation in Eq. (18) with . At the full sample size, , no subsampling is performed. As increases, the within-instance binomial-noise contribution decreases, while between-instance variability remains; accordingly, the aggregate uncertainty does not in general vanish as grows. The binomial-noise contribution scales as , so increasing the number of runs has diminishing returns, particularly when is large. The normal-approximation intervals should be interpreted cautiously for small evaluation sets, where they can understate uncertainty [15]. This analysis illustrates the trade-off between computational cost and the statistical precision of scenario-level performance estimates.
References
- [1]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024.arxiv.org/abs/2406.12045
- [5]Zhuochun Li, Youngmin Ko, Ali Keramati, Tuhin Kundu, Liang-Chun Tsai, Nicola Ferri, Mirco Milletari, Jiaxiang Liu, Susana Palmaz Lopez Pelaez, Yuepeng Wang, Vadim Smolyakov, Xiang Jiang, Kjartan Ólafsson, and Tommy Guy. One success isn’t reliability: Thinkingbox, a sandbox and benchmark for agents in stateful business workflows. arXiv preprint arXiv:2608.19741, 2026.arxiv.org/abs/2608.19741
- [6]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chanzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.arxiv.org/abs/2107.03374
- [7]Anonymous. Unbiased pass^k and pass^k evaluation with fewer rollouts, 2026. Submitted to ICLR 2027. OpenReview: https://openreview.net/forum?id=SWs3hMG9IC.
- [8]Anonymous. Are agent reliability rankings real? a statistical treatment of the pass^k metric, 2026. Submitted to ICLR 2027. OpenReview: https://openreview.net/forum?id=NH98Pcbbwd.
- [9]Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024.arxiv.org/abs/2411.00640
- [10]Bjarni Haukur Bjarnason, André Silva, and Martin Monperrus. On randomness in agentic evals. arXiv preprint arXiv:2602.07150, 2026.arxiv.org/abs/2602.07150
- [11]Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, and Arvind Narayanan. Towards a science of AI agent reliability. arXiv preprint arXiv:2602.16666, 2026.arxiv.org/abs/2602.16666
- [12]Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, and Vipin Chaudhary. Don’t Pass@k: A Bayesian framework for large language model evaluation. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2510.04265
- [13]Pahan Dewasurendra. Extrapolation is an assumption: Sharp limits for Pass@k forecasting. https://doi.org/10.5281/zenodo.21698609, 2026.
- [14]Mehran Kardar. Statistical Physics of Particles. Cambridge University Press, 2007.
- [15]Sam Bowyer, Laurence Aitchison, and Desi R. Ivanova. Position: Don’t use the clt in llm evals with fewer than a few hundred datapoints. In Proceedings of the 42nd International Conference on Machine Learning, Vancouver, Canada, PMLR 267, 2025.arxiv.org/abs/2503.01747
- [16]Terence Tao. An Introduction to Measure Theory, volume 126 of Graduate Studies in Mathematics. American Mathematical Society, 2011.DOI
- [17]Cédric Villani. Topics in Optimal Transportation, volume 58 of Graduate Studies in Mathematics. American Mathematical Society, 2003.DOI
- [18]Filippo Santambrogio. Optimal Transport for Applied Mathematicians: Calculus of Variations, PDEs, and Modeling, volume 87 of Progress in Nonlinear Differential Equations and Their Applications. Birkhäuser, 2015.DOI
- [19]Wikipedia contributors. Law of total variance. https://en.wikipedia.org/wiki/Law_of_total_variance. Wikipedia.
- [20]Kohsei Matsutani, Shota Takashiro, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, and Yutaka Matsuo. Rl squeezes, sft expands: A comparative study of reasoning llms. arXiv preprint arXiv:2509.21128, 2025.arxiv.org/abs/2509.21128
- [21]N. Lambert. Reinforcement learning from human feedback. https://rlhfbook.com/, 2025.arxiv.org/abs/2504.12501
- [22]Wassily Hoeffding. A class of statistics with asymptotically normal distribution. The Annals of Mathematical Statistics, 19(3):293–325, 1948.
- [23]Robert J. Serfling. Approximation Theorems of Mathematical Statistics. Wiley, 1980.
- [24]Qwen Team. Qwen3.8-27B model card. https://huggingface.co/Qwen/Qwen3.8-27B, 2026.
- [25]Joseph K. Blitzstein and Jessica Hwang. Introduction to Probability. Chapman and Hall/CRC, 2 edition, 2019.
- [26]Luca Leuzzi, Enzo Marinari, and Giorgio Parisi. Probability Theory for quantitative scientists. Cambridge University Press, 2025.DOI