betaFeedback

WorksComputer Science

Beyond Leaderboard Ranks: Selecting Software Engineering Agents Under Budget Constraints

Abstract

Teams that adopt LLM agents for issue resolution must decide which agent to pay for, and how much to spend finding out. Leaderboard ranks give weak evidence for this decision. Accuracy differences may be statistically unresolved, reported costs may be inconsistent, and evaluating candidate agents can itself be expensive. We investigate budget-constrained agent selection using 58,197 run–task records extracted from the Holistic Agent Leaderboard (HAL), focusing on SWE-bench and six additional benchmarks. Across seven benchmarks, 27 of 35 steps along the accuracy–cost Pareto frontiers are statistically unresolved. On SWE-bench, a run costing 34.16trailsthe34.16 trails the 170.78 leader by 14.6 percentage points, but the available tasks cannot statistically distinguish their accuracy. Reported costs also differ from standardized token-based estimates by up to 11.7x, while selective task evaluation can bias estimates of full-benchmark execution cost. We therefore compare evaluation strategies under explicit evaluation budgets and deployment-cost caps. Under advance knowledge of task costs, successive halving reaches our uncertainty-aware selection target on SWE-bench using 6.6% of exhaustive evaluation spend, achieving the lowest cost on six of seven benchmarks and tying on the seventh. Its advantage narrows when task costs are unknown before execution or exact-best selection is required. Our findings motivate decision-oriented agent evaluation that accounts for accuracy uncertainty, validates execution costs, and compares complex selection methods against simple adaptive allocation.

Introduction

Software teams now use autonomous agents to resolve issues in their repositories [1, 2, 3]. Each agent combines a language model with a scaffold that navigates the code, edits files, and runs tests. A team that adopts such an agent faces a maintenance budget question: which agent gives the most resolved issues per dollar on its backlog, and how much should the team spend to find that out? Two budgets apply. The evaluation budget pays for trial runs of candidate agents. The deployment budget pays for the chosen agent on future issues. Both matter, because one multi-turn repair attempt can consume many thousands of tokens, and small per-issue cost differences grow large over a backlog of hundreds of issues.

Existing evaluations primarily compare agents through leaderboard rankings. Prior work has advocated cost-controlled agent benchmarking [4] and proposed task-subset sampling methods that preserve leaderboard rankings while reducing evaluation effort [5]. However, reproducing a ranking does not necessarily identify the best agent for a given budget. Observed accuracy differences may be statistically unresolved, while reported or sampled execution costs may not reflect the expense of running an agent across the full benchmark. This raises a practical question: How can we select a competitive, affordable software engineering agent when evaluation itself is constrained by a dollar budget?

We investigate this question using execution traces from the Holistic Agent Leaderboard (HAL) [6], focusing on SWE-bench Verified Mini [7], [8]. Across 48 shared tasks, the highest-scoring run costs $170.78, while a $34.16 contender trails by 14.6 percentage points. Yet their paired accuracy difference is statistically unresolved, making it unclear whether the fivefold cost premium is justified. Figure 1 illustrates this trade-off and shows how inconsistent cost accounting can further distort agent comparisons.

Figure 1. Reconstructed API cost and accuracy of 34 SWE-bench Verified Mini runs on 48 shared tasks. The teal line traces the observed accuracy–cost Pareto frontier. At deployment cap d=$170.78d = \$170.78, the affordable unresolved set Ud\mathcal{U}_d contains the leader and four cheaper runs whose paired accuracy differences from the leader remain unresolved after BH correction (q=0.05q = 0.05). The $34.16 contender costs 5.0× less than the leader. Shading indicates the observed accuracy range. The inset shows a separate 50-task run whose published cost ($489.29) is 11.7× its reconstructed cost ($41.76) due to prompt-cache accounting discrepancies.

Motivated by this observation, we study agent selection as a budget-constrained decision problem rather than a leaderboard-ranking problem.

For repository-maintenance agents, this decision is shaped by the software repair workflow itself. Agents may repeatedly inspect source files, modify patches, execute tests, and diagnose failures, making execution cost dependent on the task and the agent’s scaffold. Consequently, selecting an agent requires comparing complete model-scaffold configurations under uncertain accuracy and execution costs, rather than ranking language models in isolation.

We address three research questions:

  • RQ1: Which accuracy improvements among more expensive agents can benchmark evaluations statistically resolve? Across seven benchmarks, 27 of 35 steps along the reconstructed accuracy–cost Pareto frontiers remain statistically unresolved. This limits what leaderboard rankings can establish about the benefits of paying for higher-ranked agents.

  • RQ2: How reliable are reported and sampled estimates of agent execution cost? Leaderboard costs differ from standardized token-based estimates by as much as 11.7×11.7 \times, while selectively evaluating medium-difficulty tasks can underestimate full-benchmark execution costs

by 20.4% on SWE-bench. Both effects complicate cost-aware agent selection.

  • RQ3: Which evaluation strategies efficiently identify affordable, competitive agents under a dollar budget? We compare simple allocation strategies with more complex acquisition and stopping methods. Under advance knowledge of individual task costs, successive halving reaches our uncertainty-aware selection target using 6.6% of exhaustive SWE-bench evaluation spend. It achieves the lowest spend on six of seven benchmarks and ties on the seventh, although its advantage narrows when task costs are unknown before execution.

To answer these questions, we extract 58,197 run–task records from HAL, reconstruct costs using standardized token accounting, analyze paired accuracy differences, and replay evaluation strategies under varying dollar budgets and cost-visibility assumptions. Rather than introducing a new optimization algorithm, our contribution is an empirical assessment of the evidence and evaluation strategies needed for budget-constrained agent selection.

Scope. SWE-bench is our primary case. It is the only repository-level maintenance benchmark in HAL with complete binary outcomes and recoverable token traces. To check whether our findings depend on this one benchmark, we repeat the analysis on three programming benchmarks (CORE-Bench [9], SciCode [10], USACO [11]) and summarize three general tool-use benchmarks (GAIA [12], TAU-bench [13], AssistantBench [14]). Our software maintenance conclusions rest on SWE-bench alone.

Software-Maintenance Benchmarks

Repository-level benchmarks evaluate software engineering agents by testing whether generated patches resolve real defects [7]. Repair success depends not only on the underlying language model but also on the agent’s scaffold and tool-use strategy. SWE-agent demonstrates the importance of agent–computer interfaces [2], while Agentless shows that a fixed localization-and-repair pipeline can remain competitive without autonomous tool loops [1]. SWE-bench Verified contains 500 human-screened repository issues [8], and the Holistic Agent Leaderboard (HAL) [6] uses a 50-task Verified Mini subset to reduce evaluation costs.

These benchmarks support comparisons between agent designs, but their rankings alone do not establish whether additional accuracy justifies higher execution costs. We study this selection problem using the 48 SWE-bench tasks shared by all complete runs in our retained HAL data (Section III-A).

Effort-Aware Evaluation in Software Engineering

Software engineering has a long history of evaluation that weighs benefit against the effort needed to obtain it. Effort-aware defect prediction ranks modules by predicted defects per line inspected, not by raw defect probability [15], [16]. Under this measure, simple unsupervised predictors can match or beat complex supervised ones [17], [18]. We apply the same view to agent selection. The “effort” is the dollar cost of agent execution, and the “benefit” is the number of resolved issues. As in effort-aware defect prediction, we test whether simple allocation rules match more complex ones once cost enters the evaluation.

Prior work also questions whether leaderboard ranks give enough evidence for model selection [19, 20, 21, 22].

Efficient Benchmark Evaluation

Existing approaches reduce evaluation effort by selecting informative benchmark examples. Anchor Points identifies representative subsets that preserve leaderboard rankings [23], [24], while Active Testing adaptively selects examples and corrects for sampling bias [25]. Closer to our setting, Ndzomga et al. [5] study task-subset selection across agent scaffolds and models, including mid-range filtering that prioritizes tasks with intermediate historical pass rates.

Other approaches predict unobserved benchmark outcomes. Li et al. [26] use learned dependencies between examples to guide evaluation acquisition, while Zhang et al. [27] find that simple random sampling remains a strong baseline, particularly for unseen systems.

These studies primarily emphasize ranking preservation, score estimation, or reducing the number of evaluations. Our objective differs: each purchased evaluation incurs a dollar cost, and the resulting observations must support selection of an agent that satisfies a deployment-cost cap. We measure whether the selected agent is truly affordable and whether its accuracy difference from the best affordable contender remains statistically unresolved, rather than whether the full leaderboard ordering is reproduced.

Budget allocation methods provide an alternative to selecting individual informative tasks. Successive halving repeatedly allocates evaluation resources among candidates and eliminates weaker performers [28], building on sequential elimination methods [29]. Hyperband extends this approach to configuration search [30]. We adapt successive halving to agent evaluations with unequal per-task dollar costs.

We also compare learned and heuristic acquisition strategies. EZR uses lightweight active learning to identify promising software configurations from limited measurements [31], [32]. Our adaptation acquires complete agent runs, while a cell-level Distance-to-Heaven (D2H) [33] sampler prioritizes individual agent–task evaluations using observed accuracy and cost. These approaches differ in their acquisition granularity and use of observed information.

By evaluating these strategies under common dollar budgets, we investigate whether more complex acquisition and stopping rules improve affordable agent selection over simple allocation. The policy adaptations and information constraints are detailed in Section III-F.

Data and Study Design

Trace Reconstruction and Inclusion Criteria

We reconstruct 58,197 run–task records from 379 of the 380 public HAL trace archives, spanning 11 benchmarks. Each run represents an executed agent configuration, including its model, scaffold, and parameter settings. We organize each benchmark into a matrix where rows represent runs, columns represent tasks, and each cell (i,j)(i,j) contains the outcome and execution cost of run ii on task jj. Repeated executions of the same configuration remain separate rows to preserve run-to-run variation.

We exclude four benchmarks with incomplete outcome or token representations. For the remaining seven, we retain tasks shared across runs and remove runs whose recorded outcomes cover fewer than half of their traced tasks or whose reconstructed costs are nonpositive. Missing or nonpositive individual cell costs are imputed using the corresponding run’s median positive cell cost. The resulting complete matrices contain 20,633 cells across 262 runs (Table I).

Our primary software engineering benchmark, SWE-bench Verified Mini, retains 34 runs evaluated on 48 of its original 50 tasks. Recomputing accuracy on these shared tasks changes individual scores by at most 2.33 percentage points without reversing any strict pairwise ranking. For other benchmarks, binary outcomes sometimes simplify native grading: Assistant-Bench uses exact-match outcomes, while CORE-Bench and SciCode use full task-level success rather than finer subtask scores.

We distinguish three benchmark categories. SWE-bench is our primary repository-maintenance benchmark, evaluating agents that repair issues in existing software repositories. CORE-Bench, SciCode, and USACO provide programming-oriented comparisons, while GAIA, TAU-bench Airline, and AssistantBench serve as general tool-use stress tests. The additional benchmarks examine whether the statistical and budget-allocation findings extend beyond repository maintenance.

Cost Reconstruction

We reconstruct API execution costs from HAL token logs using a standardized price sheet. For each model call, let II, RR, WW, and OO denote inclusive input, cache-read, cache-write, and output token counts, with corresponding per-token prices pIp_I, pRp_R, pWp_W, pOp_O. The reconstructed cost is

C=(I−R−W)pI+RpR+WpW+OpO.(1)C = (I - R - W)p_I + Rp_R + Wp_W + Op_O. \tag*{(1)}

We normalize provider-specific token fields before applying Equation (1). When cache usage is available only at the run level, we allocate cached tokens across tasks proportionally to their prompt-token counts.

Because provider prices change over time, published leaderboard costs may reflect different historical rate cards. We recover token rates from single-model runs where possible and standardize each model to its latest fitted pricing period, using assumptions, documented in our artifact, when exact recovery is not possible. This enables comparisons on a common price basis rather than reproducing historical invoices.

Our reconstructed costs represent API token spend only; they exclude hosting, external tools, and human-review costs. Incomplete cache records and multi-model scaffolds introduce additional uncertainty into task-level cost attribution.

BenchmarkkknnUnres. (%)StepsFloor (pp)
Repository maintenance
SWE-bench Mini344840.13/412.5
Programming-oriented tasks
USACO1530721.90/42.0
SciCode3865100.02/29.2
CORE-Bench614567.57/813.3
General tool-use
GAIA3516532.47/93.6
TAU Airline475079.86/612.0
AssistantBench3233100.02/218.2

Table I. Retained matrices and ranking resolution, grouped by benchmark domain. kk: run rows; nn: tasks; Unres.: unresolved pairs after BH correction; Steps: unresolved/all accuracy–cost frontier steps; Floor: best-case exact-test gap (100(6/n)100(6/n) pp, uncorrected; Section IV).

Paired Ranking Analysis

Let Yij∈{0,1}Y_{ij} \in\{0, 1\} indicate whether run ii succeeds on task jj. We compare run accuracy using the exact two-sided McNemar test [34], which accounts for paired outcomes on the same tasks. We apply Benjamini–Hochberg (BH) correction [35] at false discovery rate q=0.05q = 0.05 separately within each benchmark, covering 5,371 pairwise comparisons.

We call a pair unresolved when its corrected test does not reject equal success probabilities. This indicates insufficient evidence to distinguish the runs, not proof of equivalent performance.

We also construct each benchmark’s accuracy–cost Pareto frontier, retaining runs for which no cheaper run achieves equal or higher observed accuracy. A frontier step connects adjacent runs ordered by increasing cost; it is unresolved when their paired accuracy difference is not statistically significant.

We assess robustness using alternative tests, bootstrap confidence intervals, and Bonferroni correction [36]. We additionally examine whether bootstrap intervals fall within specified accuracy margins to distinguish statistical non-rejection from evidence of practical equivalence [37, 38].

Evaluation Budget and Deployment Cap

We simulate budget-constrained evaluation using the retained historical outcome and cost matrices. Purchasing cell (i,j)(i,j) reveals its outcome YijY_{ij} and incurs its reconstructed cost CijC_{ij}. Let E=∑i,jCijE = \sum_{i,j} C_{ij} denote the cost of exhaustively evaluating every run on every task. We test evaluation budgets of 1%, 2.5%, 5%, 10%, and 25% of EE, rounded to the nearest dollar.

We distinguish two constraints: the evaluation budget limits spending on comparing candidate agents, while the deployment cap dd limits the reconstructed API cost of executing the selected agent across all retained benchmark tasks. We use this complete-benchmark cost as a workload-level proxy for agent operating expense.

For each benchmark, we consider three deployment caps at the 25th, 50th, and 100th percentiles of true reconstructed complete-run costs. The 100th-percentile cap admits all retained runs.

After evaluation, each policy recommends the run with the highest estimated accuracy whose estimated complete-run cost does not exceed dd. Unless otherwise specified, accuracy is estimated from observed task outcomes, and complete-run cost is estimated as nn times the observed mean cell cost. Runs without observations receive an accuracy prior of 0.5 and a pooled mean cost estimate from observed cells.

Selection Quality and Uncertainty

Our objective is to select an affordable agent whose accuracy is not statistically distinguishable from the best affordable contender.

For deployment cap dd, let bdb_d denote the most accurate run whose actual complete-run cost does not exceed dd. Let Ud\mathcal{U}_d contain all truly affordable runs whose BH-corrected accuracy differences from bdb_d are unresolved, including bdb_d itself.

If a policy selects run i^d\hat{i}_d, its selection score is

Q=1∣Dvalid∣∑d∈Dvalid1[i^d∈Ud],(2)Q = \frac{1}{|\mathcal{D}_{\mathrm{valid}}|} \sum_{d \in\mathcal{D}_{\mathrm{valid}}} \mathbf{1}[\hat{i}_d \in\mathcal{U}_d], \tag*{(2)}

where Dvalid\mathcal{D}_{\mathrm{valid}} contains deployment caps for which the policy makes a recommendation. Caps without a recommendation are omitted, and QQ is missing if no recommendation is made.

Because QQ conditions on a recommendation, we also report an all-caps score that counts omitted recommendations as failures. To assess the quality of selected agents beyond statistical non-rejection, we additionally measure true affordability, accuracy regret relative to bdb_d, and an exact-best score that requires selecting a truly affordable run attaining the highest observed accuracy.

These complementary metrics are important because a high QQ on a low-resolution benchmark does not establish that the selected agent matches the best agent on future tasks.

Our primary experiments use 300 randomized replicates per condition. We compare policy differences using paired bootstrap intervals. To estimate the spending required to reach selection quality Q=0.80Q = 0.80, we additionally run 200 replicates across ten budget levels from 0.5% to 40% of EE and interpolate the first crossing of each policy’s mean quality curve. Additional thresholds and sensitivity analyses are reported in the replication artifact.

Evaluation Policies and Information Boundaries

We compare simple allocation baselines, adaptive selection strategies, and cost-aware extensions.

Simple allocation. Our non-adaptive baselines include random-cell sampling, random-task sampling (evaluating all runs on sampled tasks), round robin across runs, and cheapest-first acquisition.

We also adopt successive halving (SH) [28], which assigns equal dollar-spending targets to surviving runs within each round. Each run purchases unseen tasks until its spending reaches its target, with individual purchases allowed to exceed the local target if sufficient global evaluation budget remains.

Unused funds remain available globally rather than carrying over as run-specific allocations.

After each round, SH eliminates the lower-scoring half of surviving runs based on observed accuracy. Runs without observations rank below observed runs, with ties preserving the current survivor order. Consequently, at tight budgets, some candidates may be eliminated without an accuracy observation.

Selective task sampling. We adapt mid-range filtering (MR) [5], which initially spends 10% of its evaluation budget on random-cell probes to estimate task success rates. Subsequent purchases prioritize tasks with estimated success rates between 0.30 and 0.70, using remaining tasks when necessary. An oracle variant uses true full-matrix task success rates as a diagnostic upper-information condition.

Learned and heuristic acquisition. Our row-level EZR adaptation [31], [32] uses a best-versus-rest [39] Naive Bayes classifier [40] over model identifier, model family, provider, reasoning effort, scaffold, and token price rates. It begins with up to four complete runs and subsequently acquires one unobserved run at a time.

Cell-level Distance-to-Heaven (D2H) instead prioritizes individual agent–task evaluations. It first attempts two randomly selected task purchases per run, subject to budget feasibility. For each run ii, D2H computes

di=(1−a~i)2+c~i22,si=wini+1,(3)d_i = \sqrt{\frac{(1-\tilde{a}_i)^2+\tilde{c}_i^2}{2}}, \qquad s_i = \frac{w_i}{\sqrt{n_i+1}}, \tag*{(3)}

where a~i\tilde{a}_i and c~i\tilde{c}_i are min–max normalized observed accuracy and log⁡(1+c^i)\log(1+\hat{c}_i) across candidate runs. Here, c^i\hat{c}_i estimates complete-run cost as the observed mean cell cost multiplied by the benchmark task count, and nin_i is the number of purchased cells.

The weight wiw_i is 1 for the max⁡(1,⌊k⌋)\max(1,\lfloor\sqrt{k}\rfloor) runs with smallest did_i, and 0.15 otherwise, where kk is the number of candidate runs. D2H visits runs in descending acquisition score sis_i, breaking ties randomly, and selects an unseen task uniformly at random. Under advance cost checks, unaffordable purchases are skipped in favor of the next candidate.

We also test additional acquisition heuristics, cost estimators, and early-stopping rules. Detailed variants and ablations are provided in the replication artifact.

Cost visibility. Our primary replay assumes advance cost checks: policies can inspect the exact historical cost CijC_{ij} before purchasing a cell and skip purchases exceeding the remaining budget. This is an idealized setting because an agent’s actual token usage is generally unknown before execution.

We therefore also evaluate a prospective setting without advance cost checks. Here, policies observe execution costs only after purchasing cells and may exceed their nominal budgets. All oracle conditions are separated from deployable policies that rely only on purchased observations.

RQ1: Statistical Support for Accuracy Gains

Resolution of Accuracy-Cost Trade-offs

Paying more for a higher-ranked agent does not necessarily yield a statistically distinguishable accuracy improvement. On SWE-bench Verified Mini, three of four steps along the accuracy-cost Pareto frontier remain statistically unresolved. Among 34 runs evaluated on 48 shared tasks, 225 of 561 pairwise comparisons (40.1%) are unresolved after BH correction.

Figure 1 illustrates this trade-off. The cheapest contender statistically unresolved from the leader costs $34.16, compared with $170.78 for the leader, despite trailing by 1.6 percentage points in accuracy. The available evidence cannot establish whether this fivefold cost premium is justified. This does not imply equivalent performance, but identifies a cheaper candidate worth further evaluation.

The limitation extends beyond software maintenance. Across seven HAL benchmarks, 27 of 35 accuracy-cost frontier steps remain unresolved (Table I). Six benchmarks contain at least one unresolved step, with USACO the only exception. Across all 5,371 pairwise comparisons, unresolved fractions range from 21.9% on USACO to 100% on SciCode and AssistantBench. Thus, leaderboard rankings often imply finer distinctions than the available statistical evidence supports.

Why Small Benchmarks Cannot Resolve Accuracy Gaps

Limited statistical resolution is partly a mathematical consequence of benchmark size [19]. Consider two runs evaluated on nn shared tasks. If they disagree on mm tasks and every disagreement favors the same run, the exact two-sided McNemar test yields p=2(1/2)mp = 2(1/2)^m. At significance level α=0.05\alpha= 0.05, at least six such disagreements are needed to reject equal success probabilities. Therefore, the smallest observable accuracy gap that an uncorrected exact test can resolve is 6/n6/n [41]. Multiple-testing correction and disagreements in both directions can only increase this requirement.

For SWE-bench (n=48n = 48), this best-case resolution floor is 12.5 percentage points. In practice, competing agents often succeed on different subsets of tasks, further reducing statistical power. Simulations incorporating observed disagreement patterns estimate that SWE-bench requires a true accuracy gap of 20.7 percentage points to achieve 80% rejection power at uncorrected α=0.05\alpha= 0.05 (Table II).

The same limitation is more severe on smaller or lower-variance benchmarks. AssistantBench (n=33n = 33) has an exact-test floor of 18.2 percentage points, exceeding its entire observed accuracy range of 15.2 points. Consequently, no pair can pass even an uncorrected exact test under the retained binary outcomes. On SciCode (n=65n = 65), the 9.2-point resolution floor matches the observed score range, and no comparisons survive BH correction.

Robustness and Practical Interpretation

Low ranking resolution is not solely an artifact of BH correction. Without multiple-testing correction, 21.0% to 100% of run pairs remain unresolved under exact McNemar tests, with similar results from unadjusted bootstrap intervals (Table II). Alternative tests and Bonferroni correction provide additional sensitivity checks.

Importantly, statistical non-rejection does not establish practical equivalence. On SWE-bench, only 4.8% of run pairs have 90% bootstrap accuracy-difference intervals entirely within ±10\pm10 percentage points. Thus, many unresolved comparisons still permit differences large enough to matter in deployment.

RQ1 Takeaway. Most accuracy–cost frontier improvements remain statistically unresolved due to limited benchmark size and disagreements between agents. Preserving leaderboard order alone cannot justify paying more for a higher-ranked agent; selection must account for accuracy uncertainty and cost.

RQ2: Reliability of Agent Cost Estimates

Reliable agent selection requires comparable execution costs and accurate estimates of full-benchmark deployment spend. We examine two sources of uncertainty: discrepancies in published leaderboard costs and bias introduced when deployment costs are estimated from sampled tasks.

Discrepancies in Reported Execution Costs

Across 333 runs with positive published and reconstructed costs, the published-to-reconstructed cost ratio has a median of 1.00, a 95th percentile of 5.00, and a maximum of 11.7×. These discrepancies reflect accounting inconsistencies and historical pricing differences.

Prompt-cache accounting. In two Anthropic agent runs, logged token totals are consistent with cached tokens being counted twice, producing recorded-to-reconstructed cost ratios of 11.7× and 9.9×. The inset of Figure 1 illustrates the 11.7× cache-accounting discrepancy. For one run, HAL reports $1,789.67, compared with $180.13 under our standardized reconstruction. An execution of the same configuration approximately five hours earlier has substantially different reported

BenchmarkExactBoot.Bonf.WithinMDE
Repository maintenance
SWE-bench Mini36.933.262.64.820.7
Programming-oriented tasks
USACO21.020.036.221.96.6
SciCode97.783.4100.075.712.2
CORE-Bench58.752.995.42.124.9
General tool-use
GAIA31.129.452.110.913.3
TAU Airline68.164.096.40.230.2
AssistantBench100.094.6100.047.822.8

Table 2. Ranking-resolution sensitivity checks grouped by benchmark domain. Exact and Boot.: unresolved percentages without correction under exact McNemar and 95% bootstrap intervals. Bonf.: Bonferroni-corrected unresolved percentages. Within: percentage of 90% intervals inside ±10\pm10 percentage points. MDE: simulated 80%-power accuracy gap in percentage points.

costs despite similar reconstructed spend. This illustrates how inconsistent cache accounting can distort comparisons between agent runs.

Historical pricing. Leaderboard costs also reflect different provider price periods. For 16 runs using OpenAI’s o3, recorded costs reflect an earlier $10/$40 per-million-token input/output price sheet, compared with the $2/$8 rates used in our standardized reconstruction. This produces a 5.0× difference attributable to pricing rather than token consumption. Some runs also applied legacy rates after the price change, further complicating historical comparisons.

Reconstruction limits. Cost reconstruction is constrained by incomplete traces and multi-model scaffolds. Forty-two extracted runs lack recoverable token records, while multi-model executions do not always preserve sufficient information for precise cache attribution. Regression checks recover exact input/output rates for 14 model identifiers, but cache-rate assumptions remain necessary where the available logs are insufficient. Thus, reconstructed costs provide a consistent comparison basis rather than exact historical invoices.

Sampling Bias in Deployment-Cost Estimates

Even with standardized prices, evaluating only a subset of tasks can bias estimates of complete-run cost. We compare the estimated-to-true cost ratio across sampling strategies at a 10% evaluation budget (Table III).

Under oracle mid-range filtering, which selects tasks with full-matrix pass rates between 0.30 and 0.70, the median estimated-to-true cost ratio falls to 0.796 on SWE-bench and 0.834 on CORE-Bench. These correspond to underestimates of 20.4% and 16.6%, respectively. By contrast, random-task sampling yields ratios of 0.985 and 0.933 on the same benchmarks, despite purchasing fewer cells.

Tasks selected for accuracy discrimination may not represent the full distribution of execution costs. Across 34 retained SWE-bench runs, the 20 tasks with pass rates below 0.30 incur a mean cost of $4.38 per run–task cell, compared with $2.59 for the 27 mid-range tasks (pass rates 0.30–0.70). Thus, the hardest issues cost 1.69× more per attempt and fail more often. For a maintenance team, the cost of failed attempts falls on the hardest items in the backlog. A sampler that skips these

Estimator / samplerSWEUSACOSciCodeCORE
Oracle MR0.7960.9860.8220.834
Probed MR0.9730.9940.9910.934
Random tasks0.9850.9920.9090.933
Oracle MR + probe0.7930.9880.8270.827
Oracle MR + cost model0.7950.9650.7140.805
Oracle MR + both0.9820.9810.8841.055
Oracle MR, separate costs1.0061.0070.8481.037

Table 3. Median estimated/true complete-run cost ratio at a 10% evaluation budget (100 replicates). Values below 1 indicate underestimation. Oracle conditions use full-matrix task pass rates.

issues to sharpen accuracy comparisons also skips the most expensive part of the workload.

The practical mid-range sampler, which includes an initial random probe and purchases tasks outside the preferred difficulty band when necessary, shows substantially less bias on SWE-bench (0.973). Separating cost estimation from accuracy-driven sampling also improves cost ratios: a dedicated uniform probe yields 1.006 on SWE-bench and 1.037 on CORE-Bench, although residual bias remains on SciCode (0.848). These results suggest that representative cost observations are important when task selection is optimized for accuracy discrimination.

RQ2 Takeaway. Leaderboard costs can be wrong by up to 11.7× due to cache accounting and old price sheets. Hard issues cost more and fail more, so samplers that skip them underestimate deployment cost by up to 20%. Teams need standardized costs and a separate, uniform cost sample.

RQ3: Budgeted Agent Selection

Evaluation Cost of Competitive Selection

Successive halving (SH) provides a strong baseline for selecting affordable, competitive agents under limited evaluation budgets. When policies can check individual task costs before execution, SH reaches the uncertainty-aware selection targets Q=0.80Q = 0.80 and Q=0.90Q = 0.90 on all seven benchmarks. For Q=0.80Q = 0.80, it achieves the lowest interpolated evaluation spend on six benchmarks and ties on the seventh (Table IV).

On SWE-bench Verified Mini, exhaustively evaluating 34 runs across 48 tasks costs $5,399.89. SH reaches Q=0.80Q = 0.80 after spending $353.62, just 6.55% of exhaustive evaluation cost. Cell-level D2H follows closely at $379.19, whereas row-level EZR requires $942.20. These differences describe crossings of interpolated mean curves rather than statistically established differences in minimum evaluation cost.

The result extends across the programming-oriented benchmarks. SH achieves the lowest interpolated spend on SWE-bench, USACO, and CORE-Bench, and ties with other policies on SciCode. On CORE-Bench, SH requires $744.44 compared with $1,292.27 for random-task sampling. Its advantage is smaller on USACO, where the two policies require nearly identical budgets ($72.69 and $72.93). Among general tool-use benchmarks, SH reaches Q=0.80Q = 0.80 on GAIA for $1,171.54, while random-task sampling does not reach the target within 40% of exhaustive evaluation spend (Figure 2).

Figure 2. Mean selection score QQ versus reconstructed evaluation spend on three benchmarks (200 replay replicates per point). The dotted line marks the target Q=0.80Q = 0.80.

However, selection quality also depends on benchmark resolution. On SciCode, all corrected pairwise accuracy comparisons are unresolved, so any truly affordable selection receives full credit under QQ. Several policies consequently reach the target at the first tested budget level. Their apparent tie therefore reflects the tested budget grid and limited statistical resolution, not necessarily identical minimum evaluation costs. This illustrates why selection scores must be interpreted alongside ranking uncertainty (Section IV).

Selection Quality Beyond Statistical Ties

Our primary selection score QQ becomes less discriminative when many accuracy comparisons are statistically unresolved. We therefore assess whether high QQ corresponds to genuinely affordable selections with small observed accuracy regret (Table V).

At a 25% evaluation budget, SH selects truly affordable runs in 91.2%–100% of cap decisions across benchmarks, with mean accuracy regret between 0.0 and 3.2 percentage points. On SWE-bench, all selections are affordable, with mean regret of 1.4 percentage points, although only 60.5% select the exact highest-scoring affordable run.

These results indicate that SH often identifies affordable candidates with low observed regret, even when it does not identify the exact-best configuration.

Evaluation Without Advance Cost Checks

The preceding results assume advance knowledge of each cell’s execution cost. In live evaluations, however, token consumption is generally unknown until an agent completes its task. We therefore repeat the replay without advance cost checks, allowing purchases to exceed the nominal evaluation budget.

SH remains competitive but no longer leads on nearly every benchmark. Under the all-caps Q=0.80Q = 0.80 criterion, it achieves the lowest mean actual spend on four of seven benchmarks. Random-task sampling leads on USACO, mid-range filtering on SciCode, and D2H on GAIA.

Unknown execution costs also introduce budget overruns. At a 10% SWE-bench evaluation budget, every SH replicate exceeds its allocation, although the mean overrun is only $3.04. Row-level EZR incurs larger overruns because acquiring a complete run commits the evaluator to many task executions at once. Cell-level policies offer finer budget control by purchasing individual agent–task evaluations.

These results indicate that both cost visibility and acquisition granularity influence the effectiveness of budgeted agent selection.

Do More Complex Acquisition Heuristics Help?

Variants of row-level EZR yield mixed results across benchmarks. Although expected-improvement (EI) acquisition improves selection quality in several settings, it reduces SWE-bench QQ by 0.132 at the 25% budget (95% CI [−0.169, −0.097]). Alternative token-price feature variants likewise provide no consistent benefit.

Additional heuristics also fail to consistently improve cell-level D2H. Adding cost-normalized acquisition weights and mixed-outcome task prioritization reduces QQ at a 25% budget on SWE-bench (0.925 versus 0.968 for D2H), USACO (0.841 versus 0.964), and CORE-Bench (0.769 versus 0.838). Thus, simpler acquisition rules can outperform more elaborate alternatives.

Cost-estimation choices also affect selection. Using median rather than mean cell costs can underestimate complete-run costs for right-skewed execution distributions. Under median-based estimation, D2H reaches Q=0.80Q = 0.80 at lower spend than SH on SWE-bench ($449.04 versus $477.87), while no policy reaches the target on GAIA within the tested range. By contrast, stress tests of token rates and cache allocation leave SH’s leading position largely unchanged in the examined cases.

BenchmarkSHRandomMRD2HEZR
Repository maintenance
SWE-bench Mini353.62755.98655.06379.19942.20
Programming-oriented tasks
USACO72.6972.9383.12173.55–
SciCode25.2925.2925.2925.2936.73
CORE-Bench744.441,292.271,528.52931.82–
General tool-use
GAIA1,171.54–4,174.731,214.072,371.57
TAU Airline156.11235.16218.63160.55280.07
AssistantBench81.87200.61158.50140.32124.48

Table 4. Interpolated mean evaluation spend (USD) required to reach Q=0.80Q = 0.80 under advance cost checks, grouped by benchmark domain. Bold indicates the lowest spend among five policies. A dash indicates the target was not reached within 40% of exhaustive evaluation cost.

BenchmarkAff. (%)Regret (pp)Exact best (%)
SWE-bench Mini100.01.460.5
USACO99.80.591.3
SciCode100.00.099.7
CORE-Bench93.83.258.3
GAIA91.20.879.3
TAU Airline99.82.558.0
AssistantBench100.00.874.8

Table 5. Successive halving selection quality at a 25% evaluation budget. Aff.: percentage of selections that are truly affordable; Regret: mean accuracy regret in percentage points among affordable selections; Exact best: percentage selecting a truly affordable run with the highest observed accuracy.

Early Stopping and Selection Quality

Finally, we examine whether stopping evaluation when predicted improvements become small can reduce spending without compromising selection.

Our stopping rule halts evaluation when every candidate’s predicted one-step accuracy improvement falls below λ(6/n)\lambda(6/n), where 6/n6/n is the benchmark’s uncorrected exact-test resolution floor. We vary the threshold scale λ\lambda to examine the trade-off between evaluation spending and selection quality.

Early stopping produces a clear cost–quality trade-off. At the 25% SWE-bench budget, stopping at a threshold scale of 0.25 reduces budget use by 16.1 percentage points but lowers QQ from 0.756 to 0.614. On benchmarks with limited ranking resolution, stopping can save substantially more budget with little change in selection score.

However, this apparent efficiency can reflect the inability of a benchmark to distinguish candidate agents rather than successful identification of the best candidate. Aggressive stopping is therefore difficult to justify when meaningful accuracy differences remain unresolved. RQ3 Takeaway. Under advance cost checks, successive halving reaches the uncertainty-aware SWE-bench selection target using 6.55% of exhaustive evaluation spend. Its advantage narrows under unknown execution costs and stricter selection criteria, while more complex heuristics offer inconsistent improvements.

Threats to Validity

Outcome validity. We treat SWE-bench’s test-based “resolved” label as ground truth. Prior work shows that some test-passing patches are incorrect or only plausible [43, 44]. Such errors add noise to accuracy estimates and can further reduce ranking resolution. Our cost findings do not depend on this label.

Dataset validity. HAL provides a convenience sample of agent executions, and our completeness filters exclude benchmarks and runs with insufficient traces or outcomes. SWE-bench analysis uses 48 shared tasks rather than the full 50-task subset. Other benchmarks sometimes use simplified binary outcomes instead of native grading metrics. These choices limit generalizability, particularly beyond repository-level maintenance tasks.

Statistical validity. Paired tests treat benchmark tasks as evaluation units, potentially overlooking dependencies between issues from the same repository [20, 45]. Because comparisons share runs and tasks, BH correction does not guarantee false discovery rate control under arbitrary dependence [46]. We therefore report alternative tests and correction procedures as sensitivity checks. Moreover, statistical non-rejection does not establish practical equivalence. Only four repeated-configuration groups are available, limiting conclusions about agent execution variability [47].

Cost validity. Reconstructed costs rely on standardized pricing, cache-allocation assumptions, and occasional imputation. Incomplete traces and multi-model scaffolds further limit exact attribution. Our estimates capture API token spending but exclude hosting, external tools, and human-review costs. Consequently, reconstructed dollar amounts support comparative analysis rather than exact historical invoices or future deployment quotes.

Replay and selection validity. Budget policies operate on fixed historical outcomes rather than fresh agent executions. The primary replay assumes advance knowledge of individual task costs, which is generally unavailable in live evaluations. Our prospective replay relaxes this assumption but can exceed nominal budgets. Furthermore, the selection score QQ credits unresolved accuracy differences and omits caps without a recommendation, potentially favoring policies on low-resolution benchmarks. We address these limitations through affordability, all-caps, regret, and exact-best analyses, but prospective validation remains necessary.

Comparison validity. Evaluated strategies differ in acquisition granularity, initialization, features, and cost estimators. Their observed performance therefore reflects complete policy implementations rather than isolated algorithmic components.

Because all strategies were explored on the same historical corpus, our findings establish comparative evidence within this setting, not universal superiority of successive halving.

Our SH implementation also uses soft per-run spending targets and does not guarantee an initial observation for every candidate. At tight budgets, runs receiving no observations can be eliminated before their accuracy is measured, potentially disadvantaging expensive configurations.

Discussion and Practical Implications

Our findings suggest that evaluating software engineering agents should focus on supporting reliable, affordable selection decisions rather than reproducing leaderboard rankings alone. We discuss three implications for agent evaluation and software maintenance practice.

From Leaderboard Rankings to Agent Selection

Report supported accuracy–cost trade-offs. Agent leaderboards should report accuracy uncertainty alongside execution costs, rather than relying solely on point estimates and ranks. As RQ1 demonstrates, higher-ranked agents frequently lack statistically distinguishable gains over cheaper alternatives. For software teams selecting maintenance agents, an observed accuracy improvement should therefore be weighed against its additional execution cost and the uncertainty of the measured difference. Statistically unresolved comparisons do not establish equivalent patch quality; instead, they identify candidates that warrant further evaluation before committing to a more expensive configuration.

Reliable Cost Measurement and Estimation

Separate evaluation budgets from deployment costs. Agent evaluation involves deciding both how much to spend comparing candidate systems and how much to spend executing the selected agent. These costs should be reported separately, with task-level token usage, cache accounting, and model pricing preserved for reproducibility. Standardized price sheets can prevent historical pricing differences from distorting comparisons.

Validate costs independently of accuracy sampling. Tasks selected for their ability to distinguish agent accuracy may not represent the execution costs of the complete workload. As RQ2 shows, difficulty-based filtering can underestimate full-benchmark costs, while separate uniform cost probes reduce this bias in several cases. Evaluators should therefore check whether accuracy-oriented task subsets also provide representative cost estimates before using them to enforce deployment budgets.

Toward Budget-Aware Agent Evaluation

Establish simple adaptive allocation baselines. Successive halving provides a strong starting point for evaluating multiple agent configurations under limited budgets. By allocating initial resources broadly and progressively concentrating evaluation on promising candidates, it can substantially reduce evaluation spending without requiring complex acquisition models. Our results suggest that additional cost-aware heuristics and early-stopping rules should be justified by consistent improvements in selection quality over such simple baselines.

However, budget-aware evaluation must also reflect the unpredictability of agent execution. In practice, multi-turn tool-use trajectories have unknown costs before completion, and successive halving’s advantage narrows when advance cost checks are unavailable. Practical evaluation systems should therefore track realized spending, account for possible overruns, and assess selection quality beyond statistical ties.

Validate selection on live backlogs. A natural next step is to run budgeted selection on a team’s own issue backlog, with patch review as the outcome measure. Our analysis considers complete agent configurations, including models and scaffolds, but measures historical benchmark outcomes and API token costs rather than the downstream quality of deployed patches. Future work should validate budgeted selection through fresh repository-level executions that account for agent stochasticity, patch correctness, maintainability, and operational costs. This would establish whether efficient benchmark selection translates into reliable software maintenance decisions.

Closing perspective. Across seven HAL benchmarks, 27 of 35 accuracy-cost frontier steps remain statistically unresolved, while successive halving reaches our uncertainty-aware selection target on SWE-bench using only 6.6% of exhaustive evaluation spend under advance cost checks. Together with the cost-accounting and sampling-bias findings, these results motivate a decision-oriented approach to agentic software engineering evaluation. Progress should be measured not only by higher benchmark scores, but by how reliably and economically those scores support agent-selection decisions.

Data Availability

The source execution traces are publicly available in the HAL dataset [6]: https://huggingface.co/datasets/agent-evals/hal_traces. The study uses the released task outcomes and token-usage records to construct its analysis matrices.

References

References

  1. [1]C. S. Xia et al., “Agentless: Demystifying LLM-based software engineering agents,” 2024. arxiv.org/abs/2407.01489arxiv.org/abs/2407.01489
  2. [2]J. Yang et al., “SWE-agent: Agent-computer interfaces enable automated software engineering,” 2024. arxiv.org/abs/2405.15793arxiv.org/abs/2405.15793
  3. [3]X. Wang et al., “OpenHands: An open platform for AI software developers as generalist agents,” in Proc. ICLR, 2025. arxiv.org/abs/2407.16741arxiv.org/abs/2407.16741
  4. [4]S. Kapoor et al., “AI agents that matter,” 2024. arxiv.org/abs/2407.01502arxiv.org/abs/2407.01502
  5. [5]F. Ndzomga, “Efficient benchmarking of AI agents,” 2026. arxiv.org/abs/2603.23749
  6. [6]S. Kapoor et al., “Holistic agent leaderboard: The missing infrastructure for AI agent evaluation,” in Proc. ICLR, 2026. arxiv.org/abs/2510.11977arxiv.org/abs/2510.11977
  7. [7]C. E. Jimenez et al., “SWE-bench: Can language models resolve real-world GitHub issues?” in Proc. ICLR, 2024. arxiv.org/abs/2310.06770arxiv.org/abs/2310.06770
  8. [8]OpenAI, “Introducing SWE-bench Verified,” 2024. openai.com/index/introducing-swe-bench-verified
  9. [9]Z. S. Siegel et al., “CORE-Bench: Fostering the credibility of published research through a computational reproducibility agent benchmark,” 2024. arxiv.org/abs/2409.11363arxiv.org/abs/2409.11363
  10. [10]M. Tian et al., “SciCode: A research coding benchmark curated by scientists,” in Proc. NeurIPS, 2024. arxiv.org/abs/2407.13168arxiv.org/abs/2407.13168
  11. [11]Q. Shi et al., “Can language models solve olympiad programming?” 2024. arxiv.org/abs/2404.10952arxiv.org/abs/2404.10952
  12. [12]G. Mialon et al., “GAIA: A benchmark for general AI assistants,” in Proc. ICLR, 2024. arxiv.org/abs/2311.12983arxiv.org/abs/2311.12983
  13. [13]S. Yao et al., “τ-bench: A benchmark for tool-agent-user interaction in real-world domains,” in Proc. ICLR, 2025. openreview.net/forum?id=roNSXZpUDNDOI
  14. [14]O. Yoran et al., “AssistantBench: Can web agents solve realistic and time-consuming tasks?” in Proc. EMNLP, 2024. arxiv.org/abs/2407.15711arxiv.org/abs/2407.15711
  15. [15]T. Mende and R. Koschke, “Effort-aware defect prediction models,” in 2010 14th European Conference on Software Maintenance and Reengineering. IEEE, 2010, pp. 107–116.
  16. [16]Y. Kamei et al., “A large-scale empirical study of just-in-time quality assurance,” IEEE Transactions on Software Engineering, vol. 39, no. 6, pp. 757–773, 2012.DOI
  17. [17]Y. Yang et al., “Effort-aware just-in-time defect prediction: simple unsupervised models could be better than supervised models,” in Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, 2016, pp. 157–168.
  18. [18]W. Fu and T. Menzies, “Revisiting unsupervised learning for defect prediction,” in Proceedings of the 2017 11th joint meeting on foundations of software engineering, 2017, pp. 72–83.DOI
  19. [19]D. Card et al., “With little power comes great responsibility,” in Proc. EMNLP, 2020. doi.org/10.18653/v1/2020.emnlp-main.745arxiv.org/abs/2010.06595
  20. [20]E. Miller, “Adding error bars to evals: A statistical approach to language model evaluations,” 2024. arxiv.org/abs/2411.00640arxiv.org/abs/2411.00640
  21. [21]P. Rodriguez et al., “Evaluation examples are not equally informative: How should that change NLP leaderboards?” in Proc. ACL, 2021. doi.org/10.18653/v1/2021.acl-long.346DOI
  22. [22]Y. Perlitz et al., “Efficient benchmarking (of language models),” in Proc. NAACL, 2024. doi.org/10.18653/v1/2024.naacl-long.139
  23. [23]R. Vivek et al., “Anchor points: Benchmarking models with much fewer examples,” in Proc. EACL, 2024. aclanthology.org/2024.eacl-long.95
  24. [24]F. M. Polo et al., “TinyBenchmarks: Evaluating LLMs with fewer examples,” 2024. arxiv.org/abs/2402.14992arxiv.org/abs/2402.14992
  25. [25]J. Kossen et al., “Active testing: Sample-efficient model evaluation,” in Proc. ICML, 2021. proceedings.mlr.press/v139/kossen21a.htmlarxiv.org/abs/2103.05331
  26. [26]Y. Li et al., “Active evaluation acquisition for efficient LLM benchmarking,” 2024. arxiv.org/abs/2410.05952arxiv.org/abs/2410.05952
  27. [27]G. Zhang, F. E. Dorner, and M. Hardt, “How benchmark prediction from fewer data misses the mark,” 2026. arxiv.org/abs/2506.07673arxiv.org/abs/2506.07673
  28. [28]K. Jamieson and A. Talwalkar, “Non-stochastic best arm identification and hyperparameter optimization,” in Proc. AISTATS, 2016. proceedings.mlr.press/v51/jamieson16.html
  29. [29]Z. Karnin, T. Koren, and O. Somekh, “Almost optimal exploration in multi-armed bandits,” in Proc. ICML, 2013. proceedings.mlr.press/v28/karnin13.html
  30. [30]L. Li et al., “Hyperband: A novel bandit-based approach to hyperparameter optimization,” J. Mach. Learn. Res., vol. 18, 2018. jmlr.org/papers/v18/16-558.html
  31. [31]T. Menzies and S. Srinivasan, “Can AI be easy? lessons learned from the EZR.py toolkit,” 2024. arxiv.org/abs/2606.03640
  32. [32]T. Menzies, “EZR: Explanation system for semi-supervised multi-objective optimization,” 2026. github.com/timm/ezr
  33. [33]T. Menzies et al., “MOOT: A repository of many multi-objective optimization tasks,” in Proc. MSR, 2026. github.com/timm/mootarxiv.org/abs/2511.16882
  34. [34]Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, 1947. doi.org/10.1007/BF02295996pubmed.ncbi.nlm.nih.gov/20254758
  35. [35]Y. Benjamini and Y. Hochberg, “Controlling the false discovery rate: A practical and powerful approach to multiple testing,” J. R. Stat. Soc. B, vol. 57, 1995. doi.org/10.1111/j.2517-6161.1995.tb02031.xDOI
  36. [36]C. E. Bonferroni, “Teoria statistica delle classi e calcolo delle probabilità,” Pubbl. R. Ist. Sup. Sci. Econ. Comm. Firenze, vol. 8, 1936.
  37. [37]K. F. Phillips, “Power of the two one-sided tests procedure in bioequivalence,” J. Pharmacokinet. Biopharm., vol. 18, 1990. doi.org/10.1007/BF01068703pubmed.ncbi.nlm.nih.gov/2348380
  38. [38]D. Lakens, “Equivalence tests: A practical primer for t tests, correlations, and meta-analyses,” Soc. Psychol. Personal. Sci., vol. 8, 2017. doi.org/10.1177/1948550617697177
  39. [39]L. C. Tiao et al., “BORE: Bayesian optimization by density-ratio estimation,” in Proc. ICML, 2021. proceedings.mlr.press/v139/tiao21a.htmlarxiv.org/abs/2102.09009
  40. [40]G. I. Webb, “Naïve Bayes,” in Encyclopedia of Machine Learning. Springer, 2010. doi.org/10.1007/978-0-387-30164-8_576
  41. [41]R. J. Connor, “Sample size for testing differences in proportions for the paired-sample design,” Biometrics, vol. 43, 1987. doi.org/10.2307/2531961pubmed.ncbi.nlm.nih.gov/3567305
  42. [43]Y. Wang, M. Pradel, and Z. Liu, “Are “solved issues” in swe-bench really solved correctly? an empirical study,” in Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering, 2026, pp. 169–181.DOI
  43. [44]R. Aleithan et al., “Swe-bench+: Enhanced coding benchmark for llms,” arXiv preprint arXiv:2410.06992, 2024.arxiv.org/abs/2410.06992
  44. [45]J. Du et al., “DependEval: Benchmarking LLMs for repository dependency understanding,” in Findings ACL, 2025. aclanthology.org/2025.findings-acl.373
  45. [46]Y. Benjamini and D. Yekutieli, “The control of the false discovery rate in multiple testing under dependency,” Ann. Stat., vol. 29, 2001. doi.org/10.1214/aos/1013699998DOI
  46. [47]S. Ouyang et al., “An empirical study of the non-determinism of ChatGPT in code generation,” ACM Trans. Softw. Eng. Methodol., vol. 34, no. 2, 2025. doi.org/10.1145/3697010DOI

Paper details

vlnc:W_zq2ubfhp
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents