WorksComputer Science
HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference
Abstract
Evaluating software agents on a full benchmark is expensive. A small task subset can preserve agent rankings while yielding inaccurate scores. Historical responses provide useful structure, but tasks with identical source responses can have different outcomes for a new target agent. We introduce Hierarchical Score Posterior Recovery (HSPR), which combines source-fitted response factors with uncertain group success rates. Observed target outcomes update the posteriors over the target latent state and group rates; posterior-guided acquisition selects subsequent tasks. Its score-based criterion equals the expected one-step reduction in full-score posterior variance. Observed successes and posterior-predictive counts for unqueried tasks yield a full-score posterior. The final estimate uses its median with source-selected calibration. At a 10% task budget on four task banks from the SWE-bench family, HSPR achieves the lowest mean absolute error (MAE) on three banks and the lowest maximum absolute error (MaxErr) on all four among the methods compared. Removing group uncertainty or replacing adaptive queries increases macro MAE in the respective controls. Macro MAE decreases across 5-20% budgets. Code and project resources are publicly available.
Introduction
Evaluating a software agent requires repository exploration, code editing, tool use, and execution across hundreds or thousands of tasks [8, 4]. A small execution budget makes this evaluation more practical and creates an inference problem: recovering the agent’s score on the full benchmark from a few observed task outcomes. Accurate score recovery
is essential for comparing progress and assessing capability.
At a 10% budget on SWE-bench Pro, Mid-Range selection (Ndzomga, 2026) preserves model ordering well (), yet its raw subset mean has mean absolute error (MAE) of 22.798 percentage points (pp) (Table 1). Using the same Mid-Range outcomes, a source-fitted Ridge estimator reduces MAE to 3.434 pp. Estimation accuracy also varies across task banks: ATLAS-port has MAE of 0.957 pp on Full and 7.736 pp on Pro. Low-budget evaluation depends on both task selection and the estimator applied to the observed outcomes.
The source-response signature of a task records its binary outcomes across historical agents. Tasks with identical signatures may have different outcomes for a new target agent, while still sharing a response tendency. Modeling this tendency as an uncertain group success rate lets one observation inform other group members without fixing their outcomes. We ask: How can we recover a target agent’s full-benchmark score from a few executions while accounting for uncertainty in how source response patterns transfer to the target? We introduce Hierarchical Score Posterior Recovery (HSPR), illustrated in Figure 1. HSPR uses source-fitted response factors to model a target latent response state and uncertain success rates within source-signature groups. Each observed outcome updates the latent-state posterior and the group-rate posteriors. Posterior-guided acquisition selects the next task using these dependencies. Observed successes and posterior-predictive counts for unqueried tasks yield a distribution over the full-benchmark score. Its median minimizes model-conditional expected absolute loss; source-only validation selects calibration for the final estimate.
Figure 1. Task-efficient software-agent evaluation. Representative strategies estimate full-benchmark scores from limited target executions. HSPR combines hierarchical response modeling with adaptive querying. The displayed median is the decision before source-selected calibration.
At a 10% task budget, HSPR achieves the lowest MAE point estimates on Lite, Full, and Pro and the lowest MaxErr on all four SWE-bench task banks compared with the baselines in Table 1. On Verified, its MAE is within 0.055 pp of ATLAS-port. In the respective controls, removing group uncertainty increases macro MAE, as does replacing adaptive queries with the estimator held fixed. Across 5–20% task budgets, equal-bank macro MAE decreases as more outcomes are observed.
| Method | Lite (30/300) | Verified (50/500) | Full (229/2294) | Pro (73/730) | ||||||||
| MAE | MaxErr | MAE | MaxErr | MAE | MaxErr | MAE | MaxErr | |||||
| Uniform raw | 6.632 | 19.300 | 0.654 | 5.482 | 15.460 | 0.785 | 2.227 | 5.342 | 0.923 | 3.832 | 10.233 | 0.757 |
| Stratified | 4.468 | 13.354 | 0.723 | 3.302 | 10.768 | 0.843 | 1.636 | 3.877 | 0.938 | 3.111 | 7.508 | 0.837 |
| RSL ((Zhang et al., 2025)) | 4.911 | 16.064 | 0.683 | 3.654 | 14.590 | 0.823 | 2.277 | 8.547 | 0.930 | 4.467 | 12.822 | 0.714 |
| FAQ-port ((Wu et al., 2026)) | 6.681 | 26.263 | 0.638 | 4.951 | 25.989 | 0.765 | 1.807 | 5.049 | 0.921 | 5.433 | 15.330 | 0.673 |
| Mid-Range ((Ndzomga, 2026)) | 18.448 | 49.333 | 0.830 | 16.449 | 32.600 | 0.915 | 24.709 | 50.239 | 0.862 | 22.798 | 39.041 | 0.967 |
| Mid-Range + Ridge (control) | 4.717 | 19.223 | 0.725 | 3.653 | 18.734 | 0.844 | 1.532 | 10.394 | 0.890 | 3.434 | 10.980 | 0.868 |
| ATLAS-port ([13]) | 2.689 | 8.989 | 0.851 | 1.861 | 7.406 | 0.914 | <u>0.957</u> | <u>2.945</u> | 1.000 | 7.736 | 14.855 | 0.398 |
| PTA-IRT ([4]) | 2.724 | 11.703 | 0.816 | 2.177 | 8.310 | 0.887 | 2.280 | 6.174 | 0.956 | <u>2.731</u> | 11.729 | 0.890 |
| HSPR (ours) | 1.927 | 5.683 | 0.912 | <u>1.915</u> | 6.000 | <u>0.914</u> | 0.717 | 2.638 | <u>0.978</u> | 2.098 | 4.601 | <u>0.895</u> |
Table 1. Main results at a matched 10% task budget. MAE and MaxErr are in percentage points (pp); is unscaled. Metrics pool complete target cohorts before averaging query seeds. Best and second-best point estimates are bold and underlined, respectively, using unrounded values. § 4.1 describes the baselines and ports.
Our contributions are threefold:
We introduce HSPR, a hierarchical full-score model that combines source-fitted response factors with uncertain target success rates within source-signature groups.
We derive posterior-guided acquisition for score recovery, including a criterion equal to the expected one-step reduction in finite-score variance. Source-only validation selects the deployed acquisition objective before target evaluation.
We provide a controlled empirical study across four SWE-bench task banks, including end-to-end hierarchy controls, estimation and acquisition ablations, paired statistical analysis, mechanism diagnostics, and 5–20% budget sweeps.
Related Work
Efficient benchmark prediction. Active Testing combines informative queries with sampling correction [11]; Active Surrogate Estimators learn a surrogate for unobserved errors [10]. Random-Sampling-Learn (RSL) uses random-core regression; learned benchmark predictors can fail under extrapolation (Zhang et al., 2025). tinyBenchmarks uses compact calibrated subsets (Maia Polo et al., 2024), while SparseEval optimizes anchors and weights from the response matrix (Zhang et al., 2026). Factorized Active Querying (FAQ) combines historical factors with adaptive querying and finite-population inference (Wu et al., 2026). HSPR uses a model-conditional finite-score posterior with global factors and uncertain group rates.
Psychometric evaluation. Item-response models calibrate NLP test items [12]. Amortized model-based evaluation predicts item difficulty from question content to support adaptive testing (Truong et al., 2025). Agent Psychometrics models task-level agent responses [5]. ATLAS uses item-response models with Fisher-guided acquisition [13]; PTA-IRT uses historical trajectory information for task selection and reconstruction [4]. HSPR builds on historical item calibration but adds residual response dimensions and conditional target-specific group rates. ATLAS’s Fisher-guided acquisition targets latent ability. HSPR’s score-variance criterion targets the full-benchmark score, accounting for dependence among unobserved outcomes.
Subsets and objectives. Historical pass-rate selection can preserve software-agent rankings (Ndzomga, 2026); subset quality depends on task banks and representations [6]. SubLIME predicts subset-to-full rank correlation from anchor models and dataset characteristics (Saranathan et al., 2025). Ranking measures relative ordering (Liu et al., 2025), while score recovery requires an accurate numerical scale.
Agent evaluation context. HELM, BIG-bench, MMLU, and Dynabench cover heterogeneous evaluation settings [14, 7, 9] (Srivastava et al. 2023). AgentBench and AgentBoard evaluate tool use and multi-turn agents (Liu et al., 2024; Ma et al., 2024). Agent² RL-Bench evaluates agents on end-to-end reinforcement-learning post-training engineering [2]. EarlyEval reduces within-task execution through early outcome prediction (Shi et al., 2026). Trajectory-based selection [1] and process-aware adaptive testing (van der Linden, 2008) use additional observations. HSPR uses source binary responses and observed target outcomes.
Hierarchical Score Posterior Recovery
Figure 2 summarizes HSPR from historical response modeling to adaptive target queries and full-score recovery. Algorithm 1 specifies the evaluation procedure.
Figure 2. HSPR evaluation pipeline. Source responses provide fitted response factors and source-signature groups. Sequential queries update the posteriors over the target latent state and group success rates. Observed successes and posterior-predictive success counts for unqueried tasks define the full-score posterior used for the final calibrated estimate.
Finite-Score Estimation
Let be the source response matrix of agents on a fixed bank of tasks. The target outcome vector defines the full-benchmark score
The evaluator observes at most distinct target-task outcomes and returns an estimate of . Let be the queried task set and the observed history. In archived replay, is fixed and only queried entries are revealed. HSPR models the unobserved entries probabilistically. We condition on throughout; acquisition at step uses only and .
The score separates into observed and unobserved outcomes:
The first term is exact. HSPR places a joint predictive distribution on the second, preserving dependence among unobserved outcomes.
Historical Factors and Groups
Source response representation. We fit a regularized logistic item model on historical agents. For task , is an intercept and an ability slope. We obtain source-agent ability values by standardizing the logits of their smoothed full-benchmark scores. From source responses, we fit a prior over task success rates that mixes a point mass at zero with a Beta distribution. It supplies initial probabilities used to initialize and regularize the item intercepts.
After item fitting, a centered SVD of the source response residuals provides task directions. Scaled residual loadings describe departures from the scalar ability pattern. The source-fitted item parameters are fixed during target inference. The target latent response state combines scalar ability with residual coordinates :
Here is the logistic function and the diagonal prior precision matrix. The residual rank and prior precision are source-selected. Appendix § 8 specifies initialization, scaling, and the configuration grid.
Source response signatures. We define source-signature groups by exact equality of source responses: if for every source agent . Each group contains tasks. Tasks sharing a signature are observationally indistinguishable in the source matrix, so we average their source-fitted intercepts, slopes, and residual loadings. With collecting the averaged loadings, the group baseline success probability is
We model target outcomes within each group as exchangeable draws from a shared uncertain success rate. Unobserved outcomes within a group need not be identical.
Target-Specific Group Uncertainty
The group success rate has conditional prior mean :
Group rates are independent conditional on . Given these rates, outcomes for are independent Bernoulli variables with rate . Integrating couples tasks within a group even after conditioning on the global state: they share an unknown target-specific rate. For a fixed , smaller allows greater prior dispersion around this mean. A single source-selected applies to all groups. The probability-dependent concentration gives smaller prior variance when the baseline probability is near zero or one.
Let and count observed outcomes and successes in group . Conditional conjugacy gives
where and . The observed history also updates the target latent-state posterior:
Here is the Beta function; the ratio gives the likelihood of the observed binary sequence within each group. An observation updates the group-rate and latent-state posteriors, sharing information within and across groups, respectively.
With unqueried tasks in group , the unobserved success count has distribution
The finite score satisfies . Conditional on the group counts are independent; marginally they share uncertainty about . The predictive score distribution retains these dependencies through joint integration over the global state and group rates.
Posterior-Guided Acquisition
For a group with , let denote the outcome of a task sampled uniformly from . The score-variance acquisition rule selects the group with maximum
Proposition 1 (Score variance reduction) Under the HSPR predictive model, for any unobserved binary response with , Equation 10 equals the expected reduction in posterior variance of the full-benchmark score:
This greedy criterion includes resolving the queried outcome and updating predictions for other unqueried tasks. The variance identity follows from the law of total variance (Appendix § 9); it does not establish optimality for final absolute error.
The covariance shows how hierarchy enters task selection. Let and . Conditional on the global state,
The terms represent resolving the queried outcome and sharing evidence with the other group members. Larger unqueried groups amplify the latter contribution at a fixed rate posterior. Integrating over adds dependencies across groups. The latent-state acquisition alternative uses
It equals the expected reduction in the sum of posterior variances of the latent coordinates under the same model. Source-only validation chooses the acquisition objective before target replay: 13 of the 16 deployed folds select Score and three select Latent. Both use the same hierarchical posterior; the final absolute-loss decision is specified separately below.
After choosing a group, we sample uniformly from its unqueried tasks. Exhausted groups are excluded. The selected task is evaluated once, are updated, and the posterior is recomputed before the next query. Uniform within-group selection treats tasks with the same source signature symmetrically.
Predictive Score Decision
Under absolute score loss, a median of minimizes posterior expected loss:
We sample from its weighted approximation, then each conditional , and finally . Every score draw includes the observed successes exactly. The median of these complete score draws estimates .
For optional calibration, let and define
Source-only validation selects for the median decision. The factor limits correction to the unobserved fraction. This plug-in calibration uses the same observed outcomes as posterior fitting. The absolute-loss optimality in Equation (14) applies to the unadjusted predictive median. At full budget, .
Source-Only Configuration
Source-only leave-one-agent-out validation proceeds in stages. The global stage jointly evaluates rank, precision, acquisition, and calibration, retaining the selected rank and precision. With these fixed, the hierarchical stage selects group
Algorithm 1 HSPR evaluation of a target agent
Require: Source responses M, target execution oracle O, budget B
1: Select (r, P, κ, ρ, λ) by staged source-only validation, ρ ∈ {S, z}
2: Fit item parameters and source-signature groups on M
3: Initialize joint posterior; D₀, C₀ ← ∅
4: for t = 0, . . . , B − 1 do
5: g ← arg max_{h:U_h>0} G_h^ρ(D_t)
6: i ∼ Uniform(G_g \ C_t)
7: D_{t+1} ← D_t ∪ {(i, O(i))}
8: C_{t+1} ← C_t ∪ {i}
9: Update global and conditional group posteriors
10: end for
11: Construct p(S | D_B) from observed successes and predictive unobserved success counts
12: return Calibrated score estimate $\widehat{S}$ ({xref}`Eq. 16`)
uncertainty strength and acquisition jointly with posterior-mean calibration. The final stage selects median calibration on the resulting query paths. Each validation row is removed before fitting item parameters, residual factors, and groups. We refit on all source agents and freeze the configuration before target evaluation. Appendix § 8 gives the grids and fitting details.
We approximate the continuous posterior in Equation (8) by importance sampling with a Gaussian proposal centered at an auxiliary logistic-normal MAP estimate. The weights account for the target-model likelihood, prior, and proposal density. Predictive draws estimate the finite-score median; Appendix § 12 reports numerical validation.
Experiments
Experimental Setup
We evaluate held-out agent configurations on four archived task banks from the SWE-bench family. We use public PTA-IRT response matrices [4] for Lite, Verified, Full [8], and Pro [3]. The four task banks contain 300, 500, 2,294, and 730 tasks and 35, 70, 14, and 14 agent configurations, respectively. A configuration is an evaluated agent setup, represented by one row of the response matrix. Each benchmark uses four folds. Source and target configuration identifiers are disjoint within a fold; all configurations are targets once. This configuration-level split permits shared base models or related scaffolds between sources and targets.
Source folds contain 26–27 configurations on Lite, 52–53 on Verified, and 10–11 on Full and Pro. At the primary 10% checkpoint, budgets are : 30, 50, 229, and 73 distinct tasks. HSPR model settings, acquisition objectives, and calibration are frozen after source-only validation and before target replay. Each target has five HSPR query paths (seeds 0–4), each containing distinct tasks. Budgets apply separately to each method and query path. Predictions are recorded before full target outcomes are accessed for scoring. The historical response bank is treated as given, and counts distinct target-task outcomes revealed during replay. This budget represents task executions, without rerunning the archived agents.
We compare Uniform raw means, difficulty stratification, RSL (Zhang et al., 2025), FAQ-port (Wu et al., 2026), Mid-Range raw means (Ndzomga, 2026), Mid-Range with Ridge, ATLAS-port [13], and frozen PTA-IRT [4]. RSL applies source-fitted RidgeCV with penalties (.1, 1, 10) to uniformly sampled tasks, sharing its query sets with Uniform raw. Mid-Range selects the tasks whose source pass rates are closest to .5. Mid-Range with Ridge shares the Mid-Range raw query set: a source-fitted regression maps the selected outcomes to full scores. FAQ-port uses without-replacement queries and caps factor rank for the available source cohort. ATLAS-port retains Fisher-guided selection and p-IRT estimation, using a source-fold 4PL item bank. PTA-IRT uses frozen released predictions on the same configurations. Appendix § 10 details these adaptations.
Table 1 compares complete procedures under matched target-task budgets, allowing their query policies and estimators to differ. Table 2 examines group uncertainty, adaptive querying, and the predictive decision under controlled settings.
| Ablation / control | Lite | Verified | Full | Pro | Macro |
| HSPR (ours) | 1.927 | 1.915 | 0.717 | 2.098 | 1.664 |
| A. Hierarchical modeling: source-selected acquisition objective | |||||
| w/o group uncertainty | 1.895 | 1.970 | 1.331 | 2.034 | 1.808 |
| B. Query adaptivity: frozen HSPR reconstruction | |||||
| w/o adaptive queries (Uniform) | 4.573 | 3.288 | 1.413 | 3.066 | 3.085 |
| w/o adaptive queries (Mid-Range) | 3.961 | 3.059 | 2.741 | 1.975 | 2.934 |
| C. Predictive decision: identical HSPR query paths | |||||
| w/o median (posterior mean) | 1.941 | 1.927 | 0.722 | 2.108 | 1.674 |
Table 2. Ablations at a 10% budget; MAE in pp, with equal-bank Macro. A retains each fold’s acquisition objective, global configuration, and median calibration; only group uncertainty changes, and both variants generate their own query paths. B freezes HSPR reconstruction and calibration, using five Uniform seeds or one deterministic Mid-Range path. C shares HSPR query paths, with decision-specific calibration selected on sources.
Metrics. For each task bank, let denote the set of held-out agent configurations pooled across all four folds. For each method and query seed, we compute
Kendall’s uses all cohort pairs and accounts for ties. We then average the pooled metrics over query seeds. Sampling/RSL baselines have ten saved seeds, ATLAS-port and HSPR five, and deterministic or frozen methods one prediction per configuration. MAE and MaxErr are reported in pp throughout.
Appendix § 14 reports HSPR evaluator overhead.
Main Results
At a 10% task budget, HSPR achieves the lowest MAE point estimates on Lite, Full, and Pro and remains within 0.055 pp of ATLAS-port on Verified (Table 1). It has the lowest seed-averaged MaxErr on all four banks and ranks among the top two in all 12 benchmark–metric combinations, leading eight. Equal-bank macro MAE is 1.664 pp, compared with 2.478 pp for PTA-IRT, the strongest baseline on this aggregate.
On Pro, raw Mid-Range preserves ordering () but has MAE 22.798 pp versus 2.098 pp for HSPR. This difference motivates reporting absolute score error alongside ranking fidelity.
Holm-corrected Wilcoxon tests on paired, seed-averaged absolute errors favor HSPR over RSL on all four banks, over Stratified on three, and over ATLAS-port on Pro. Appendix § 11 reports the complete bootstrap intervals and corrected tests.
Ablation Study
Hierarchical modeling. Hierarchical uncertainty affects the evaluation procedure beyond its final reconstruction step. Table 2A holds the source-selected acquisition objective fixed within each fold and retains the adopted global configuration and median calibration. The two variants generate their own query paths, so removing group uncertainty changes both the evidence collected and its interpretation. Removing hierarchy increases equal-bank macro MAE from 1.664 to 1.808 pp; the paired configuration bootstrap interval for the 0.143 pp difference is [−0.012, 0.302]. The largest gain occurs on Full, where hierarchy reduces MAE from 1.331 to 0.717 pp. Appendix § 12 reports the execution protocol and a separate reconstruction control holding query paths fixed.
Query adaptivity. Informative query allocation provides a large aggregate gain with the reconstructor held fixed. Table 2B freezes the HSPR model, predictive median, and calibration and replaces adaptive queries with Uniform or fixed Mid-Range queries. Macro MAE rises from 1.664 to 3.085 and 2.934 pp, respectively. HSPR updates acquisition scores after each observed outcome, allowing subsequent queries to reflect the current target posterior.
Predictive decision. Table 2C uses identical HSPR query paths and replaces the median with the posterior mean, with calibration selected on sources for each decision family. The median-based estimate has slightly lower macro MAE (1.664 versus 1.674 pp).
Mechanism Analysis
Source-signature groups support local evidence sharing. Full’s 2,294 tasks form approximately 276 source-signature groups on average across targets, giving (Table 3). An observed outcome updates the posterior over the shared group success rate, informing predictions for unqueried group members. Verified retains 426 groups among 500 tasks, leaving less within-group sharing to exploit. This structural contrast is consistent with the larger end-to-end hierarchical gain on Full. Larger groups allow more within-group pooling; its usefulness also depends on how the historical structure transfers to the target.
| Bank | Tasks | Groups | Source-zero tasks | ||
| Share (%) | Target success (%) | ||||
| Lite | 300 | 201.5 | 1.49 | 24.6 | 0.87 |
| Verified | 500 | 426.0 | 1.17 | 8.3 | 0.74 |
| Full | 2294 | 275.9 | 8.32 | 41.3 | 1.84 |
| Pro | 730 | 222.0 | 3.29 | 36.7 | 2.07 |
Table 3. Historical response structure. averages source group counts over targets; divides the task count by this average. Source-zero tasks have no successes among the source agents. Target success rates within these sets are averaged over targets, using complete outcomes after prediction.
Source-zero tasks are those with no successes among the agents in a given source fold. On Full and Pro, 41.3% and 36.7% of tasks have all-zero source responses, yet target success rates within these sets are 1.84% and 2.07%, respectively.
These post-hoc target rates use complete outcomes after estimation. The tasks share a source signature within a fold while admitting different target outcomes. An uncertain target-specific rate represents this departure and lets observed evidence revise predictions for remaining group members.
Hierarchy also changes where evidence is acquired. In the controlled Full comparison, source-zero tasks receive 13.2% of queries with hierarchy, compared with 0.19% without it.
Budget Sensitivity
Across 5–20% task budgets, HSPR’s equal-bank macro MAE decreases from 2.562 to 1.262 pp (Figure 3). Its 10% source-selected settings remain fixed at all checkpoints, with nested distinct-task paths and five query seeds per target. Baselines retain their saved budget-specific settings.
Figure 3. Budget sensitivity across four SWE-bench task banks at 5–20% task budgets. HSPR uses fixed 10% source-selected settings and nested query paths; baselines retain budget-specific settings. MAE and MaxErr are in pp; metrics pool target configurations before averaging query seeds.
Discussion
Conditional on the target latent state, the second term in Equation (12) scales with the number of other unqueried group members and the posterior variance of their common rate. At a fixed task-bank size and rate posterior, an observation can inform more remaining outcomes in a larger group. Full’s larger source-signature groups (Table 3) permit more within-group sharing than Verified’s, consistent with its larger hierarchical gain (Table 2). As queries reveal outcomes, the remaining group sizes and rate posteriors change, altering this contribution to Score acquisition.
Conclusion
HSPR combines source-fitted response factors, uncertain group success rates, and posterior-guided acquisition to estimate full-benchmark scores from limited observations. Its final estimate uses a posterior median with source-selected calibration. At a 10% task budget, HSPR achieves the lowest MAE point estimates on three of four SWE-bench task banks and the lowest MaxErr on all four among the compared methods. Controlled experiments show substantial aggregate gains from adaptive querying. The contribution of group uncertainty varies across task banks, with the largest observed improvement on SWE-bench Full. These results highlight the role of historical response structure in task-efficient score recovery.
Limitations
We evaluate held-out agent configurations on four task banks from the SWE-bench family. Generalization under substantial shifts in model families or agent scaffolds remains to be evaluated. HSPR’s response factors and source-signature groups are derived from the historical source bank and therefore depend on its coverage. Posterior uncertainty is conditional on the assumed response model, including within-group exchangeability.
We measure evaluation budgets by the number of distinct target-task executions and report evaluator inference overhead in Appendix § 14. Token, wall-clock, and monetary savings depend on per-task execution costs and cannot be inferred directly from task-count reduction. Historical data collection, source fitting, and configuration selection incur additional costs.
Ethical Considerations
This study uses existing software-agent response matrices and does not involve human participants or private user data. Treating recovered scores as fully observed results may mislead model comparisons, particularly when historical source data poorly represent a target configuration. We therefore recommend reporting the evaluation budget and clearly distinguishing recovered scores from full-benchmark measurements.
Implementation and Source-Only Configuration
Item initialization. For task let . Source-only marginal likelihood fits a mixture to historical task rates. For , the initial rate is . For a zero column it is multiplied by the posterior probability of the continuous component. Rates are clipped to before taking their logits. The zero mass is estimated continuously from source responses.
A source ability is the logit of its Jeffreys-smoothed full score, standardized using the source mean and a standard deviation floored at .25. Sixteen regularized Newton updates fit task intercepts and positive slopes. Intercepts have unit quadratic regularization around their initial logits; slopes have precision two around one. Updates are clipped to .5 in magnitude, intercepts to , and slopes to . These numerical settings are fixed for source fitting.
Residual factors. Let be the fitted source probability, and let denote centered across sources. Define the mean fitted Bernoulli variance as . For , residual loading column is
The item model and residual factors are refitted whenever a validation row is removed. At most eight directions are computed; the residual-rank grid is with precision , plus scalar with precision . For , the ability precision is and residual coordinates have precision . For , ability precision is . Source-signature groups average the intercepts, slopes, and residual directions, avoiding numerical differences among source-identical items.
Source-only configuration grid. HSPR uses staged leave-one-source-agent-out validation over finite configuration grids. The global grid combines the eleven rank/precision pairs above with logistic score-variance or latent-state acquisition and calibration (44 settings). This stage retains the selected rank and precision. With these held fixed, the hierarchical grid uses , Score or Latent acquisition, and posterior-mean calibration . The selected group uncertainty strength and acquisition rule are then held fixed while median calibration is chosen on the resulting query paths.
The global stage uses deterministic query paths and averages errors over held-out source agents. The hierarchical and median stages also average over query seeds 0, 1, and 2. Ties in source MAE use the largest per-agent validation error, averaged across seeds when applicable, then fixed grid order. Each held source is excluded before refitting the response model and task groups. We refit the model on all source agents using the selected configuration and freeze it before target replay. Table 4 lists the deployed settings for every fold.
| Bank | Fold | Policy | ||||
| Lite | 0 | 1 | 4 | 1 | Score | 0.5 |
| Lite | 1 | 4 | 4 | 4 | Score | 0.5 |
| Lite | 2 | 4 | 8 | 0.25 | Score | 0.5 |
| Lite | 3 | 1 | 8 | 4 | Latent | 0 |
| Verified | 0 | 4 | 1 | 1 | Score | 0 |
| Verified | 1 | 1 | 8 | 1 | Score | 0.5 |
| Verified | 2 | 1 | 8 | 4 | Latent | 0.5 |
| Verified | 3 | 2 | 4 | 1 | Score | 0.5 |
| Full | 0 | 0 | 0.25 | 1 | Score | 0 |
| Full | 1 | 2 | 4 | 1 | Score | 0 |
| Full | 2 | 4 | 4 | 1 | Score | 0.5 |
| Full | 3 | 4 | 8 | 1 | Score | 0 |
| Pro | 0 | 4 | 1 | 4 | Score | 0 |
| Pro | 1 | 2 | 8 | 16 | Latent | 0 |
| Pro | 2 | 2 | 4 | 4 | Score | 0 |
| Pro | 3 | 4 | 4 | 16 | Score | 0.5 |
Table 4. Frozen source-selected settings. For , the scalar ability precision is .25 and residual precision is ; for , ability precision is . Score and Latent denote the acquisition rules in § 3.4.
Posterior proposal and updates. An auxiliary logistic-normal group MAP supplies a Gaussian proposal for . Its local offsets have quadratic precision and are solved alongside the global coordinates using a Schur complement. The proposal is refreshed after each eight observed outcomes. Between refreshes, conjugate updates reweight the existing nodes and update observed counts exactly within the weighted-node model.
If node is drawn from proposal density , its normalized weight satisfies
Gaussian normalization constants common to the nodes cancel. The implementation computes Beta-integrated ordered likelihoods with rising-factorial terms to avoid cancellation when concentrations are large. Probabilities used by that likelihood are clipped to .
The 8,192 nodes use a scrambled Sobol sequence mapped through a standard normal inverse CDF, with simulation seed 20261005. The predictive median uses 131,072 complete finite-score draws. Predictive decision sampling uses the same fixed numerical seed but a separate generator from the query RNG. Separate generators keep predictive sampling independent of within-group query randomization. Query seeds 0–4 supply the five target paths.
Derivation of the Acquisition Criterion
Let and . Conditional on ,
Integrating the conditional variance and adding the variance of the conditional score mean gives the predictive finite-score variance. Both the Bernoulli outcome variance and the shared rate uncertainty remain in this calculation.
For a particular unqueried item in group , the conditional covariance with is
Marginalizing over takes the posterior expectation of this conditional covariance and adds the covariance between the conditional score mean and , with held fixed. This additional term captures global transfer between groups. Omitting the direct item term would instead optimize information about other unqueried tasks.
For any random variable and binary query outcome with , let . Then
The law of total variance therefore yields
Taking proves Proposition 1; summing coordinatewise with gives Equation (13). This identity is exact for the assumed distribution, including a finite weighted-node approximation.
Evaluation Protocol and Reproducibility
Algorithm 1 operates on a fixed task bank and reveals target outcomes through its execution oracle. During source validation, the entire held-out row is removed before fitting initialization rates, item parameters, residual factors, and source signatures. The model is then refitted on all source rows using the selected configuration. All model, query-policy, and calibration choices are frozen before target replay.
At each step, acquisition uses sources and observed history only. Exhausted groups are excluded, and one remaining task is sampled uniformly within the selected group before revealing its binary outcome. Every path contains distinct executions. Each method and query path has its own -execution budget. Predictive draws use a numerical generator separate from query selection.
The score estimate and queried task identities are recorded before full target outcomes are accessed. Full truth then supplies MAE, ranking, and maximum error. The complete target folds are pooled within each query seed before metrics are averaged, as defined in § 4.1. This ordering separates inference information from scoring information. Historical bank collection and source fitting are additional costs outside .
Adapted baseline protocols. FAQ-port uses the without-replacement variant and caps its logistic factor rank at the source cohort size minus one. It retains posterior factor updates and the PAI estimator; this variant does not inherit the with-replacement coverage theorem. ATLAS-port retains sequential expected-a-posteriori ability estimation, maximum-Fisher-information selection with top-five randomization, and p-IRT score estimation. It uses the source-fold 4PL bank in place of the original R/mirt 3PL fitting and applies fixed distinct-task budgets. Thus these rows report adapted procedures on the available banks. RSL uses the Uniform query set; Mid-Range raw and Mid-Range with Ridge share their source-selected task set. Neither shares HSPR’s adaptive query paths. PTA-IRT retains its released frozen predictions, including its use of historical trajectory information. The matched target-task budget does not equalize historical information or source-fitting costs.
Complete Paired Statistical Comparisons
The contrasts in Table 5 all use the same held-out configuration keys within a benchmark. Baseline seed counts differ, so seed-averaged errors supply one paired observation per configuration. The bootstrap measures variability over the archived configuration sample, conditional on the observed task bank. Wilcoxon tests use the same global 16-contrast Holm correction as § 4.
| Benchmark | Comparator | MAE (pp) | Paired 95% CI (pp) | Holm |
| Lite | ATLAS-port | -0.762 | 0.081 | |
| Lite | PTA-IRT | -0.797 | 1.000 | |
| Lite | Stratified | -2.541 | < .001 | |
| Lite | RSL | -2.984 | < .001 | |
| Verified | ATLAS-port | +0.055 | 1.000 | |
| Verified | PTA-IRT | -0.262 | 1.000 | |
| Verified | Stratified | -1.387 | < .001 | |
| Verified | RSL | -1.738 | < .001 | |
| Full | ATLAS-port | -0.240 | 1.000 | |
| Full | PTA-IRT | -1.562 | 0.116 | |
| Full | Stratified | -0.919 | 0.001 | |
| Full | RSL | -1.559 | 0.017 | |
| Pro | ATLAS-port | -5.638 | 0.003 | |
| Pro | PTA-IRT | -0.633 | 1.000 | |
| Pro | Stratified | -1.012 | 0.122 | |
| Pro | RSL | -2.369 | 0.036 |
Table 5. Paired MAE differences: HSPR minus comparator, so negative favors HSPR. Seed-averaged agent errors are paired. The 95% percentile bootstrap intervals use 10,000 configuration resamples, conditional on the archived task bank, and are unadjusted. Wilcoxon -values are Holm-adjusted across all 16 comparisons.
Additional Controlled Analyses and Numerical Validation
End-to-end hierarchy controls. Table 2A retains each fold’s source-selected acquisition objective (13 Score and three Latent folds), global rank and precision, and adopted median calibration. Removing hierarchy sets the group rate to its global response probability, eliminating the conditional Beta variability. Each control updates its own posterior and selects its own distinct-task path. Both retain all 133 target configurations and five query seeds. The equal-bank macro difference uses 10,000 paired configuration bootstrap samples within each bank, after averaging seeds. Removing hierarchy increases macro MAE from 1.664 to 1.808 pp; the 0.143 pp difference has a conditional 95% interval of [−0.012, 0.302] pp, which includes zero. Individual-bank Wilcoxon tests do not remain significant after correcting the four hierarchy comparisons with Holm’s method. The macro interval summarizes aggregate variation across the archived configuration cohorts.
Predictive precision. Source validation checks that predictive sample means are within six Monte Carlo standard errors of analytic posterior means. For the median’s numerical interval, binomial order-statistic quantiles identify the central 95% indices among 131,072 sorted score draws. Its width is checked against one finite task increment . This interval measures Monte Carlo precision of the posterior median.
Integration resolution. Source-only checks compare 8,192 and 32,768 nodes on common query histories, with a score tolerance of .0005 on the 0–1 scale (.05 pp). The Verified draw-resolution check compares 65,536 and 131,072 draws using the same frozen query policy and factor convention. These comparisons assess numerical accuracy at the reported operating settings.
Matched-path reconstruction. The reported same-path no-group control retains the same global configuration and selected tasks while integrating the global posterior without a Beta rate residual. It is evaluated as a posterior mean. Comparing it with the hierarchical mean separates the effect of conditional group uncertainty on these paths. Table 2C compares the posterior mean with the median on identical query paths, with calibration selected separately for each decision family. Table 6 reports the complete matched-path comparison. Its effect varies across banks: the largest improvement is on Full, while Pro slightly favors the no-group mean.
| Bank | MAE w/o groups | MAE with groups | Group gain | |||
| Lite | 300 | 201.5 | 1.49 | 1.949 | 1.941 | +0.009 |
| Verified | 500 | 426.0 | 1.17 | 1.928 | 1.927 | +0.001 |
| Full | 2294 | 275.9 | 8.32 | 0.902 | 0.722 | +0.180 |
| Pro | 730 | 222.0 | 3.29 | 2.101 | 2.108 | -0.006 |
Table 6. Historical group structure and matched-path reconstruction at 10% budgets. Both MAE columns use posterior means with identical query paths and calibration. is the target-averaged source group count. Group gain is MAE without minus with group uncertainty, computed before rounding (pp). Bold marks the largest and gain.
Acquisition controls. Table 2B retains all HSPR model settings and median calibration. Uniform reuses saved baseline permutations for seeds 11, 23, 37, 53, and 71; Mid-Range sorts source rates by distance to .5 with stable task-order ties. All 133 targets yield 665 Uniform and 133 deterministic Mid-Range predictions.
Thirty-two source-only checks compare integration resolutions on identical paths, excluding held sources before fitting and selection. Predictive sampling for the static acquisition controls uses a pre-defined numerical-precision rule: start at 131,072 draws and double, up to 2,097,152, until sample means agree within six Monte Carlo standard errors and median numerical intervals span at most . This rule uses numerical error estimates before full target scoring. All adaptive references meet the criteria; 797 static predictions use 131,072 draws and one uses 524,288. Recorded static paths reproduce their baseline selectors.
The budget sweep preserves the primary 10% predictions and checks all source checkpoints at the same numerical tolerances.
Historical Response Structure and Transfer Diagnostics
Table 7 gives the complete per-bank support statistics underlying § 4.4. Entries average configurations equally, using each target’s frozen source fold. Target success is measured within that target’s source-zero tasks; query coverage is the fraction of its queried tasks drawn from this set, averaged over five seeds. Query coverage reflects the deployed source-selected acquisition policy.
| Bank | Source groups | Source-zero tasks (%) | Target success on zeros (%) | Queries to zeros (%) |
| Lite | 201.5 | 24.6 | 0.87 | 0.248 |
| Verified | 426.0 | 8.3 | 0.74 | 0.000 |
| Full | 275.9 | 41.3 | 1.84 | 13.225 |
| Pro | 222.0 | 36.7 | 2.07 | 0.020 |
Table 7. Post-hoc support diagnostic at 10% budgets. Each target uses its frozen source fold; entries average targets equally, with queries averaged over five seeds. Source-zero tasks have zero successes among all source agents. Target success rates are computed after prediction for this diagnostic.
Computational Overhead
We profiled HSPR at the 10% task budget over all 133 target configurations and five query seeds (665 runs). Experiments used an AMD EPYC 7352 server with 96 logical CPUs and approximately 566 GiB of system memory, running Python 3.11.3, NumPy 1.26.4, and SciPy 1.17.1. We ran 24 evaluations concurrently, each in a separate process with one CPU thread, without a GPU. Timing includes posterior initialization, sequential querying and updates, and the final predictive decision and calibration. It excludes data loading, source fitting and configuration selection, and software-agent task execution.
Table 8 reports per-run wall times on a shared server. The complete batch took 510.1 seconds with concurrent evaluation; the timed inference intervals consumed 3.31 CPU hours in total. All replayed query paths matched the archived primary results, and the largest absolute prediction difference was 2.22 × on the unit score scale. These measurements quantify deployment inference overhead, not the total computational cost of source-only configuration selection or end-to-end agent evaluation savings.
| Task bank | Runs | Median (s) | P90 (s) |
| Lite | 175 | 6.19 | 8.57 |
| Verified | 350 | 20.31 | 32.35 |
| Full | 70 | 30.96 | 61.67 |
| Pro | 70 | 7.79 | 14.12 |
Table 8. HSPR evaluator wall time per target and query seed at the 10% task budget. P90 is the 90th percentile across runs within each bank.
Score Recovery Across Model Families
Figure 4 compares full-benchmark scores with HSPR estimates at the 10% task budget. Each point represents one target configuration, with model families distinguished by color and marker. The diagonal shows exact agreement, making overestimation and underestimation visible across the score range.
Figure 4. Full-benchmark scores and HSPR estimates by model family at the 10% task budget. Points show mean predictions over five query seeds; bars show one sample standard deviation across those seeds. The dashed line marks exact agreement, and the shaded reference band marks pp. Panel MAE averages absolute errors over configurations and seeds. Family labels follow archived configuration names; Mixed denotes multiple named families and Unspecified denotes no identified family.
The relative errors of the represented families vary across task banks. On Lite, GPT/o-series configurations have lower mean error than Claude configurations (1.71 versus 2.15 pp), while the ordering reverses on Verified (1.70 versus 1.52 pp). Full shows consistently small errors across its configurations: the GPT/o-series, Claude, and unspecified groups have mean errors of 0.39, 0.80, and 0.74 pp, respectively, and every configuration’s mean absolute error is below 1.71 pp. The individual points also reveal variation among configurations within a family that is hidden by family averages.
References
- [1]Mahmoud Ayyad, Zehao Wang, Jiho Shin, Ying Zou, and Bram Adams. 2026. Trajectory-aware benchmark subset selection for cost-efficient software engineering agent regression testing. arXiv preprint arXiv:2609.24928.arxiv.org/abs/2609.24928
- [2]Wanyi Chen, Xiao Yang, Xu Yang, Tianming Sha, Qizheng Li, Zhuo Wang, Bowen Xian, Fang Kong, Weiqing Liu, and Jiang Bian. 2026. Agent2 RL-Bench: Can LLM agents engineer agentic RL post-training? arXiv preprint arXiv:2604.10547.arxiv.org/abs/2604.10547
- [3]Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa R Kundurthy, Sean M. Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. 2026. SWE-bench pro: Can AI agents solve long-horizon software engineering tasks? In Proceedings of the 43rd International Conference on Machine Learning, pages 23917–23934.arxiv.org/abs/2509.16941
- [4]Kefeng Duan, Dewu Zheng, Yanlin Wang, Xiwen Wang, Ensheng Shi, Xilin Liu, Yuchi Ma, Jiachi Chen, Mingwei Liu, and Zibin Zheng. 2026. Efficient SWE agent benchmarking via trajectory-aware evaluation. arXiv preprint arXiv:2609.01603.arxiv.org/abs/2609.01603
- [5]Chris Ge, Daria Kryosheieva, Daniel Fried, Uzay Girit, and Kaivalya Hariharan. 2026. Agent psychometrics: Task-level performance prediction in agentic coding benchmarks. In Proceedings of the Conference on Language Modeling (COLM).arxiv.org/abs/2604.00594
- [6]Rostislav Gusev and Alexey Zaytsev. 2026. Benchmarking on tasks that matter: Dataset selection for preserving model rankings. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, pages 8954–8965.DOI
- [7]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.
- [8]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations.arxiv.org/abs/2310.06770
- [9]Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, et al. 2021. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
- [10]Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Thomas Rainforth. 2022. Active surrogate estimators: An active learning approach to label-efficient model evaluation. In Advances in Neural Information Processing Systems, volume 35.arxiv.org/abs/2202.06881
- [11]Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. 2021. Active testing: Sample-efficient model evaluation. In Proceedings of the 38th International Conference on Machine Learning, pages 5753–5763.arxiv.org/abs/2103.05331
- [12]John P. Lalor, Hao Wu, and Hong Yu. 2016. Building an evaluation scale using item response theory. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 648–657.arxiv.org/abs/1605.08889
- [13]Peiyu Li, Xiuxiu Tang, Si Chen, Ying Cheng, Ronald Metoyer, Ting Hua, and Nitesh V. Chawla. 2026. Adaptive testing for LLM evaluation: A psychometric alternative to static benchmarks. In Proceedings of the 43rd International Conference on Machine Learning, pages 70246–70269.arxiv.org/abs/2511.04689
- [14]Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, et al. 2023. Holistic evaluation of language models. Transactions on Machine Learning Research.



