WorksComputer Science
Continue or Restart? The Limits of Early Failure Prediction in Coding Agents
Abstract
Early stopping promises to reduce compute costs for autonomous coding agents by terminating unpromising runs early. Deciding when to abandon a run requires separating baseline issue difficulty from the trajectory of an individual attempt. We study the distinction in 147,002 execution traces from two public corpora spanning four model-framework settings. Comparing pre-execution issue context against early execution prefixes, we evaluate linear count models, initial-turn text embeddings, task-conditional rankers, and a fine-tuned Qwen3-0.6B monitor. Although early prefixes modestly improve cross-task outcome ranking, within-task discrimination remains near chance across early landmarks (5k-20k tokens): initial-turn embeddings and 20k-token task-conditional models yield AUCs around 0.50, and three-seed neural evaluations across 32,406 attempts at 20k tokens achieve a mean within-task AUC of 0.50 [0.48, 0.51] across tasks of varying solve rates. To evaluate how these predictions translate into stopping decisions, we replay score-guided and matched random restarts under shared token budgets alongside uninterrupted continuation. Under a 75k-token cap, learned restarting improves resolution over matched random restarting by up to 1.98 percentage points. An exact algebraic decomposition resolves this paradox: the gain over random restarting stems almost entirely from between-task stopping allocation (avoiding stops on easy issues), whereas attempt-level selection within a task contributes near zero (-0.11 to +0.24 percentage points). Across all four settings at the 75k cap (and 216 of 224 tested budget-threshold configurations overall), uninterrupted continuation achieves higher solve rates at lower mean token cost than both restart policies. Practical evaluations of early stopping must therefore establish attempt-level discrimination and demonstrate net compute savings relative to continuation.
Introduction
Consider a coding agent that has consumed 20,000 (20k) transcript tokens investigating a repository issue. By this point, the agent has inspected repository files and invoked initial test or edit commands, yet whether its trajectory will ultimately resolve the issue remains unknown. Suppose a failure monitor assigns the active attempt a low probability of success. Under a 75,000 (75k) token budget, aborting the run to start a fresh attempt leaves only 55k tokens for the replacement and discards all prior repository exploration. Whenever an aborted run would have succeeded within the full budget, or whenever its replacement exhausts the residual 55k budget before finishing, restarting degrades overall resolution. Evaluating early stopping therefore requires determining whether early trajectory signals justify abandoning an active attempt.
Autonomous coding agents pair language models with execution scaffolds to navigate codebases, modify source files, and run verification tests [1, 2, 3]. Throughout this study, a task denotes a repository issue, and a run (or attempt) denotes a single execution trajectory on that task. At any decision landmark, the prefix comprises the interaction sequence of assistant turns and tool observations recorded up to that transcript-token count, prior to final patch evaluation.
Evaluating early failure predictors presents two methodological challenges. First, baseline task solvability confounds attempt-level progress. If an agent consistently resolves one issue and fails another, a monitor that estimates static task difficulty will achieve strong corpus-wide ranking even if it assigns identical scores to every attempt on the same issue. We therefore separate pooled prediction, which ranks outcomes across all eligible runs, from within-task discrimination, which compares successful and failed attempts on the exact same task using area under the ROC curve (AUC, where 0.50 indicates random ranking).
Second, predictive ranking does not directly establish the resource efficiency of a stopping policy. Mid-execution interventions can follow two distinct regimes: abort-only termination, which abandons low-scoring runs without replacement to save tokens at the expense of forfeiting any late resolutions, and restart-based stopping, which terminates a stalled attempt to reallocate the remaining budget to a fresh attempt on the same issue. If an early monitor captures only static issue difficulty (AUC$_{\mathrm{within}} \approx 0.50), abort-only termination at 20k tokens merely executes a delayed task filter after burning 20k tokens (which is strictly dominated by pre-execution filtering at zero token cost), so we focus our policy evaluation on whether trajectory-based stopping improves resolution via cold restarts. Aborting an active run to restart improves outcomes only when a replacement attempt is more likely to solve the task within the residual budget than the original run is via continuation. Stopping rules must therefore be evaluated against both matched random restarting (aborting the same fraction of active runs without using predictions) and uninterrupted continuation under a shared token cap.
While prior work proposes monitors based on token probabilities, behavioral states, internal activations, and state-preserving recovery [4, 5, 6, 7], recent studies show that task identity confounds pooled trajectory metrics [8, 9] and that continuation often outperforms cold restarts in mathematical reasoning [10]. However, prior work has not tested whether task-conditional training or end-to-end neural monitors recover attempt-level signal, nor explained why score-guided policies still beat random restarts in replay. We mine 147, 002 released execution traces from Nebius SWE-rebench and NVIDIA Open-SWE-Traces [11, 12] across four model-framework combinations (settings). Across landmarks from 5k to 60k tokens (focusing on 20k, where nearly all runs remain active), we compare pre-execution issue features against early prefixes using action-count regressions, frozen text embeddings, task-conditional rankers, and a fine-tuned Qwen3-0.6B monitor adapted from prior work [7]. Using recorded-attempt replay under shared token caps, we answer three research questions:
RQ1. How much does observing an early execution prefix improve outcome prediction beyond pre-execution task context?
RQ2. Do early execution prefixes distinguish successful from failed attempts on the same task?
RQ3. What drives the performance difference between learned and random restarting, and how do both policies compare with uninterrupted continuation under a shared token budget?
Across all four settings, we make three main contributions. First, we show via unseen-repository cross-validation, permutation controls, and equivalence testing (TOST) that while static context supports cross-task ranking, early prefixes yield near-chance within-task discrimination () across 5k embeddings as well as 20k count models, task-conditional rankers, and the three-seed neural monitor, even though 87.51% of primary-corpus runs have already invoked tests and 32.80% have edited code by 20k tokens. Second, in budget-constrained replay under a 75k cap, we derive an exact decomposition proving that learned stopping’s gain over matched random restarting (up to 1.98 percentage points) stems from between-task stopping allocation rather than within-task attempt selection. Third, we show across 224 budget-threshold configurations that uninterrupted continuation achieves higher solve rates at lower mean token cost than both restart policies across the 75k and 100k frontiers (111 of 112 configurations, and 216 of 224 overall).
Related Work
a) Mining coding-agent execution traces: Benchmarks such as SWE-bench evaluate autonomous agents on real-world repository issues [1], using execution frameworks like SWE-agent [2] and OpenHands [3] to orchestrate repository exploration and patch verification. Large-scale public corpora now capture complete execution traces from these workflows [11, 12]. These datasets extend classical mining of developer interaction histories [13] to autonomous software engineering, recording fine-grained sequences of tool invocations and environment observations. Recent empirical studies mining these traces show that issue-level properties strongly shape trajectory metrics. Mentiyev and Assunção demonstrate that the apparent relationship between trajectory length and failure shifts substantially once task identity is controlled [8]. Complementary analyses examine how failure cascades develop during execution [14] and whether behavioral features carry consistent predictive weights across issues and repeated trials [15]. These findings motivate mining multi-attempt corpora to test whether early trajectory observations capture attempt-specific progress beyond static issue difficulty.
b) Early failure prediction in agent runs: Prior work proposes diverse signals to anticipate unsuccessful agent runs before completion. Trajectory-based approaches mine historical execution experience to terminate unpromising patch generation [16] or model agent behavior through state transitions and late-emerging failure indicators [5, 17]. A second line of work monitors execution sequentially by estimating residual resource requirements [18] or applying sequential statistical testing to intermediate verifier scores [19]. A third family extracts signals directly from the underlying language model, using generated token probabilities [4] or probing internal representations to predict edit outcomes and likely failures [6, 20]. Whereas existing monitors are primarily evaluated on pooled benchmark accuracy or total token reduction, we evaluate whether early prefixes rank competing attempts on the exact same software issue.
c) Restart strategies and state recovery: Restart policies originate in randomized algorithms, where Luby et al. established optimal and universal restart schedules for heavy-tailed runtime distributions based on elapsed execution time [21]. In autonomous coding workflows, mid-episode interventions can either start fresh or transfer accumulated context. Scouting uses an initial repository exploration phase to route the issue and its gathered context to a downstream solver [22]. FailFast–RestartSmart pairs a 0.6B failure monitor, trained on terminal outcomes and intermediate test supervision, with a recovery mechanism that retains code edits from aborted runs [7]. Across four coding-agent policies, their system reports 14.6–20.4% execution-token savings at a 5% false-positive target. Notably, at a 25% false-positive target with Qwen3.6-27B, transferring saved edits raises resolution from 66.6% to 71.8%, whereas restarting cold reaches only 66.8%. That sharp gap between state-preserving recovery and cold restarting highlights the cost of discarding prior exploration. Building on this insight, we adapt their 0.6B terminal-outcome monitor to evaluate cold replacement attempts under shared token budgets and isolate how sunk exploration tokens affect the restart tradeoff.
d) Disentangling task difficulty from stopping policies: Two concurrent studies examine the gap between problem difficulty and attempt-level intervention. Investigating the Nebius OpenHands corpus [11] alongside additional benchmarks, EsfandyariDoulabi et al. show that high pooled AUROC largely reflects cross-task difficulty even when within-task discrimination remains near chance, concluding that cross-task budget allocation is more effective than abort-only stopping in this regime [9]. In mathematical reasoning, Bulut similarly finds that early prefix predictions offer minimal within-problem discrimination and that continuing an active trace outperforms cold restarts under matched token budgets [10].
Our work extends this emerging perspective across four coding-agent settings by connecting within-task evaluation directly to the mechanics of restart policies. First, we evaluate whether task-conditional training or a fine-tuned neural monitor recovers attempt-level signal that count features and frozen embeddings miss, including across tasks stratified by empirical solvability. Second, we explain the apparent paradox in which a learned stopping rule beats a matched random restart baseline despite near-chance within-task ranking. By decomposing replay outcomes into between-task and within-task selection and accounting for both replacement credit and forfeited original successes, we show how task selection drives the gain over random restarting while uninterrupted continuation dominates both policies across the shared-budget frontier.
Data and Method
Our methodology comprises four stages: curating multi-attempt execution traces across four model-framework settings, extracting prefix representations at fixed token landmarks, evaluating cross-task and within-task predictive discrimination, and replaying stopping policies under shared token budgets.
Released runs and labels
We analyze 147,002 execution traces drawn from two public corpora of autonomous coding agents solving benchmark repository issues. Each trace represents a single run (or attempt) by an agent assigned to resolve a specific repository issue, which we term a task , belonging to a source code repository. We define a setting as a unique combination of an underlying large language model and an agent execution framework.
Our primary corpus, released by Nebius [11], executes OpenHands with Qwen3-Coder-480B-A35B-Instruct [23] on SWE-rebench tasks [24]. To evaluate generalization across architectures and scaffolding, we incorporate three slices from NVIDIA Open-SWE-Traces [12]: SWE-agent with Qwen3.5-122B-A10B [25], OpenHands with Qwen3.5-122B-A10B, and OpenHands with MiniMax-M2.5 [26]. Table I summarizes the corpus statistics across all four settings. Because individual benchmark issues can appear in multiple corpora, task counts overlap across settings.
| Setting | Runs | Tasks | Repositories | ICC |
| OH / Qwen3-Coder | 67,074 | 6,306 | 1,823 | 0.797 |
| SA / Qwen3.5 | 15,098 | 7,885 | 1,815 | 0.768 |
| OH / Qwen3.5 | 30,957 | 14,142 | 2,234 | 0.713 |
| OH / MiniMax-M2.5 | 33,873 | 14,716 | 2,275 | 0.815 |
Table I. Summary of released execution-trace settings. OH: OpenHands; SA: SWE-agent. ICC measures within-task outcome clustering on the binary scale.
Each completed run terminates with a binary outcome label produced by the benchmark evaluation harness, which applies the agent’s final code patch to the repository and executes the verification test suite. We set if the patch resolves the issue and if the issue remains unresolved, excluding unlabelled records from our analysis.
Trace representation and decision landmarks
During execution, an agent interacts with its environment through an alternating sequence of assistant turns and tool observations. To establish a uniform measure of computational progress within each setting, we measure trajectory length in transcript tokens, counted across all recorded messages using that setting’s native model tokenizer, and denote the total transcript length of completed run by . We also normalize raw tool invocations into standard behavioral categories covering file inspection, codebase search, code editing, and test execution.
A decision landmark is a fixed transcript-token threshold at which a monitor evaluates an ongoing run. At landmark , the cohort of eligible (or active) runs consists exclusively of attempts whose total token length exceeds the threshold (), ensuring that a stopping intervention remains actionable. For each eligible run, the predictor observes only the execution prefix completed up to token . We evaluate predictions across seven landmarks ( tokens). Because shorter runs terminate as increases, we report the surviving fraction (the proportion of initial runs satisfying ) at each landmark to characterize the active population. We designate tokens as our primary decision point because agents have already engaged in substantial environment interaction by this stage while nearly all runs remain active (Figure 1, right). On the primary corpus at 20k tokens, 87.51% of active runs have already executed a detected test command and 32.80% have a detected edit action.

Figure 1. Prediction across fixed token landmarks. Left: within-task AUC for issue-plus-prefix count models, with 95% task-bootstrap confidence intervals at each landmark. The dashed line indicates random ranking (0.50). Right: surviving fraction of initial runs still active at each landmark. OH: OpenHands; SA: SWE-agent.
Evaluation metrics and task clustering
Before assessing individual predictors, we quantify the degree to which trial outcomes cluster by task using the intraclass correlation (ICC), computed via an analysis-of-variance estimator for binary outcomes [27] across tasks with repeated attempts. Values of ICC approaching 1.0 indicate that attempts on the same issue predominantly share the same terminal outcome.
At a given landmark , a failure monitor assigns each eligible run a scalar success score , where higher values indicate a higher predicted probability of resolving the task. We evaluate the calibration and sharpness of probability estimates using log loss [28] (the mean negative log-likelihood of the true outcome , where lower values reflect superior probabilistic accuracy) and evaluate ranking discrimination using two formulations of the area under the ROC curve (AUC) [29]:
1) Pooled AUC pairs every successful eligible run with every failed eligible run across the entire setting, computing the fraction of pairs in which the successful run receives a higher score (with half credit for ties, where 0.50 represents random ranking). Because pooled pairs span different issues, pooled AUC captures both cross-task difficulty and attempt-level differences.
2) Within-task AUC () isolates attempt-level discrimination by restricting comparisons to pairs of runs executed on the exact same task. We define a mixed-outcome task at landmark as a task whose active runs include at least one success and at least one failure. Let and denote the sets of successful and failed active runs on task , respectively. Aggregating across all mixed-outcome tasks, within-task AUC is defined as:
where is the indicator function. Each valid within-task pair carries equal weight in Eq. (1); for instance, a mixed-outcome task with two active successes and three active failures contributes pairs.
To account for hierarchical dependence in our data during uncertainty estimation, we compute 95% cluster-bootstrap confidence intervals that resample whole clustering units with replacement [30] from saved out-of-fold predictions. Specifically, pooled and replay metrics resample repositories (and training seeds for the neural monitor), whereas within-task AUC resamples tasks.
Prediction models for RQ1 and RQ2
To measure how much early execution trajectories improve outcome prediction over static issue descriptions (RQ1) and whether they distinguish competing attempts on the same task (RQ2), we train linear and conditional models using five-fold unseen-repository cross-validation. In this splitting protocol, all tasks belonging to a given repository are assigned to the same fold, ensuring that models are evaluated exclusively on repositories held out during training.
Our issue baseline extracts eight pre-execution count features from the problem statement covering text length, code blocks, error and reproduction indicators, URLs, and referenced file paths. Our issue-plus-prefix model augments these eight issue features with 18 behavioral count features extracted from the observed prefix up to landmark , capturing distinct files inspected, repeated shell commands, error outputs, cumulative text volume, code edits, and test invocations (two fine-grained test-output features are specific to the primary corpus trace schema, yielding 16 prefix features for the Open-SWE settings). Both models use L2-regularized logistic regression fit on log-transformed, standardized feature counts. Comparing the issue-plus-prefix model against the issue baseline isolates the predictive gain contributed by observing execution.
Because pooled logistic regression fits weights to variation both across tasks and within tasks, we also train a task-conditional logistic model [31] restricted to mixed-outcome tasks. By conditioning on task identity, this formulation eliminates the task-level intercept and estimates feature weights solely from differences between successful and failed attempts on the same issue using prefix features. We also evaluate cross-setting transfer at 20k tokens by applying the 16-feature conditional model trained on one setting directly to another after holding out overlapping repositories and tasks. To verify that our conditional estimation pipeline remains unbiased under the null, we validate the estimator against a permutation control that shuffles outcome labels within each task. To quantify how tightly within-task AUC concentrates around chance (0.50), we evaluate equivalence bounds of [0.45, 0.55] and [0.47, 0.53] using two one-sided tests (TOST) at the level via 90% task-bootstrap confidence intervals [32, 33].
Beyond standard count regressions, we evaluate three supplementary designs to test alternative sources of predictive signal:
a) Task metadata and solvability stratification: We test whether augmenting count models with Open-SWE-Traces task categories (bug fix, enhancement, feature request, other) [12] and programming language under unseen-repository cross-validation improves cross-task and within-task discrimination. To test whether attempt-level discrimination is stronger on easier issues, we also stratify mixed-outcome tasks at the 20k landmark into low (< 25%), medium (25–75%), and high (> 75%) empirical solve-rate tiers (Table III).
b) Frozen text embeddings: To test whether dense lexical representations capture early trajectory signals missed by summary counts, we encode the truncated issue description, the initial assistant turn, and the first tool observation on the primary corpus using the frozen all-MiniLM-L6-v2 sentence encoder [34, 35], concatenating these embeddings with initial prompt-context metadata as inputs to a regularized linear classifier.
c) Task-rate oracle: To establish an upper bound on task-level predictability and test whether prefix features add value when baseline task solvability is already known, we construct a diagnostic task-rate oracle on tasks with at least four active attempts. Across repeated random splits, we hold out half of a task’s attempts to compute its empirical solve fraction and supply that fraction as a feature when scoring the remaining half, both standalone and combined with prefix counts.
Adapted neural monitor
To test whether end-to-end representation learning extracts fine-grained within-task trajectory cues that count models and frozen embeddings miss (RQ2), we adapt the neural failure monitor architecture from FailFast–RestartSmart [7]. Specifically, we fine-tune Qwen3-0.6B [36] using low-rank adaptation (LoRA) [37] and a linear classification head to predict terminal task resolution at the 20k-token landmark under a 4,096-token input limit. We compare two input configurations trained and evaluated on identical data splits: an issue-only arm that receives the problem description, and an issue-plus-prefix arm that receives the problem description alongside the most recent transcript messages completed by 20k tokens. When present in the preceding prefix, the issue-plus-prefix input also includes the latest earlier message matching an editor-tool heuristic (which captures editor-tool invocations including fileviewing and modification operations, and is simply omitted when no prior message matches).
To prevent static task difficulty from dominating gradient updates and isolate attempt-level discrimination for RQ2, each optimization step samples a matched pair consisting of one successful and one failed attempt from the same mixed-outcome task, minimizing a composite objective that combines binary cross-entropy (BCE) loss with a pairwise ranking loss [38] that rewards assigning a higher score to the successful attempt. By construction, balanced within-task pair sampling fixes the training prevalence of every task at 0.50. For the issue-only arm, where both attempts in a pair share identical input text, the BCE objective has an analytical minimum at a constant prediction of 0.50 (yielding a baseline log loss of and because all attempts on a task receive identical scores), providing a null reference for measuring attempt-level signal in the issue-plus-prefix arm.
We conduct our neural evaluation across all primary-corpus tasks with at least 11 active attempts at the 20k landmark, sampling 11 attempts per task uniformly without outcome stratification to form a cohort of 32,406 attempts across 2,946 tasks. We partition this cohort using three-fold unseen-repository cross-validation so that every task is scored out-of-fold, and we train three random seeds per fold with checkpoint selection governed by validation within-task AUC. Because the three seeds evaluate on identical held-out repository folds, we report mean metrics aggregated across seeds and construct joint repository-and-seed bootstrap intervals.
Budget-constrained restart replay for RQ3
To determine how early failure scores translate into operational stopping decisions and resource consumption (RQ3), we perform recorded-attempt replay across all tasks with at least two recorded attempts. In each replay trial, an initial attempt executes until landmark . If a stopping policy terminates run at landmark , the system initiates a cold replacement attempt selected uniformly at random from the remaining recorded attempts on the same task, and we compute expected outcomes over all valid replacement choices while giving each starting attempt equal weight. Using out-of-fold success scores from the pooled issue-plus-prefix count logistic regression model (which spans all four settings and reflects the combined task and prefix signal deployed in standard monitors), we compare three primary policies: (1) Learned restart (), which terminates an active run at landmark if its out-of-fold success score falls below a fixed percentile threshold (evaluated across seven thresholds spanning the 2nd to 98th score percentiles); (2) Matched random restart (), which terminates the exact same overall fraction of active runs at landmark as the corresponding learned threshold, selecting runs uniformly at random without consulting ; and (3) Continuation, which permits the initial attempt to proceed to completion without intervention. (Terminating active runs at landmark without replacement, fixed truncation, corresponds to setting replacement credit , which upper-bounds solve rate by the pre-landmark completion rate and cannot improve resolution over continuation.)
We first evaluate policies in an uncapped replay regime where initial and replacement runs execute until their recorded completion lengths. To model realistic compute constraints, we then enforce a shared hard token cap governing the combined token expenditure of the initial run and any subsequent replacement. When a policy aborts run at landmark , the replacement attempt (with recorded length and outcome ) receives a residual token budget of . Its resulting replacement success credit and combined transcript-token cost are given by:
Under Eqs. (2)–(3), a replacement attempt earns resolution credit only if it solves the issue and completes within the remaining tokens. For example, stopping at under a cap leaves 55k tokens: a successful replacement requiring 50k tokens earns full credit () at a total cost of tokens, whereas a replacement requiring 60k tokens exhausts the cap at and earns . A policy’s overall solve rate is the mean resolution credit achieved across all starting attempts and replacement draws.
a) Decomposing learned versus random stopping: When a learned restart policy achieves a higher solve rate than matched random restart (), that advantage can arise from two distinct mechanisms: between-task selection (concentrating stops on tasks where restarting yields a more favorable expected return) and within-task selection (preferentially aborting the weaker attempts on a given task). To isolate how much of the policy difference stems from attempt-level failure detection [39], we derive an exact algebraic decomposition of .
Under a fixed cap , let indicate whether continuing starting attempt resolves the task within budget, and let denote the expected replacement credit obtained by averaging in Eq. (2) over all other recorded attempts on the same task. Defining as the net expected solve-rate gain of restarting attempt , and letting denote policy ’s stopping indicator (or stopping probability under randomized restart, with for runs that finished prior to ), the replay solve rate across all starting attempts is:
(4) shows that the net effect of stopping attempt equals the replacement success credited () minus the original continuation success forfeited (). Let denote the stopping differential between the learned and matched random policies on attempt , and let denote the set of active starting attempts on task at landmark , with and denoting the tasklevel means of and over . The solve-rate differential decomposes exactly into:
Here, quantifies the solve-rate difference attributable to task-level stopping rates (), whereas captures the covariance between policy stopping decisions and attempt-level restart value within the same task.
b) Cost-solve efficiency frontier: To evaluate practical resource efficiency at the 20k landmark, we compare the mean solve rate and mean transcript-token cost of every policy across all four settings, four token caps, and seven stopping thresholds. Within a given setting and token cap , a policy configuration is Pareto-dominated if an alternative policy achieves at least as high a solve rate at less than or equal mean token cost, with strict improvement on at least one dimension. Mapping dominance across the complete 224-configuration grid characterizes the empirical efficiency frontier of cold restarts relative to uninterrupted continuation.
Results
Baseline task clustering across settings
Across all four settings, terminal outcomes cluster strongly by task, with observed-scale intraclass correlation (ICC) reaching 0.71 or higher in every corpus (Table I). Consequently, repeated attempts on the same repository issue predominantly share the same outcome.
Comparing the two Qwen3.5 settings across their 7,102 shared tasks further illustrates how intrinsic issue difficulty persists across agent scaffolding. The empirical task solve rates of SWE-agent and OpenHands exhibit a Spearman rank correlation of 0.67, confirming that issues that are more tractable under one framework remain more tractable under the other. Weighting each shared task equally, SWE-agent achieves a mean solve rate of 47.30% compared to 33.24% for OpenHands, yielding a framework difference of 14.06 percentage points (95% repository-bootstrap CI: [12.26, 15.95]). Together, high within-task outcome clustering and cross-framework rank agreement establish that static task properties account for a substantial share of outcome variation.
RQ1: Value of observing an early prefix
To evaluate whether early execution trajectories improve outcome prediction beyond pre-execution context (RQ1), we compare pooled performance at early landmarks across models trained on the empirical task distribution. At 5k tokens on the primary corpus, augmenting the issue baseline with prefix action counts increases pooled AUC from 0.56 to 0.58, while log loss remains nearly unchanged (0.69 in both arms). By 20k tokens, prefix action counts increase pooled AUC from 0.57 to 0.61, reflecting that cumulative early interaction rates correlate with underlying issue difficulty across tasks. For frozen text embeddings of the initial turn at 5k tokens, pooled AUC remains at 0.62 in both the issue-only and issue-plus-prefix arms (with the paired gain interval spanning zero and log loss remaining at 0.67).
Supplementary checks confirm that static task context drives this cross-task ranking while leaving attempt-level discrimination unchanged. Even when restricting comparisons to pairs of runs from different tasks within the same repository on the primary corpus, eliminating repository-level confounding, issue-plus-prefix scores rank cross-task pairs with an AUC of 0.56 [0.55, 0.59] at 20k tokens (0.62 [0.59, 0.66] at 40k), compared to 0.49 (0.54 at 40k) for attempts on the exact same task. Similarly, adding Open-SWE task categories under unseen-repository cross-validation increases issue-baseline pooled AUC by approximately 0.02 across landmarks while leaving within-task AUC unchanged (, with no statistically significant gains across task-category or language subgroups after Holm–Bonferroni multiplicity correction [40]).
Answer to RQ1: Early action counts and static metadata yield modest gains in cross-task pooled ranking (0.02–0.04 AUC), whereas initial-turn text embeddings provide no consistent pooled improvement over pre-execution issue text alone.
RQ2: Ranking attempts at the same task
Early action counts and frozen embeddings
We next evaluate whether prefix representations separate successful from failed attempts on the same issue. At 5k tokens on the primary corpus, the issue-plus-prefix count model achieves a within-task AUC of 0.50 [0.49, 0.51], indicating chance-level attempt discrimination despite the pooled gain observed in RQ1. At the primary 20k decision point, within-task AUC remains near 0.49–0.50 across all four settings (Figure 1, left). At intermediate landmarks of 30k and 40k tokens, within-task AUC remains indistinguishable from chance across all three Open-SWE settings and rises only modestly on the primary corpus to 0.52 [0.51, 0.53] at 30k (93% of runs active) and 0.54 [0.53, 0.55] at 40k (69% active), a stage at which aborting a run forfeits 30k–40k sunk tokens and compresses the residual budget available for a replacement attempt. By 60k tokens, within-task AUC on the primary corpus reaches 0.56, but the surviving fraction drops to 23% of initial runs (Figure 1, right), reflecting both longer observation horizons and selective attrition of shorter trajectories.
The diagnostic task-rate oracle demonstrates how near-perfect pooled discrimination can coexist with chance-level within-task ranking. Because task solvability is heavily polarized across issues (ICC ), predicting outcomes solely from the empirical solve fraction of a task’s reserved attempts at 5k tokens achieves a mean pooled AUC of 0.97 while yielding a within-task AUC of 0.50 by construction. Combining prefix action counts with the task-rate oracle leaves pooled AUC at 0.97 and yields a within-task AUC of 0.50 at 5k tokens (rising to 0.53 at 60k tokens), confirming that early prefix features add negligible discrimination once baseline task solvability is known. Figure 2 contrasts pooled and within-task AUC across four predictors at 5k tokens: whereas pooled AUC spans 0.58 to 0.97, within-task AUC clusters tightly around 0.50, including for frozen text embeddings (0.49 [0.48, 0.50]).

Figure 2. Comparison of pooled AUC and within-task AUC at 5k tokens on the primary corpus. Each horizontal line connects a predictor’s pooled and within-task performance. Counts: issue features plus prefix action counts. Embeddings: frozen sentence embeddings of the issue and initial step. Task rate: empirical solve fraction computed from a reserved half of each task’s attempts (evaluated across tasks with at least four attempts and averaged over three random splits). The dashed line marks random ranking (0.50).
Task-conditional and cross-setting training
To test whether removing between-task variance during training uncovers attempt-level signal, we evaluate task-conditional logistic regression fit directly on mixed-outcome tasks. At 20k tokens on the primary corpus, the conditional model achieves a within-task AUC of 0.50 [0.49, 0.51] across 38,447 successful-failed pairs drawn from 1,737 mixed-outcome tasks. The conditional model’s 90% task-bootstrap interval [0.49, 0.51] falls entirely inside both the [0.45, 0.55] and [0.47, 0.53] TOST equivalence bounds, while the within-task label-permutation control remains centered at chance (0.50 [0.49, 0.51]).
Across the three Open-SWE settings at 20k tokens, conditional within-task AUC is 0.50 [0.48, 0.52] for OpenHands/Qwen3.5, 0.51 [0.49, 0.53] for OpenHands/MiniMax-M2.5, and 0.48 [0.45, 0.52] for SWE-agent/Qwen3.5, with all three 95% confidence intervals spanning 0.50. Both OpenHands settings pass the within-task permutation control and have 90% confidence intervals falling strictly within the [0.47, 0.53] equivalence bound (we refrain from formal TOST equivalence claims on SWE-agent/Qwen3.5 because its permutation control deviated from 0.50). Evaluating cross-setting transfer at 20k tokens, with overlapping repositories and task identifiers held out, likewise yields within-task AUCs of at most 0.50 across all three transfer pairs (SWE-agent/Qwen3.5 to OpenHands/Qwen3.5, and the primary setting to both SWE-agent/Qwen3.5 and OpenHands/MiniMax-M2.5).
Adapted neural monitor
Table 2 reports the three-seed evaluation of the LoRA-adapted Qwen3-0.6B monitor across 32,406 attempts at the 20k landmark. Across 16,818 successful-failed pairs from 859 mixed-outcome tasks, the issue-plus-prefix monitor achieves a mean within-task AUC of 0.50 [0.48, 0.51]. Performance remains near 0.50 within every individual repository fold and training seed, and weighting each mixed-outcome task equally produces identical results. As expected under balanced within-task pair sampling, the issue-only reference arm converges to the analytical baseline (AUC = 0.500, log loss = 0.693).
| Fold | Input | Pooled AUC | Within-task AUC | Log loss |
| 1 | Issue | 0.526 | 0.500 [0.499, 0.501] | 0.693 |
| 1 | Issue + prefix | 0.523 | 0.500 [0.476, 0.526] | 1.788 |
| 2 | Issue | 0.513 | 0.500 [0.499, 0.500] | 0.693 |
| 2 | Issue + prefix | 0.504 | 0.496 [0.475, 0.517] | 1.012 |
| 3 | Issue | 0.527 | 0.500 [0.497, 0.501] | 0.693 |
| 3 | Issue + prefix | 0.469 | 0.490 [0.467, 0.513] | 0.879 |
Table 2. Three-seed LoRA-adapted Qwen3-0.6B evaluation at 20k tokens across three repository-disjoint folds (311, 285, and 263 mixed-outcome tasks). Metrics are averaged across seeds rather than pooled across seed scores; brackets show 95% repository-and-seed bootstrap intervals for AUC. Lower log loss is better.
The neural monitor’s 90% repository-and-seed bootstrap interval [0.48, 0.51] lies strictly within the [0.45, 0.55] equivalence bound.
Stratification by task solvability and late-trajectory signals
To test whether early attempt-level discrimination emerges on easier issues where failures might reflect avoidable early missteps, Table 3 stratifies mixed-outcome tasks at 20k tokens into low (< 25%), medium (25–75%), and high (> 75%) empirical solve-rate tiers. Across all three tiers, within-task AUC ranges from 0.49 to 0.51 for the count-based model and from 0.49 to 0.50 for the three-seed neural monitor, with every confidence interval containing 0.50. High-solvability tasks therefore show the same near-chance early ranking as harder tasks.
| Predictor | Stratum | Tasks | Pairs | AUC [95% interval] |
| Count | Low-pass | 373 | 5,545 | 0.513 [0.484, 0.541] |
| Count | Medium-pass | 800 | 23,378 | 0.488 [0.476, 0.502] |
| Count | High-pass | 564 | 9,524 | 0.496 [0.475, 0.519] |
| Neural | Low-pass | 170 | 2,188 | 0.504 [0.464, 0.544] |
| Neural | Medium-pass | 405 | 10,966 | 0.491 [0.473, 0.509] |
| Neural | High-pass | 284 | 3,664 | 0.503 [0.475, 0.532] |
Table 3. Within-task AUC at 20k tokens stratified by empirical task solve rate : low (), medium (), and high (). Count denotes the primary-corpus issue-plus-prefix count model (95% task-bootstrap CIs); Neural denotes the three-seed Qwen3-0.6B mean (95% repository-and-seed CIs). All intervals contain 0.50. Strata are descriptive post-hoc tiers.
To verify that successful and failed trajectories on the same task do eventually diverge by completion, we compute an oracle end-of-run diagnostic measuring reference-hunk recall (the fraction of ground-truth patch diff hunks overlapped by the agent’s final edits, scored as 0 when no patch is produced). At run termination, reference-hunk recall achieves a within-task AUC of 0.62 [0.61, 0.64], confirming that attempt-level divergence materializes over the full episode even though it remains undetectable from prefix representations at 20k tokens.
Answer to RQ2: Across initial-turn frozen embeddings (5k tokens) as well as 20k-token action-count models, task-conditional rankers, cross-setting transfers, and the three-seed Qwen3-0.6B neural evaluation (AUC = 0.50 [0.48, 0.51]), early prefixes provide near-chance discrimination between successful and failed attempts on the same task across all solvability tiers.
RQ3: Evaluating restart policies under shared token budgets
To understand how early failure scores behave when deployed as stopping rules (RQ3), Table 4 compares learned restart (stopping active runs scoring below the median eligible score at 20k tokens) against matched random restart. In the uncapped replay regime, solve-rate differences between learned and random restarting are negligible across all four settings (within 0.20 percentage points): because unconstrained replacement attempts run to completion without budget truncation and within-task ranking is near chance, replacing an active attempt on a given task preserves the task’s expected solve rate. Under a shared 75k-token cap, however, the solve-rate differential spans to percentage points, favoring learned restart in both Qwen3.5 settings (+1.98 and +0.77 percentage points) while favoring random restart in the primary and MiniMax settings.
| Setting | Continue | Learned | [95% interval] |
| OH / Qwen3-Coder | 46.15 | 41.53 | −0.458 [−0.677, −0.219] |
| SA / Qwen3.5 | 28.71 | 20.66 | +1.984 [+1.313, +2.600] |
| OH / Qwen3.5 | 27.19 | 20.16 | +0.768 [+0.302, +1.221] |
| OH / MiniMax-M2.5 | 41.05 | 35.76 | −0.663 [−1.014, −0.333] |
Table 4. Restart replay at the 20k-token landmark using the median eligible-score threshold. Solve rates are percentages; is learned minus matched random restart in percentage points, with 95% repository-bootstrap confidence intervals.
Figure 3. Solve-rate difference (in percentage points) between median-threshold learned restart and matched random restart (blue), and between learned restart and uninterrupted continuation (red), evaluated at the 20k-token landmark under a shared 75k-token cap. Error bars indicate 95% repository-bootstrap confidence intervals (5,000 draws), with positive values favoring learned restart. While learned restart outperforms random restart in the two Qwen3.5 settings, uninterrupted continuation outperforms learned restart across all four settings.
Between-task selection drives the 75k learned-versus-random differential
Applying the exact decomposition from Eqs. (4)–(6) at the median threshold explains how learned restarting can outperform matched random restarting despite near-chance within-task AUC. As shown in Table V, the between-task selection term (−0.70 to +1.75 percentage points) matches the sign of the total differential and exceeds the magnitude of the within-task term in every setting. By contrast, remains small across all four corpora (−0.11 to +0.24 percentage points) and never determines the direction of the overall policy difference.
The task-level net gain in (5) clarifies why changes sign across settings. Because the learned policy assigns lower scores to harder tasks, it concentrates stops on tasks where both the replacement success rate within the remaining 55k tokens () and the original continuation success rate () are lower than the corpus average. Consequently, learned stopping yields less replacement credit than random stopping, but it also forfeits fewer original continuation successes. In the two Qwen3.5 settings, the reduction in forfeited continuation successes outweighs the reduction in replacement credit, producing a net positive purely through task-level stopping allocation.
| Setting | Total | Between | Within |
| OH / Qwen3-Coder | −0.458 | −0.494 | +0.036 |
| OH / MiniMax-M2.5 | −0.663 | −0.704 | +0.042 |
| OH / Qwen3.5 | +0.768 | +0.881 | −0.113 |
| SA / Qwen3.5 | +1.984 | +1.749 | +0.235 |
Table V. Exact decomposition of learned minus matched random solve rate (in percentage points) at the 20k landmark under a 75k-token cap. Task means center over attempts active at 20k and normalize by all starting attempts. Between-task selection () governs the sign in every setting; components are rounded independently.
Uninterrupted continuation dominates the efficiency frontier
Whereas comparing learned against random restart evaluates how a fixed stopping quota is distributed across runs, comparing against uninterrupted continuation tests whether cold restarting saves compute or improves resolution overall. Extending replay across all seven score-percentile thresholds, four settings, and four hard token caps yields 224 learned and random restart configurations at the 20k decision point (Table VI). Under the primary 75k-token cap, where replacement runs receive 55k residual tokens, uninterrupted continuation achieves both a strictly higher solve rate and a strictly lower mean transcript-token cost than every learned and random restart configuration across all four settings (56 of 56 configurations). In SWE-agent/Qwen3.5, for example, continuation resolves 28.71% of starting attempts within the 75k cap (down from 47.30% uncapped across shared tasks, as many SWE-agent runs exceed 75k tokens) at a mean expenditure of 68.10k tokens, whereas median learned restart resolves 20.66% at a mean expenditure of 69.93k tokens. This dominance persists at the 100k-token cap (80k residual tokens), where continuation Pareto-dominates 55 of 56 configurations, leaving only a single boundary point for OpenHands/MiniMax (2nd-percentile threshold, which adds less than 0.01 percentage points in solve rate at an added cost of 376 mean tokens). Across all 224 tested configurations spanning , continuation Pareto-dominates 216 configurations, with the other seven non-dominated cases being exact ties at the 30k cap. Two coupled budget mechanics explain why cold restarts simultaneously lower solve rates and increase mean token consumption under a shared cap . First, terminating a run at landmark restricts the replacement attempt to a residual budget of tokens (), whereas continuation permits the initial run to utilize the full cap . At tighter caps ( and , leaving 10k and 30k residual tokens), Figure 1 (right) shows that 91%–100% of coding trajectories require more than tokens to finish, rendering cold replacements nearly guaranteed to truncate prior to patch submission. Even at and , replacement trajectories that would have succeeded between and tokens are forfeited. Second, aborting an active run at forfeits the initial 20k tokens invested in repository exploration and forces the replacement attempt to repeat setup and localization from scratch, driving combined token expenditure above the cost of allowing moderate-length original attempts to finish naturally.
| Cap | Dominated | Tied | Distinct trade-offs |
| 30k | 49 | 7 | 0 |
| 50k | 56 | 0 | 0 |
| 75k | 56 | 0 | 0 |
| 100k | 55 | 0 | 1 |
Table VI. Cost-solve comparisons against uninterrupted continuation at the 20k-token landmark (56 restart configurations per cap across seven thresholds, four settings, and learned vs. matched random policies). Ties match continuation in both cost and solve rate. The sole non-dominated point is learned restart for OH/MiniMax at 100k (2nd-percentile threshold: +0.0033 percentage points at +376 mean tokens).
Answer to RQ3: Under a shared 75k-token cap, learned stopping outperforms matched random restart by up to 1.98 percentage points through between-task stopping selection, while attempt-level selection contributes minimally. Against uninterrupted continuation, however, cold restarts achieve lower solve rates at higher mean token costs across the 75k and 100k frontiers (111 of 112 configurations, and 216 of 224 configurations across all tested caps), showing that improvements over random restart do not establish compute savings.
Discussion
Evaluating runtime interventions in autonomous software engineering requires connecting task-level difficulty with attempt-level discrimination and budget-constrained policy mechanics. Our findings explain why strong cross-task ranking does not translate into effective cold restarts under shared token budgets.
The timing bottleneck of trajectory-level signals
Comparing early prefixes with later checkpoints clarifies why attempt-level discrimination is difficult in repository-level coding. On the primary corpus, 87.51% of active runs have already executed a detected test command and 32.80% have a detected edit action by 20k transcript tokens. Thus, the near-chance within-task discrimination observed at 20k tokens (AUC) cannot be explained simply by an absence of early test execution. Rather, detecting an early test invocation or edit action does not establish whether the interaction was diagnostic or whether its outcome distinguished eventual task resolution from failure. Measurable within-task separation emerges only at later landmarks, rising from 0.52 at 30k tokens and 0.54 at 40k tokens to 0.56 at 60k tokens, and at run completion, where reference-hunk recall reaches a within-task AUC of 0.62. By those later stages, however, either a substantial fraction of the token budget has been spent or most runs have already terminated. Early failure prediction therefore faces a fundamental timing tension: at landmarks early enough to leave ample residual budget for a replacement attempt, observed prefix signals do not separate competing runs on the same issue.
b) Why task-level selection confounds stopping baselines: When a monitor primarily captures static issue difficulty, score thresholding acts as an implicit task filter. Under a shared cap, learned stopping concentrates aborts on hard tasks where continuation rarely succeeds, whereas matched random stopping aborts easy and hard tasks alike, forfeiting impending successes on tractable issues () despite near-chance attempt ranking (). For the same reason, mid-run abort-only termination wastes 20k tokens executing a task filter that could be applied pre-execution at zero token cost. Against uninterrupted continuation, forfeiting initial exploration while compressing the residual budget causes cold restarts to underperform across the budget frontier.
c) Designing compute-efficient agent interventions: Separating task difficulty from attempt progress points to two compute-management strategies:
Pre-execution routing and budgeting: Because static signals rank difficulty before execution, they naturally support up-front token provisioning [9] and solver routing [22]. On shared Qwen3.5 tasks, scaffold choice alone shifts solve rates by 14.06 percentage points (SWE-agent 47.30% vs. OpenHands 33.24%), dwarfing the at most 1.98 percentage-point gain of mid-run restarts and motivating pre-execution routing across frameworks.
State-preserving recovery: Because cold prefixes lack attempt-level signal, retaining intermediate edits and test diagnostics is essential to avoid repeating exploration, explaining why state-preserving recovery substantially outperforms cold restarts [7] and motivating cross-framework context transfer when runs stall.
d) Evaluation standards for mining agent traces: As public corpora of agent trajectories grow [11], [12], runtime monitoring studies should evaluate both attempt-level discrimination and net compute efficiency. Specifically, predictive evaluations should report within-task AUC () across repeated attempts alongside pre-execution issue baselines and permutation controls. Policy evaluations under shared token budgets should compare proposed stopping rules against both matched random stopping and uninterrupted continuation, decomposing outcomes into between-task and within-task selection to verify genuine attempt-level value.
Threats to Validity
a) Internal validity: Because our study mines public execution corpora [11], [12], outcome labels and trajectory contents reflect the original benchmark harnesses and execution environments. To guard against data-cleaning artifacts or estimation bias, we audited source task identifiers for text consistency and evaluated within-task label-permutation controls across all conditional modeling pipelines; permutation controls centered at chance (0.50) in the primary and OpenHands settings and identified the calibration deviation in SWE-agent/Qwen3.5, where we refrained from formal equivalence claims.
b) Construct validity: We measure computational progress in transcript tokens to standardize comparisons across corpora; however, transcript length abstracts away provider-level caching discounts and wall-clock tool execution latency. Our neural monitor adapts the single-landmark terminal-outcome classifier from FailFast–RestartSmart [7], omitting intermediate test supervision and state-preserving edit transfer to isolate the predictive signal of raw execution prefixes and the mechanics of cold restarts; additionally, the editor-tool heuristic used to include an earlier prefix message can match file-view operations alongside code modifications. Recorded-attempt replay evaluates binary task resolution under hard token caps, assigning zero credit to partial edits that exceed the residual budget, and released task-category labels in Open-SWE-Traces may reflect post-hoc curation rather than metadata available at issue creation.
c) External validity: Our findings are scoped to four open-weight model-framework settings on Python-centric repository benchmarks evaluated with linear count and embedding classifiers alongside a 0.6B-parameter neural monitor. Within-task discrimination requires mixed-outcome tasks by definition. Monitors that probe internal model representations [6], [20] or incorporate sequential verifier feedback and state-preserving recovery [7], [19] may exhibit different discrimination profiles.
d) Conclusion validity: Cluster-bootstrap confidence intervals resample out-of-fold predictions at the cluster level rather than refitting models per bootstrap draw. Subgroup and multi-landmark sensitivity checks apply Holm–Bonferroni multiplicity adjustments [40], whereas confidence intervals at the primary 20k landmark are reported unadjusted. Finally, Pareto dominance across the 224 budget-threshold configurations is evaluated on empirical replay point estimates.
Conclusion
Early failure prediction benefits autonomous coding agents only when it helps decide whether to abandon an active attempt, rather than merely identifying which software issues are difficult. Mining 147,002 execution traces across four model-framework settings, we find that early execution prefixes add minimal predictive signal beyond static task context. Across early decision points, initial-turn frozen embeddings (5k tokens) as well as 20k-token action-count models, task-conditional rankers, and a three-seed LoRA-adapted Qwen3-0.6B monitor (32,406 attempts) all yield within-task discrimination near chance (), including across high-solvability tasks.
Evaluating stopping rules through budget-constrained replay shows why predictive ranking and policy efficiency must be assessed together. Under a shared 75k-token cap, learned stopping outperforms matched random restart by up to 1.98 percentage points in two settings, but exact decomposition reveals that this gain is driven by between-task stopping allocation rather than within-task failure detection. More importantly, uninterrupted continuation achieves higher solve rates at lower mean token cost than cold restarts across the 75k and 100k frontiers (111 of 112 configurations, and 216 of 224 configurations across all tested caps). Future early-stopping and recovery methods should therefore evaluate discrimination within repeated attempts on the same task and demonstrate net compute savings relative to uninterrupted continuation.
References
- [1]C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” in International Conference on Learning Representations, vol. 2024, 2024, pp. 54 107–54 157.arxiv.org/abs/2310.06770
- [2]J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024.arxiv.org/abs/2405.15793
- [3]X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh et al., “Openhands: An open platform for ai software developers as generalist agents, 2024,” URL https://arxiv.org/abs/2407.16741, vol. 2, no. 5, 2025.
- [4]D. Pham, K. Katevas, A. S. Shamsabadi, and H. Haddadi, “AgentStop: Terminating local AI agents early to save energy in consumer devices,” arXiv:2605.15206v1, 2026. [Online]. Available: https://arxiv.org/abs/2605.15206v1
- [5]S. Cho, F. C. Fernandez, U. Mohammed, Z. Wu, K. D. Costa, I. Wicaksono, and A. Kos hiyama, “Automata from agent traces: Failure and next-step prediction,” arXiv:2608.23670v1, 2026. [Online]. Available: https://arxiv.org/abs/2608.23670v1
- [6]A. Silva, H. Tu, and M. Mompurrus, “Latent programming horizons in coding agents,” arXiv:2607.05188v1, 2026. [Online]. Available: https://arxiv.org/abs/2607.05188v1
- [7]C. Wang, Y. Lyu, J. He, Z. Yang, C. Zhong, Y. Harel, and D. Lo, “Fail-fast, restart-start: Early failure prediction and restart for SWE agentic tasks,” arXiv:2608.03222v1, 2026. [Online]. Available: https://arxiv.org/abs/2608.03222v1
- [8]T. Mehtiyev and W. Assunção, “Beyond resolution rates: Behavioral drivers of coding agent success and failure,” arXiv:2604.02547v1, 2026. [Online]. Available: https://arxiv.org/abs/2604.02547v1
- [9]M. EsfandyariDoulabi, L. Arkoh, B. Tadesse, V. Patel, M. Sharma, M. d’Amorim, and W. Assunção, “Disentangling task difficulty from run-level failure in agent failure prediction,” arXiv:2610.05572v1, 2026. [Online]. Available: https://arxiv.org/abs/2610.05572v1
- [10]Y. U. Bulut, “It’s the problem, not the path: Budget and difficulty confounds in LLM reasoning trajectories,” arXiv:2609.03436v1, 2026. [Online]. Available: https://arxiv.org/abs/2609.03436v1
- [11]Nebius, “SWE-rebench OpenHands Trajectories,” Hugging Face dataset, 2026. [Online]. Available: https://huggingface.co/datasets/nebius/SWE-rebench-openhands-trajectories
- [12]NVIDIA, “Open-SWE-Traces,” Hugging Face dataset, 2026. [Online]. Available: https://huggingface.co/datasets/nvidia/Open-SWE-Traces
- [13]R. Robbes, T. Matricon, T. Degueule, A. Hora, and S. Zacchiroli, “Promises, perils, and (timely) heuristics for mining coding agent activity,” in Proceedings of the 23rd International Conference on Mining Software Repositories, 2026, pp. 496–507.
- [14]X. Zhao, H. Li, S. Li, T. Zhao, E. T. Barr, F. Sarro, and H. Ye, “Failure as a process: An anatomy of CLI coding agent trajectories,” arXiv:2607.09510v1, 2026. [Online]. Available: https://arxiv.org/abs/2607.09510v1
- [15]A. Banerjee and P. Raichura, “Measuring cross-task behavioral consistency in language model agents,” arXiv:2608.13598v1, 2026. [Online]. Available: https://arxiv.org/abs/2608.13598v1
- [16]Y. Guo, Y. Xiao, J. M. Zhang, H. Marman, Y. Lou, Y. Liu, and Z. Chen, “EET: Experience-driven early termination for cost-efficient software engineering agents,” arXiv:2601.05777v2, 2026. [Online]. Available: https://arxiv.org/abs/2601.05777v2
- [17]A. Baidya, X. Liang, R. Guo, X. Gao, and K. Das, “When evidence is sparse: Weakly supervised early failure alerting in dialogs and LLM-agent trajectories,” arXiv:2606.05414v1, 2026. [Online]. Available: https://arxiv.org/abs/2606.05414v1
- [18]Y. Lin, Z. Wang, M. Liu, Y. Shan, L. Bai, J. Zhang, X. Jin, B. Chen, J. Su, X. Wang, J. Pei, and M. Li, “BAGEN: Are LLM agents budget-aware?” arXiv:2606.00198v1, 2026. [Online]. Available: https://arxiv.org/abs/2606.00198v1
- [19]S. Sadhuka, D. Prinster, C. Fannjiang, G. Scalia, B. Berger, A. Regev, and H. Wang, “E-valuator: Reliable agent verifiers with sequential hypothesis testing,” arXiv:2512.03109v2, 2026. [Online]. Available: https://arxiv.org/abs/2512.03109v2
- [20]K. Ruan, Z. Huang, Z. Zhou, Q. Wei, J. Lin, X. Wang, and H. Sun, “Doomed from the start: Early abort of LLM agent episodes via a recall-controlled forecast cascade,” arXiv:2607.06503v2, 2026. [Online]. Available: https://arxiv.org/abs/2607.06503v2
- [21]M. Luby, A. Sinclair, and D. Zuckerman, “Optimal speedup of the vegas algorithms,” Information Processing Letters, vol. 47, no. 4, pp. 173–180, 1993.
- [22]I. Bhola, A. Krishnan, and Mukunda NS, “Scouting: Cost-aware routing of coding agents by scouting the repository first,” arXiv:2608.04804v1, 2026. [Online]. Available: https://arxiv.org/abs/2608.04804v1
- [23]Qwen Team, “Qwen3-Coder-480B-A35B-Instruct,” Hugging Face model card, 2025. [Online]. Available: https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct
- [24]I. Badertdinov, A. Golubev, M. Nekrashevich, A. Shevtsov, S. Karasik, A. Andriushchenko, M. Trofimova, D. Livintseva, and B. Yangel, “Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,” Advances in Neural Information Processing Systems, vol. 38, 2026.
- [25]Qwen Team, “Qwen3.5-122B-A10B,” Hugging Face model card, 2026. [Online]. Available: https://huggingface.co/Qwen/Qwen3.5-122B-A10B
- [26]MiniMax, “MiniMax-M2.5,” Hugging Face model card, 2026. [Online]. Available: https://huggingface.co/MiniMaxAI/MiniMax-M2.5
- [27]M. S. Ridout, C. G. Demétrio, and D. Firth, “Estimating intraclass correlation for binary data,” Biometrics, vol. 55, no. 1, pp. 137–148, 1999.pubmed.ncbi.nlm.nih.gov/11318148
- [28]T. Gneiting and A. E. Raftery, “Strictly proper scoring rules, prediction, and estimation,” Journal of the American Statistical Association, vol. 102, no. 477, pp. 359–378, 2007.
- [29]J. A. Hanley and B. J. McNeil, “The meaning and use of the area under a receiver operating characteristic (roc) curve,” Radiology, vol. 143, no. 1, pp. 29–36, 1982.pubmed.ncbi.nlm.nih.gov/7063747
- [30]C. A. Field and A. H. Welsh, “Bootstrapping clustered data,” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 69, no. 3, pp. 369–390, 2007.DOI
- [31]N. E. Breslow, N. E. Day, and E. Heseltine, “Statistical methods in cancer research,” 1980.
- [32]D. J. Schuirmann, “A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability,” Journal of pharmacokinetics and biopharmaceutics, vol. 15, no. 6, pp. 657–680, 1987.pubmed.ncbi.nlm.nih.gov/3450848
- [33]D. Lakens, “Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. social psychological and personality science, 8 (4), 355–362,” 2017.
- [34]N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), 2019, pp. 3982–3992.arxiv.org/abs/1908.10084
- [35]W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,” Advances in neural information processing systems, vol. 33, pp. 5776–5788, 2020.
- [36]A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
- [37]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021.arxiv.org/abs/2106.09685
- [38]D. Sculley, “Combined regression and ranking,” in Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, 2010, pp. 979–988.DOI
- [39]E. M. Kitagawa, “Components of a difference between two rates,” Journal of the american statistical association, vol. 50, no. 272, pp. 1168–1194, 1955.DOI
- [40]S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics, pp. 65–70, 1979.DOI
