WorksComputer Science
BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
Abstract
Enabling AI systems to autonomously improve AI models requires agents capable of performing machine learning engineering (MLE). In practice, designing, training, and evaluating complex models demands modifying multi-file repositories to reproduce results, debug training runs, integrate datasets, and port frameworks. However, existing MLE benchmarks limit agents to single-file edits without full repository or shell access, and common repository-level software engineering benchmarks have rapidly saturated. To address this, we introduce the Code Complexity Index (CCI), a difficulty metric based on ten structural features spanning repository scale, dependency depth, and patch size. CCI strongly predicts average solve rates across eight public coding benchmarks (), identifying patch size and complexity as top predictors. Guided by CCI, we develop BuggyMLE, a scalable framework where agents inject controlled mutations into working ML repositories, synthesizing over 6,600 bug-injected MLE repository variants and nearly 260K lines of injected code based on 222 source repositories. Using BuggyMLE, we make two contributions. First, we construct BuggyMLE-Bench-Lite, the first difficulty-graded, repository-level variant of MLE-Bench Lite; across 14 evaluated models, the top solve rate drops from 95% to 9% as difficulty is scaled from the easiest to the hardest tier. Second, we synthesize a training curriculum across three difficulty levels over 200 additional ML repositories, fine-tuning a Qwen3.5-9B model to rival frontier models on BuggyMLE-Bench-Lite. Our trained model transfers beyond ML, raising its SWE-bench Lite resolve rate from 24% to 33% and its LiveCodeBench v6 pass@1 from 69% to 71%.
Introduction
Autonomous AI systems are increasingly asked to carry out the open-ended work of research and engineering rather than isolated question answering, a direction broadly framed as AI-for-Research [12]. One subset of AI-for-Research, the study of AI developing AI training pipelines and models, has emerged as a cornerstone of recursive self-improvement [11], as agents that autonomously conduct ML research can improve the foundations on which future research agents are built. Machine learning engineering (MLE) tasks are a natural proving ground for this ambition: here, success requires navigating real codebases, running training pipelines, and iteratively improving measurable metrics. Benchmarks such as MLE-Bench [9] and FrontierCS [30] measure how far frontier models have progressed on these open-ended problems, yet they face three recurring limitations. First, they predominantly grade single-file solutions, sidestepping the multi-file code repositories that characterize real-world MLE projects. Second, many existing suites are largely saturated, with frontier models consistently outperforming human reference solutions [30, 45]. Third, manual curation is costly: each task demands a vetted dataset, an execution harness capable of running and scoring training, and a human-verified reference outcome. As frontier models advance, hand-built suites saturate within months, driving a surge of interest in synthetically generating MLE tasks at scale [37].
Synthetic task generation aims to break this curation bottleneck through mutation-based fault injection into existing codebases [41, 44] or generation-based synthesis, where new tasks and their evaluation setups are generated directly from underlying datasets ([37] (He et al. 2026)). While both paradigms have scaled available task quantity by an order of magnitude, from hundreds ([35]) to thousands ([37, 45]), generating large volumes of tasks at flat or unmeasured complexities may leave benchmarks saturated and offer no principled curriculum from easy to hard tasks. Because existing pipelines lack a systematic measure of what makes a task hard, they cannot tell whether newly generated benchmarks remain useful or further scale difficulty as the frontier moves, underscoring the need to both measure complexity and control task difficulty (Table 1).
Table 1. Work on agentic MLE falls into three lines: synthetic data generators, which emit new tasks; training frameworks, which provide the harness in which an agent is trained or evaluated; and sandbox environments, which define what an agent may touch while solving. A check mark indicates the capability is present. BuggyMLE spans all three, and is the only work giving the agent an unrestricted shell over a multi-file repository together with explicit control over task difficulty.
Existing MLE agent frameworks compound this limitation by relying on highly specialized, ML-specific code-generation harnesses, such as evolutionary program search ([33, 45]) or multi-agent orchestrators ([36]), designed to rewrite single standalone scripts. Although these custom scaffolds may attain state-of-the-art benchmark scores via heavy test-time search, they make it difficult to disambiguate gains driven by the harness from the model’s intrinsic capabilities. Moreover, regenerating a single script poorly reflects modern MLE practice, where engineers diagnose and modify interconnected multi-file repositories in an open shell environment. Together, these limitations raise three central questions: *what makes a repository-level MLE task challenging for frontier models, how can this be measured, and how can we then systematically control its difficulty?
To this end, we develop the Code Complexity Index (CCI), a principled metric that quantifies the difficulty of a coding benchmark using only structural properties of the code. The CCI predicts a benchmark’s average solve rate from three feature groups (repository scale, dependency complexity, and bug/patch complexity) using a four-parameter logistic regression. Across eight established coding benchmarks, CCI explains 95.1% of the variance in average solve rate (; Figure 1), confirming the intuition that as required solution complexity (CCI) increases, solve rates decline.
Figure 1. Code complexity predicts task difficulty across BuggyMLEBench-Lite and eight public coding benchmarks. Top: An agent injects bugs into the clean repositories of all 22 BuggyMLEBench-Lite tasks to form the 1 Bug, 5 Bugs, 10 Bugs, and 50 Bugs tiers. (a) A representative BuggyMLEBench-Lite repository (Tabular Playground Series) is plotted at the CCI of its code after each injected bug. Its ROC AUC falls with each bug, dropping below the Kaggle competition leaderboard median after the first injection, with dots colored by the pipeline stage each bug modifies. (b) On the same CCI axis, gray points show eight public coding benchmarks at their average solve rate across all evaluated models, along with their linear fit (; CCI uses no model information). The blue band spans judge-verified Full Repair across nine baseline models from Table 2 (excluding Qwen3.5-9B and its trained variants) on the four BuggyMLEBench-Lite tiers (all 22 tasks per tier, each plotted at its median CCI), highlighting the median model (solid) and best model (dashed; Claude Opus 5.5). Solve rates decline with rising CCI along the same trajectory as the public benchmarks. We further verify this trend on 200 additional BuggyMLE tasks in Appendix § 8.§ 5.
Inspired by this finding, we present BuggyMLE, a synthetic, mutation-based generation framework for MLE tasks. BuggyMLE uses a large language model (LLM) agent to explore an existing ML task’s repository and inject breaking changes, then measures whether frontier LLM solvers can diagnose and fix the injected faults. Unlike prior mutation-based approaches that focus primarily on software engineering tasks, BuggyMLE targets the ML domain specifically, where evaluation requires running training pipelines and measuring task-specific performance metrics. We validate our framework on both synthetically generated repositories from MLE-Smith (Qiang et al., [37]) and human-engineered repositories from PyHealth (Wu et al., 2026), demonstrating that our approach can produce realistic, difficulty-controlled MLE tasks regardless of the source repository’s origin. Bug-injected tasks follow the same difficulty trajectory as human-curated benchmarks: as we inject more bugs into a repository, its CCI rises and all models solve fewer tasks, matching the public-benchmark trend (Figure 1b).
In summary, we make the following contributions:
Model-Agnostic Complexity Metric (CCI): We develop CCI, a ten-feature structural metric that predicts average benchmark solve rates from code properties alone, explaining 95.1% of the variance across eight public coding benchmarks () without using any features of the evaluated model.
Curriculum Learning through Scaling Task Difficulty (BuggyMLE): We introduce BuggyMLE, a scalable environment providing multi-file repository access and an unrestricted shell where bug-injection agents scale task difficulty. At scale, BuggyMLE synthesizes 6,674 execution-verified, bug-injected repository variants (K lines of injected code) from 222 source repositories, predictably lowering solve rates across models while providing dense, verifiable rewards for supervised fine-tuning (SFT) and reinforcement learning (RL).
Difficulty-Graded Repository Benchmark (BuggyMLEBench-Lite): We construct BuggyMLE-Bench-Lite, which, to the best of our knowledge, is the first repository-level variant of MLE-Bench Lite (Chan et al., [9]): 88 oracle-verified tasks spanning four controlled difficulty tiers (1 to 60 bugs per repository) over 22 multi-file MLE repositories. Across 14 models, the top solve rate falls from 95% to 9% from the easiest to the hardest tier.
Out-of-Distribution (OOD) Validation: Using BuggyMLE to generate a difficulty-graded curriculum across 200 ML repositories, we train Qwen3.5-9B and carry its gains beyond ML repair, lifting SWE-bench Lite resolve rates from 24% to 33% and LiveCodeBench v6 pass@1 from 69% to 71%.
Related Work
Coding and ML Benchmarks
The rapid advancement of LLM-based coding agents has driven the development of increasingly challenging evaluation benchmarks across several domains. In software engineering, SWE-bench Jimenez et al. [25] introduced the paradigm of evaluating agents on real-world GitHub issues, spawning a family of successors including SWE-Gym [34] for agent training, SWE-bench Pro [14] for long-horizon enterprise-scale issues, DeepSWE [23] and FrontierSWE [13] for frontier-difficulty tasks, SWE-Marathon [15] for long-horizon software engineering challenges, ProgramBench [46], and WebGen-Bench (Lu et al., 2025) for generating entire applications and websites from scratch. In the computer science task space, FrontierCS [30] evaluates algorithmic and research-style code generation and FrontierCode (Lu et al., 2026) probes the limits of LLMs for competitive programming.
For machine learning specifically, MLAgentBench [22] introduced a curated set of ML experimentation tasks, followed by MLGym [32] with a gym-style RL interface, MLE-Dojo [35] with over 200 executable ML tasks, and MLS-Bench [29], which provides additional coverage of ML system building. In the debugging and defect detection space, DSDBench [47] targets data science debugging, MLDebugging [21] covers multi-library debugging scenarios, and curated defect repositories such as Defect4ML [31] and DenchMark [26] provide collections of real-world ML framework bugs.
A common thread across all of these benchmarks is their reliance on human curation: each task, test case, and evaluation criterion is manually authored or verified, limiting scalability and making it difficult to systematically vary difficulty. Our work addresses this gap by providing both a scalable generation pipeline and a quantitative model of what makes tasks hard.
Environment Scope and Tooling
Frameworks for ML engineering differ in the scope of code an agent may touch and in the generality of the tools it is given. AlphaEvolve [33] and OpenMLE [45] improve on benchmarks by iteratively regenerating candidate solutions under an evolutionary search, in which the model proposes a program and the harness scores it. Matryoshka [36] distributes the work across sub-LLMs that generate and rewrite code, coordinated by an orchestrator with specialized tooling. MLE-Dojo [35], MLGym [32], and MLAgentBench [22] expose structured task interfaces over which such loops run. In each case the agent acts through a fixed set of operations, and in most cases on a single solution artifact. BuggyMLE instead places the agent in a container with an unrestricted shell, where it may run any command, inspect and patch any file in the repository, and decide for itself whether an experiment is worth its remaining budget (Table 1). This freedom mirrors real ML engineering, where practitioners inherit a working repository and change one part of it at a time while leaving the rest of the pipeline intact. Evolutionary harnesses produce impressive ML performance, but repeatedly regenerating a single code artifact under a scoring loop ignores the complexity of real-world ML workflows, which span multiple files.
Synthetic Task and Data Generation
An emerging line of work addresses the curation bottleneck through automated task generation. Mutation-based approaches take existing codebases and systematically introduce faults. SERA [41] generates reproducible assessment environments through code mutation, and SWE-smith [44] scales bug generation to thousands of Python repositories using automated fault injection. These approaches inherit the realism of their source codebases but typically lack principled control over the resulting task difficulty.
Generation-based approaches synthesize entirely new tasks and evaluation setups. MLE-Smith [37] converts raw datasets into competition-style ML challenges using a multi-agent pipeline, FrontierSmith (He et al., 2026) transforms closed-form solution tasks into open-ended problems, TaskCraft [42] generates multi-tool agentic tasks through compositional extensions, and AutoCodeBench [48] creates multilingual code problems via LLM-driven reverse synthesis. Complementary to both paradigms, self-play methods such as Self-Questioning Language Models [10] and Self-Challenging Language Models [49] use LLMs to generate challenges for themselves, framing task generation as asymmetric self-play.
BuggyMLE sits in the mutation-based camp but introduces three key differentiators. First, rather than sampling from a fixed fault catalog, our Injector agent explores each repository and introduces breaking changes freely, and we control task difficulty directly by scaling the magnitude of the injected change. Second, CCI offers a quantitative predictor of benchmark difficulty computed from code structure alone, and is, to our knowledge, the first such metric that relates structural code properties to expected solver performance with high explanatory power (). Third, agents solve tasks within a general-purpose terminal environment across the full repository rather than through a fixed action set on a single file, mirroring real-world ML engineering workflows.
Methodology
Code Complexity Index
What makes a coding benchmark hard? We answer with CCI, a composite score that maps structural properties of a repository and its patch to expected solver performance. We fit our CCI across eight established coding benchmarks whose leaderboard results are public.
Raw features. For each benchmark, we compute the task-level median (p50) of ten structural features and group them into three conceptual dimensions:
Repo scale () captures codebase size and directory structure. Dependency complexity () measures the depth and connectivity of the inter-file dependency graph, and bug/patch complexity () describes how far the required fix reaches across files, lines, and hunks. A hunk is a contiguous block of modified lines in a unified diff, so a single fix can span several hunks in one or more files.
Preprocessing. All quantities below are computed per benchmark. Let be a benchmark’s value of feature . The raw features are right-skewed counts, so we log-transform each one, z-score it across benchmarks, and score each group by the mean of its features’ z-scores:
where and are the mean and standard deviation of across benchmarks. Each group score is rescaled to unit variance across benchmarks; for benchmark , we write for the score of , and likewise for the other two.
Modeling benchmark difficulty. We regress model ’s continuous solve rate on benchmark onto its three group scores via logistic regression:
where , which keeps predicted solve rates between 0 and 1. Each (model, benchmark) pair is weighted by the number of tasks in the benchmark. Pairs from the same benchmark are correlated, so we cluster standard errors by benchmark.
CCI derivation. A benchmark is harder when its features lower the predicted solve rate, so its raw complexity score is the weighted sum of its group scores with the sign flipped:
Finally, we min-max scale the raw scores across the eight reference benchmarks:
New repositories and tiers (including BuggyMLEBench-Lite) are projected onto this scale using the fixed , , and min–max bounds from the eight reference benchmarks without refitting.
This four-parameter specification survives every ablation we tried: alternative preprocessing transformations, alternative link functions, and leave-one-benchmark-out (LOBO) cross-validation (Figure 2). It also beats both single-group subsets and richer parameterizations among specifications that predict valid solve rates (Appendix § 8.§ 2). Adding model identity, whether as fixed effects or as an external intelligence index, makes out-of-sample predictions worse, because some benchmarks are completely saturated and others not at all (Appendix § 8.§ 4).
Figure 2. The CCI robustly predicts average solve rates across eight public coding benchmarks. (a) Linear regressions of average solve rate on CCI: both the full fit (blue, ) and all eight leave-one-benchmark-out refits (lavender) slope downward consistently. (b) Each held-out benchmark’s actual pass rate versus the pass rate predicted from the remaining seven benchmarks stays close to the identity line (MAE = 0.048); each benchmark retains the same color and marker across both panels.
Remark 1 (Patch features dominate benchmark difficulty). Bug/patch complexity is the primary driver of benchmark difficulty. Its coefficient is the largest in magnitude of the three, ahead of repo scale and about three times that of dependency complexity in CCI (Appendices A.2 and A.3; Appendix Figure 10). Intuitively, even a large repository remains tractable when a fix is localized, whereas larger patches require diagnosing and coordinating edits across multiple files and hunks.
Remark 2 (BuggyMLE leverages CCI to control task difficulty). All models follow the CCI trend (Figure 1b). Regardless of model scale, solve rates drop predictably as we increase CCI with our BuggyMLE framework, not only validating our CCI model but also enabling the construction of tasks at any target difficulty.
BuggyMLE
BuggyMLE is a four-stage, configuration-driven pipeline that turns clean machine learning repositories into graded debugging tasks and evaluates agent repairs (Figure 3). YAML configurations set task selection, mutation budgets, model backends, and grading criteria for each stage, and stages run in parallel for large-scale generation:
Figure 3. BuggyMLE data curation workflow: every multi-bug repository is built from a vetted clean repository and decomposed into verified, individually impactful bugs. (a) Repository curation: an auditor keeps candidate ML repositories that run fast, give a stable metric across runs, and have a responsive metric (one that moves when the code changes). (b) Bug injection: an injector agent turns the clean repository into a mutated one at a target complexity (the CCI slider); a mutation is kept only if automated checks confirm it changes ML performance. (c) Bug attribution: in two passes, an oracle first cumulatively reverts mutations until baseline performance recovers (identifying a minimal reverting set), then applies each mutation alone onto the clean repository to measure isolated ML performance against the clean baseline (dashed), confirming impactful bugs and marking no-effect edits as decoys. (d) Evaluation: a solver’s fixed repository is graded against the clean repository and the oracle’s findings for correctness, recovered ML performance, and cheating (anti-bypass checks and an oracle-grounded LLM judge).
Injector: An LLM agent explores a clean ML repository inside a Docker sandbox and introduces breaking changes across the pipeline, subject to automated verification checks.
Oracle: Starting from the diff between the clean and buggy repositories, an ablation procedure isolates which mutations actually drive performance degradation.
Solver: A debugging agent receives the buggy codebase and tries to diagnose and repair the bugs without the clean reference code or any bug specification.
Evaluator: The repaired candidate solution is executed in a clean container and graded on quantitative performance recovery and on an oracle-guided code review.
Bug injection and oracle attribution. The Injector works without a fixed bug taxonomy. It inspects the target repository and introduces plausible bugs wherever they fit, in data loading, preprocessing, model architecture, training, or evaluation code. Difficulty comes from the mutation budget (the required number of modified files, hunks, and lines), so task complexity can be raised while the dataset and codebase stay fixed (Appendix § 8.§ 5).
Deterministic scripts in the Docker sandbox validate every candidate injection against three criteria. (a) The diff against the clean repository must meet the mutation budget across files, hunks, and lines. (b) End-to-end execution must either crash or degrade the evaluation metric by more than three standard deviations of the clean repository’s run-to-run variance, or by at least 10% when that variance is unmeasured. (c) If execution completes, it must still produce a valid metric. A failed injection goes back to the agent with diagnostic feedback, up to five times.
A multi-hunk injection can mix benign edits with true bugs. The Oracle separates them in two passes (Algorithm 1, Appendix § 9.§ 6). It first reverts candidate mutations in ranked order until baseline performance returns, which identifies a minimal reverting set. It then applies each mutation alone to the clean repository and measures its isolated effect on the metric. The result is a verified checklist of impactful mutations, kept apart from inert decoy mutations, for downstream grading.
Agent harness. Injector, Oracle, and Solver share one execution harness inside a GPU-enabled Docker container. Agents interact through a multi-turn tool-calling loop with unrestricted bash and file-editing tools (read_file, write_file, modify_file, and list_dir), enabling arbitrary multi-file edits and script execution within a fixed turn budget (Figure 3; Appendix § 12.§ 1).
Evaluation framework. Candidate repairs are graded by execution-based checks and oracle-grounded semantic verification. The execution check runs the solver’s modified repository in a fresh container and records the output metric (scoring 0 if execution fails). The metric check then measures the recovery ratio:
where and denote the performance of the clean and repaired repositories. A candidate achieves full repair only if it satisfies three conditions:
Metric recovery: recovery .
Routing integrity: The repository’s entry point still executes the previously mutated modules, so an agent cannot bypass broken components.
Per-bug verification: Grounded in the Oracle’s verified bug records (ignoring decoys), an LLM judge inspects each bug site against its ground-truth diff and mechanism, confirming it was resolved by a genuine fix or refactor rather than a symptom-masking workaround. BuggyMLEBench-Lite grading requires every verified bug to pass, whereas RL rewards use the fraction of resolved sites as a dense training signal (§ 3.3; judge configurations in Appendix § 12.§ 3).
Beyond binary solve rates, we report progress , the fraction of the degradation recovered, and the fraction of tasks where strictly improves over . For BuggyMLEBench-Lite tasks we also map to its Kaggle leaderboard percentile.
Curriculum Learning with BuggyMLE
Because BuggyMLE can synthesize tasks at any target difficulty by scaling the injected hunk budget on the same underlying codebase, we can construct a difficulty-graded curriculum while holding the repository distribution and modalities fixed across 200 training repositories (Figure 4). Using this capability, our All Difficulties recipe trains Qwen3.5-9B (Qwen Team, [38]) with low-rank adaptation (LoRA) adapters in two sequential phases across the full complexity spectrum: SFT followed by RL (Appendix § 11). The SFT phase grounds the model in the sandbox harness using 2,724 evaluator-verified repair trajectories across all difficulty bands from five frontier teacher models, including Gemini 3.6 Flash and Gemini 3.7 Flash (Google DeepMind, [17]), supplemented via rejection-sampling fine-tuning (RFT) on 411 verified rollouts from a teacher-tuned Qwen3.8-27B (Qwen Team, 2026d) model to incorporate explicit reasoning traces (Appendix § 9). In practice, because issuing separate per-bug judge calls across thousands of multi-bug rollouts is prohibitively expensive when curating training data at scale, we filter candidate SFT trajectories (as well as our 200-task scaling sweep; Appendix § 8.§ 5) with a single repository-level LLM review conditioned on the full Oracle report (Appendix § 12.§ 3), rejecting patches that edit only decoys, bypass faulty logic, or introduce new defects.
Figure 4. Distribution of tasks across data modalities. Proportion of tasks by modality for (a) SFT + RFT training, (b) RL training, and (c) the BuggyMLEBench-Lite evaluation benchmark, where image tasks form the largest share. Detailed recipe statistics are provided in Appendix Figure 15.
Initialized from this SFT checkpoint, we apply Group Relative Policy Optimization (GRPO) [40] across a three-stage curriculum of increasing difficulty on a fixed pool of 40 repositories: Easy (1–5 hunks), Medium (10–12), and Hard (30–34). Each rollout runs as a single interactive episode without test-time search and is scored using all evaluation criteria from § 3.2: continuous metric progress, execution and routing-integrity gates, and per-bug resolution judged against the injected defects (formalized in Appendix § 11). Incorporating every criterion into the reward prevents superficial workarounds and enables the policy to keep climbing on residual or silent bugs even when raw ML performance practically saturates.
BuggyMLEBench-Lite
Because MLE-Bench Lite [9] provides only a raw dataset and grading script for each of its 22 competitions rather than an existing codebase, we first construct and verify a modular baseline repository for every competition prior to bug injection. Using BuggyMLE, we instantiate four complexity tiers across these repositories. The 1 Bug, 5 Bugs, and 10 Bugs tiers contain the corresponding number of verified defects per repository, whereas the 50 Bugs tier first expands each repository to roughly twice its original size before injecting 23–60 bugs (Appendix § 9). For the first three tiers, oracle verification confirms that reverting all injected mutations exactly restores the baseline grading score on every task. To prevent data contamination, no evaluation datasets or repositories appear in the training corpus.
Empirical validation confirms that nearly all injected bugs significantly degrade test performance and span every stage of the ML pipeline (Figure 5). Furthermore, the vast majority of defects are rated as realistic by automated and human judges, with very few flagged as implausible (Appendix Figure 17). In practice, many injected defects resemble routine engineering modifications accompanied by plausible justifying comments (Figure 6); Appendix § 10.§ 5 presents nine additional examples spanning the full spectrum from overt self-sabotage to subtle, realistic modeling flaws.
Figure 5. Distribution of injected bug impacts across complexity tiers, pipeline stages, and defect categories. Proportion of bugs stratified by their isolated impact on the competition test metric. (a) Breakdown across complexity tiers, showing that mild and strong impacts dominate the 1, 5, and 10 Bugs tiers, whereas catastrophic failures are rare and execution crashes occur only at the 50 Bugs tier. (b) Distribution of unique bugs across the 1, 5, and 10 Bugs tiers by ML pipeline stage, and (c) across the six most frequent defect categories (total counts in parentheses; individual bugs may span multiple categories). A bug is classified as impactful if applying it in isolation shifts the graded test score beyond the task’s significance threshold, and is categorized by magnitude as mild (), strong (), or catastrophic (). A crash fails to produce a valid submission file, whereas a decoy alters at most the validation score.
Figure 6. Representative examples of injected benchmark bugs. (a)–(c) Three injected defects from BuggyMLEBench-Lite spanning image, audio, and tabular tasks, showing removed lines (red) and added lines (green), including the inline comments generated during injection to rationalize each modification. Reported scores reflect each defect’s isolated impact on the competition test metric, alongside its 1–5 realism rating assigned by the LLM judge (RQ6, § 5).
Experiments
Trained models. To isolate the effect of curriculum difficulty, we compare our All Difficulties Qwen3.5-9B agent (§ 3.3) against three controlled ablations trained under the same LoRA setup: Easy Only (1–5 hunks; 936 teacher SFT trajectories) and Hard Only (30–34 hunks in RL and in SFT; 628 teacher SFT trajectories), each of which restricts both SFT and GRPO to its respective single difficulty subset, and SFT Only (the SFT + RFT checkpoint of All Difficulties prior to RL). From each RL run, we evaluate the exported checkpoint achieving the highest held-out validation score on its target difficulty band (Appendix § 11.§ 2); corpus breakdowns and hyperparameters appear in Appendix Figure 15 and Appendix § 11.
Baselines. External baselines fall into two tiers by model scale and inference cost (Appendix § 12.§ 2). Budget Models are each provider’s lowest-cost commercial tier or open-weight models of comparable size: Claude Haiku 5.5, Claude Haiku 4.5, Gemini 3.5 Flash-Lite, Qwen3.8-27B without thinking, and our untuned Qwen3.5-9B base model ([1, 2, 16, 38 b]). They are the primary comparison class for our 9B agent. Frontier Models are larger flagship systems (Claude Opus 5.5, Claude Sonnet 5.5, Claude Sonnet 5, Claude Sonnet 4.6, and Gemini 3.8 Flash) ((Anthropic, 2026d,e,f,g; Google DeepMind, 2026d)) and serve as upper-bound reference points.
Evaluation. All models, trained and hosted, run in the same sandbox with the same task prompt (Appendix § 12.§ 1) on the 22 tasks of each BuggyMLEBench-Lite tier. We report three metrics. Full Repair (solve rate) is the percentage of tasks that reach full repair (§ 3.2). Recovery is the mean recovery ratio clipped to [0, 1]. Percentile is the mean Kaggle leaderboard percentile of the repaired code’s submission, scored 0 when no valid submission is produced.
OOD evaluation. Transfer beyond ML debugging is measured on two general coding benchmarks that share nothing with our training corpus. SWE-bench Lite ([25]) poses 300 repository issues, scored by pass@1. LiveCodeBench v6 ([24]) poses 1,055 competitive programming problems, answered in a single turn without tools, with two samples per problem. To probe transfer across a broader task spectrum, Appendix § 13.§ 3 evaluates four additional settings: three that synthesize ML solutions from scratch (as a standalone script without tools, a single file in the sandbox, or a full repository) and DeepSWE ([23]), which benchmarks longer-horizon software issues in general Python repositories.
Results
We organize our empirical evaluation around seven research questions examining how agent performance scales with benchmark difficulty (RQ1), how training curriculum difficulty shapes agent capability (RQ2, RQ3), the distinct contributions of SFT and RL (RQ4), OOD generalization (RQ5), and how bug realism and observability govern task difficulty (RQ6, RQ7).
RQ1: How do agents fare as BuggyMLEBench-Lite gets harder? Consistent with CCI predictions, solve rates decline sharply across difficulty tiers (monotonically for 13 of the 14 models), with the top solve rate dropping from 95% on the easiest tier to 9% on the hardest (Table 2). Model separation is clearest on the 10 Bugs tier, where our 9B All Difficulties agent outperforms Claude Opus 5.5 in end-to-end task solves and trails only Claude Sonnet 5.5. This ordering stems from strict per-bug verification: although Opus 5.5 recovers the target test metric on most tasks, it frequently leaves individual injected bugs unresolved, resulting in a per-bug resolution rate comparable to our 9B agent (Figure 8). At the 50 Bugs tier, end-to-end task resolution nearly vanishes, matching the trajectory predicted by human benchmarks at this CCI level; only Claude Opus 5.5 and Gemini 3.8 Flash complete any task in full. While our agent still resolves the second-highest share of individual bugs behind Sonnet 5.5, residual unfixed bugs suppress its overall metric recovery relative to Opus 5.5, as every 50 Bugs episode of our agent terminates at the turn or context limit (often before producing a valid submission).
| Model | 1 Bug: Full Repair | 1 Bug: Recovery | 1 Bug: Percentile | 5 Bugs: Full Repair | 5 Bugs: Recovery | 5 Bugs: Percentile | 10 Bugs: Full Repair | 10 Bugs: Recovery | 10 Bugs: Percentile | 50 Bugs: Full Repair | 50 Bugs: Recovery | 50 Bugs: Percentile |
| Reference | ||||||||||||
| Clean repository (no bugs) | – | – | 49.1 | – | – | 49.1 | – | – | 49.1 | – | – | 52.0 |
| Frontier Models | ||||||||||||
| Claude Opus 5.5 | 95.5 | 99.8 | 57.2 | 31.8 | 88.4 | 44.6 | 13.6 | 92.2 | 46.7 | 9.1 | 77.8 | 43.9 |
| Claude Sonnet 5.5 | 90.9 | 95.5 | 54.0 | 27.3 | 85.5 | 42.4 | 31.8 | 86.4 | 43.6 | 0.0 | 64.3 | 30.6 |
| Claude Sonnet 5 | 63.6 | 80.2 | 39.2 | 4.5 | 67.5 | 24.2 | 0.0 | 53.7 | 12.6 | 0.0 | 30.2 | 3.5 |
| Claude Sonnet 4.6 | 27.3 | 30.9 | 17.5 | 0.0 | 25.6 | 5.8 | 0.0 | 30.1 | 4.5 | 0.0 | 0.0 | 0.0 |
| Gemini 3.8 Flash | 27.3 | 27.3 | 15.6 | 13.6 | 41.2 | 21.9 | 13.6 | 18.5 | 14.1 | 4.5 | 22.4 | 9.2 |
| Budget Models | ||||||||||||
| Claude Haiku 5.5 | 77.3 | 89.5 | 46.1 | 18.2 | 75.2 | 33.4 | 0.0 | 71.1 | 19.3 | 0.0 | 41.3 | 8.8 |
| Claude Haiku 4.5 | 9.1 | 33.3 | 11.2 | 0.0 | 17.3 | 4.1 | 0.0 | 18.7 | 2.6 | 0.0 | 5.6 | 5.5 |
| Gemini 3.5 Flash-Lite | 4.5 | 8.2 | 3.8 | 0.0 | 8.0 | 2.8 | 0.0 | 6.9 | 1.3 | 0.0 | 7.0 | 5.2 |
| Qwen3.8-27B | 40.9 | 72.7 | 34.9 | 4.5 | 44.0 | 14.8 | 0.0 | 53.1 | 11.0 | 0.0 | 9.8 | 1.6 |
| Qwen3.5-9B | 13.6 | 62.1 | 24.7 | 0.0 | 49.3 | 14.5 | 0.0 | 48.8 | 8.1 | 0.0 | 4.1 | 0.5 |
| Ours: Curriculum Learning with BuggyMLE (9B) | ||||||||||||
| SFT Only (9B) | 50.0 | 70.7 | 36.9 | 4.5 | 62.7 | 22.0 | 0.0 | 43.4 | 9.0 | 0.0 | 7.2 | 1.5 |
| SFT+RL Curriculum: Easy Only (1–5 Hunks) | 50.0 | 73.9 | 35.6 | 0.0 | 54.6 | 12.2 | 0.0 | 48.0 | 8.2 | 0.0 | 9.9 | 5.5 |
| SFT+RL Curriculum: Hard Only (30–34 Hunks) | 22.7 | 69.4 | 34.2 | 18.2 | 67.3 | 31.0 | 9.1 | 59.6 | 19.1 | 0.0 | 12.3 | 1.4 |
| SFT+RL Curriculum: All Difficulties | 72.7 | 81.5 | 43.0 | 22.7 | 77.4 | 37.0 | 18.2 | 81.1 | 40.8 | 0.0 | 14.1 | 5.1 |
Table 2. Evaluation results on the held-out BuggyMLEBench-Lite benchmark (22 tasks per difficulty tier; all values in percentages). Reference denotes the clean baseline repository prior to bug injection. The 50 Bugs tier expands each repository to roughly twice its original size with 23–60 injected bugs per task. Arrows indicate the direction of better performance. Shading, bold (best), and underline (second best) highlight rankings within the budget and 9B model comparison class. Model tiers and metrics are defined in § 4.
Figure 8. Impact of metric recovery, fault localization, observability, and realism on bug resolution. Each row corresponds to one of the ten models audited at the per-bug level, ordered as in Table 2, showing frontier models (lavender), budget baselines (gray), and our trained 9B models (blue). (a) 10 Bugs task solve rate under test-score recovery alone (open circles) versus judge-verified full repair requiring all injected bugs to be fixed (filled circles); the gap indicates tasks where the test metric recovered despite remaining bugs, which is widest for Claude Opus 5.5 and Sonnet 5.5. (b) Proportion of injected bugs modified (light) versus fully resolved (dark), showing that top agents (All Difficulties, Opus 5.5, and Sonnet 5.5) localize nearly all bugs and resolve most modified sites, whereas the 9B base model modifies roughly one-third. (c) Resolution rate for visible bugs (which degrade validation metrics) versus silent bugs (which affect only test predictions), demonstrating consistently lower resolution on silent bugs across all models. (d) Resolution rate stratified by LLM-judged realism score (4–5, 2–3, and 1). Panels (b)–(d) pool tasks across the 1, 5, and 10 Bugs tiers; all x-axes report percentages.
RQ2: Do harder training tasks improve performance on easier tasks? Incorporating high-complexity tasks improves single-bug repair, provided those tasks are embedded within a full curriculum. lum. Specifically, adding high-complexity RL stages raises the 1 Bug solve rate of the All Difficulties agent substantially above both the SFT Only and Easy Only baselines (72.7% vs. 50.0%). Conversely, training exclusively on hard tasks hurts performance on the 1 Bug tier: because of distribution shift, the Hard Only model achieves fewer end-to-end 1 Bug solves (22.7% vs. 50.0%) despite resolving more individual bugs (59.1% vs. 50.0%) than the SFT baseline.
RQ3: Is there any benefit to training on easier tasks? Although training solely on easy tasks fails to generalize upward (with Easy Only solving 0% of tasks on the 5 and 10 Bugs tiers), easy stages serve as an essential foundation for curriculum learning. Including easy stages allows the All Difficulties model to consistently outperform Hard Only on complex tiers, doubling the solve rate on 10 Bugs. Furthermore, easy stages enable a gradual reduction in episode length. However, the sharpest efficiency gains emerge during the budget-constrained hard stage, where the agent learns to complete repairs in substantially fewer turns without sacrificing reward (Figure 7).
Figure 7. High-complexity curriculum stages drive turn efficiency. Training dynamics of the All Difficulties curriculum across Easy (1–5 hunks), Medium (10–12), and Hard (30–34) RL stages (Hard stage shaded). (a) Per-step training reward (gray), 8-step moving average (blue), held-out evaluation score on the corresponding difficulty band (lavender), and the selected evaluation checkpoint (dashed). Reward recovers within each stage and remains stable through the Hard stage. (b) Mean turns per episode with stage-wise linear trends relative to the turn budget (dashed). Episode length remains nearly flat during the Easy and Medium stages but drops sharply during the Hard stage without degrading reward. An earlier turn-budget adjustment during the Hard stage is analyzed in Appendix Figure 22.
RQ4: What do SFT and RL each contribute? SFT primarily establishes reliable tool use, eliminating syntax errors in tool calls and replacing the base model’s full-file rewrites with targeted edits. Subsequent RL training improves fault localization and exploration efficiency. Across the 1, 5, and 10 Bugs tiers, the All Difficulties agent modifies the majority of bug-containing files (compared to fewer than half for SFT Only), executes substantially fewer exploratory throwaway scripts, and achieves higher mean metric recovery at every tier. Pipeline re-execution frequency does not distinguish the models, as the base, SFT Only, and All Difficulties agents all re-run the pipeline following their final edit in most episodes. Appendix Figure 23 illustrates this qualitative shift on a representative 5 Bugs task: whereas the base model repeatedly trains auxiliary models and leaves two bug sites untouched, the RL agent reverts four bugs simultaneously, diagnoses a fifth from a chance-level validation AUC, and resolves it.
RQ5: Does training on BuggyMLE transfer beyond ML debugging? Training on BuggyMLE transfers positively to general software repair: the All Difficulties model resolves significantly more SWE-bench Lite issues than the untuned Qwen3.5-9B base model, whereas the narrower Easy Only and Hard Only curricula underperform the base model (Table 3). On single-turn algorithmic coding (LiveCodeBench v6), SFT initially reduces pass@1 due to domain specialization, but subsequent GRPO training recovers this loss and slightly surpasses the base model. Taken together, these results show that reinforcement learning recovers general capabilities from a degraded SFT checkpoint, and that more diverse curricula can even improve those abilities beyond the base model. However, transfer remains bounded by task paradigm. When tasked with generating ML solutions from scratch, we observe mixed results on Kaggle leaderboards, with our model scoring slightly higher than the base model in some settings and lower in others, though tool-use reliability improves in every interactive setting (Appendix § 13.§ 3). Consistently improving from-scratch ML generation likely requires direct optimization on solution quality, such as a leaderboard-percentile reward.
| Model | Resolved (%) ↑ | Fail-to-Pass Tests (%) ↑ | Tokens / Ep. ↓ | Tool-Call Success (%) ↑ | Δ vs. Qwen3.5-9B | Pass@1 All (%) ↑ | Pass@1 Easy (%) ↑ | Pass@1 Medium (%) ↑ | Pass@1 Hard (%) ↑ | Δ vs. Qwen3.5-9B |
| Base Model | ||||||||||
| Qwen3.5-9B | 24.3 | 26.4 | 794 | 75.5 | – | 69.3 | 97.4 | 76.5 | 35.7 | – |
| Ours: Curriculum Learning with BuggyMLE (9B) | ||||||||||
| SFT Only (9B) | 18.3 | 23.0 | 361 | 95.1 | – | 61.3 | 94.7 | 65.7 | 25.9 | – |
| SFT+RL Curriculum: Easy Only (1–5 Hunks) | 17.0 | 21.7 | 776 | 93.6 | −7.3† | 68.3 | 97.4 | 74.3 | 35.0 | −1.0 |
| Hard Only (30–34 Hunks) | 22.0 | 29.2 | 1428 | 99.2 | −2.3 | 69.8 | 96.9 | 78.2 | 35.6 | +0.4 |
| All Difficulties | 33.0 | 39.5 | 616 | 98.8 | +8.7† | 71.3 | 96.9 | 80.5 | 37.7 | +2.0† |
Table 3. OOD generalization on general coding benchmarks (experimental settings in § 4). Arrows indicate the direction of better performance, with shading, bold, and underline ranked across all rows. Δ denotes the paired difference relative to the untuned base model, where † indicates a 95% bootstrap confidence interval excluding zero.
RQ6: How realistic is our setup, and how does realism affect difficulty? Both automated and expert evaluations confirm that our injected bugs closely mirror real-world ML defects. An LLM judge rates nearly half of the evaluated bugs at 4 or 5 out of 5 for realism, and a domain expert judged 27 of 30 randomly sampled bugs to be plausible in production ML codebases (Appendix Figure 17). Higher realism directly increases repair difficulty, with bugs rated 4 or 5 exhibiting the lowest resolution rates across all evaluated models (Figure 8d). These realistic defects typically represent plausible but suboptimal modeling choices, such as misconfigured label smoothing or aggressive negative subsampling, that degrade performance on a specific dataset without introducing syntactic or logical errors detectable via static inspection.
RQ7: Which bug types are hardest to resolve, and does task modality matter? Bug observability is the primary determinant of repair difficulty. Because official evaluation uses held-out test data, an agent’s sandbox validation metric does not necessarily reflect test performance. We distinguish between visible bugs, which directly degrade the printed validation metric, and silent bugs, which leave validation scores intact and manifest only by corrupting test-time predictions (for instance, a label-indexing off-by-one error that preserves a 0.953 validation score while reducing the test score to 0.017). Across all models, silent bugs are resolved at substantially lower rates than visible ones (Figure 8c). Categorized by pipeline stage, output-formatting and submission defects prove most difficult for all models except those trained on high-complexity curricula (All Difficulties and Hard Only; Appendix Figure 28). By contrast, data modality has minimal impact on frontier agents, with Claude Opus 5.5 and Sonnet 5.5 resolving comparable shares of individual bugs across all four BuggyMLEBench-Lite modalities (Appendix Figure 28c).
Conclusion
We show that code structure, rather than model identity, drives coding benchmark difficulty. The CCI captures this relationship using ten structural features and four parameters, accurately tracking solve rates across eight diverse public benchmarks. Operationalizing this insight, BuggyMLE scales the difficulty of fixed repositories via a controllable mutation budget, yielding BuggyMLEBench-Lite, a repository-level benchmark that remains unsaturated at its highest tiers. Training a 9B agent on an easy-to-hard BuggyMLE curriculum allows it to rival frontier models on 10 Bugs and generalize to broader code repair.
Limitations. While CCI is stable under LOBO evaluation across eight human-curated benchmarks, validating it as a general law requires broader coverage, including independently generated synthetic sets (as BuggyMLE tasks were excluded to prevent circularity). Additionally, CCI relies purely on structural features under agentic evaluation; it does not capture semantic complexity and is less predictive on standalone-program benchmarks (e.g., FrontierCS Algo) than on the repository-scale tasks BuggyMLE targets. Finally, while BuggyMLE training strengthens software engineering and code repair, transfer to from-scratch ML solution generation yields mixed results.
Acknowledgments
We thank the open-source benchmark community for making leaderboard data publicly available.
CCI Validation and Scaling Sweep
Feature Definitions and Benchmark Statistics
To complement § 3.1, Table 4 defines the ten task-level median (p50) structural features grouped into repository scale, dependency complexity, and bug/patch complexity.
| Group | Feature (p50) | Description |
| Repo Scale | lines_of_code | Total source lines |
| Repo Scale | file_count | Number of source files |
| Repo Scale | max_dir_depth | Deepest directory nesting |
| Dependency | dependency_count | Number of inter-file dependencies |
| Dependency | max_dep_chain | Longest dependency chain |
| Patch | files_changed | Files modified by the bug fix |
| Patch | hunks | Number of diff hunks |
| Patch | lines_added | Lines added in the fix |
| Patch | lines_removed | Lines removed in the fix |
| Patch | total_lines_delta | Net line change |
Table 4. Raw features used in CCI.
Table 5 reports the resulting CCI score, task count, task type, language, and mean frontier-model solve rate for each of the eight public coding benchmarks used to fit the index.
| Benchmark | Tasks | Task type | Language | CCI | Solve rate |
| SWE-bench Verified | 500 | Repo issue repair | Python | 0.00 | 0.71 |
| FrontierCS Research | 61 | Research code generation | Python | 0.31 | 0.46 |
| FrontierCS Algo | 188 | Algorithmic | C++ | 0.38 | 0.39 |
| SWE-bench Pro | 731 | Long-horizon repo repair | Multi | 0.48 | 0.31 |
| FrontierSWE | 17 | From-scratch engineering | Multi | 0.58 | 0.30 |
| DeepSWE | 113 | Long-horizon engineering | Multi | 0.68 | 0.27 |
| SWE-Marathon | 17 | Long-horizon project engineering | Multi | 0.69 | 0.11 |
| ProgramBench | 201 | From-scratch reconstruction | Multi | 1.00 | 0.00 |
Table 5. Benchmarks ordered by CCI. Solve rates average all evaluated frontier models. SWE-Marathon counts the 17 of its 20 tasks with readable reference code.
Pipeline Ablations and LOBO Stability
We ablate each design choice of the CCI pipeline across all eight public benchmarks (105 model–benchmark pairs) and verify that alternative specifications leave our conclusions unchanged.
Log transformation. Applying reduces skewness on all 10 raw features (Table 6). Refitting the regression on untransformed z-scores preserves the sign of every group coefficient, confirming that log transformation improves numerical conditioning without altering qualitative findings.
| Feature () | Raw skew | skew | Improved? |
| lines_of_code | 1.70 | −0.64 | yes |
| file_count | 1.46 | −0.05 | yes |
| max_dir_depth | 0.27 | −0.07 | yes |
| dependency_count | 1.14 | −0.10 | yes |
| max_dep_chain | 2.23 | 1.26 | yes |
| files_changed | 2.66 | 1.32 | yes |
| hunks | 2.25 | 0.38 | yes |
| lines_added | 1.94 | −0.02 | yes |
| lines_removed | 0.79 | 0.44 | yes |
| total_lines_delta | 1.94 | −0.01 | yes |
Table 6. Skewness before and after .
Feature grouping. Because features within the same structural dimension covary strongly, fitting an ungrouped ten-feature logistic regression suffers from severe multicollinearity. Averaging z-scores within the three conceptual groups reduces the maximum pairwise correlation from 1.00 to 0.82 (Figure 9), with balanced variance contributions across constituent features.
Figure 9. Grouping the ten structural features into three dimensions eliminates severe within-group collinearity. Pairwise Pearson correlations across the eight public coding benchmarks (log-transformed task medians; blue: positive, lavender: negative). (a) Raw features exhibit near-collinearity within conceptual blocks (e.g., file_count vs. max_dir_depth, ; files_changed vs. hunks, ), which destabilizes an ungrouped regression. (b) Aggregating into Repo Scale, Dependency, and Bug Patch z-score averages lowers the maximum off-diagonal correlation to while keeping Bug Patch nearly orthogonal to both repository-level groups ( and −0.03).
Link-function robustness. Both weighted least squares (WLS) with an identity link and a binomial generalized linear model (GLM) with a probit link yield identical coefficient signs to our logistic specification. Because binomial counts across tasks exhibit over-dispersion (Pearson ), naive parametric standard errors underestimate uncertainty; we therefore evaluate generalization via non-parametric LOBO cross-validation rather than in-sample -values.
LOBO coefficient stability. Given eight reference benchmarks, a natural concern is whether a single outlier benchmark drives the regression. LOBO refits rule this out: holding out any benchmark preserves all three coefficient signs and relative magnitudes (Figure 10), and held-out pass-rate predictions remain tightly aligned with the identity line (Figure 2).
Figure 10. All three CCI group coefficients preserve their sign and relative magnitude across every LOBO refit. Binomial GLM log-odds coefficients when each of the eight benchmarks is held out in turn (lavender dots and horizontal offsets, exact values at right) compared against the full eight-benchmark fit (vertical blue line; zero shown dashed) for (a) Repo Scale (+0.64), (b) Dependency Complexity (−0.34), and (c) Bug/Patch Complexity (−0.99). No single benchmark drives the fit or flips any coefficient’s sign, and Bug/Patch Complexity remains the dominant predictor in every refit (−0.91 to −1.18).
Model-specification ablation. Table 7 compares nine regression specifications by held-out LOBO mean absolute error. Among specifications whose predictions remain valid solve rates in [0, 1], the four-parameter, complexity-only logistic model achieves the lowest out-of-sample error (MAE = 0.115). OLS reaches 0.105 only because 10 of its 82 held-out predictions (over models appearing in at least two benchmarks) fall below zero and are clipped to the 0% observed on ProgramBench. Adding per-model fixed effects more than doubles held-out error (0.239), underperforming even an intercept-only baseline (0.227), because leaderboard model subsets vary across benchmarks and overfit benchmark-level intercepts; we therefore exclude model identity from CCI. Because CCI is model-agnostic, per-(model, benchmark) pair MAE in Table 7 (0.115) reflects within-benchmark spread across models, whereas benchmark-average pass rates in Figure 2b achieve MAE = 0.048; likewise, (Figure 10a) acts as a conditional adjustment for suites with large repositories alongside small localized patches (such as SWE-bench Verified).
| Approach | Params | Predicts in [0, 1] | LOBO MAE |
| 3 complexity groups (logistic regression) | 4 | ✓ | 0.115 |
| + model-family fixed effects | ∼17 | ✓ | 0.130 |
| Bug patch only | 2 | ✓ | 0.166 |
| Patch + dependency | 3 | ✓ | 0.167 |
| + intelligence index | 5 | ✓ | 0.173 |
| Intercept only (baseline) | 1 | ✓ | 0.227 |
| + per-model fixed effects | ∼54 | ✓ | 0.239 |
| OLS + 3 groups | 4 | ✗ | 0.105 |
| WLS + 3 groups | 4 | ✗ | 0.133 |
Table 7. LOBO prediction error by model specification. MAE: mean absolute error of the held-out solve rate; OLS: ordinary least squares; WLS: weighted least squares. Linear specifications can predict solve rates outside [0, 1] and are clipped before scoring. Bold: lowest error among specifications whose predictions stay within [0, 1].
Feature Group Contributions
To isolate the predictive power of each structural dimension in § 3.1, Figure 11 regresses average benchmark solve rate onto each group score individually.
Figure 11. Bug/patch complexity alone explains three-quarters of benchmark difficulty variance, and combining all three groups reaches . Average benchmark pass rate across evaluated models (gray points, directly labeled) regressed via OLS (blue line) onto each min–max scaled score in [0, 1]: (a) Repo Scale (), (b) Dependency Complexity (), (c) Bug/Patch Complexity (), and (d) the full three-group CCI (). Repository size and dependency structure are weak standalone predictors because small single-file algorithmic tasks (e.g., FrontierCS Algo) can still be difficult, whereas combining them with patch scope aligns all eight benchmarks tightly along a single linear frontier.
Bug/patch complexity accounts for the largest share of variance (), reflecting the difficulty of coordinating edits across multiple files and diff hunks. Dependency complexity and repository scale are weak standalone predictors ( and 0.047) because compact single-file algorithmic benchmarks remain challenging, yet both provide complementary signal that raises the joint fit to .
Model Capability vs. Benchmark Difficulty
We next test whether cross-benchmark pass-rate variation can be explained by differences in the evaluated models’ general capability rather than by code complexity. Within any single benchmark, stronger models achieve higher pass rates (Figure 12), but pooled across benchmarks an external Model Intelligence Index explains virtually none of the variance ().
Figure 12. Model capability discriminates within individual benchmarks but explains none of the variance across benchmarks (). Each point plots one (model, benchmark) pair ( pairs with an available index) across the eight public coding benchmarks (distinct marker and color per benchmark), comparing external Model Intelligence Index against benchmark pass rate. Although higher-intelligence models generally score higher within a fixed benchmark, the pooled least-squares trend (dashed black line) is flat because benchmark-level floor and ceiling effects (from near-saturated SWE-bench Verified down to unsaturated ProgramBench) dominate cross-benchmark variation.
Floor and ceiling regimes. Tasks requiring small, localized patches saturate across nearly all frontier models, whereas repairs spanning large, coupled repositories remain near zero even for the strongest systems. Structural code complexity sets these benchmark-level ceilings and floors; model capability primarily differentiates performance within the intermediate regime.
Controlled 200-Task Scaling Sweep
To verify that CCI predicts difficulty when repository identity and bug injector are controlled, we evaluate six commercial solvers across 200 BuggyMLE repositories (190 synthetic Kaggle tasks and 10 PyHealth tasks, with 90 PyHealth rollouts per cell) injected at target budgets of 1, 10, and 30 diff hunks by a fixed Claude Opus 4.8 injector (Figure 13). Increasing the hunk budget monotonically lowers the fitted solve rate of every solver. Four solvers track above the public-benchmark trend and two track below it, providing a calibrated reference curve for setting generation budgets.
Figure 13. Scaling the injected hunk budget on a fixed 200-task pool predictably lowers solve rates across six frontier and budget solvers. Solve rate versus projected CCI at 1, 10, and 30 injected diff hunks (1h, 10h, 30h vertical guides), with Claude Opus 4.8 held fixed as injector, oracle, and reviewer. Black line and gray dots show the eight public human benchmarks and their linear fit (); colored markers and least-squares lines track each solver across hunk budgets (solid: Claude family; dashed: Gemini family). A task counts as solved only when it achieves full metric recovery and the oracle-grounded reviewer confirms that the verified bugs are fixed. Gemini 3.6 Flash’s 30-hunk cell is omitted due to incomplete coverage (Appendix § 8.§ 5).
Run completeness. Throughout the solver sweep, Claude Opus 4.8 serves as the fixed injector, oracle, and code reviewer so that only the solver varies across runs. Cells truncated early by provider rate limits or safety filters are evaluated over completed trajectories with conclusive verdicts, and Gemini 3.6 Flash’s incomplete 30-hunk synthetic cell is omitted from regression fits.
Criterion sensitivity. To test whether downward CCI scaling depends on how strictly repairs are graded, Figure 14 evaluates the solver sweep under three nested success criteria: metric recovery alone, metric recovery plus oracle-grounded confirmation that the patch fixes the verified bugs, and additionally requiring a principally valid repair (§ 3.2). Requiring causal bug resolution removes trajectories that restore the target metric without repairing the underlying defects; for example, at 10 synthetic hunks under strict baseline recovery (Figure 14d–Figure 14f), 69% of Claude Opus 5 runs recover the clean baseline metric, whereas 53% neutralize the verified mutations (and 98% vs. 78% under a relaxed threshold). Further requiring principal validity changes solve rates by only a few percentage points and leaves solver rankings unchanged, confirming that low solve rates at high CCI reflect genuine repair difficulty rather than overly strict grading.
Figure 14. Negative CCI scaling and solver rankings hold across evaluation strictness levels and repository sources. Solver solve rates at 1, 10, and 30 injected hunks across human-authored PyHealth repositories (top, (a)–(c); 90 tasks per cell) and synthetic Kaggle repositories (bottom, (d)–(f); 190 tasks per cell) as the success criterion is progressively tightened: (a, d) metric recovery alone (Metric Recovery), (b, e) additionally requiring the oracle-grounded reviewer to confirm that the patch fixes the verified bugs (+ Fix Addresses Bug), and (c, f) further requiring a principally valid repair (+ Principally Valid; § 3.2). Filtering for causal bug resolution removes superficial metric recoveries without altering the downward CCI slope, while requiring principal validity changes solve rates minimally and preserves solver ordering. Black line and gray dots denote the human-benchmark reference frontier; incomplete runs are excluded (Appendix § 8.§ 5).
Dataset Splits and Oracle Audit
Across this study, BuggyMLE generates 6,674 bug-injected variants across 222 distinct repositories (the 200 training repositories plus the 22 BuggyMLEBench-Lite benchmark repositories), including 4,431 solver-sweep, 2,152 SFT-data-generation, and 88 BuggyMLEBench-Lite variants. In total, these mutations comprise nearly 260K lines of injected code (259,583 net code lines, excluding comments, docstrings, blank lines, and the injector’s scratch test.py scripts and generated artifacts). All training trajectories are generated from the 200 training repositories (190 synthetic Kaggle tasks and 10 PyHealth tasks), which share no repository, code, or dataset with BuggyMLEBench-Lite. Sixteen base repositories were excluded prior to training because automated evaluator audits flagged every trajectory on them for suspected target leakage, evidenced by anomalously high metric scores (Figure 15).
Figure 15. Training corpus statistics: most repositories supply SFT data, and trajectories span all three difficulty bands. (a) Allocation of the 200 training repositories: 142 supply SFT trajectories, 37 are held out for validation, 5 are used exclusively in RL, and 16 are excluded prior to training. (b) Distribution of the 3,135 SFT + RFT trajectories across bug-difficulty bands: 1,328 easy (≤5 hunks; 42%), 1,100 medium (6–29 hunks; 35%), and 707 hard (≥30 hunks; 23%); the Easy Only (936 rows) and Hard Only (628 rows) corpora each train on a single band.
Held-Out Partition
Tasks are ranked by sha256(task name) within each run family, and the lowest 20% per family are held out (37 tasks in total; Table 8). We partition strictly at the repository level rather than the trajectory level because multiple bug instances are generated per base repository; our dataset build pipeline programmatically enforces zero task overlap with both the 37 held-out tasks and BuggyMLEBench-Lite.
| task | source |
| airlines-dataset-to-predict-a-delay | Synthetic Kaggle |
| arabic-company-reviews | Synthetic Kaggle |
| bbc-news-articles | Synthetic Kaggle |
| car-dataset | Synthetic Kaggle |
| civil-engineering-global-project-dataset | Synthetic Kaggle |
| corona-virus-report | Synthetic Kaggle |
| covid19-in-usa | Synthetic Kaggle |
| covid19cxr_resnet_dev | PyHealth |
| daily-power-generation-in-india-20172020 | Synthetic Kaggle |
| drug_rec_mimic4_retain | PyHealth |
| e-coli-resistance-dataset | Synthetic Kaggle |
| ecommerce-order-dataset | Synthetic Kaggle |
| ethereum-price-data-binance-api-2017now | Synthetic Kaggle |
| fakenews-csv | Synthetic Kaggle |
| formula-1-race-data | Synthetic Kaggle |
| fragrantica-com-fragrance-dataset | Synthetic Kaggle |
| global-superstore-dataset | Synthetic Kaggle |
| google-2019-cluster-sample | Synthetic Kaggle |
| hourly-weather-surface-brazil-southeast-region | Synthetic Kaggle |
| icc-mens-odi-world-cup-2003-2023 | Synthetic Kaggle |
| indian-agriculture-dataset | Synthetic Kaggle |
| legal-citation-text-classification | Synthetic Kaggle |
| london-homes-energy-data | Synthetic Kaggle |
| los_regression_mimic4_rnn | PyHealth |
| medquad | Synthetic Kaggle |
| mlcourse | Synthetic Kaggle |
| mortality | Synthetic Kaggle |
| p2p-bob-exchange | Synthetic Kaggle |
| patient | Synthetic Kaggle |
| russia-real-estate-2021 | Synthetic Kaggle |
| sf-bay-area-bike-share | Synthetic Kaggle |
| tmdb-movies-daily-updates | Synthetic Kaggle |
| tmdb-movies-dataset-2023-930k-movies | Synthetic Kaggle |
| twitter-climate-change-sentiment-dataset | Synthetic Kaggle |
| ufc-datasets-1994-2025 | Synthetic Kaggle |
| us-collegiate-sports-dataset | Synthetic Kaggle |
| weather-long-term-time-series-forecasting | Synthetic Kaggle |
Table 8. The 37 held-out tasks and the source family each comes from.
SFT Corpora
Each SFT example is a complete multi-turn repair trajectory with tool outputs truncated identically to the RL harness (Appendix § 12.§ 1). A teacher rollout is retained only if it passes independent Evaluator verification (metric recovery and preserved execution routing through the mutated modules), terminates via an explicit finalize call, and originates from a non-held-out repository. The RFT corpus is sampled from Qwen3.8-27B after SFT on training tasks under the same filtering criteria, capped at four trajectories per bug instance. Easy Only and Hard Only replace both SFT stages with a single teacher corpus restricted to their target hunk band (Table 9 and Table 10). Validation sets are drawn exclusively from the training side of the held-out partition.
| corpus | trains | source | rows | tasks | hunks | reasoning | gate | validation |
| Teacher | All Difficulties, step 1 | teacher models | 2,724 | 128 | 1–52 (10) | 0 | 98,304 | 85 / 14 |
| RFT | All Difficulties, step 2 | 27B sibling | 411 | 61 | 1–34 (5) | 250 | 98,304 | 34 / 19 |
| Easy Only ( hunks) | Easy Only | teacher models | 936 | 130 | 1–5 (1) | 0 | 49,152 | 63 / 12 |
| Hard Only ( hunks) | Hard Only | teacher models | 628 | 107 | 30–52 (31) | 0 | 98,304 | 59 / 34 |
Table 9. The four SFT corpora. Hunks: range (median) per trajectory. Reasoning: trajectories with a reasoning block. Gate: longest trajectory kept, in tokens. Validation: trajectories / tasks.
Table 10. Teacher trajectories by teacher model for the Teacher and Hard Only corpora (the Easy Only corpus draws from the same five teacher models).
RL Pools
RL training uses three difficulty-stratified instance pools constructed over the same 40 base repositories, holding repository identity and data modality constant across difficulty bands (Table 11 and Table 12). Each band’s held-out validation pool comprises distinct bug injections on a subset of those repositories, measuring generalization to unseen defects within familiar codebases. Thirty-two of the 40 repositories also contribute distinct bug instances to the teacher SFT corpus; none belong to the held-out tasks or BuggyMLEBench-Lite.
| band | train instances | train hunks | eval instances | eval repositories | eval hunks |
| 1–5 hunks | 40 | 1–5 (1) | 31 | 26 | 1–5 |
| 10–12 hunks | 40 | 10–12 (10) | 32 | 23 | 10–11 |
| 30–34 hunks | 40 | 30–34 (32) | 29 | 25 | 30–32 |
Table 11. The three RL pools. Hunks: range and median of injected diff hunks per instance.
| repository | modality | 1–5 | 10–12 | 30–34 |
| agriculture-crop-images | image | 1 | 11 | 32 |
| agriculture-crop-yield | tabular | 1 | 10 | 31 |
| arabic-customer-reviews | text | 5 | 10 | 32 |
| arabic-sentiment-twitter-corpus | text | 1 | 12 | 32 |
| bbc-news | text | 1 | 10 | 31 |
| bitcoin-tweets | text | 1 | 10 | 32 |
| blinkit-sales-dataset | time series | 1 | 11 | 34 |
| capgemini-employee-reviews-dataset | text | 1 | 10 | 31 |
| cardiovascular-disease | tabular | 3 | 10 | 30 |
| cell-towers-worldwide-location-data-by-continent | geospatial | 1 | 10 | 32 |
| chatgpt-reddit | text | 1 | 11 | 32 |
| city-bike-travels-dataset | tabular | 1 | 12 | 31 |
| continuous-groundwater-level-measurements-2023 | signal | 1 | 11 | 30 |
| cow-muzzle-dataset | image | 5 | 10 | 31 |
| cyclistic-rideshare-data-2024 | geospatial | 1 | 10 | 32 |
| environmental-sensor-data-132k | signal | 5 | 10 | 33 |
| european-flights-dataset | time series | 1 | 10 | 33 |
| facenet-keras | image | 5 | 10 | 32 |
| flight-delay-and-causes | tabular | 2 | 10 | 33 |
| football-players-data | tabular | 1 | 10 | 31 |
| heart-attack-prediction-in-indonesia | tabular | 1 | 10 | 32 |
| himalayan-expeditions | tabular | 1 | 11 | 31 |
| hospital-length-of-stay-dataset-microsoft | tabular | 1 | 10 | 32 |
| ibovespa-stocks | time series | 1 | 10 | 32 |
| indian-pharmaceutical-products | tabular | 1 | 10 | 32 |
| indian-premier-league-ipl-all-seasons | tabular | 1 | 10 | 32 |
| it-service-ticket-classification-dataset | text | 1 | 10 | 31 |
| jena-climate | time series | 1 | 10 | 32 |
| lg-18650hg2-liion-battery-data | signal | 5 | 10 | 30 |
| lianjia | geospatial | 5 | 11 | 31 |
| medium-post-titles | text | 1 | 10 | 32 |
| mens-t20i-cricket-complete-dataset | tabular | 5 | 10 | 31 |
| paper-keyword | text | 1 | 11 | 32 |
| philippine-major-cities-air-quality-data | time series | 1 | 10 | 33 |
| pizza-sales | time series | 1 | 10 | 30 |
| presidential-elections-poland-2025-official-data | time series | 1 | 10 | 30 |
| skin-cancer-malignant-vs-benign | image | 1 | 10 | 32 |
| social-media-surveillance | text | 2 | 11 | 33 |
| stanford-background-dataset | image | 1 | 12 | 31 |
| wesad-full-dataset | signal | 5 | 11 | 31 |
Table 12. The 40 RL repositories and the injected diff hunks of the training instance each contributes to the 1–5, 10–12, and 30–34-hunk pools.
BuggyMLEBench-Lite Difficulty Ladder
Each of the 22 BuggyMLEBench-Lite competitions begins from a modular clean repository paired with an initial multi-bug injection (Lite-Original). We construct controlled easier tiers by iteratively reverting one bug at a time, re-evaluating after each revert, and snapshotting the states where 10, 5, and 1 verified impactful bugs remain, forming the 10 Bugs, 5 Bugs, and 1 Bug tiers (Table 13). On 11 of the 22 competitions, the clean reference repository exceeds the human Kaggle leaderboard median; on the remaining 11, full repair restores the clean baseline without surpassing that median. The 50 Bugs tier is constructed separately by expanding each clean repository to approximately twice its original codebase size and injecting 23–60 bugs per competition, so its clean reference scores are reported separately.
| competition | Lite-Original | 10 Bugs | 5 Bugs | 1 Bug |
aerial-cactus-identification{sup}†` | 18 | 10 | 5 | 1 |
aptos2019-blindness-detection | 24 | 13 | 5 | 1 |
denoising-dirty-documents{sup}†` | 22 | 15 | 8 | 2 |
detecting-insults-in-social-commentary{sup}†` | 19 | 12 | 6 | 2 |
dog-breed-identification | 33 | 23 | 9 | 2 |
dogs-vs-cats-redux-kernels-edition | 27 | 17 | 6 | 2 |
histopathologic-cancer-detection{sup}†` | 27 | 15 | 8 | 1 |
jigsaw-toxic-comment-classification-challenge | 25 | 17 | 9 | 2 |
leaf-classification{sup}†` | 32 | 20 | 9 | 2 |
mlsp-2013-birds | 17 | 11 | 5 | 1 |
new-york-city-taxi-fare-prediction | 23 | 11 | 5 | 1 |
nomad2018-predict-transparent-conductors-v2{sup}†` | 28 | 17 | 10 | 2 |
plant-pathology-2020-fgvc7{sup}†` | 30 | 17 | 8 | 2 |
random-acts-of-pizza{sup}†` | 26 | 17 | 8 | 3 |
ranzcr-clip-catheter-line-classification | 26 | 19 | 8 | 4 |
siim-isic-melanoma-classification | 51 | 28 | 12 | 1 |
spooky-author-identification{sup}†` | 27 | 16 | 9 | 1 |
tabular-playground-series-dec-2021 | 35 | 23 | 10 | 1 |
tabular-playground-series-may-2022{sup}†` | 28 | 19 | 8 | 2 |
text-normalization-challenge-english-language | 19 | 14 | 8 | 3 |
text-normalization-challenge-russian-language | 18 | 13 | 6 | 1 |
the-icml-2013-whale-challenge-right-whale-redux{sup}†` | 25 | 17 | 6 | 1 |
| total | 580 | 364 | 168 | 38 |
Table 13. BuggyMLEBench-Lite tasks and injected diff hunks per tier. †: clean repository above the leaderboard median.
Split Disjointness
Table 14 verifies pairwise task disjointness between every training and validation split, the held-out task set, and BuggyMLEBench-Lite. For Easy Only SFT, disjointness is enforced and verified at corpus construction time from its task manifest and validation split.
| set | tasks | held-out | BuggyMLEBench-Lite |
| Teacher SFT (train) | 128 | 0 | 0 |
| Teacher SFT (validation) | 14 | 0 | 0 |
| RFT (train) | 61 | 0 | 0 |
| RFT (validation) | 19 | 0 | 0 |
| Easy Only SFT (validation) | 12 | 0 | 0 |
| Hard Only SFT (train) | 107 | 0 | 0 |
| Hard Only SFT (validation) | 34 | 0 | 0 |
| RL pools (training) | 40 | 0 | 0 |
| RL pools (evaluation) | 37 | 0 | 0 |
| held-out tasks | 37 | – | 0 |
Table 14. Task overlaps with the held-out tasks and BuggyMLEBench-Lite.
Oracle Attribution and Repair Audit
Algorithm 1 details the two-pass Oracle verification procedure summarized in § 3.2, including diff harvesting, hallucination filtering, the greedy reverting-set search (Phase A), and per-mutation forward ablation against the clean baseline equivalence band (Phase B).
Reverting all injected patches yields the oracle repair, which serves as the ceiling reference for metric recovery; we therefore verify that oracle repairs reproduce clean baseline performance across both corpora.
Training corpus. Each candidate instance enters the training corpus only after passing two containerized checks: reverting all injected bugs must restore the clean baseline metric to within three standard deviations of run-to-run variance, and applying each bug in isolation must measurably degrade the metric (non-impactful mutations are discarded). All 212 RL instances passed both checks.
BuggyMLEBench-Lite. Reverting a tier’s injected patches reconstructs the clean repository, and the official MLE-Bench grader reproduces the clean baseline score across all 22 competitions to within 0.007% (Table 15).
Table 15 | Oracle repair on BuggyMLEBench-Lite. Clean: the graded score of the clean repository. Oracle: the graded score after every injected bug is reversed. : their relative difference, in percent. Arrows give the metric’s better direction.
| competition | clean | oracle | (%) | |
| aerial-cactus-identification | ↑ | 0.99992 | 0.99992 | 0.000 |
| aptos2019-blindness-detection | ↑ | 0.83104 | 0.83104 | 0.000 |
| denoising-dirty-documents | ↓ | 0.04195 | 0.04195 | 0.002 |
| detecting-insults-in-social-commentary | ↑ | 0.80907 | 0.80907 | 0.000 |
| dog-breed-identification | ↓ | 3.06187 | 3.06187 | 0.000 |
| dogs-vs-cats-redux-kernels-edition | ↓ | 0.15686 | 0.15686 | 0.000 |
| histopathologic-cancer-detection | ↑ | 0.95185 | 0.95185 | 0.000 |
| jigsaw-toxic-comment-classification-challenge | ↑ | 0.97989 | 0.97989 | 0.000 |
| leaf-classification | ↓ | 0.07109 | 0.07109 | 0.001 |
| mlsp-2013-birds | ↑ | 0.83505 | 0.83505 | 0.001 |
| new-york-city-taxi-fare-prediction | ↓ | 4.10941 | 4.10941 | 0.000 |
| nomad2018-predict-transparent-conductors-v2 | ↓ | 0.06294 | 0.06294 | 0.007 |
| plant-pathology-2020-fgvc7 | ↑ | 0.95744 | 0.95744 | 0.000 |
| random-acts-of-pizza | ↑ | 0.62741 | 0.62741 | 0.000 |
| ranzcr-clip-catheter-line-classification | ↑ | 0.82456 | 0.82456 | 0.001 |
| siim-isic-melanoma-classification | ↑ | 0.86334 | 0.86334 | 0.001 |
| spooky-author-identification | ↓ | 0.41563 | 0.41563 | 0.000 |
| tabular-playground-series-dec-2021 | ↑ | 0.95296 | 0.95296 | 0.000 |
| tabular-playground-series-may-2022 | ↑ | 0.98903 | 0.98903 | 0.000 |
| text-normalization-challenge-english-language | ↑ | 0.98813 | 0.98813 | 0.000 |
| text-normalization-challenge-russian-language | ↑ | 0.97314 | 0.97314 | 0.000 |
| the-icml-2013-whale-challenge-right-whale-redux | ↑ | 0.97356 | 0.97356 | 0.000 |
Table 15. Oracle repair on BuggyMLEBench-Lite. Clean: the graded score of the clean repository. Oracle: the graded score after every injected bug is reversed. : their relative difference, in percent. Arrows give the metric’s better direction.
50 Bugs. On the expanded 50 Bugs repositories, each clean baseline was verified to score deterministically across two independent runs, and all 1,068 injected bugs were ablated individually on the clean codebase, with 1,042 (97.6%) independently degrading the test metric or causing an execution failure (Figure 17e). Re-applying the recorded bug patches reconstructs every 50 Bugs repository, and this tier retains crash-inducing mutations alongside metric-degrading defects (Figure 5a).
Figure 17. Automated realism audit, human expert calibration, oracle decoy filtering, and downstream solver resolution on BuggyMLEBench-Lite. (a) LLM-judged 1–5 realism distribution across all 364 raw Lite-Original patches (mean 3.40/5), ranging from plausible oversights (score 5) to overt sabotage (score 1). (b) Realism distribution across the 222 unique evaluated bugs in the 1 Bug, 5 Bugs, and 10 Bugs tiers, where 93% score ≥ 2 and stratum shares remain consistent across tiers. (c) Breakdown across the four task modalities (patch count and mean realism in parentheses), showing the share accompanied by a plausible rationalizing comment versus overt sabotage (1/5). (d) Blind human calibration by an ML practitioner on 30 randomly sampled defects across the evaluated tiers, reporting real-repository plausibility and agreement with the LLM stratum (90% on both). (e) Oracle-measured test impact versus decoys and below-threshold shifts across Lite-Original and all four BuggyMLEBench-Lite tiers. (f) Line-level bug resolution by Claude Sonnet 5 pooled across the 1, 5, and 10 Bugs tiers, stratified by realism score.
Algorithm 1 Two-pass Oracle attribution. , : clean and mutated repositories; : Injector trajectory; : baseline equivalence band ( when run-to-run variance is measured, else relative tolerance , with by default and in the Appendix § 8.§ 5 sweep); , : Phase A and Phase B turn budgets.
1: procedure OracleAttribution() 2: ▹ pair file diffs with Injector turns 3: ▹ discard hallucinated records
// Phase A: greedy minimal reverting set and baseline recovery check
4: clear cached artifacts; ; ; recovered False; turns 5: rank by expected pipeline impact (objective/optimizer/labels before logging/dead code) 6: while and turns do 7: next highest-ranked mutation(s) in (or all remaining near ) 8: ; ▹ accumulating pass; never re-apply 9: ; turns turns ▹ execute run.sh in sandbox 10: if then 11: recovered True; break ▹ reverting set; rest marked no_effect 12: end if 13: end while
// Phase B: per-mutation forward ablation on the clean repository
14: ; turns 15: for do 16: if turns then 17: ; continue 18: end if 19: ; ; turns turns ▹ inject alone 20: if run crashed or produced no valid metric then 21: status verified (code_crash) 22: else if degrades beyond then ▹ deterministic check against 23: status verified (mlperf_degradation) 24: else if improves beyond then 25: status improves_performance 26: else 27: status no_effect ▹ confirmed decoy 28: end if 29: 30: end for 31: return (recovered, , ) 32: end procedure
BuggyMLEBench-Lite Bug Anatomy and Examples
To characterize defect quality prior to tier filtering, we audit all 364 injected patches (spanning 580 diff hunks) in the unpruned Lite-Original pool across isolated metric impact, LLM-judged realism, and engineering pedagogy (complementing the summary of evaluated tiers in § 3.4).
Unit and measurements. Each bug corresponds to one injected patch, which may span multiple diff hunks. We apply each patch in isolation to the clean reference repository and evaluate the resulting submission with the official competition grader. A patch is classified as impactful if it shifts the graded test score beyond the competition’s significance threshold, a crash if execution fails to produce a valid submission, a decoy if at most the local validation score changes, and below threshold otherwise. Impactful bugs are further stratified by absolute test-score shift into catastrophic (), strong (), and mild (). Because thresholds are defined in each competition’s native metric units, severity magnitudes are interpreted within rather than across tasks. Independently, Claude Opus 5 scores each unique bug’s realism and pedagogical value against a fixed rubric.
Isolated Metric Impact
Of the 364 patches in Lite-Original, 97 (26.6%) are validation-only decoys (94) or below-threshold perturbations (3) that do not materially shift the graded test score (Figure 17e). Constructing the 1 Bug, 5 Bugs, and 10 Bugs tiers via oracle filtering removes all non-impactful patches except 4 decoys in 10 Bugs (and retains 97.6% impactful or crash-inducing mutations across the 1,068 patches in 50 Bugs), with most evaluated defects inducing mild or strong degradation rather than outright crashes (Figure 16).
Figure 16. Nearly three-quarters of raw Lite-Original injections independently degrade graded test performance across every pipeline stage and defect category. Isolated test-metric impact of all 364 bugs in the unpruned Lite-Original pool, stratified by (a) ML pipeline layer and (b) the nine most frequent defect types (total bug counts in parentheses; a multi-aspect bug may appear in multiple types). Horizontal bars partition bugs into catastrophic (, dark blue), strong (, medium blue), and mild (, light blue) test-score degradation, below-threshold shifts (pale blue), validation-only decoys (lavender), and execution crashes (charcoal). Filtering out decoys when constructing the 1 Bug, 5 Bugs, and 10 Bugs tiers removes nearly all non-impactful patches (Figure 17e).
Realism and Expert Calibration
Claude Opus 5 rates each patch from 1 to 5 according to how plausibly a competent ML engineer could introduce it as an unintentional mistake. A score of 5 denotes an organic engineering oversight (for example, calling model.train() once outside the epoch loop so that subsequent epochs execute in evaluation mode after the first validation pass), whereas a score of 1 denotes overt sabotage, such as hard-coding a fabricated assumption about the dataset schema.
The distribution centers on scores 3 and 4 (mean 3.40, with 93% of the 222 evaluated defects scoring ; Figure 17a–Figure 17c): most mutations implement syntactically natural ML logic, frequently accompanied by a plausible rationalizing comment (most prevalent in text competitions at 46.9%). Across Lite-Original, the 125 patches that are both impactful and rated 4 or 5 form a realistic core spanning all 22 competitions; for instance, on dog-breed-identification, invoking model.train() only prior to the epoch loop leaves batch-normalization statistics frozen for 44 of 45 training epochs. Conversely, the 21 Lite-Original score-1 patches reflect adversarial sabotage rather than organic developer errors; filtering to scores (or to the realistic core) isolates naturalistic debugging scenarios.
Expert check. As a blind human calibration check on the automated realism rubric, an ML practitioner independently rated 30 defects randomly sampled from the 222 unique bugs across the 1 Bug, 5 Bugs, and 10 Bugs tiers prior to viewing LLM scores. The expert judged 27 of 30 defects
(90%) plausible in real-world ML repositories and agreed with the LLM rating stratum on 27 of 30 (Figure 17d).
Transferable Engineering Practices
We also evaluate whether reverting each defect reinforces a generalizable ML engineering principle rather than correcting an isolated typographical or constant tweak. Across Lite-Original, 79% of patches encode a transferable practice spanning data splitting, index alignment, training state management, metric alignment, or output calibration (Table 16); the remaining 21% consist primarily of inert decoys and narrow hyperparameter perturbations.
| Stage | Practice |
| Data | Keep a true holdout; fit scalers on training data only; apply the same preprocessing to train and test. |
| Alignment | Permute features and labels together; keep ids and predictions in one ordering; match a positional submission’s row order. |
| Training | Re-assert train mode in every training loop and eval mode for inference; rescale a scheduler’s horizon when stepping it per batch; zero gradients every step; freeze only pretrained weights; size the batch to the update budget. |
| Evaluation | Optimize the loss the metric grades; match the checkpoint comparison to the metric’s direction; compute the metric on the full holdout; pass the explicit label set. |
| Output | Match the output activation and clipping to the target range; write predictions with enough precision. |
Table 16. Practices encoded by reverting the injected bugs.
Resolution by Realism Stratum
As a line-level complement to the judge-based resolution analysis in § 5, we inspect which injected lines survive in Claude Sonnet 5’s final repositories across the 1 Bug, 5 Bugs, and 10 Bugs tiers. Here a patch is marked fixed if all of its added lines are removed, partly fixed if a subset remains, and unresolved otherwise (deletion-only patches are excluded, and each bug is counted once per tier).
Claude Sonnet 5 fully removes high-realism defects (scores 4–5) at roughly half the rate of lower-realism defects (16% vs. 27–28%; Figure 17f), and this gap persists within multi-bug repositories containing both strata (for example, 10/46 vs. 15/49 on 5 Bugs and 12/93 vs. 29/104 on 10 Bugs) rather than arising solely from tier difficulty. All ten models evaluated at the per-bug level exhibit the same ordering (Figure 8d); because individual bugs within a shared repository are not statistically independent, we emphasize the consistent directional trend across models rather than exact point differences.
Representative Injected Bugs
To illustrate what each realism tier looks like at the code level, Figure 18–Figure 20 present nine evaluated defects from BuggyMLEBench-Lite, with three per realism stratum (scores 1, 2–3, and 4–5), stratified by data modality and sampled with random seed 2027 (excluding the examples in Figure 6). Each entry reports the competition and evaluation metric, the injector’s summary, isolated test-metric impact, and a brief code-level annotation summarizing why the defect falls into its realism stratum.
Figure 18. Injected bugs rated 1/5 for realism represent overt sabotage justified by fabricated comments. (a)–(c) Three randomly sampled score-1 defects from BuggyMLEBench-Lite spanning audio, tabular, and text tasks (Bird Songs, Tabular May 2022, and Detecting Insults). Left column reports task modality, pipeline stage, bug summary, the code-level rationale for its score of 1 (Rating), and isolated test-metric degradation (blue pill; validation unchanged marks silent bugs). Right column shows the unified diff (pink: removed; green: added), where each mutation relies on an invented premise about the dataset or library API.
Figure 20. Injected bugs rated 4–5/5 closely resemble subtle, honest ML mistakes. (a)–(c) Three randomly sampled score 4–5 defects across image, audio, and text tasks (Histopathology, Right Whale Calls, and Text Norm. Russian), formatted as in Figure 18. Examples include emitting single-argsort sort indices in place of prediction ranks, replacing ReLU with saturating Sigmoid activations across convolutional blocks, and a shared additive smoothing constant that silently alters a threshold gate in another module; all three compile cleanly and mirror real research bugs.
Figure 19. Injected bugs rated 2–3/5 apply recognizable ML operations in questionable contexts. (a)–(c) Three randomly sampled score 2–3 defects across image, tabular, and text tasks (Aerial Cactus, Tabular May 2022, and Spooky Authors), formatted as in Figure 18. Each mutation invokes a real ML concept (batch shuffling for batch normalization, rounding probabilities to match binary labels, or strided subsampling), but applies it in a way a careful code review would flag (e.g., permuting inputs without labels or rounding predictions under a ranking metric).
(a) Histopathology · RANKS
Predicts whether a 96x96 tissue patch contains metastatic cancer. Scored by AUC.
Image Output Realism 5/5
The submission column is np.argsort(probabilities) scaled to the unit interval, which is the sorted-order index rather than the rank, so the column is a permutation of the ranks and not the ranks.
Rating: Rank-transforming predictions for AUC is sound, and confusing argsort's sort indices with ranks is a classic NumPy slip.
AUC 0.952 → 0.501
validation unchanged
histo/emit.py
- The probability column is written as the float it is, with no
- rounding and no clipping. ROC AUC only reads the order of the column,
- so rounding cannot improve the score and can only lose it: every pair
- of patches that rounds to the same value becomes a tie, and ties are
...
+ import numpy as np
- submission[config.TARGET_COLUMN] = probabilities
+ order = np.argsort(probabilities)
+ submission[config.TARGET_COLUMN] = order / float(len(order))
(b) Right Whale Calls · SIGACT
Predicts whether an audio clip contains a right-whale up-call. Scored by AUC.
Audio Model Realism 4/5
The three convolutional blocks use nn.Sigmoid() in place of nn.ReLU, so gradients decay by a factor of four per block.
Rating: Bounding activations against spectrogram outliers is a real consideration, though Sigmoid saturates across three conv blocks.
AUC 0.974 → 0.934
whale/detector.py
+ The activation is bounded rather than rectifying. A rectifier passes the whole positive
+ tail through, so one loud transient in a spectrogram arrives at the next block's
+ normaliser as an outlier that moves the statistics for everything sharing its batch; a
...
- nn.ReLU(inplace=True),
+ nn.Sigmoid(),
(c) Text Norm. (Russian) · GATEPRIOR
Predicts the spoken form of each written Russian token. Scored by accuracy.
Text Training Realism 4/5
enough_agreement() divides by (total + 39), an additive prior that silently converts the 0.5 agreement gate into a requirement of about forty sightings.
Rating: Additive smoothing is a real technique; that a second gate shares the prior is easy to miss across two files.
Accuracy 0.973 → 0.936
library/gates.py
+ # Additive prior on the denominator of the agreement ratio. Without it
+ # a key seen once with a single reading scores an agreement of exactly
+ # 1.0 -- the same number a key seen four hundred times unanimously
+ # scores -- and the gate cannot tell the two apart. The prior pulls
...
+ AGREEMENT_PRIOR = 39
- return count / total >= threshold
+ return count / (total + AGREEMENT_PRIOR) >= threshold
Curriculum Training Details
Pipeline, Rewards, and Hyperparameters
Table 17 summarizes the five-stage training pipeline for our All Difficulties Qwen3.5-9B agent, where every stage updates rank-32 LoRA adapters () across all linear layers, and Table 18 details the GRPO hyperparameters executed via SkyRL on the Megatron backend.
| Step | Data | Length | Objective |
| SFT, teacher | 2,724 evaluator-verified repair trajectories from frontier teacher models, 128 tasks, no reasoning | 682 optimizer steps (2 epochs, batch size 8), lr | next-token loss, 98,304-token context |
| SFT, RFT | 411 evaluator-verified trajectories sampled from our SFT’d Qwen3.8-27B, with reasoning (41% of trained tokens inside <think>), 61 tasks | 103 optimizer steps (2 epochs), lr | same adapter, continued |
| RL, 1–5 hunks | 40 buggy repositories, one instance each with 1–5 diff hunks (median 1) | 3 rounds 48 = 144 steps | mlperf_converge |
| RL, 10–12 hunks | the same 40 repositories, each re-injected with 10–12 hunks | 100 steps | judge_progress |
| RL, 30–34 hunks | the same 40 repositories, re-injected with 30–34 hunks | 72 steps | judge_progress, KL 0.001 |
Table 17. The five training steps of the All Difficulties model.
| Model | Stage | Steps | Turns | Context | Reward | KL |
| All Difficulties | 1–5 hunks, rounds 1–3 | 40 | 40,960 | mlperf_converge | 0 | |
| 10–12 hunks | 100 | 40 | 40,960 | judge_progress | 0 | |
| 30–34 hunks | 72 | 80, then 60 | 81,920 | judge_progress | 0.001 | |
| Easy Only | 1–5 hunks | 150 | 40 | 40,960 | judge_progress | 0 |
| Hard Only | 30–34 hunks | 400 | 60 | 49,152 | judge_progress | 0 |
Table 18. Settings of every RL stage.
GRPO objective and reward functions. At each RL step, for every sampled buggy repository we roll out a group of interactive trajectories and compute group-normalized episode advantages , where and are the group mean and standard deviation of . Policy parameters are updated under the clipped GRPO objective with token-level KL penalty coefficient (Table 18).
Let denote the clipped metric progress ratio closing the gap between the buggy and clean baseline scores (§ 3.2), gated to fixed negative penalties on crashes, no-op submissions, or routing-integrity violations. Early in our study, Stage 1 of All Difficulties (1–5 hunks) trained under mlperf_converge:
where matches the metric-recovery threshold in § 3.2, measures file-level similarity toward the clean reference repository, and measures edit overlap with injected bug sites. All subsequent curriculum stages (Medium and Hard of All Difficulties, as well as both Easy Only and Hard Only) optimize judge_progress:
where is the fraction of injected bug sites judged resolved by Gemini 3.8 Flash (counting partial fixes as 0.5, and falling back to if judge evaluation is unavailable). Because rewards resolving every injected defect individually, continues to supply gradient signal even when fixing a subset of bugs already saturates .
Training regimes. Easy Only replaces the two-stage SFT phase with 4 epochs of SFT (468 optimizer steps at batch size 8) on the 936 teacher trajectories containing 1–5 hunks, while Hard Only trains for 2 epochs (158 optimizer steps; 49,152 RL context) on the 628 trajectories containing at least 30 hunks; each regime then executes GRPO exclusively on its corresponding difficulty band. Notably, SFT Only already includes the 411 RFT trajectories yet achieves modest downstream performance (50.0%/4.5%/0.0% on 1/5/10 Bugs and 18.3% on SWE-bench Lite), indicating that the RFT warmup itself plays a minor role compared to progressive curriculum RL.
Reward design across stages. On easy tasks (1–5 hunks, median 1), metric recovery coincides with causal bug resolution (Figure 26a), so mlperf_converge provides a reliable, judge-free training signal in Stage 1 without per-rollout LLM judge overhead. At higher bug counts (10–12 and 30–34 hunks), however, partial or dominant-bug repairs can saturate raw ML performance while leaving secondary or silent defects intact; Stages 2 and 3 of All Difficulties (as well as the single-band Easy Only and Hard Only ablations) therefore optimize per-site judge_progress (Table 18). Notably, although Easy Only and Hard Only train end-to-end with judge_progress, Easy Only still solves 0% of tasks on the 5 and 10 Bugs tiers and Hard Only underperforms across tiers and OOD benchmarks, confirming that downstream capability is driven by progressive difficulty scaling rather than the Stage 1 reward choice.
Figure 26. Test-metric recovery substantially overstates true bug repair as defect count grows, especially for frontier models. Task solve rate on BuggyMLEBench-Lite under metric recovery alone (recovery , open circles) versus strict judge-verified full repair requiring every injected defect to be resolved (filled circles), across (a) 1 Bug, (b) 5 Bugs, (c) 10 Bugs, and (d) 50 Bugs (22 tasks per panel; lavender: frontier models, gray: budget baselines, blue: our trained 9B models). The two criteria coincide on 1 Bug, where fixing the sole bug is necessary to recover the score, but diverge sharply from 5 Bugs onward as strong models compensate for unfixed or silent defects via modeling improvements elsewhere (e.g., Claude Opus 5.5 reaches 64% score recovery on 10 Bugs vs. 14% verified full repair).
Table 18 | Settings of every RL stage. Steps: the checkpoint used. All stages use 10 tasks × 8 rollouts per step, temperature 1.0, learning rate , and 4,096 output tokens per turn.
Training Dynamics and Checkpoint Selection
Figure 21 plots training reward and held-out validation trajectories for all three RL curricula evaluated in Table 2. Checkpoints are selected by a single rule across all runs: the exported checkpoint maximizing held-out validation score on that curriculum’s target difficulty band (note that validation scores are comparable within, rather than across, curricula because target bands and reward functions differ). Across all three regimes, prolonged optimization eventually degrades held-out performance: Easy Only declines during a 50-step extension past step 150, All Difficulties declines after Hard-stage step 72, and Hard Only drops sharply past step 400.
Figure 21. RL training reward and held-out validation curves across all three curriculum ablations show consistent learning followed by late-stage overfitting. Per-step training reward (thin gray), 8-step moving average (blue), held-out evaluation score on each run’s target difficulty band (lavender markers), and the exported checkpoint selected for final evaluation (vertical dashed line) for (a) Easy Only (1–5 hunks, peaking at step 150 before a 50-step unused extension), (b) Hard Only (30–34 hunks, evaluated at step 400 prior to late collapse), and (c) All Difficulties across cumulative Easy, Medium, and Hard stages (selected at Hard step 72). In (c), the Easy stage uses mlperf_converge while Medium and Hard use judge_progress, so vertical reward levels are comparable within, rather than across, that stage boundary.
Case study: lowering the turn budget mid-stage. To bound KV-cache memory over long multi-bug rollouts during the All Difficulties Hard stage, the first 20 steps ran with an 80-turn budget and all subsequent steps ran at 60 turns (Figure 22); Figure 7 omits these initial 20 steps so that main-text curves reflect fixed stage settings. Under the 80-turn cap, episodes averaged 67 turns and 42% reached the turn limit. Lowering the cap to 60 turns initially raised the cap-hit rate to 81% over the next ten steps and briefly depressed reward because truncated episodes cannot invoke finalize. Training quickly adapted, however: reward recovered prior to the selected evaluation checkpoint, held-out validation rose from 0.44 to 0.63 at the selected checkpoint, while mean episode length fell to 45 turns and the cap-hit rate dropped to 19% by the end of the run, indicating that tighter turn budgets impose only transient reward loss while encouraging turn-efficient repair (Figure 7b).
Figure 22. Cutting the turn budget from 80 to 60 mid-stage costs reward briefly, then the agent learns to finish within it. The full hard stage of the All Difficulties run, including its first 20 steps at an 80-turn budget (shaded). (a) Mean turns per episode with each budget (dashed). (b) Share of episodes that hit the turn cap: it jumps when the budget drops, then falls steadily. (c) Training reward per step (gray) and 8-step mean (blue), held-out score (lavender), and the checkpoint we evaluate (dashed).
Execution Harness and Evaluation Setup
Sandbox Harness and Episode Protocol
A unified container harness executes all interactive episodes across RL rollouts, BuggyMLEBench-Lite evaluations, and commercial baselines. At each turn, the agent emits free-form reasoning and XML-wrapped tool calls, which the harness parses, executes inside a GPU-enabled Docker container holding the repository and read-only dataset, and returns as structured observations (Algorithms 2 and 3).
Tools and protocol. Agents interact with the container through six tools (Table 19). Each assistant response may invoke multiple tool calls formatted as
| Tool | Effect |
bash(command, workdir) | runs a shell command in the container, under a per-call timeout |
read_file(path) | returns a file’s contents |
write_file(path, content) | creates or overwrites a file |
modify_file(path, old, new) | replaces an exact substring of a file, so a single function can be patched without rewriting the file |
list_dir(path) | lists a directory |
finalize(verdict, description) | ends the episode; the verdict is recorded but not used in scoring |
Table 19. The agent’s tools. Searching the repository is done through bash.
<tool_call>{"name": "read_file", "input": {"path": "main.py"}}</tool_call>
whose outputs are concatenated in the subsequent observation. Unparseable calls are recorded as malformed turns and return a syntax error. Tool outputs retain their first 1,000 and last 3,000 characters, and the dataset mount is read-only so candidate repairs cannot modify evaluation data.
Prompt. The system prompt specifies the task objective, tool schemas, and XML call syntax. On training instances, the initial user message includes the clean baseline metric alongside cached stdout/stderr from the failing buggy run; on BuggyMLEBench-Lite, it provides the competition description, submission contract, and expected clean baseline score while omitting cached stdout/stderr logs, requiring the agent to execute python main.py to observe runtime behavior.
Algorithm 2 One episode. : turn budget; : context budget in tokens; : consecutive malformed-turn limit (3 in training; in evaluation, so only an episode of malformed turns is cut short).
1: procedure Episode(task, , , , ) 2: start the task’s container; mount the data read-only; remove answer-key files 3: hashes of the source files 4: [system prompt with tools, task prompt]; 5: for do 6: if the episode clock has run out then break 7: ; tool calls parsed from 8: if contains a call that cannot be parsed then 9: ; if then break 10: an error message 11: else if then break ▷ a reply without a tool call ends the episode 12: else if finalize then break 13: else 14: ; the truncated result of each call in , executed in order 15: end if 16: “[BUDGET] turn ; turns, context tokens remaining . . .” 17: 18: if then break 19: end for 20: return Score(container, , , ) 21: end procedure
Episode termination. An episode terminates when the agent invokes finalize, emits a response containing no tool call, reaches the malformed-turn limit , or exhausts its turn, context-token, or wall-clock budget; in all cases, the container state at termination is evaluated identically by Algorithm 3.
Algorithm 3. Scoring the repository at the end of an episode, whichever way it ended.
procedure
the first that holds: malformed turns; source hashes equal (no change); code does not compile; entry point no longer reaches the injected code (bypassed)
if gate then return the gate’s fixed reward
re-run the entry point (
run.sh; on BuggyMLEBench-Lite,python main.py, then the official MLE-Bench grader onsubmission.csv)if the run crashes or reports no metric then return the crash reward
( when lower is better)
return ; reward variant, Appendix § 11
end procedure
Settings. Table 20 summarizes shared harness budgets across training and evaluation, and Appendix § 12.§ 2 details per-model configurations for hosted baselines.
| RL 1–5, 10–12 hunks | RL 30–34 hunks | BuggyMLEBench-Lite 1 / 5 / 10 / 50 Bugs | |
| turns per episode | 40 | 60§ | 40 / 60 / 80 / 80‡ |
| context / output per turn (tokens) | 40,960 / 4,096 | 81,920 / 4,096† | 98,304 / 4,096 |
| malformed limit | 3 | 3 | the turn budget |
| sampling | temperature 1.0 | temperature 1.0 | temperature 0.7 |
Table 20. Harness settings. Table 21 gives per-model budgets.
Baseline Model Configurations
Table 21 details the evaluation configuration for every baseline and trained checkpoint (22 episodes per model and tier, one trajectory per task). Unless noted otherwise, models share a 98,304-token context window and turn caps of 40, 60, 80, and 80 on the 1 Bug, 5 Bugs, 10 Bugs, and 50 Bugs tiers. All episodes run under multi-hour wall-clock timeouts (at most 8 hours per episode) sufficient for repeated pipeline executions.
| Model | Turns (1 / 5 / 10 / 50 Bugs) | Context and output |
| Claude Opus 5.5 ([5]) | 80 / 80 / 80 / 80 | shared |
| Claude Sonnet 5.5 ([8]) | 80 / 80 / 80 / 80 | shared |
| Claude Sonnet 5 ([7]) | 80 / 80 / 80 / 80 | shared |
| Claude Sonnet 4.6 ([6]) | 40 / 60 / 80 / 80 | 262k; 40,960 per turn |
| Gemini 3.8 Flash ([19]) | 40 / 60 / 80 / 80 | 262k; 40,960 per turn |
| Claude Haiku 5.5 ([2]) | 40 / 60 / 80 / 80 | shared |
| Claude Haiku 4.5 ([1]) | 40 / 60 / 80 / 80 | 150k; 32,000 per turn |
| Gemini 3.5 Flash-Lite ([16]) | 40 / 60 / 80 / 80 | 262k; 40,960 per turn |
| Qwen3.8-27B ([39]) (thinking off) | 40 / 60 / 80 / 80 | shared |
| Qwen3.5-9B ([38]) | 80 / 80 / 80 / 80 | shared |
| SFT Only (9B) | 80 / 80 / 80 / 80 | shared |
| SFT+RL, Easy Only | 40 / 60 / 80 / 80 | shared |
| SFT+RL, Hard Only | 40 / 60 / 80 / 80 | shared |
| SFT+RL, All Difficulties | 80 / 80 / 80 / 80 | shared |
Table 21. Evaluation setup. Context and output list total context and per-turn output limits where they differ from the shared harness (Table 20).
Turn and termination behavior. On 10 Bugs and 50 Bugs, turn caps are uniform (80 turns) across all models, while 1 Bug and 5 Bugs use 40- and 60-turn caps under the default schedule and 80 turns under the uniform schedule (Table 21); these lower caps do not bind in practice, as Easy Only, Hard Only, and Claude Haiku 5.5 hit the turn cap on 0% of 1 Bug episodes and at most 4.5% of 5 Bugs episodes (Table 22). Termination behavior instead distinguishes our single-band curricula from All Difficulties: Hard Only reaches its episode timeout on 12 of 22 episodes on 1 Bug, and Easy Only terminates prematurely without emitting a tool call on 17 of 22 episodes on both 5 Bugs and 10 Bugs.
| Variant | Model | Recovery (%) | Full Repair (%) | Bugs Fixed (%) | Improved Repo (%) | Ended with Finalize Tool (%) | Tool-Call Success (%) | Hit Turn Cap (%) | Leaderboard Percentile (%) | Tasks Above Clean Repo |
| Reference | Clean repository (1, 5, 10 Bugs) | – | – | – | – | – | – | – | 49.1 | – |
| 1 Bug | Claude Opus 5.5 | 99.8 | 95.5 | 95.5 | 100.0 | 100.0 | 96.7 | 0.0 | 57.2 | 14 |
| 1 Bug | Claude Sonnet 5.5 | 95.5 | 90.9 | 90.9 | 95.5 | 100.0 | 91.8 | 0.0 | 54.0 | 11 |
| 1 Bug | Claude Sonnet 5 | 80.2 | 63.6 | 63.6 | 86.4 | 100.0 | 89.9 | 0.0 | 39.2 | 5 |
| 1 Bug | Claude Sonnet 4.6 | 30.9 | 27.3 | 27.3 | 27.3 | 72.7 | 95.9 | 0.0 | 17.5 | 1 |
| 1 Bug | Gemini 3.8 Flash | 27.3 | 27.3 | 27.3 | 27.3 | 90.9 | 97.1 | 0.0 | 15.6 | 3 |
| 1 Bug | Claude Haiku 5.5 | 89.5 | 77.3 | 77.3 | 86.4 | 100.0 | 98.7 | 0.0 | 46.1 | 5 |
| 1 Bug | Claude Haiku 4.5 | 33.3 | 9.1 | 9.1 | 27.3 | 77.3 | 89.8 | 0.0 | 11.2 | 1 |
| 1 Bug | Gemini 3.5 Flash-Lite | 8.2 | 4.5 | 4.5 | 9.1 | 63.6 | 85.0 | 0.0 | 3.8 | 0 |
| 1 Bug | Qwen3.8-27B | 72.7 | 40.9 | 45.5 | 63.6 | 50.0 | 84.9 | 9.1 | 34.9 | 5 |
| 1 Bug | Qwen3.5-9B | 62.1 | 13.6 | 18.2 | 50.0 | 68.2 | 93.3 | 18.2 | 24.7 | 1 |
| 1 Bug | Ours (9B SFT) | 70.7 | 50.0 | 50.0 | 72.7 | 81.8 | 99.2 | 13.6 | 36.9 | 3 |
| 1 Bug | Ours: SFT+RL Curriculum: Easy Only (1–5 Hunks) | 73.9 | 50.0 | 50.0 | 68.2 | 68.2 | 98.1 | 0.0 | 35.6 | 3 |
| 1 Bug | Hard Only (30–34 Hunks) | 69.4 | 22.7 | 59.1 | 72.7 | 40.9 | 98.4 | 0.0 | 34.2 | 4 |
| 1 Bug | All Difficulties (1–5, 10–12, then 30–34) | 81.5 | 72.7 | 81.8 | 81.8 | 72.7 | 98.9 | 9.1 | 43.0 | 6 |
| 5 Bugs | Claude Opus 5.5 | 88.4 | 31.8 | 82.7 | 95.5 | 100.0 | 99.1 | 0.0 | 44.6 | 10 |
| 5 Bugs | Claude Sonnet 5.5 | 85.5 | 27.3 | 77.3 | 95.5 | 95.5 | 93.0 | 0.0 | 42.4 | 7 |
| 5 Bugs | Claude Sonnet 5 | 67.5 | 4.5 | 47.3 | 90.9 | 100.0 | 91.4 | 0.0 | 24.2 | 0 |
| 5 Bugs | Claude Sonnet 4.6 | 25.6 | 0.0 | 13.6 | 22.7 | 77.3 | 95.3 | 0.0 | 5.8 | 0 |
| 5 Bugs | Gemini 3.8 Flash | 41.2 | 13.6 | 33.6 | 40.9 | 68.2 | 95.3 | 4.5 | 21.9 | … |
| 5 Bugs | Claude Haiku 5.5 | 75.2 | 18.2 | 75.5 | 86.4 | 100.0 | 97.5 | 0.0 | 33.4 | 3 |
| 5 Bugs | Claude Haiku 4.5 | 17.3 | 0.0 | 7.3 | 27.3 | 63.6 | 90.1 | 4.5 | 4.1 | 0 |
| 5 Bugs | Gemini 3.5 Flash-Lite | 8.0 | 0.0 | 4.5 | 9.1 | 63.6 | 82.8 | 0.0 | 2.8 | 0 |
| 5 Bugs | Qwen3.8-27B | 44.0 | 4.5 | 31.8 | 54.5 | 18.2 | 69.9 | 27.3 | 14.8 | 1 |
| 5 Bugs | Qwen3.5-9B | 49.3 | 0.0 | 29.1 | 72.7 | 50.0 | 92.7 | 40.9 | 14.5 | 0 |
| 5 Bugs | Ours (9B SFT) | 62.7 | 4.5 | 37.3 | 90.9 | 50.0 | 97.6 | 22.7 | 22.0 | 1 |
| 5 Bugs | Ours: SFT+RL Curriculum: Easy Only (1–5 Hunks) | 54.6 | 0.0 | 31.8 | 72.7 | 18.2 | 96.4 | 4.5 | 12.2 | 0 |
| 5 Bugs | Hard Only (30–34 Hunks) | 67.3 | 18.2 | 63.6 | 81.8 | 50.0 | 98.0 | 0.0 | 31.0 | 3 |
| 5 Bugs | All Difficulties (1–5, 10–12, then 30–34) | 77.4 | 22.7 | 71.8 | 90.9 | 81.8 | 97.7 | 0.0 | 37.0 | 5 |
| 10 Bugs | Claude Opus 5.5 | 92.2 | 13.6 | 80.9 | 95.5 | 100.0 | 99.0 | 0.0 | 46.7 | 10 |
| 10 Bugs | Claude Sonnet 5.5 | 86.4 | 31.8 | 84.1 | 95.5 | 100.0 | 95.6 | 0.0 | 43.6 | 6 |
| 10 Bugs | Claude Sonnet 5 | 53.7 | 0.0 | 38.6 | 68.2 | 95.5 | 91.7 | 0.0 | 12.6 | 0 |
| 10 Bugs | Claude Sonnet 4.6 | 30.1 | 0.0 | 16.4 | 40.9 | 68.2 | 95.2 | 0.0 | 4.5 | 0 |
| 10 Bugs | Gemini 3.8 Flash | 18.5 | 13.6 | 22.3 | 22.7 | 86.4 | 96.3 | 0.0 | 14.1 | 3 |
| 10 Bugs | Claude Haiku 5.5 | 71.1 | 0.0 | 73.2 | 81.8 | 95.5 | 98.8 | 0.0 | 19.3 | 1 |
| 10 Bugs | Claude Haiku 4.5 | 18.7 | 0.0 | 5.5 | 31.8 | 77.3 | 97.1 | 0.0 | 2.6 | 0 |
| 10 Bugs | Gemini 3.5 Flash-Lite | 6.9 | 0.0 | 1.4 | 9.1 | 50.0 | 88.6 | 0.0 | 1.3 | 0 |
| 10 Bugs | Qwen3.8-27B | 53.1 | 0.0 | 36.8 | 59.1 | 45.5 | 82.3 | 18.2 | 11.0 | 1 |
| 10 Bugs | Qwen3.5-9B | 48.8 | 0.0 | 15.0 | 50.0 | 0.0 | 91.9 | 40.9 | 8.1 | 0 |
| 10 Bugs | Ours (9B SFT) | 43.4 | 0.0 | 25.0 | 59.1 | 27.3 | 97.9 | 59.1 | 9.0 | 0 |
| 10 Bugs | Ours: SFT+RL Curriculum: Easy Only (1–5 Hunks) | 48.0 | 0.0 | 36.8 | 54.5 | 22.7 | 95.2 | 0.0 | 8.2 | 0 |
| 10 Bugs | Hard Only (30–34 Hunks) | 59.6 | 9.1 | 66.8 | 68.2 | 59.1 | 97.3 | 0.0 | 19.1 | 1 |
| 10 Bugs | All Difficulties (1–5, 10–12, then 30–34) | 81.1 | 18.2 | 80.9 | 86.4 | 100.0 | 96.5 | 0.0 | 40.8 | 6 |
| Reference | Clean repository (50 Bugs) | – | – | – | – | – | – | – | 52.0 | – |
| 50 Bugs | Claude Opus 5.5 | 77.8 | 9.1 | 54.9 | 81.8 | 90.9 | 99.1 | 0.0 | 43.9 | 9 |
| 50 Bugs | Claude Sonnet 5.5 | 64.3 | 0.0 | 86.3 | 86.4 | 100.0 | 96.0 | 0.0 | 30.6 | 2 |
| 50 Bugs | Claude Sonnet 5 | 30.2 | 0.0 | 49.0 | 59.1 | 90.9 | 96.6 | 0.0 | 3.5 | 0 |
| 50 Bugs | Claude Sonnet 4.6 | 0.0 | 0.0 | 6.2 | 0.0 | 72.7 | 91.2 | 0.0 | 0.0 | 0 |
| 50 Bugs | Gemini 3.8 Flash | 22.4 | 4.5 | 33.5 | 27.3 | 81.8 | 95.6 | 0.0 | 9.2 | 3 |
| 50 Bugs | Claude Haiku 5.5 | 41.3 | 0.0 | 54.6 | 68.2 | 31.8 | 98.7 | 0.0 | 8.8 | 1 |
| 50 Bugs | Claude Haiku 4.5 | 5.6 | 0.0 | 9.3 | 9.1 | 59.1 | 93.0 | 0.0 | 5.5 | 1 |
| 50 Bugs | Gemini 3.5 Flash-Lite | 7.0 | 0.0 | 6.5 | 0.0 | 50.0 | 87.8 | 9.1 | 5.2 | 1 |
| 50 Bugs | Qwen3.8-27B | 9.8 | 0.0 | – | 22.7 | 45.5 | 77.2 | 45.5 | 1.6 | 0 |
| 50 Bugs | Qwen3.5-9B | 4.1 | 0.0 | 14.0 | 9.1 | 0.0 | 90.2 | 100.0 | 0.5 | 0 |
| 50 Bugs | Ours (9B SFT) | 7.2 | 0.0 | 16.3 | 13.6 | 4.5 | 96.8 | 95.5 | 1.5 | 0 |
| 50 Bugs | Ours: SFT+RL Curriculum: Easy Only (1–5 Hunks) | 9.9 | 0.0 | 25.1 | 9.1 | 0.0 | 96.2 | 40.9 | 5.5 | 1 |
| 50 Bugs | Hard Only (30–34 Hunks) | 12.3 | 0.0 | 58.0 | 22.7 | 9.1 | 94.7 | 72.7 | 1.4 | 0 |
| 50 Bugs | All Difficulties (1–5, 10–12, then 30–34) | 14.1 | 0.0 | 64.9 | 22.7 | 0.0 | 94.1 | 31.8 | 5.1 | 0 |
Table 22. All metrics on BuggyMLEBench-Lite, 22 episodes per model and tier. Arrows, shading, and the Reference rows follow Table 2. Bugs fixed: injected bugs judged fixed, whether or not the repository still runs (not judged for Qwen3.8-27B on 50 Bugs). Improved repo: the repository beats its injected start. Ended with finalize tool: episodes terminating via an explicit finalize call. Tool-call success: failed/attempted actions. Hit turn cap: the episode used its whole turn budget. Tasks above clean repo: tasks where the repaired code ranks above the clean repository.
Expanded context and output caps. Four hosted baselines (Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.8 Flash, and Gemini 3.5 Flash-Lite) are evaluated with expanded context windows and higher per-turn output limits (Table 21) to accommodate longer per-turn reasoning traces under the shared prompt and XML tool interface (Appendix § 12.§ 1).
Inference backends. Anthropic baselines execute via the Claude Code CLI (claude -p), invoked once per turn with built-in tools and extended thinking disabled; Gemini baselines execute via the Vertex AI API within our harness loop at default thinking settings; and Qwen models are served locally.
Model tiers. As defined in § 4, Budget Models comprise each commercial provider’s lowest-cost tier (Claude Haiku 4.5/5.5 and Gemini 3.5 Flash-Lite) alongside open-weight models up to 27B parameters (Qwen3.8-27B and Qwen3.5-9B); all larger commercial flagships constitute Frontier Models.
Qwen3.8-27B. Qwen3.8-27B is evaluated with thinking disabled on every tier, because an initial thinking-enabled run failed to invoke finalize across all 66 episodes on the 1, 5, and 10 Bugs tiers.
LLM Judges and Code Review
§ 3.3’s repository-level semantic review is implemented with Claude Opus 4.8 (temperature 0) to avoid the cost of issuing separate per-site judge calls across thousands of multi-bug rollouts on the 200-task pool (Appendix § 8.§ 5), operating in a single pass from code diffs and Oracle ablation records without executing an external test suite.
Inputs. The reviewer receives three unified diffs (clean-to-fixed, buggy-to-fixed, and clean-to-buggy, each truncated at 50,000 characters), together with the Oracle’s structured ablation report listing verified causal mutations (affected files, defect mechanism, and isolated metric impact), confirmed no-effect decoys, and any metric-improving mutations, grounding the evaluation in measured causal attribution rather than subjective code style.
Verdicts. Parallel sub-checks evaluate two binary criteria: whether the candidate patch actually fixes the verified causal mutations (Fix Addresses Bug in Figure 14), and whether the repair is principally valid rather than bypassing the faulty logic, leaving the underlying bug intact, or introducing a new defect in its place (Principally Valid). A candidate patch that modifies only decoy mutations fails the first check even if the evaluation metric recovers.
Judge models and scope. Across our pipeline, judge backbones are chosen to match call volume and latency constraints: single-pass offline repository reviews on the 200-task sweep use Claude Opus 4.8 (temperature 0), whereas concurrent per-site checks use low-latency models, specifically Claude Haiku 5.5 for BuggyMLEBench-Lite full-repair grading (§ 3.2) and Gemini 3.8 Flash for online judge_progress RL rollouts (Appendix § 11; the RL environment’s auxiliary repository-reviewer penalty remained disabled across all reported training runs). § 5 and Appendix § 10 further validate bug-level LLM judgments against blind human review.
Extended Experimental Results
Representative Repair Trajectories
To illustrate the behavioral shift quantified in RQ4 (§ 5), Figure 23–Figure 25 compare episode traces of the untuned Qwen3.5-9B base model and our All Difficulties RL agent on representative 5 Bugs (Aerial Cactus, discussed in RQ4; Figure 23), 1 Bug (Detecting Insults; Figure 24), and 10 Bugs (Spooky Authors; Figure 25) repositories.
Figure 23. Curriculum RL replaces blind pipeline rewrites with targeted multi-file diagnosis and verification on a 5 Bugs task. Side-by-side episode trace on Aerial Cactus (each model’s reported sample; T: turn; quotes verbatim from model reasoning). (a) Untuned Qwen3.5-9B spends 80 turns rewriting files and standalone scripts, fixing 3/5 bugs without finalizing (recovery 0.48; red boxes mark failure modes). (b) All Difficulties diagnoses four suspects across modules, reverts them in one turn, catches the fifth bug from a chance-level validation AUC, and finalizes in 14 turns with 5/5 bugs fixed (recovery 1.00; green boxes mark diagnostic reasoning and repairs). (c) Per-bug judge audit and fixing turn.
Figure 24. On a silent 1 Bug task, the RL agent inspects submission code directly whereas the base model overfits validation noise. Side-by-side episode trace on Detecting Insults (layout as in Figure 23). (a) Untuned Qwen3.5-9B misinterprets a 0.005 validation gap as the defect, tunes vectorizer hyperparameters across 43 turns without inspecting submit.py, and finalizes with 0/1 bugs fixed (recovery 0.72). (b) All Difficulties locates and removes the injected test-score Gaussian noise in submit.py by turn 6, verifies on main.py, and finalizes in 12 turns with full recovery (1.00). (c) Per-bug judge audit.
Figure 25. On a 10 Bugs repository, the RL agent systematically clears all ten defects across library modules while the base model rewrites main.py around them. Side-by-side episode trace on Spooky Authors (layout as in Figure 23). (a) Untuned Qwen3.5-9B repeatedly rewrites main.py from scratch over 60 turns without editing any underlying library/module, fixing 0/10 bugs (recovery 0.60). (b) All Difficulties patches seven bugs across eight files in a single turn (T6), uses residual validation log-loss gaps against the baseline to isolate the remaining three subtle hyperparameter and data-slice bugs (T13–T16), and resolves all 10/10 bugs in 20 turns (recovery 0.99). (c) Per-bug judge audit across all ten injected defects.
On detecting-insults-in-social-commentary (1 Bug; Figure 24), the sole injected defect adds Gaussian noise to test-time predictions in submit.py without affecting validation scores. The base model misinterprets the small gap between its local validation score (0.804) and the reference baseline (0.809) as the defect, tunes a TF-IDF vectorizer flag, and terminates without inspecting submit.py. By contrast, the All Difficulties agent audits the submission path, removes the injected noise within 12 turns, and restores full test performance (alongside a superfluous gradient-scaling edit that does not harm recovery).
On spooky-author-identification (10 Bugs; Figure 25), the base model repeatedly rewrites main.py into a standalone cross-validation script without modifying the underlying library/ modules that contain the defects, incidentally bypassing one bug and resolving none. The All Difficulties agent patches seven defects across eight library files in a single turn, uses the residual gap to the baseline log-loss to isolate the remaining three subtle hyperparameter and data-slice defects, and resolves all ten bugs in 20 turns.
Full BuggyMLEBench-Lite Breakdown
Table 22 expands Table 2 with the complete suite of outcome and behavioral metrics across all four BuggyMLEBench-Lite tiers, accompanied by three diagnostic breakdowns: Figure 26 contrasts metric-only recovery against judge-verified full repair across tiers, Figure 27 characterizes episode lengths and tool-use distributions on 10 Bugs, and Figure 28 disaggregates per-bug resolution rates across pipeline layers, defect categories, and data modalities.
Figure 27. Curriculum RL halves episode length and shifts tool usage from exploratory shell commands toward structured file reads and patches. Episode-level execution statistics across the 22 tasks of the 10 Bugs tier (lavender: frontier models, gray: budget baselines, blue: our trained 9B models). (a) Turns per episode and (b) total tool calls per episode (shaded bar: interquartile range; vertical tick: median; dot: mean; 80-turn budget). Whereas Qwen3.5-9B Base and SFT Only routinely exhaust the 80-turn cap, All Difficulties finishes in a median of turns, comparable to Claude Haiku 5.5 and Sonnet 5.5. (c) Share of tool calls by tool type: Claude Code baselines execute predominantly through bash (67–96%), whereas All Difficulties balances bash (25%), read_file (32%), and modify_file (36%) while eliminating the whole-file write_file calls seen in the untuned base model.
Figure 28. High-complexity RL closes the gap with frontier models across pipeline stages, defect categories, and data modalities, particularly on output-layer bugs. Percentage of individual injected bugs judged resolved, pooled across the 1 Bug, 5 Bugs, and 10 Bugs tiers (darker blue indicates higher resolution rate), broken down by (a) ML pipeline layer, (b) the eight most frequent defect categories, and (c) competition data modality. Output-formatting defects are the hardest layer for base, SFT, and most external models (4% for Qwen3.5-9B Base, 19% for Claude Sonnet 5, 69% for Sonnet 5.5), whereas Hard Only (78%) and All Difficulties (83%) match or exceed Claude Opus 5.5 (76%). Across modalities (c), All Difficulties transfers evenly across image, signal, tabular, and text repositories (66–82%).
Extended OOD Benchmarks
Table 23 expands the OOD evaluation of § 5 with four additional benchmarks that progressively depart from repository-level bug repair, comparing our All Difficulties checkpoint, the same one reported in every other table, against the untuned 9B base model and Claude Sonnet 5 across evaluation tasks with zero training overlap.
| Task | Metric | Claude Sonnet 5 | Qwen3.5-9B | Ours (9B SFT+RL) | Ours − Qwen3.5-9B |
| Code only (no tools, up to 10 debug attempts) | 66 episodes per model | ||||
| Quality | Leaderboard Percentile | 61.2 | 16.4 | 17.9 | +1.5 |
| Quality | Valid Submission (%) | 100 | 61 | 50 | −10.6 |
| Efficiency | Input + Output Tokens / Ep. (k) | 13 | 78 | 65 | −12.7{sup}†` |
| Efficiency | Tool-Call Success (%) | – | – | – | – |
Sandbox: write solution.py | 22 episodes per model | ||||
| Quality | Leaderboard Percentile | 73.9 | 23.8 | 17.7 | −6.2 |
| Quality | Valid Submission (%) | 100 | 68 | 59 | −9.1 |
| Efficiency | Input + Output Tokens / Ep. (k) | 1022 | 2112 | 1191 | −921.3{sup}†` |
| Efficiency | Tool-Call Success (%) | 98.2 | 74.9 | 92.7 | +17.4{sup}†` |
| Sandbox: build a repository | 22 episodes per model | ||||
| Quality | Leaderboard Percentile | 72.0 | 20.0 | 22.1 | +2.1 |
| Quality | Valid Submission (%) | 100 | 41 | 50 | +9.1 |
| Efficiency | Input + Output Tokens / Ep. (k) | 1213 | 2259 | 1410 | −849.5{sup}†` |
| Efficiency | Tool-Call Success (%) | 97.6 | 69.9 | 87.6 | +18.5{sup}†` |
| Sandbox: DeepSWE issues | 34 episodes per model | ||||
| Quality | Resolved (%) | 14.7 | 0.0 | 0.0 | +0.0 |
| Quality | Fail-to-Pass Tests (%) | 68.5 | 3.3 | 12.2 | +8.9{sup}†` |
| Efficiency | Input + Output Tokens / Ep. (k) | 3668 | 956 | 1382 | +425.9{sup}†` |
| Efficiency | Tool-Call Success (%) | 96.6 | 67.0 | 95.1 | +28.2{sup}†` |
Table 23. Table 23 | OOD tasks. Top rows of each task measure quality, bottom rows efficiency. Tokens: input plus output per episode. Shading and bold (best) compare the base model and ours. : paired per-task difference from the base; †: its 95% task-bootstrap interval excludes zero.
Tasks. In code only, the model generates a standalone training script directly from the competition description without tool access across up to ten attempts. In solution.py, the agent authors a single-file pipeline inside the interactive sandbox starting from an empty workspace, while Repository requires constructing a modular multi-file repository from scratch. All three ML synthesis settings evaluate across the 22 MLE-Bench Lite competitions using the official Kaggle grader and report mean leaderboard percentile. Finally, DeepSWE evaluates long-horizon software issue resolution across 34 general Python repository tasks.
LiveCodeBench v6. We evaluate under the same sampling parameters as BuggyMLEBench-Lite with reasoning traces capped at 4,096 tokens; responses from which no executable program can be extracted (13–25% across models) are scored as incorrect.
RL improves tool-call reliability across interactive environments. In both ML sandbox settings, tool-call success rises by 17 to 19 points over the untuned base model, and tokens per episode fall by about 40% (Table 23). On DeepSWE, tool-call success rises by 28 points. Tokens per episode rise there by 45%, because our model wastes 9 fewer turns per episode on malformed calls and each valid call returns tool output.
From-scratch ML synthesis shows mixed leaderboard results. Our model edges ahead of the untuned base model in standalone-script (+1.5) and full-repository (+2.1) synthesis while trailing on single-file solution.py (−6.2), with all differences within 7 points and none statistically significant, whereas Claude Sonnet 5 demonstrates substantial headroom (61–74th percentile). Consistently improving open-ended ML synthesis from scratch likely requires direct optimization on synthesis objectives, such as leaderboard-percentile rewards.
References
- [1]Anthropic. Claude Haiku 4.5 System Card, October 2025. URL https://www-cdn.anthropic.com/7aad69bf12627d42234e01ee7c36305dc2f6a970.pdf.
- [2]Anthropic. Claude Haiku 5.5 System Card, October 7, 2026a. URL https://www.anthropic.com/claude-haiku-5-5-system-card.
- [3]Anthropic. Claude Opus 4.8 System Card, May 28, 2026b. URL https://www.anthropic.com/claude-opus-4-8-system-card.
- [4]Anthropic. Claude Opus 5 System Card, July 24, 2026c. URL https://www.anthropic.com/claude-opus-5-system-card.
- [5]Anthropic. Claude Opus 5.5 System Card, September 22, 2026d. URL https://www.anthropic.com/claude-opus-5-5-system-card.
- [6]Anthropic. Claude Sonnet 4.6 System Card, February 17, 2026e. URL https://www.anthropic.com/claude-sonnet-4-6-system-card.
- [7]Anthropic. Claude Sonnet 5 System Card, June 30, 2026f. URL https://www.anthropic.com/claude-sonnet-5-system-card.
- [8]Anthropic. Claude Sonnet 5.5 System Card, September 28, 2026g. URL https://www.anthropic.com/claude-sonnet-5-5-system-card.
- [9]J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. Mądry. MLE-bench: Evaluating machine learning agents on machine learning engineering. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.07095
- [10]L. Chen, M. Prabhudesai, K. Fragkiadaki, H. Liu, and D. Pathak. Self-questioning language models. arXiv preprint arXiv:2508.03682, 2025a.arxiv.org/abs/2508.03682
- [11]M. Chen, L. Wang, and B. Qu. Recursive self-improvement in AI: From bounded self-refinement to autonomous research loops. arXiv preprint arXiv:2607.07663, 2026.arxiv.org/abs/2607.07663
- [12]Q. Chen, M. Yang, L. Qin, J. Liu, Z. Yan, J. Guan, D. Peng, Y. Ji, H. Li, M. Hu, Y. Zhang, Y. Liang, Y. Zhou, J. Wang, Z. Chen, and W. Che. AI4Research: A survey of artificial intelligence for scientific research. arXiv preprint arXiv:2507.01903, 2025b. URL https://arxiv.org/abs/2507.01903.
- [13]E. Chu, R. Agarwal, A. Thangamuthu, B. Graham, and J. Mattern. FrontierSWE: Benchmarking coding agents at the limits of human abilities, 2026. URL https://frontierswe.com.
- [14]X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, V. Bharadwaj, J. Holm, R. Aluri, C. B. C. Zhang, N. Jacobson, B. Liu, and B. Kenstler. SWE-bench Pro: Can AI agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025. URL https://arxiv.org/abs/2509.16941.
- [15]R. Desai, J. Hu, J. Cabezas, N. Harsola, P. Shukla, R. B. Chaim, A. E. Assadi, O. M. Kamath, F. Faldu, P. Hebbar, et al. SWE-Marathon: Can agents autonomously complete ultra-long-horizon software work? arXiv preprint arXiv:2606.07682, 2026.arxiv.org/abs/2606.07682
- [16]Google DeepMind. Gemini 3.5 Flash-Lite Model Card, July 21, 2026a. URL https://deepmind.google/models/model-cards/gemini-3-5-flash-lite/.
- [17]Google DeepMind. Gemini 3.6 Flash Model Card, July 21, 2026b. URL https://deepmind.google/models/model-cards/gemini-3-6-flash/.
- [18]Google DeepMind. Gemini 3.7 Flash Model Card, August 13, 2026c. URL https://deepmind.google/models/model-cards/gemini-3-7-flash/.
- [19]Google DeepMind. Gemini 3.8 Flash Model Card, September 2, 2026d. URL https://deepmind.google/models/model-cards/gemini-3-8-flash/.
- [21]J. Huang, X. Feng, Q. Chen, H. Zhao, Z. Cheng, J. Bai, J. Zhou, M. Li, and L. Qin. MLDebugging: Towards benchmarking code debugging across multi-library scenarios. In Findings of the Association for Computational Linguistics: ACL 2025, pages 5866–5879, 2025.arxiv.org/abs/2506.13824
- [22]Q. Huang, J. Vora, P. Liang, and J. Leskovec. MLAgentBench: Evaluating language agents on machine learning experimentation. arXiv preprint arXiv:2310.03302, 2023.arxiv.org/abs/2310.03302
- [23]W. Huang, C. Lee, L. Tng, and S. Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks. arXiv preprint arXiv:2607.07946, 2026.arxiv.org/abs/2607.07946
- [24]N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=chfJJYJc3iL.arxiv.org/abs/2403.07974
- [25]C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024.arxiv.org/abs/2310.06770
- [26]M. Kim, Y. Kim, and E. Lee. Denchmark: A bug benchmark of deep learning-related software. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 540–544. IEEE, 2021.DOI
- [29]B. Lyu, Y. Yang, S. Huang, J. Zhang, Q. Xu, X. Li, X. Han, Y. Zhang, H. Zhang, R. Huang, et al. MLS-Bench: A holistic and rigorous assessment of AI systems on building better AI. arXiv preprint arXiv:2605.08678, 2026.arxiv.org/abs/2605.08678
- [30]Q. Mang, W. Chai, Z. Li, H. Mao, S. Zhou, A. Du, H. Li, S. Liu, E. Chen, Y. Wang, et al. FrontierCS: Evolving challenges for evolving intelligence. arXiv preprint arXiv:2512.15699, 2025.arxiv.org/abs/2512.15699
- [31]M. M. Morovati, A. Nikanjam, F. Khomh, and Z. M. Jiang. Bugs in machine learning-based systems: a faultload benchmark. Empirical Software Engineering, 28(3):62, 2023.arxiv.org/abs/2206.12311
- [32]D. Nathani, L. Madaan, N. Roberts, N. Bashlykov, A. Menon, V. Moens, A. Budhiraja, D. Magka, V. Vorotilov, G. Chaurasia, et al. MLGym: A new framework and benchmark for advancing AI research agents. arXiv preprint arXiv:2502.14499, 2025.arxiv.org/abs/2502.14499
- [33]A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P.-S. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Novozin, P. Kohli, and M. Balog. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025. URL https://arxiv.org/abs/2506.13131.
- [34]J. Pan, X. Wang, G. Neubig, N. Jaitly, H. Ji, A. Suhr, and Y. Zhang. Training software engineering agents and verifiers with SWE-Gym. arXiv preprint arXiv:2412.21139, 2024.arxiv.org/abs/2412.21139
- [35]R. Qiang, Y. Zhuang, Y. Li, D. Sagar VK, R. Zhang, C. Li, I. Wong, S. Yang, P. Liang, C. Zhang, and B. Dai. MLE-Dojo: Interactive environments for empowering LLM agents in machine learning engineering. Advances in Neural Information Processing Systems, 38, 2025.arxiv.org/abs/2505.07782
- [36]R. Qiang, C. Li, H. Sun, Y. Zhuang, C. Zhang, and B. Dai. Matryoshka Agent: Unfolding sub-agents for long-horizon machine learning engineering. arXiv preprint arXiv:2607.25090, 2026a.arxiv.org/abs/2607.25090
- [37]R. Qiang, Y. Zhuang, A. Singh, P. Liang, C. Zhang, S. Yang, and B. Dai. MLE-Smith: Scaling MLE tasks with automated multi-agent pipeline. In International Conference on Learning Representations, volume 2026, pages 5963–5990, 2026b.arxiv.org/abs/2510.07307
- [38]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026a. URL https://qwen.ai/blog?id=qwen3.5.
- [39]Qwen Team. Qwen3.8-Max: A new bar for coding and cowork, August 2026b. URL https://qwen.ai/blog?id=qwen3.8.
- [40]Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. URL https://arxiv.org/abs/2402.03300.
- [41]E. Shen, D. Tormoen, S. Shah, A. Farhadi, and T. Dettmers. SERA: Soft-verified efficient repository agents. arXiv preprint arXiv:2601.20789, 2026.arxiv.org/abs/2601.20789
- [42]D. Shi, J. Cao, Q. Chen, W. Sun, W. Li, H. Lu, F. Dong, T. Qin, M. Liu, Y. Jiang, et al. TaskCraft: Automated generation of agentic tasks. In International Conference on Learning Representations, volume 2026, pages 43714–43734, 2026.arxiv.org/abs/2506.10055
- [44]J. Yang, K. Lieret, C. Jimenez, A. Wettig, K. Khandpur, Y. Zhang, B. Hui, O. Press, L. Schmidt, and D. Yang. SWE-smith: Scaling data for software engineering agents. Advances in Neural Information Processing Systems, 38, 2025a.arxiv.org/abs/2504.21798
- [45]J. Yang, C. Jiang, Y. Fu, T. Luo, C. Ren, W. Wang, K. Zhao, H. Liu, Y. Zuo, Y. Wang, et al. Frontis-MAI1: Training an AI4AI model towards recursive self-improvement in machine learning engineering. arXiv preprint arXiv:2607.28568, 2026a.arxiv.org/abs/2607.28568
- [46]J. Yang, K. Lieret, J. Ma, P. Thakkar, D. Pedchenko, S. Sootla, E. McMilin, P. Yin, R. Hou, G. Synnaeve, et al. ProgramBench: Can language models rebuild programs from scratch? arXiv preprint arXiv:2605.03546, 2026b.arxiv.org/abs/2605.03546
- [47]Z. Yang, S. Wang, Y. Yan, and Y. Deng. Why stop at one error? Benchmarking LLMs as data science code debuggers for multi-hop and multi-bug errors. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17377–17392, 2025b.arxiv.org/abs/2503.22388
- [48]C. Zhou, A. Liu, Y. Deng, Z. Zeng, T. Zhang, H. Zhu, J. Cai, Y. Mao, C. Zhang, L. Tan, et al. AutoCodeBench: Large language models are automatic code benchmark generators. In International Conference on Learning Representations, volume 2026, pages 79116–79148, 2026.arxiv.org/abs/2508.09101
- [49]Y. Zhou, S. Levine, J. Weston, X. Li, and S. Sukhbaatar. Self-challenging language model agents. Advances in Neural Information Processing Systems, 38:113959–113991, 2025.arxiv.org/abs/2506.01716



























