WorksComputer Science
BOXBOX: Recursive Harness Self-Improvement for Long-Running Performance Optimization
Abstract
Automated harness optimization improves agents by learning from trajectories and task feedback. Existing offline methods use reference answers to optimize harnesses for reuse on held-out tasks. However, many real-world tasks are open-ended, while long runs expose new difficulties that offline-optimized harnesses may not address. To study harness optimization in these settings, we introduce PerfRace, a benchmark for long-running performance optimization of software projects, and investigate whether agents can use task-specific experience to recursively improve their harnesses during an ongoing task. To this end, we introduce BOXBOX, which enables agent-triggered harness revision and kernel-managed hot reloading while preserving execution context. We evaluate BOXBOX on PerfRace under matched test-time budgets, comparing it with Frozen Harness, offline Harness Evolution, and Harness Scaling. BOXBOX achieves higher speedup with more efficient use of task time and model cost, reaching a Speedup@4h of 1.70x, compared with 1.54x for the strongest baseline.
Introduction
Large language model (LLM) agents interact with their environment through harness that determines both the available actions and the observations they receive [43 Merrill et al.,] 2026; Lin et al., [17]). Recent studies have shown that harness design can substantially affect agent performance and behavior, even when the underlying model is fixed (Lin et al., [17]; Wang et al., [40]). These findings have motivated automatic optimization of agent harness, in which an optimizer analyzes agent trajectories and evaluation feedback, proposes modifications to the harness, and evaluates the resulting agent on benchmark tasks.
Existing offline harness optimization methods (Park et al., [30]; Zhang et al., [47]; Lin et al., [17]; Wang et al., [40]) often use reference answers or task-completion feedback to diagnose failures, and revise harnesses to improve performance on held-out tasks. This setup aligns with existing agent benchmarks that evaluate whether an agent completes a specified task (Jimenez et al., [11]; Merrill et al., [24]; Froger et al., [6]).
However, many real-world tasks are open-ended and long-running, making end-state evaluations alone insufficient to capture how agents progress throughout these runs. Moreover, this setting poses two challenges for offline harness optimization. First, open-ended tasks make it difficult to identify useful harness revisions, as they may lack clear criteria for diagnosing what needs improvement. Prior evaluations show that harness evolution does not reliably improve performance without unit-test feedback (Wang et al., [40]). Second, continued execution exposes new task-specific difficulties that may limit the generalization of harness revisions learned from other tasks (Nie et al., [28]; Wang et al., [40]).
To study harness optimization in these settings, we introduce PERFRACE, a benchmark for optimizing performance of real-world software projects. Our experiments show that offline Harness Evolution (HE) underperforms Frozen Harness on PERFRACE (Figure 1), with no consistent improvement during training or validation (Appendix § 9.§ 2), consistent with prior findings of unreliable gains and limited generalization (Wang et al., [40]).
Figure 1. Performance of BOXBOX and baselines on 50 PERFRACE test tasks, with three runs per method and task. Curves show aggregate best-so-far correct speedup against task time (a) and mean cumulative model cost per run (b), sampled at matching task times, with shaded bands indicating 95% fixed-task run-resampling intervals. BOXBOX achieves higher speedup with greater time and cost efficiency than the baselines.
Nevertheless, the current task’s trajectory can reveal difficulties encountered during execution and provide task-specific evidence for harness revision (Nie et al., [28]). We therefore investigate whether an agent can use this experience to improve its own harness during the ongoing task, making harness optimization an online, task-specific process. Existing methods allocate budget between task execution and harness optimization through an outer loop, with harness updates scheduled by predefined triggers or a separate meta-agent (Wang et al., [40]; Liu et al., [19]; Zhang et al., [49]). However, external control limits the task-solving agent’s ability to respond to emerging difficulties. First, despite observing task progress, the agent cannot decide whether to spend its remaining budget on task execution or harness revision. Second, updates occur between task attempts or execution segments, delaying their use in the ongoing task.
To address these limitations, we introduce BoxBox1 to support agent-triggered revision of the full executable harness through kernel-managed hot reloading. When requested, the kernel pauses task execution and launches an agent review session to diagnose difficulties from the trajectory and revise the harness. After validation, it resumes the task with preserved execution context under the revised harness, which governs both subsequent execution and future reviews, enabling recursive harness improvement. We evaluate BoxBox on PERFRACE against Frozen Harness, offline Harness Evolution and externally controlled Harness Scaling under matched task budgets, comparing best-so-far correct speedup and model cost. BoxBox achieves a Speedup@4h of , compared with for Frozen Harness, for Harness Evolution, and for Harness Scaling.
Our main contributions are summarized as follows:
We introduce PERFRACE, a benchmark for evaluating agents on open-ended performance optimization tasks, tracking correctness-preserving improvements to real-world software throughout task execution.
We present BoxBox, which combines agent-triggered, full-harness revision from task-specific experience with kernel-managed hot reloading of the runtime while preserving execution context.
Our experiments show that BoxBox achieves higher best-so-far correct speedup with greater time and cost efficiency than the evaluated baselines, supporting harness optimization as part of an agent’s own capabilities.
Related Work
Automated harness optimization. Prior work has extensively studied the automated optimization of prompts for LLM-based systems [52, 32, 45, 13, 1]. Subsequent work extends automated optimization to agents, using execution trajectories and task feedback to improve their memory, skills, and workflows [51, 50, 27, 8, 48]. Meta-Harness and AHE further extend automated optimization to executable harness code, using trajectories and evaluation feedback to guide iterative revisions [16, 17].
Building on this setting, offline methods revise harnesses using reference answers or task-completion feedback from training tasks, aiming to improve held-out performance [30, 47, 17]. However, an evaluation finds unreliable gains without unit-test feedback and limited transfer from reusable harnesses, but reports gains from Harness Scaling, which alternates task attempts and harness revisions at test time [40].
Following this task-specific direction, JIT-Agent synthesizes a tailored harness for each task before execution begins [46]. TTHE further explores harness optimization without reference answers, using trajectories and execution-derived feedback to revise executable harnesses on unlabeled test batches [28]. Continual Harness moves harness optimization into ongoing task execution through an outer loop that periodically invokes a refiner while preserving the task state [12]. In BOXBOX, the task-solving agent decides when to allocate its remaining budget to full-harness revision, while a separate kernel manages runtime hot reloading and preserves execution context.
Long-Horizon and Open-Ended Agent Evaluation. Existing agent benchmarks cover diverse capabilities but commonly evaluate tasks with explicit end states, reporting task success, pass rates, or final artifact quality [25, 20, 11, 41, 24, 36]. Recent iterative optimization benchmarks evaluate solution quality but still use reference implementations, human performance, or reference outputs for task construction or scoring [2, 21, 10, 42, 53] (Mang et al. 2026). Closer to our setting, Frontier-Eng and FrontierSWE support iterative optimization of engineering designs and software projects, with tasks curated from established problems and expert contributions [3, 4].
In contrast, PerfRace constructs long-running performance optimization tasks from real-world software projects and workloads without requiring reference optimizations. Its construction pipeline provides a scalable way to continually source new tasks from public software projects.
Benchmarks for Performance Optimization. Shypula et al. [35] introduce PIE for evaluating LLMs’ ability to make performance-improving edits to single-file competitive programming solutions while preserving functional correctness. Subsequent benchmarks broaden the optimization domains to GPU kernels, general-purpose numerical programs, and algorithmic redesign, while retaining function- or file-level optimization tasks [5, 9, 18, 38, 29, 31].
GSO and SWE-Perf extend evaluation to repository-level performance optimization, deriving tasks and human reference optimizations from GitHub commits and pull requests [34, 7]. SWEfficiency and FormaCode further incorporate a targeted workload per task and multiple community-maintained workloads, respectively [22, 33].
These repository-level benchmarks construct tasks around a targeted performance test, whereas real performance engineering starts from an end-to-end workload and discovers bottlenecks through profiling. PerfRace adopts this workload-first formulation, leaving agents to identify and implement correctness-preserving improvements without a known target patch.
PerfRace
PerfRace benchmarks real-world software performance optimization with three design principles: (1) Live: projects are continually sourced from public GitHub repositories; (2) Open-ended: tasks use project workloads without reference optimizations or known target patches; and (3) Long horizon: agents iteratively optimize code and choose when to submit for feedback within a fixed budget.
Task Formulation
Each PERFRACE task provides the agent with an editable project workspace, a fixed workload that the agent can run locally, and access to an external Judge. For official evaluation, the Judge uses a separate copy of this workload and additional held-out correctness cases.
During the evaluation loop, the agent iteratively modifies the project and submits changes to the Judge for feedback. Let denote the evaluation start time of submission and its score, with for incorrect submissions. Scores are attributed retrospectively to evaluation start times; verification time also counts toward the task budget. We include the unmodified project as a baseline submission with and . We define the best-so-far correct speedup as , which the agent aims to maximize within a fixed task budget .
Construction
We construct PERFRACE by identifying real projects with optimization potential and building reliable tasks around realistic workloads (Figure 2a). Each PERFRACE task adopts the Harbor Task Format ([24]), as illustrated in Figure 2b.
Figure 2. Overview of PERFRACE. (a) Construction discovers projects from GitHub coding-agent activity, applies a two-stage Filter comprising a deterministic precheck and source-based assessment, and uses Agent–Human Pair Verification to produce a PERFRACE task. (b) Task format separates the agent workspace from hidden evaluation files. (c) The evaluation loop repeatedly submits workspace snapshots to the Judge and tracks best-so-far correct speedup.
Discovery. To identify less-explored optimization opportunities beyond mature, heavily maintained repositories, we search GitHub coding-agent activity for candidate projects. Specifically, we identified 103,475 candidate projects from public GitHub pull requests and branch pushes made by known coding agents between May and August 2026 (Appendix § 10.§ 3).
Filter. We filter candidate projects in two stages. First, deterministic prechecks verify repository and source-revision accessibility, license redistributability, and repository size and popularity, retaining 7,257 candidates. We then use GPT-5.6-Luna to assess each candidate’s setup process, evaluable workload, runtime feasibility, and optimization potential from its source code. Of these candidates, 239 pass this assessment and proceed to Pair Verification (Appendix § 10.§ 3).
Pair Verification. We use agent–human Pair Verification to construct runnable tasks and assess their quality against three criteria (Appendix § 10.§ 4):
Realism. The workload and metric represent meaningful project use.
Reliability. Repeated equivalent runs support the same correctness and performance conclusions.
Integrity. The score reflects genuine task completion rather than an invalid or shortcut solution.
Specifically, we first let the agent derive workload inputs and scale from real project use and propose correctness checks, measurement boundaries, and performance metrics for human review. Second, the agent configures warmup and runs repeated tests to detect flakiness and distinguish genuine speedups from noise; the reviewer uses these results to approve or revise the measurement protocol. Finally, the agent conducts red-team tests against reward hacking using invalid, incomplete, hard-coded, and work-skipping solutions; the reviewer requests verifier fixes as needed before approving the task for packaging and release. Eventually, we curated 100 tasks, each requiring an average of 9.1 hours of curation, 12 agent–human interactions, and $11.35 in agent cost; detailed statistics are provided in Appendix § 10.§ 3.
Benchmark Composition
The resulting benchmark contains 100 tasks across seven domains (Figure 3): developer tools (30 tasks) compiles, builds and validates software; data systems (25 tasks) ingests, indexes and queries data; scientific computing (15 tasks) runs simulations and numerical solvers; media processing (9 tasks) renders graphics and audio; geometry (8 tasks) processes maps and 3D models; machine learning (7 tasks) trains models and runs inference; and security (6 tasks) encrypts data and analyzes attack logs. Each task pairs a real project with a fixed workload; for example, one task optimizes CCC while it compiles SQLite. Appendix § 10.§ 2 provides a detailed characterization of the tasks.
Figure 3. Composition of PerfRace: seven workload domains (inner ring), 100 tasks (outer ring), and representative project–workload pairs (right).
Evaluation
In the evaluation loop, the agent chooses when to submit snapshots for correctness and performance feedback from the Judge (Figure 2(c)). The score is the speedup over the unmodified project for correct submissions and zero otherwise. Runs without a correctness-preserving improvement receive a score of . During the final five minutes, the agent is prompted to end optimization and leave the project in a coherent state for the Judge’s final evaluation.
Prior studies show that agents can inflate scores by tampering with evaluation code or accessing reference outputs ([39] (Von Airx et al. 2025)). We therefore use shadow verification, evaluating project snapshots in separate containers while keeping verifier code and held-out test data out of the agent’s workspace. Time spent in shadow verification counts toward . Appendix § 10.§ 5 provides further details on shadow verification.
BoxBox
We now present BoxBox for agent-triggered harness revision during long-running tasks. The following subsections present the problem formulation, explain the kernel-managed harness update mechanism, and describe a simple self-review strategy for diagnosis and revision.
Problem Formulation
Consider a fixed language model and an executable harness that may change during a task. Let , , and denote the active harness, environment state, and accumulated trajectory at interaction step , respectively ([44]). Task execution follows
Here, is a tool call from the action space exposed by the active harness, and is its distribution induced by constructing model inputs from and invoking . resolves and executes through the harness’s tool implementation, updates the environment state, and processes the resulting feedback into ; appends the action–observation pair to the history.
When a revision is attempted, the agent revises based on the interaction history , producing a candidate . If activated, it becomes the active harness for the next interaction; otherwise, execution continues under . Subsequent action selection and execution therefore use .
Kernel-Managed Harness Updates
BoxBox treats the harness as a task-specific implementation, allowing a revision to be tailored to the requirements of the ongoing task. To support such adaptations, BoxBox exposes the full executable harness, allowing the agent to translate task-specific needs into direct changes to tools, control flow, and state management. Let denote the set of harness implementations compatible with the execution interface. Methods with predefined editable components restrict candidate revisions to a subset ([47, 17, 30]), whereas BoxBox allows revisions .
In BoxBox, the harness is the editable agent implementation, while the runtime is its running version that controls the agent’s execution and maintains its state. These revisions change the agent’s execution code, so a faulty update can interrupt the task and make it difficult for the agent to recover on its own. BoxBox therefore introduces a separate kernel that manages runtime hot reloading and can safely retain or restore the previous runtime if an update fails validation or activation.
Specifically, we register boxbox() as a tool that yields control to the kernel, with boxbox() . When the agent invokes boxbox(), the kernel suspends task execution and preserves the agent’s execution context, including the accumulated trajectory, terminal session, and workspace state. It then launches a review session for the agent to examine the trajectory, revise the harness as needed, and commit any changes. After the review session concludes, the kernel validates any new commits, loads the revised runtime on success, and otherwise retains the current runtime. During activation, the kernel transfers the preserved execution context to the new runtime and reattaches the existing terminal session. On successful activation, , and subsequent interactions follow Eqn. (1). Finally, the kernel returns control to the agent, allowing it to resume the ongoing task.
Beyond agent-initiated updates, runtime faults also trigger kernel-managed recovery and review based on the recorded fault evidence. Appendix § 11 provides further implementation details.
Figure 4. BoxBox mechanism. The agent requests a review on demand, triggering a trap that transfers control to the kernel. The kernel then launches the review, validates revisions, and hot-reloads the runtime with preserved execution context and rollback if activation fails.
Agent-Controlled Harness Optimization
Based on this mechanism, BoxBox lets the task-solving agent decide when to request a harness review, so it can adapt its harness in response to difficulties encountered during task execution. Unlike prior methods that use an outer loop [16, 12, 19], the agent itself allocates the shared budget between project-code optimization and harness revision to maximize .
Within the review session, we provide a default skill (Appendix § 12.§ 3) that guides the agent through a simple self-review strategy based on diagnosis and harness revision:
Diagnosis. The agent examines the trajectory for execution failures, repeated work, and successful patterns that could be automated or reused. It uses these observations to identify limitations of the current harness, such as missing capabilities or costly tool interactions.
Revision. The agent selects harness revisions based on their expected benefit within the remaining budget, such as adding guidance, improving tools, or changing execution logic. For example, repeated tool calls can be combined into a reusable tool.
We track harness revisions as Git commits so the agent can inspect prior changes alongside execution records in subsequent reviews, assess their effects, and decide whether further revisions are needed.
Experiments
Experimental Setup
Baselines. We compare BoxBox with three baselines: Frozen Harness (FROZEN) retains the initial harness throughout task execution. Harness Evolution (HE) optimizes the harness offline on training tasks and selects it by validation performance, then holds it fixed during test-task execution. Harness Scaling (HS) uses an external controller to revise the harness between attempts on the same task, using feedback from each completed attempt. Both HE and HS are adapted from Wang et al. [40]. Following Meta-Harness [16], all methods start from the same initial harness , Terminus-KIRA [14], a simple terminal-based harness that executes shell commands and returns their outputs to the model.
Evaluation protocol. We split the 100 PerFRAce tasks into 40 training and 10 validation tasks for HE, and 50 held-out test tasks for evaluating all methods. Both task execution and harness optimization are performed using GPT-5.6 Luna with high reasoning effort. Each method is evaluated with three independent runs per test task, each sharing a four-hour budget () across task execution, harness optimization, and evaluation. Detailed configurations are provided in Appendix § 9.
Metrics. To account for randomness, we summarize task ’s best-so-far correct speedup across runs as . Let be its mean model cost in US dollars, including harness optimization. Across tasks, we use harmonic mean speedup to limit the influence of extreme values [34] and arithmetic mean task-level speedup per dollar:
Reported mean model cost averages across tasks. Cost accounting and statistical details appear in Appendix § 9.
Main Results
We report performance and model cost in Table 1 and plot speedup against task time and cumulative model cost in Figure 1. Appendix § 9.§ 5 provides optimization trajectories for all test tasks.
| Method | Speedup@4h | Mean model cost | Speedup/$ ↑ |
| FROZEN | 1.442 | $23.22 | 0.2142 | |
| HE | 1.281 | $24.56 | 0.1243 | |
| HS | 1.539 | $12.98 | 0.2872 | |
| BOXBOX | 1.702 | $16.43 | 0.4886 |
Table 1. Results on 50 test tasks with three four-hour runs per task, reporting Speedup@4h, mean model cost per run, and mean task-level speedup per dollar.
Offline harness optimization does not reliably improve performance. HE reaches 1.281× at four hours, below FROZEN (Table 1). HE struggles to identify consistently effective harness revisions during training, while the revisions selected through validation show limited generalization to long-running test tasks. Appendix § 9.§ 2 provides detailed results and failure analysis. HE costs an average of $24.56 per test run. Including offline training and validation costs, shared across the 150 test runs, brings this average to at least $33.73. This additional offline expenditure does not translate into better test performance or greater cost efficiency than FROZEN.
Task-specific harness optimization supports continued improvement. Both HS and BOXBOX improve over FROZEN, reaching 1.539× and 1.702× at four hours, respectively, compared with 1.442× for FROZEN (Table 1). Both also reduce mean model cost, to $12.98 and $16.43 per run, respectively, compared with $23.22 for FROZEN. Figure 1 shows the corresponding speedup trajectories against elapsed task time and mean cumulative model cost. Over the second half of the four-hour budget, the FROZEN curve largely levels off, increasing from 1.410× to 1.442×, while HS improves from 1.431× to 1.539× and BOXBOX from 1.530× to 1.702×. These trends show that task-specific harness optimization supports further progress as execution continues.
Agent-controlled harness optimization improves performance and efficiency. Figure 1(a) shows that BOXBOX achieves higher speedups than HS at matched elapsed task time. This result compares HS’s attempt-level external control with BOXBOX’s finer-grained, agent-triggered revision during task execution. Whereas HS relies on an external controller to initiate revisions, BOXBOX lets the agent decide when to allocate budget to adapting its harness to the needs of the ongoing task. This allows the agent to address harness limitations as they arise and apply useful revisions directly to the ongoing task. Figure 1(b) further shows higher speedups for BOXBOX at matched cumulative model costs. Although HS has a lower total model cost, BOXBOX achieves greater speedup per dollar under our task-level metric. Appendix § 9.§ 3 provides further analysis.
Ablation Studies and Analysis
We further conduct ablation studies on 14 randomly sampled test tasks to isolate the contributions of key design choices to the effectiveness and efficiency of harness optimization, then analyze how agents perform harness optimization during task execution.
Ablation Studies.
Agent-controlled triggering. Following the scheduled refinement in Continual Harness (Karten et al. [12]), the w/o Agent Triggering variant schedules reviews every 30 minutes with full-harness editing. It achieves lower speedup at higher model cost than agent-triggered BOXBOX (Table 2). Scheduled reviews can interrupt the process of proposing, implementing, and evaluating code changes. They also consume model calls even when they conclude that no harness revision is needed. In contrast, BOXBOX lets the agent decide when to allocate budget to harness optimization based on the needs of the ongoing task.
| Method | Speedup@4h | Mean model cost | Speedup/$ |
| w/o Agent Triggering | 1.508 | $21.58 | 0.1338 | |
| w/o Full Editing | 1.349 | $16.07 | 0.1132 | |
| Full BOXBOX | 1.584 | $15.07 | 0.3452 |
Table 2. Ablation study of BOXBOX. We examine agent-controlled triggering and full-harness editing in terms of task performance (Speedup@4h), model cost, and cost efficiency (Speedup/$).
Full-harness editing. Following Self-Harness (Zhang et al. [47]), the w/o Full Editing variant retains agent-controlled triggering but restricts each review to a predefined scope. It achieves lower speedup than BOXBOX (Table 2). Predefined editing boundaries can constrain task-specific adaptations that require changes across components or beyond the permitted scope. Full-harness editing lets the agent decide what to modify and how extensively based on the needs of the ongoing task.
Harness Optimization Analysis
Review frequency. BOXBOX invokes a median of five reviews per run, with 92% of runs containing three to six (Figure 5(a)). These results show that boxbox() provides an effective entry point for harness optimization during task execution, allowing agents to review their harnesses based on accumulated trajectories.

Figure 5. Harness optimization analysis. (a) Number of reviews per run across 150 test runs, including agent-triggered and fault-recovery reviews. (b) Per-run review time as a share of the four-hour task budget and review cost as a share of total model cost within that budget; dots represent runs, and boxes show the median and interquartile range. (c) Modification types across a collection of 620 applied, nonempty harness updates from BOXBOX runs.
Budget allocation. Agents allocate most of their time budget to task execution. Harness reviews account for a median of 2.1% of the time budget and 13.8% of model cost per run (Figure 5(b)). Together with BOXBOX’s higher speedup and lower model cost than FROZEN, these results suggest that agents can effectively balance task execution and harness optimization within a shared budget.
Modification scope. Harness modifications span prompt rules, skills, tools, execution logic, and fault handling (Figure 5(c)). These changes show that agents use full-harness editing to adapt both their capabilities and execution process. Cases in Appendix § 9.§ 4 illustrate how these modifications add capabilities, automate waiting, and revise context management.
The ablations show that agent-controlled triggering and full-harness editing improve performance and cost efficiency, while the analysis shows that agents adapt their capabilities and execution process while allocating most of their time to task execution. Together, these findings support harness optimization as part of the task-solving agent’s own capabilities.
Conclusion
In this work, we introduced PERFRACE, a benchmark for long-running performance optimization of real-world software, and BOXBOX, which combines agent-triggered, full-harness revision from task-specific experience with kernel-managed hot reloading that preserves execution context. On PERFRACE, offline Harness Evolution yields inconsistent gains, whereas BOXBOX achieves higher best-so-far correct speedup and greater time and cost efficiency than the evaluated baselines. It achieves a Speedup@4h of , compared with for the strongest baseline, and reduces mean model cost by 29.3% relative to Frozen Harness. These results support harness optimization as an agent capability for improving how it solves a task while continuing its execution.
Security and Responsible AI Limitations
BOXBOX allows an agent to modify executable harness code during an ongoing task, which introduces security and Responsible AI risks beyond those of a fixed harness. In particular, the agent executes and modifies code that may originate from untrusted repositories, and harness revisions may introduce security regressions, weaken existing safeguards, or create unintended execution behavior. More generally, self-modifying agent mechanisms could be misused to alter controls or expand capabilities in ways not intended by system operators. Our evaluation is conducted in isolated research environments with kernel-managed validation and rollback for harness updates and separate shadow verification for task evaluation. These mechanisms reduce experimental risk but do not constitute a complete security boundary or guarantee safe behavior under adversarial conditions. BOXBOX should therefore be viewed as a research prototype rather than a production-ready self-modifying agent framework. Any production deployment would require additional sandboxing, access control, security review, monitoring, and human oversight of agent-generated harness changes before they are activated.
Experimental Details
We describe the experimental environment and evaluation details in Appendix § 9.§ 1, followed by the experiments for Harness Evolution and Harness Scaling in Appendices § 9.§ 2 and § 9.§ 3. Appendix § 9.§ 4 examines how BoxBox harness revisions support task execution, and Appendix § 9.§ 5 provides additional per-task optimization trajectories.
Experimental Setup
Experimental environment. To facilitate replication of our experimental environment, we conduct experiments on an Azure Standard_D96ads_v5 virtual machine [26] with 96 vCPUs backed by an AMD EPYC 7763 processor and 384 GiB of system memory, running Ubuntu 24.04 with Docker.
Initial harness. All methods start from Terminus-KIRA [14], a terminal-based harness that executes shell commands and returns their outputs to the model. FROZEN retains this harness throughout task execution while the agent iteratively optimizes the project code.
Time and cost accounting. Figure 1 and Table 1 use the same 600 test runs: 50 tasks, four methods, and three runs per method and task. For all methods, elapsed task time sums recorded agent-execution and verification phases. Agent execution includes harness review; environment setup is excluded. Only evaluations started within the four-hour budget contribute to Speedup@4h. We calculate model cost from recorded token usage using the rates in Table 3. Reported mean model costs per test run include all recorded model calls; cost and token curves report cumulative usage at each task-budget time point.
| Model | Uncached input | Cached input | Output |
| GPT-5.6-Luna | $1.00 | $0.10 | $6.00 | |
| GPT-5.6-Terra | $2.50 | $0.25 | $15.00 |
Table 3. Model token rates used for cost calculation (USD per million tokens).
Statistical analysis. The bands in Figure 1 show the 2.5th and 97.5th percentiles of 5,000 resampled benchmark runs over the fixed 50-task set. Each sample independently selects one of the three observed runs per task and aggregates the selected scores using the harmonic mean. The central curves instead use all three runs per task, with the geometric-then-harmonic aggregation defined in Equation (2). In the cost panel, speedup intervals are plotted at the mean cumulative cost for each task-time point; they do not represent uncertainty in cost.
Harness Evolution
Our HE baseline adapts the implementation of Wang et al. [40] to PERFRACE, retaining its core diagnosis-and-revision workflow. We detail the offline optimization procedure and examine training progress and the generalization of the selected harness to held-out tasks.
Setup. We adapt task execution, evaluation, and experiment orchestration to PERFRACE, replacing the task-success objective with best-so-far correct speedup under a fixed time budget and tailoring the diagnostic and revision prompts to guide harness improvements that increase speedup and reduce costs.
The reference HE experiments use full-batch training on Terminal-Bench 2.1 [40]. Publicly released mini-swe-agent trajectories generated with DeepSeek-V4-Flash on this benchmark [15] average 50 agent steps and 12.2 minutes per run, compared with 965 steps for our FROZEN runs under a four-hour budget on PERFRACE. We therefore limit each training run to 45 minutes and partition the 40 training tasks into mini-batches of five to keep the volume of trajectory evidence manageable for diagnosis. Across three reshuffled epochs (24 iterations), we evaluate on one mini-batch at iteration and use the resulting trajectories and evaluation outcomes to produce . For validation, we compare , , and using fresh baseline measurements. Table 4 summarizes the HE pipeline and its recorded model costs.
| Stage | Tasks | Evaluated runs | Budget per run | Cost ($) |
| Training | 40 | 24 iterations × 5 tasks | 45 min | 200.34 |
| Validation | 10 | 3 harnesses × 3 runs/task | 2 h | 1,176.12 |
| Test | 50 | h24, 3 runs/task | 4 h | 3,683.34 |
Table 4. HE configuration and recorded model costs. Training and validation costs include retries and reruns and are lower bounds where usage records are missing.
Training, validation, and test results. Training performance does not improve consistently across epochs (Figure 6(a)). Aggregate speedup decreases from 1.086× in epoch 1 to 1.062× in epoch 2 and recovers to 1.069× in epoch 3, remaining below its initial level. Validation performance does not improve consistently across iterations (Figure 6(b)), with achieving the highest score among the evaluated checkpoints. We therefore use for testing, where it achieves 1.281×, below Frozen Harness at 1.442× (Table 1). These results show that additional offline revisions do not reliably improve training performance or yield a test-time advantage, consistent with prior findings of limited transfer [40]. Including the recorded offline cost raises mean cost to at least $33.73 per test run.

Figure 6. HE training and validation performance. (a) Mini-batch training speedup, with aggregate scores for each epoch. (b) Validation speedup of harness checkpoints , , and . Speedups are aggregated as in Equation (2).
Failure analysis. To understand HE’s failure modes, we examine how a revision introduced during training affected execution on a test task. At iteration 5, diagnostic reports identified two patterns: (1) measurements showed faster execution of the baseline code, which had either remained unchanged or been restored after edits were discarded; and (2) records of faster execution did not specify which code version had been measured. These patterns made it difficult to determine whether measured gains reflected actual code improvements or measurement noise. HE therefore added a check to the harness to prevent agents from submitting code without evidence of an actual performance improvement.
Specifically, the agent had to record the current and baseline commits, at least three timing measurements for each version, and correctness evidence in .autoresearch/promotion.json. The harness checked that the record matched the current commit, the worktree was clean, the code differed from the baseline, and the measured improvement exceeded a noise threshold. If these requirements were not met, it blocked submission and returned feedback directing the agent to repair. The check read this record and inspected Git state in the working directory selected by the harness (Figure 7).

Figure 7. Selected additions to HE’s harness at iteration 5. The harness checks the repository and performance evidence before submission, blocking submission and returning repair instructions when a check fails.
The check enforced fixed requirements based on failures observed during training. However, its assumptions about repository location did not hold for some test tasks, causing it to block submissions before assessing the performance evidence. To examine this failure pattern, we selected mirage-kubernetes-triage for detailed trajectory analysis.
In the selected run, the agent prepared Git commits and performance evidence inside the editable repository, while the harness ran its submission checks in the parent workspace. The check therefore failed to resolve the candidate commit and returned cannot resolve candidate HEAD, blocking submission before assessing the performance evidence. Repeated attempts to repair the evidence in the repository did not change the directory used by the harness. The trajectory records 473 blocked submission requests spanning approximately 3 hours and 46 minutes.
This case illustrates both challenges in § 1. Difficulty in attributing measured gains motivated stricter submission requirements, but did not establish that these requirements would help the agent find better optimizations. On the selected test task, the check’s repository-location assumptions instead led to repeated repair attempts and blocked submissions, illustrating how a revision intended to address training failures can hinder execution under new task-specific conditions.
Harness Scaling
Our HS baseline adapts the implementation of Wang et al. [40] to PerfRace. We describe the execution procedure, analyze model costs, and examine a failure case.
Setup. We adapt task execution and evaluation to PERFRACE, using best-so-far correct speedup under a fixed time budget as the optimization objective. Each run shares a four-hour budget across task execution, harness optimization, and evaluation. An external controller revises the harness using feedback from each completed attempt and starts a new agent session, preserving project-code progress and resetting the execution context.
We record 2,132 HS attempts, with a median of 12 per run. Across 1,983 boundaries, the median interval between one attempt ending and the next beginning is 3.37 minutes, including harness optimization and runtime transition.
Model costs. HS’s session resets can lower input-token costs by reducing the accumulated conversation history. Although project-code progress is retained, subsequent attempts may need to reconstruct earlier decisions or repeat experiments whose outcomes are no longer available in the execution context. BOXBOX preserves this context across harness updates, allowing subsequent optimization to build directly on earlier findings. This trade-off helps explain why HS’s lower total model cost does not necessarily translate into greater optimization progress per dollar.
Failure analysis. To examine HS’s optimization process, we trace one kitops-minilm-pack run (Figure 8). During one harness optimization step, HS’s controller reported that the agent repeatedly rebuilt the project, ran tests, and reran the workload on the same code version. It therefore added a repetition check to warn after three consecutive matching command batches.
Figure 8. HS optimization trajectory on kitops-minilm-pack. Annotations show selected harness revisions and an agent session containing 113 consecutive checks that produced no new performance measurements.
However, the new repetition check did not stop the agent from performing the same checks using different command forms. The agent ran 113 consecutive commands repeating the same checks, without modifying code or collecting new performance measurements. The check only looked for consecutive repetitions of the same commands, so it could miss repeated checks written in different ways. The run reached its best correct speedup of 1.649× at checkpoint 5, with no further improvement in subsequent recorded checkpoints. The repeated checks revealed a need for further harness revision during execution, yet HS performs harness optimization only between agent sessions, leaving the current session to continue under the existing harness.
This case illustrates how session-level external control can delay harness revisions needed during ongoing task execution. The revised check missed repeated validation expressed through different commands, and the agent continued spending task time without making code changes or collecting new performance measurements. Allowing the agent to initiate further harness revision when these limitations become apparent would provide an earlier opportunity to address them.
Case Study
To analyze how BOXBOX supports effective task execution, we examine two complementary aspects of harness optimization. Appendix A.4.1 traces how agent-triggered reviews address difficulties encountered during ongoing tasks. Appendix A.4.2 examines how full-harness editing can support task performance and reduce model costs through changes to agent guidance, tools, and execution behavior.
Harness Optimization Trajectories
We examine two four-hour runs, tracing the difficulties encountered during task execution, the resulting harness revisions, and their use in subsequent optimization.
CCC. To reduce CCC’s compilation time on SQLite, the agent profiles compiler stages and tests code changes through repeated compilation runs (Figure 9). During the first review, the agent identified repeated manual processing of timing output and added a helper for repeated runs, result recording, and candidate comparisons. Using this helper, the agent established a baseline and evaluated compiler changes, but still had to extract and compare individual stage timings manually. To automate this step, the agent added stage-timing summaries to the helper in the second review, then used them to investigate optimization passes and code generation.
Figure 9. BoxBox optimization trajectory on ccc--compile--speed. Points show correct evaluation results and the line shows best-so-far speedup. Annotations mark harness review outcomes and task-code changes; numerical gains are absolute and relative increases over the preceding best-so-far speedup.
Later, a change to CCC’s source code appeared to reduce compilation time by 9.4% in separately measured batches. Yet when the agent alternated runs of the modified and unchanged compiler versions, their median compilation times were nearly identical, suggesting that the apparent gain came from measurement variation between batches rather than the code change. The agent therefore incorporated this comparison procedure into the helper during the third review to improve measurement reliability, then used it to reject several compiler changes that showed no consistent speedup. Continuing with this helper, the agent developed mnemonic-parsing and operand-splitting changes that reduced official running time from 6.321 to 6.095 and then 5.792 seconds.
FluidSynth. To optimize FluidSynth’s MIDI-to-WAV rendering performance, the agent tests code changes through repeated workload execution and output checks. Figure 10 traces harness revisions during this process. The first review noted that long-running commands required extra model calls to continue waiting, but retained the harness because a proposed waiting tool would duplicate existing functionality. As this pattern recurred, the second review extended waiting inside the existing command runner. During subsequent audio-write batching experiments, a single tool call completed a build, five renders, and output checks, allowing the agent to assess the candidate without an additional waiting call.
Figure 10. BOXBOX optimization trajectory on fluidsynth--bagatelles, using the plotting conventions of Figure 9. Reviews extend command waiting and add batch benchmarking.
Comparing batch sizes still required repeated setup of builds, timing loops, and output checks. The third review therefore added benchmark_matrix, reusing the updated command runner to execute labeled experiments and report timings and exit status. The agent then used it for six repeated renders with output-hash checks and subsequent configuration comparisons. Rendering performance improved further after the agent disabled OpenMP in the build configuration. The fourth review retained the harness, judging its existing capabilities sufficient after the agent resolved a build-state issue through a clean rebuild. The run reached a best-so-far speedup of .
Together, these cases illustrate how harness optimization can improve task performance and reduce model costs by improving the agent’s performance optimization workflow, even though the reviews do not directly modify project code.
Representative Harness Modifications
We examine five representative harness updates that change agent guidance, tools, or execution behavior. While Figure 5(c) reports the prevalence of modification categories, Table 5 summarizes the concrete changes analyzed below.
| Case | Category | Task ID | Modification |
| 1 | Review feedback | n8n-mcp--tool-batch-speed | Redirect the next experiment toward SQLJS conversion and statement overhead. |
| 2 | Prompt rules and skills | sum--text-attest | Require alternating A/B measurements when apparent gains are close to noise. |
| 3 | Tool functionality | rust-clippy--cargo-package-lint | Extend internal command polling to reduce model-directed waiting turns. |
| 4 | Execution logic | trixi-framework-trixi-jl--self-gravitating-sedov-amr | Add a message-count-based context-compaction path. |
| 5 | Fault fix | rust-clippy--cargo-package-lint | Replace an exception on shell exit with a session-recovery call. |
Table 5. Representative harness updates grouped by modification type. Case numbers refer to the detailed analyses below; task IDs follow the catalog in Table 7.
The following cases proceed from changes in agent guidance to changes in executable behavior. Cases 1–2 examine review feedback and prompt-based guidance for experiment selection and evaluation. Case 3 examines internal command polling to reduce model-directed waiting. Cases 4–5 turn to context management and terminal recovery within the agent loop. Together, they illustrate how full-harness editing can affect both the direction of the search and the process used to carry it out.
Case 1. Review feedback: redirecting an experiment
Task: n8n-mcp--tool-batch-speed
Background. The task is to reduce the end-to-end running time of the n8n-mcp server on a fixed batch of workflow-building tool calls while preserving response correctness. The agent attempted to reduce logging overhead, but local measurements showed no improvement. The review therefore used profiling evidence to identify a different direction for the next experiment.
+ The remaining profiler clue is the SQLJS adapter's repeated `all/get`
+ row/object conversions and statement lifecycle overhead. Treat this as
+ a hypothesis, not a conclusion: inspect the exact hot path and design
+ one general correctness-preserving experiment ...
+
+ ... Use a separate branch from main, build, run the exact workload with
+ paired repeated clean-vs-experiment medians, and compare response hashes.
+ Reject changes within noise or with any response mismatch.
Review feedback. The feedback directs the agent to stop revisiting logging and investigate SQLJS data-conversion and statement-management overhead. It also specifies an experimental workflow, asking the agent to preserve the current project-code version and test modifications on a separate branch through paired repeated workload measurements and response-hash comparisons. Modifications should be rejected if their measured gains fall within measurement noise or their responses differ.
Discussion. This case illustrates how review feedback can support performance optimization by guiding both experiment selection and evaluation. Redirecting effort away from an unsuccessful direction can reduce repeated exploration, while paired measurements and response checks help the agent distinguish useful code changes from measurement noise or incorrect behavior.
Case 2. Prompt rules and skills: comparing noisy measurements
Task: sum--text-attest
Background. The task is to reduce the running time of SUM’s batch text-attestation workflow while preserving its outputs. Across successive experiments, the agent encountered apparent gains that did not hold under more controlled measurements. Earlier reviews had already warned against drawing conclusions from warmup effects and uncontrolled measurement order, yet a later code change again appeared promising based on a single fast run. To address this recurring problem, the review incorporated the measurement guidance into the main prompt for use across subsequent experiments.
+ When a candidate's apparent gain is comparable to run-to-run noise, do not
+ infer from one fast first run. Keep the unchanged `main` reference and
+ candidate branch, warm both in the same controlled procedure, then run
+ alternating fresh-process A/B repetitions with identical output checks
+ and compare medians (and spread) before accepting or rejecting the hypothesis.
+ Align this local procedure with the official workload boundary before
+ using a tiny timing difference to justify submission.
Harness update. The prompt rule instructs the agent to keep an unchanged reference version and test modified code on a separate branch. It requires the same warmup procedure for both versions, followed by alternating fresh-process workload runs with identical output checks. The agent should compare median running times and their spread before accepting or rejecting the modification, with the measurement procedure aligned with the official workload.
Discussion. This case illustrates how prompt revisions can provide consistent guidance for evaluating code changes across successive experiments. More reliable comparisons can help the agent distinguish genuine improvements from measurement noise, reducing the risk of retaining ineffective changes or discarding useful ones based on unstable timing evidence.
Case 3. Tool functionality: reducing model calls for command polling
Task: rust-clippy--cargo-package-lint
Background. During performance optimization on Rust Clippy, the agent builds the project and runs benchmarks to evaluate code changes. The command tool could return before these operations finished, requiring additional model calls simply to continue waiting. The review therefore extended the tool’s internal waiting behavior to reduce these repeated interactions.
Harness update. The update extends internal waiting to 180 seconds for ordinary commands, subject to the existing block timeout. The tool continues checking for the completion marker and returns as soon as it appears, or provides a progress snapshot when the waiting window expires.
+ AUTO_COMMAND_WAIT_SEC = 180
...
+ wait_budget = min(
+ BLOCK_TIMEOUT_SEC,
+ max(command.duration_sec, AUTO_COMMAND_WAIT_SEC),
+ )
...
- while time.monotonic() - start < command.duration_sec:
+ while time.monotonic() - start < wait_budget:
The tool description is updated alongside the implementation, instructing the agent to request an additional wait only when the command outlasts the new window.
- "It is always possible to wait again if the prior output has not finished, "
- "by running empty keystrokes with a duration on subsequent requests to wait
longer. "
+ "The harness automatically continues waiting when the completion marker has "
+ "not appeared, up to 180 seconds per tool call, and returns a progress
snapshot "
+ "at that bound. "
+ "Use an empty-keystroke wait only when a command exceeds that bound; "
+ "use C-c to interrupt foreground work. "
Discussion. This case illustrates how full-harness editing allows agents to adjust tool behavior adaptively to reduce model calls spent solely on waiting. When an unfinished command returns control to the agent, another model call processes the conversation context and generates a continuation request, incurring input- and output-token costs. Internal polling checks command status without invoking the model, allowing commands that finish within the extended window to return their results without these additional calls. Fewer waiting turns also add fewer messages to subsequent context. The potential cost reduction therefore comes from reducing model interaction during command execution, while commands exceeding the window may still require further calls.
Case 4. Execution logic: context compaction
Task: trixi-framework-trixi-jl--self-gravitating-sedov-amr
Background. During performance optimization on Trixi, the agent accumulated a long interaction history through profiling and repeated experiments. The review reported that late model requests contained approximately 145,000 input tokens, making repeated processing of the accumulated history a substantial source of model cost. The existing summarization mechanism had not sufficiently limited this growth, motivating a revision to the agent loop’s context-management logic.
+ if (
+ original_instruction
+ and len(chat.messages) >= 64
+ and len(chat.messages) >= self._last_forced_compaction_messages + 24
+ ):
+ self._last_forced_compaction_messages = len(chat.messages)
+ proactive_summary_result = await self._force_proactive_compaction(
+ chat, original_instruction, self._session,
+ )
+ if proactive_summary_result:
+ prompt, subagent_refs = proactive_summary_result
+ self._pending_subagent_refs = subagent_refs
+ self._pending_handoff_prompt = prompt
if original_instruction and self._enable_summarize:
...
Harness update. The update adds a message-count trigger before the existing optional summarization check. When the original task instruction is available, the loop attempts compaction if the history contains at least 64 messages and its message count is at least 24 higher than the count recorded at the last attempt. It records the current count before invoking the existing summarizer through a helper with a timeout. Successful results supply the continuation prompt and subagent references; if summarization fails, execution continues with the existing history.
Discussion. This case illustrates how full-harness editing allows agents to adapt context-management logic during ongoing tasks. By summarizing accumulated interaction history, the revised loop can reduce the amount of context processed in subsequent model calls and the associated input-token costs.
Case 5. Fault fix: terminal recovery
Task: rust-clippy--cargo-package-lint
Background. While the agent was waiting for a long Rust Clippy build, the persistent terminal shell exited. At the next agent-loop entry, the harness raised an exception before its existing session-recovery helper could run. This failure prompted a review of how the harness handles terminal loss during ongoing execution.
- if not await self._with_block_timeout(self._session.is_session_alive()):
- raise RuntimeError("The persistent terminal shell exited")
+ # The previous episode may have outlived the persistent shell.
+ # Restart it for the next model turn without replaying any
+ # command whose completion was uncertain.
+ await self._with_block_timeout(self.ensure_agent_session())
Harness update. The update replaces the exception with a call to the existing session-recovery helper, which preserves a live terminal or starts a replacement if the shell has exited. This prepares a usable terminal for the next model turn without automatically replaying commands whose completion is uncertain. Such commands may already have modified project files, so replaying them could duplicate their effects.
Discussion. This case illustrates how full-harness editing allows agents to revise error-handling logic when runtime faults interrupt ongoing tasks. By restoring terminal access, the revised harness can enable the agent to inspect the current task state and resume performance optimization after a shell failure.
Together, these cases illustrate how full-harness editing lets agents adapt guidance, tools, and execution logic to the needs of ongoing tasks. Changes to experiment selection and measurement can support task performance by focusing effort on promising hypotheses and improving the reliability of candidate evaluation. Changes to tool execution and context management can reduce model costs by avoiding unnecessary waiting calls and repeated transmission of long histories, while fault-handling revisions support continued execution.
Per-Task Optimization Trajectories
Figures 11–15 show optimization trajectories for all 50 test tasks. Each curve geometrically averages three runs, following the scoring and time-accounting conventions of Figure 1. The vertical scale varies by task.
Figure 11. Per-task optimization trajectories (1/5).
Figure 12. Per-task optimization trajectories (2/5).
Figure 13. Per-task optimization trajectories (3/5).
Figure 14. Per-task optimization trajectories (4/5).
Figure 15. Per-task optimization trajectories (5/5).
PerfRace: Benchmark Details
The PERFRACE appendix is organized as follows: Section B.1 compares PERFRACE with related benchmarks; Section B.2 characterizes its 100 tasks; Section B.3 reports construction and curation statistics; Section B.4 describes the Pair Verification protocol and quality control; and Section B.5 details the evaluation protocol and analyzes gains from longer optimization budgets.
Benchmark Curation and Evaluation
Table 6 compares PERFRACE with representative ultra-long-horizon agent benchmarks and code-optimization benchmarks along task type, task construction, and evaluation procedure. Existing ultra-long-horizon benchmarks largely assemble fixed suites of hand-crafted or expert-curated tasks. In contrast, PERFRACE combines live GitHub discovery with agent–human Pair Verification to continually construct validated optimization tasks from real-world projects and workloads. Compared with existing code-optimization benchmarks, it does not derive tasks from known performance patches and evaluates progress repeatedly rather than only at task completion.
| Benchmark | Task type | Task construction | Evaluation |
| PERFRACE | Repository optimization on end-to-end workloads | Live GitHub discovery followed by agent–human Pair Verification | Iterative correctness and performance feedback from an external Judge |
| AutoLab [42] | Diverse metric-driven optimization tasks | Expert-curated optimization tasks | Local evaluation during the task; held-out final evaluation |
| EdgeBench [53] | Cross-domain iterative problem-solving tasks | Expert-curated day-scale tasks | Local inner loop; repeated submission-gated Judge feedback |
| FrontierSWE [4] | Ultra-long-horizon software engineering and research tasks | Real-world problems curated with academic and industry experts | Final graded submission after up to 20 hours |
| Terminal-Bench [24] | General terminal tasks | Crowdsourced tasks with multi-stage human review | Final-only test-based evaluation |
| GSO [34] | Microbenchmark-based repository optimization | Repository commit histories and generated performance tests | Final-only microbenchmark-based evaluation |
| SWE-Perf [7] | Microbenchmark-based repository optimization | Historical performance PRs and selected repository tests | Final-only microbenchmark-based evaluation |
| SWE-efficiency [22] | Microbenchmark-based repository optimization | Performance PRs and manually reconstructed workloads | Final-only microbenchmark-based evaluation |
| FormulaCode [33] | Microbenchmark-based repository optimization | Continuously ingested performance PRs and ASV workloads | Final-only microbenchmark-based evaluation |
Table 6. Comparison with related benchmarks. Microbenchmarks measure targeted operations (for example, selected API calls), whereas end-to-end workloads measure complete project-use workflows.
A workload specifies the inputs and operations used to exercise a project. A microbenchmark measures a narrowly scoped part of that work, such as a selected API call or operation. For example, SWE-fficiency includes a performance workload that repeatedly times a NumPy einsum call [22]. An end-to-end workload instead measures a complete project-use workflow. PERFRACE adopts this broader measurement boundary, leaving agents to profile the workflow and identify which components to optimize. The distinction is therefore the scope of the measured computation: targeted operations versus an end-to-end workflow, rather than whether the benchmark supplies a workload.
Detailed Benchmark Composition
We characterize the 100 PERFRACE tasks by project purpose, primary programming language, license, packaged source size, file count, and fixed workload.
Projects. The projects represented in PERFRACE span 16 primary programming languages: Rust (25 tasks), Python (18), TypeScript (15), C++ (11), C (6), C# (6), Julia (5), Go (4), Java (3), Scala (1), Lua (1), OpenSCAD (1), Solidity (1), MATLAB (1), JavaScript (1), and Zig (1). The packaged project sources range from 42 to 15,869 files and from 471.49 KiB to 689.08 MiB, with medians of 886 files and 14.56 MiB.
Figure 16. Distribution of primary programming languages, packaged source sizes, and packaged source file counts across PERFRACE projects.
Task catalog. Table 7 lists each task’s project, purpose, license, and fixed workload.
| Task | Project and purpose | License | Fixed workload |
| aiidateam-aiida-core--battery-archive-import | aiidateam/aiida-core Scientific workflow and provenance platform | MIT | AiiDA import of a public battery-cycling provenance archive |
| anyrender--openmoji | DioxusLabs/anyrender Rust 2D drawing abstraction | Apache-2.0 | AnyRender batch rasterization of the OpenMoji SVG corpus |
| apalache--zcash-sync | apalache-mc/apalache Symbolic model checker for TLA+ and Quint | Apache-2.0 | Apalache’s bounded safety check for the Zcash v2 block-download scheduler |
| arrow-rs--variant-wikidata | apache/arrow-rs Rust implementation of Apache Arrow | Apache-2.0 | apache/arrow-rs while encoding Wikidata entities as a Variant column |
| bevy-voxel-world--terrain-streaming | splashdust/bevy_voxel_world Voxel-world plugin for Bevy | Apache-2.0 | splashdust/bevy_voxel_world terrain generation and chunk meshing |
| bm25s--trec-covid-index | xhluca/bm25s BM25 search library | MIT | bm25s indexing and persistence for the TREC-COVID research collection |
| boyter-scc--linux-v6-19-json-scan | boyter/scc Source-code counting and complexity tool | MIT | SCC whole-repository JSON scan time on Linux v6.19 |
| calldiff--pi-review | tanishqkancharla/calldiff Function call-stack diff tool | MIT | Calldiff JSON call-stack diffs for a real Pi coding-agent review |
| cbcoutinho-nextcloud-mcp-server--ohr-pdf-ingest | cbcoutinho/nextcloud-mcp-server MCP server for Nextcloud | AGPL-3.0 | Nextcloud MCP Server’s born-digital PDF ingest and page-aware chunking throughput |
| Task | Project and purpose | License | Fixed workload |
| ccc--compile-speed | anthropics/claudes-c-compiler Dependency-free C compiler | CC0-1.0 | CCC compile-to-object latency on the SQLite 3.46.0 amalgamation |
| chaste--ttp06-sheet | Chaste/Chaste Biomedical simulation framework | BSD-3-Clause | Chaste cardiac simulation for a TTP06 ventricular sheet propagation |
| cmtraceopen--hadoop-logs | adamgell/cmtraceopen Cross-platform Windows log viewer | MIT | CMTrace Open while loading a real Hadoop/YARN log session |
| cocoon--oci-import | cocoonstack/cocoon MicroVM engine | MIT | local import of a supported Cocoon Ubuntu VM OCI image |
| codediff--neovim-release | esmuellert/codediff.nvim Neovim code-review and diff plugin | MIT | CodeDiff C diff latency for a real Neovim release review |
| commitizen--check | commitizen-tools/commitizen Commit and changelog automation tool | MIT | Commitizen changelog generation for Angular’s complete Git history |
| core--blob-upload | calimero-network/core Peer-to-peer application platform | Apache-2.0 | Calimero content-addressed blob ingestion for a real OpenStreetMap file upload |
| cpp2rust--woff2-compress | Cpp2Rust/cpp2rust C++-to-safe-Rust translator | MIT | Cpp2Rust’s safe Rust output for public-font WOFF2 compression |
| deepinv--kohler-kernels | deepinv/deepinv Deep-learning inverse-problems library | BSD-3-Clause | DeepInverse spatially varying blur-kernel estimation on a real Kohler camera-motion-blurred image |
| dftd4--lysozyme-gradient | dftd4/dftd4 Atomic-charge-dependent dispersion correction | LGPL-3.0 | DFT-D4 energy, analytic gradient, and virial evaluation for a real lysozyme structure |
| dither-guy--photo-batch | manoelpiovesan/dither-guy Image dithering application | GPL-3.0 | Dither Guy’s full-resolution photo batch export |
| doc-detective--redpanda-transforms | doc-detective/doc-detective Documentation testing framework | AGPL-3.0 | Doc Detective dry-run planning for Redpanda transform documentation |
| drasi-core--gtfs-delay-replay | drasi-project/drasi-core Continuous-query engine | Apache-2.0 | Drasi Core’s continuous Cypher evaluation for replayed GTFS-Realtime vehicle geofence updates |
| duckdb--taxi-ingest | Giorgi/DuckDB.NET .NET bindings for DuckDB | MIT | DuckDB.NET ingestion of NYC Yellow Taxi trip records |
| dusty--fpindex-ingest | lalinsky/dusty Zig HTTP client/server library | MIT | Dusty’s HTTP batch-ingestion path for real AcoustID fpindex updates |
| embree--chess-render | RenderKit/embree Ray-tracing kernels | Apache-2.0 | Embree triangle traversal for a public chess-scene visibility render |
| endee--esci | endee-io/endee Vector database | AGPL-3.0 | Endee HTTP dense search for a replay of Amazon Shopping Queries product searches |
| enzyme--mnist | EnzymeAD/Enzyme.jl Julia automatic-differentiation library | MIT | Enzyme.jl for Flux’s CPU LeNet MNIST training epoch |
| eole--gpt2-wikitext-generation | eole-nlp/eole Language-modeling toolkit | MIT | EOLE CPU GPT-2 batch text generation on WikiText-103 prompts |
| eucalyptus--config-render-speed | curvelogic/eucalypt Configuration language and interpreter | MIT OR Apache-2.0 | the eucalypt interpreter on a config-generation workload |
| eucalypt--eval-speed | curvelogic/eucalypt Configuration language and interpreter | MIT OR Apache-2.0 | the eucalypt interpreter on an Advent of Code workload |
| facet--venta-guild | Tim-Maes/Facet .NET DTO and mapping source generator | MIT | Facet package and source-generator compilation time for venta’s Guild service |
| flexmeasures--smard-price-import | FlexMeasures/flexmeasures Energy-management platform | Apache-2.0 | FlexMeasures import of a full year of public SMARD day-ahead prices |
| fluidsynth--fluidsynth--bagatelles-midi-render | FluidSynth/fluidsynth SoundFont software synthesizer | LGPL-2.1 | FluidSynth offline rendering of a complete CC0 MIDI performance to WAV |
| geoparquet-io--hilbert-sort | geoparquet/geoparquet-io GeoParquet tooling | Apache-2.0 | GeoParquet Hilbert ordering of an Overture Buildings shard |
| gridflock--drawer-stl | yawkat/GridFlock Gridfinity baseplate generator | MIT/CC-BY-4.0 | GridFlock’s segmented Gridfinity drawer-baseplate STL export |
| Task | Project and purpose | License | Fixed workload |
| ifcopenshell--gni-bimproject-obj-export | IfcOpenShell/IfcOpenShell — IFC library and geometry engine | LGPL-3.0 | IfcOpenShell IfcConvert OBJ export for a lighter GNI BIM Project architectural IFC delivery |
| intro-skipper--credit | intro-skipper/intro-skipper — Jellyfin intro and credits detector | GPL-3.0 | Intro Skipper’s movie credits black-frame evidence extraction |
| jagua--nesting | JeroenGar/jagua-rs — 2D collision engine for nesting | MPL-2.0 | jagua-rs collision collection replayed from a downstream nesting solver |
| javaevolved--site-build | javaevolved/javaevolved.github.io — Java developer documentation site | MIT | the full localized GitHub Pages build for java.evolved |
| jsonnet--kubernetes-mixin-speed | google/jsonnet — Data templating language | Apache-2.0 | the Jsonnet C++ interpreter on a Kubernetes monitoring configuration workload |
| juliaio-zarr-jl--oceananigans-timeseries-output | JuliaIO/Zarr.jl — Julia implementation of Zarr array storage | MIT | Zarr.jl directory-store output for a real Oceananigans tracer time series |
| julialang-julia--trixi-jeans-instability | JuliaLang/julia — Julia programming language | MIT | Julia runtime execution for a published Trixi Jeans gravitational-instability simulation |
| kitops--minilm-pack | kitops-ml/kitops — OCI packaging tool for ML artifacts | Apache-2.0 | KitOps local ModelKit packaging for a complete all-MiniLM-L6-v2 release snapshot |
| kiwifs--kubernetes-reindex | kiwifs/kiwifs — Markdown filesystem for agents and teams | BUSL-1.1 | KiwiFS SQLite FTS5 reindex for Kubernetes documentation |
| kms--batch-encrypt | Cosmian/kms — Key-management system | BUSL-1.1 | Eviden KMS KMIP BulkData AES-GCM encryption for concurrent application-record batches |
| kubb--esi | kubb-labs/kubb — OpenAPI code-generation framework | MIT | Kubb regeneration of JitaSpace’s EVE Online ESI client |
| latexsnipper--formula-batch | SakuraMathcraft/LaTeXSnipper — Formula OCR and editing tool | GPL-3.0 | resident CPU formula OCR for a public OpenStax textbook screenshot batch |
| libcachesim--twitter-lru | 1a1a11a/libCacheSim — Cache simulation library | Apache-2.0 | Four-capacity LRU cache-size sweep over a production Twitter request trace |
| lsp-tree-sitter--termux-check | neomutt/lsp-tree-sitter — Tree-sitter language-server library | GPL-3.0 | Tree-sitter validation of a real Termux package-recipe corpus |
| mai-gen-videob50--score-images | Nick-bit233/mai-gen-videob50 — Maimai score-video generation tool | MIT | batch maimai score-image rendering for a real 50-record archive |
| meshmonitor--coverage-grid | Yeraze/meshmonitor — Mesh-node monitoring web tool | BSD-3-Clause | MeshMonitor’s SQLite coverage-grid batch |
| metamorpho--reallocate | metamorpho-org/metamorpho — Tokenized lending-vault protocol | GPL-2.0 | Gas use for a historical MetaMorpho market reallocation |
| mini-qr--library-batch | lyght/mini-qr — QR-code generator and scanner | GPL-3.0 | Mini QR’s batch SVG export for a public library directory |
| mirage--kubernetes-triage | strukto-ai/mirage — Virtual terminal for coding agents | Apache-2.0 | Mirage recursive source search for Kubernetes dual-stack readiness-probe triage |
| modflow6--neversink | MODFLOW-ORG/modflow6 — Groundwater-flow simulator | CC0-1.0 | Complete steady-state simulation of the USGS Neversink–Rondout groundwater model |
| monorepo--crypto-build | commonwarexyz/monorepo — Distributed-systems primitives | Apache-2.0 | the release build latency of commonware-cryptography |
| mrst--norne-import | SINTEF-AppliedCompSc/ MRST — Reservoir-simulation toolbox | GPL-3.0 | MRST Norne field-model import and geometry construction |
| msdialworkbench--lcms-alignment | systemsomicslab/MsdialWorkbench — Mass-spectrometry analysis workbench | LGPL-3.0 | MS-DIAL Console LC-MS peak processing and alignment for a real negative-ion DDA batch |
| mssql-extension--taxi-copy | hugr-lab/mssql-extension — DuckDB extension for Microsoft SQL Server | MIT | BCP bulk import of NYC taxi records through the DuckDB MSSQL extension |
| n8n-mcp--tool-batch-speed | czlonkowski/n8n-mcp — MCP server for n8n workflow construction | MIT | n8n-mcp on a fixed MCP tool-call batch |
| Task | Project and purpose | License | Fixed workload |
| nerdbank--messagepack--earthquake-fanout | AArnott/Nerdbank.MessagePack .NET MessagePack library | MIT | SignalR MessagePack earthquake snapshot fan-out |
| ojp--taxi-export | Open-J-Proxy/ojp Database proxy and JDBC driver | Apache-2.0 | OJP JDBC export of public NYC TLC taxi records |
| omnigraph--metabolic-trials-refresh | ModernRelay/omnigraph Lakehouse-native graph engine | MIT | OmniGraph’s paginated metabolic clinical-trial refresh and lifecycle dashboard |
| onnxscript--bert-optimize | microsoft/onnxscript ONNX model authoring and optimization library | MIT | Graph optimization of a BERT-SQuAD FP32 and INT8 model bundle |
| oxidize-pdf--rag-chunking | bzsanti/oxidizePdf Rust PDF library for RAG | MIT | oxidize-pdf RAG chunking over a real CC-BY PDF corpus |
| pdal--dtm | PDAL/PDAL Point-cloud processing library | BSD-3-Clause | Ground classification and 1 m terrain-model generation from USGS 3DEP LiDAR |
| petsc--ocean-age-solve | JuliaParallel/PETSc.jl Julia wrappers for PETSc | MIT | PETSc.jl GMRES+ILU for a public OCCA ocean-age transport solve |
| physical-ai-toolchain--action-smoothness | microsoft/physical-ai-toolchain Physical-AI dataset analysis toolkit | MIT | multi-dataset LeRobot action-smoothness analysis for real ALOHA manipulation recordings |
| plonky3--ethereum-calldata-keccak-proof | Plonky3/Plonky3 Polynomial IOP toolkit | Apache-2.0 | Plonky3 Ethereum calldata Keccak trace generation and STARK proving |
| polkadot-api--descriptor-generation | polkadot-api/polkadot-api Modular Polkadot API libraries | MIT | PAPI’s multi-chain TypeScript descriptor generation |
| pytrendy--dow-index | RussellSB/pytrendy Trend-detection library | MIT | PyTrendy trend detection for a real daily Dow Jones Industrial Average price series |
| qrisp--modmul-compile | eclipse-qrisp/Qrisp Quantum-algorithm development framework | EPL-2.0 | Qrisp compilation of controlled modular multiplication |
| react-native-code-push--mattermost-release | Soomgo-Mobile/react-native-code-push React Native OTA update client | MIT | CodePush Android OTA releases for Mattermost Mobile |
| reactant--fashion-mnist-train | EnzymeAD/Reactant.jl MLIR/XLA compiler for Julia | MIT | Reactant CPU execution of a real Lux Fashion-MNIST training epoch |
| rspress--react-native-paper | web-infra-dev/rspress Static-site generator | MIT | Rspress production builds of the React Native Paper documentation site |
| rust-clippy--cargo-package-lint | rust-lang/rust-clippy Rust linter | Apache-2.0 | Clippy lint time for Cargo’s main production package |
| s3s--kubernetes-sync | s3s-project/s3s S3 service adapter | Apache-2.0 | s3s-fs recursive S3 synchronization of Kubernetes pkg source |
| senpi--image-batch | code-yongyu/senpi Coding-agent runtime | MIT | Senpi image attachment preprocessing for a real screenshot batch |
| snarkos--cdn-replay | ProvableHQ/snarkOS Decentralized operating system for ZK applications | Apache-2.0 | snarkOS CDN block bundle replay over 150 fixed Aleo Mainnet blocks |
| snarkvm--mainnet-execution-verify | ProvableHQ/snarkVM Zero-knowledge virtual machine | Apache-2.0 | snarkVM verification of a batch of accepted Aleo mainnet executions |
| solvespace--solvespace--printer-assembly-mesh-export | SolveSpace/solvespace Parametric 2D/3D CAD | GPL-3.0 | SolveSpace CLI mesh export for a complete Cartesian 3D-printer assembly |
| spectral--api-lint | stoplightio/spectral OpenAPI, JSON, and YAML linter | Apache-2.0 | Spectral linting of a production OpenAPI batch |
| sqruff--glam-etl | quarylabs/sqruff SQL formatter and linter | Apache-2.0 | Sqruff directory linting for Mozilla GLAM ETL SQL |
| strawberry-graphql--strawberry-graphql--tlc-weekly-columns | strawberry-graphql/strawberry Python GraphQL library | MIT | Strawberry GraphQL execution for Graphique’s real TLC weekly Parquet column query |
| suiteparse--livejournal-pagerank | DrTimothyAldenDavis/SuiteSparse Sparse-matrix and graph computation suite | Apache-2.0 / BSD-2-Clause (used components) | Complete PageRank computation on the LiveJournal social graph using GraphBLAS and LAGraph |
| sum--text-attest | Octo-aO/SUM Cross-runtime text-attestation library | Apache-2.0 | SUM while batch-attesting a corpus of real Wikipedia prose |
| sundials--cantera-ignition | llnl/sundials Nonlinear and differential-equation solver suite | BSD-3-Clause | SUNDIALS CVODES for a downstream Cantera methane ignition parameter sweep |
| Task | Project and purpose | License | Fixed workload |
| svf-tools-svf--nginx--andersen-wpa | SVF-tools/SVF Static value-flow analysis framework | AGPL-3.0-or-later | SVF Andersen whole-program pointer analysis on NGINX reverse-proxy bitcode |
| synalinks--kubernetes-tasks | SynaLinks/synalinks Neuro-symbolic language-model framework | Apache-2.0 | Synalinks Markdown knowledge-base ingestion for Kubernetes task documentation |
| tfhe-rs--erc7984-transfer | zama-ai/tfhe-rs Fully homomorphic encryption library | BSD-3-Clause-Clear | TFHE-rs CPU execution for confidential ERC-7984 token transfers |
| tileserver-gl--country-atlas | maptiler/tileserver-gl Vector and raster map-tile server | BSD-2-Clause | TileServer GL static PNG rendering for a full Natural Earth country atlas |
| trixi-framework-trixi-jl--self-gravitating-sedov-amr | trixi-framework/Trixi Adaptive numerical-simulation framework | MIT | a published self-gravitating Sedov blast-wave simulation with adaptive mesh refinement |
| typst--napkin | Myriad-Dreamin/typst Typst runtime for JavaScript | Apache-2.0 | Typst.ts rendering in the Napkin Community production static-site build |
| vmecpp--precise-qh | proximafusion/vmecpp Magnetohydrodynamic equilibrium solver | MIT | VMEC++ fixed-boundary precise-QH equilibrium solves |
| wiredtiger--pageviews | wiredtiger/wiredtiger Embedded storage engine | GPL-2.0 OR GPL-3.0 | WiredTiger’s durable import of a real Wikimedia pageview hour |
| xet-core--bert-download | huggingface/xet-core Content-addressed transfer and reconstruction engine | Apache-2.0 | local Xet reconstruction of a production BERT model artifact |
| yamato-security-hayabusa--external-attack-evtx-timeline | Yamato-Security/hayabusa Windows event-log forensics tool | AGPL-3.0 | Hayabusa bulk external attack-EVTX JSONL timeline generation |
| youtrackdb--osm-roads | JetBrains/youtrackdb Object-oriented graph database | Apache-2.0 | embedded YouTrackDB import of a real OpenStreetMap road network |
| zenbu--release-source | zenbu-labs/zenbu.js JavaScript application framework | MIT | Zenbu.js staging of the Zenbu desktop application’s release source |
| zio--zrk-typecheck | lalinsky/zio Zig asynchronous-I/O framework | MIT | ZIO’s downstream zrk ReleaseFast type-check latency |
Table 7. Complete catalog of the 100 PERFRACE tasks.
Construction and Curation Statistics
This section describes the implementation of Discovery and Filter and reports the candidate counts and curation effort at each construction stage.
Discovery implementation. We query the ClickHouse github_events table for opened pull requests and branch pushes by a configured list of coding-agent accounts, then group matching events by repository. For each repository, we retain the latest matching pull-request or branch URL and use the GitHub GraphQL API to record its canonical identity, default-branch revision, stars, primary language, license, disk usage, and repository status.
The resulting records are deduplicated by GitHub repository, and projects are excluded when their repository, default branch, or recorded source revision cannot be retrieved. Each step writes its results incrementally, allowing interrupted collection and metadata retrieval to resume without repeating completed work.
From GitHub activity recorded between May and August 2026, we identified 118,517 repositories, retrieved metadata for 104,428, and retained 103,475 project candidates after these checks.
| Signal | Repositories |
| Branch push by a coding-agent account | 112,127 |
| Pull request opened by a coding-agent account | 6,390 |
Table 8. GitHub activity signals used for Discovery.
Filter implementation. We first apply deterministic checks to repository metadata and the recorded Git tree. These checks require an accessible source revision, a license permitting modification and redistribution, 10–4,999 GitHub stars, and a tracked Git tree of at most 500 MiB and 25,000 entries.
Projects that pass these checks are collected at the recorded revision with their README, setup documentation, package manifests, license files, and representative source paths. We then use GPT-5.6-Luna to assess the project’s setup path, evaluable process, runtime fit, and optimization potential from this source snapshot using a structured response schema. The prompt is provided in Appendix § 12.§ 1.
We require each structured response to address the following criteria.
Setup path. Repository files support concrete commands for installing dependencies and building or running the project.
Evaluable process. The project supports a finite process with a fixed input, observable output, performance metric, and correctness check.
Runtime fit. The process can run in a Linux x86-64 container on commodity CPUs without mandatory special hardware.
Optimization potential. The project supports a substantial local computation whose cost is expected to increase with input size or repeated execution.
Using these assessments, we selected 239 high-potential project candidates for Pair Verification. Table 9 reports the first check that excludes each project from the final Pair Verification selection.
| Result | Projects |
| Fewer than 10 GitHub stars | 93,992 |
| Accessibility | 1,134 |
| Setup path | 212 |
| Runtime fit | 359 |
| Optimization potential | 4,462 |
| Evaluable process | 3,066 |
| Assessment error | 11 |
| Selected for Pair Verification | 239 |
Table 9. First failed check during Filter and the final selection result.
Curation effort. To quantify the effort required to turn selected projects into accepted tasks, Figure 17 reports the agent cost, curation time, and number of agent–human interactions for the 100 accepted tasks. The median task required 4.5 hours, $8.83 in agent cost, and eight agent–human interactions, compared with averages of 9.1 hours, $11.35, and 12 interactions.
Figure 17. Distribution of curation time, agent cost, and agent–human interactions across the 100 accepted tasks.
Pair Verification
PerfRace uses Pair Verification to ensure that each task represents meaningful project use, produces reproducible measurements, and rejects shortcut solutions; this section describes the agent–human protocol and analyzes the resulting curation trajectories.
Protocol
Pair Verification combines agent-led task construction with human review to turn each project candidate into an executable task and assess its realism, reliability, and integrity. The construction skill is provided in Appendix D.2. The agent develops the task in stages and pauses for a reviewer decision after completing the evidence required for each quality criterion.
Figure 18. Pair Verification alternates between agent-led task construction and human review at the Realism, Reliability, and Integrity gates.
Explore and Design. The agent first studies the project’s documentation, examples, code paths, and public usage evidence to understand its purpose, users, and performance-relevant operations. The agent searches public release pages for source code, models, and assets, scientific or government data portals, downstream repositories, production archives, and real user exports to identify candidate workload inputs. For each input, the agent records its download location, version, and applicable terms, and accepts it only when an explicit license, public-domain designation, or written authorization permits redistribution with the benchmark.
It uses these materials to identify candidate operations, but grounds the workload and scale in a real user or downstream application rather than repository-native tests, benchmarks, examples, or fixtures. The agent compares plausible workload candidates, then traces the selected operation from its user-facing entry point through the relevant project code and demonstrates a small input-to-output execution on the unmodified revision. During this run, it records dependency installation, project build, input preparation, and operation time to check that setup does not dominate the measurement and to estimate whether complete verification can finish within the 1,200-second budget. The agent repeats the operation on identical inputs to detect run-to-run flakiness, including sporadic failures or inconsistent correctness-relevant outputs, and to identify any dependence of runtime on random state, network access, or external services. A candidate is revised or rejected if its stochastic execution remains too variable for reproducible performance measurement after fixing available random seeds and aggregating repeated runs.
It then specifies the exact operation, workload and scale available to the agent, held-out correctness cases, observable correctness utility, performance metric, and measurement boundary. Before requesting Realism review, the agent summarizes the project’s purpose and users, the selected execution path, the basis for the input and scale, and the alternatives it considered and rejected. The reviewer approves Realism only when the operation, input, and scale represent meaningful project use, the metric measures the stated utility, and the workload inputs can be redistributed as part of the benchmark.
Build and Calibrate. After Realism approval, the agent derives the installation, build, and run commands from upstream documentation, CI configuration, and package manifests, and verifies them on the pinned source revision. The agent implements the verifier, the task’s test program that measures the performance metric on a separate copy of the workload and checks additional held-out cases for correctness. It then constructs the Harbor task files with instruction.md, task.toml, an environment/ directory containing the editable project and workload, and a separate tests/ directory containing the verifier code, workload copy, held-out cases, and expected outputs. The agent pins the project revision, base image, language toolchain, dependencies, workload assets, and expected outputs so that the image can be rebuilt and the task rerun without downloading project source or workload inputs. The task instruction tells the agent how to build and run the project, what it may edit, which behavior it must preserve, and which performance metric to improve.
The agent then builds a fresh image and runs the verifier on the unmodified project before calibrating performance measurements. The agent selects the smallest resource profile that completes the workload without CPU, memory, or storage pressure and sets the verifier timeout within 1,200 seconds. The agent uses pilot and repeated runs to choose the workload scale, warmups, repetitions, and aggregation, requiring the final measurements to have a coefficient of variation below 10% while keeping build and setup outside the timed operation. The agent checks that the timing boundary covers the complete operation requested in the task while excluding one-time environment setup and verifier preparation that are unrelated to the optimization objective.
For the timed measurements retained after warmup, , we compute
After fixing the workload, correctness checks, resources, and measurement protocol, the agent builds the final image and evaluates the unmodified baseline and one optimization candidate under the same conditions. The reviewer examines the fresh-build correctness results, repeated measurements and CVs, verification time, resource allocation, and candidate trajectory. The reviewer approves Reliability only when fresh builds and repeated runs return the same correctness verdict and stable performance measurements with the chosen warmups, repetitions, aggregation, and measurement boundary.
Validate Integrity. To identify reward-hacking paths that could receive a valid score without completing the task, the agent tests deliberately incorrect, partial, hard-coded, and work-skipping solutions and checks that workload inputs, expected outputs, and verifier-owned data cannot be replaced by agent-controlled files. The agent inspects the evaluation image for leaked expected outputs, verifier code, or construction notes that could reveal the hidden checks. For example, in rust-clippy-cargo-package-lint, removing Cargo dependency tracking made later runs reuse a cached lint result, which the verifier detects by touching clippy-driver, running the workload twice against the same warm target, and requiring each Cargo JSON trace to report a non-fresh compiler artifact for cargo.
The reviewer approves Integrity only when every requirement in the task instruction is checked, every verifier check follows from the declared task, and none of the tested shortcut solutions can receive a valid score.
Finalize. After all three reviews, the reviewer chooses ACCEPT, REVISE, or REJECT; accepted tasks pass final compatibility checks before publication as a task package comprising an immutable Docker image and task artifact.
Quality Control
We analyze the recorded agent–human Pair Verification trajectories for evidence of the Realism, Reliability, and Integrity of the resulting tasks. Of the 239 candidates, 132 were rejected during Explore and Design, six during Build and Calibrate, none during Validate Integrity, and one during Finalize, leaving 100 accepted tasks.
Figure 19. Candidates rejected at each Pair Verification stage, with candidates lacking a recorded transition into Build & Calibrate assigned to Explore & Design.
Realism. Realism checks cover the workload operation and scale, the source and redistribution terms of its inputs, and the alignment between the declared utility and performance metric. Table 10 groups the workload inputs into released source code, models, or assets (26 tasks), public scientific or government datasets (21), real downstream applications (21), production traces or archives (16), real user exports (6), and other independently grounded inputs (10).
| Input source | Examples | Tasks |
| Released source code, model, or asset | Linux source tree, OpenMoji assets, released ML models | 26 |
| Public scientific or government dataset | TREC-COVID, NYC taxi records, LC-MS data | 21 |
| Real downstream application | React Native Paper, Headlamp, Mattermost Mobile | 21 |
| Production trace or archive | Hadoop logs, GTFS delay feeds, GitHub events | 16 |
| Real user export | MIDI collection, BIM project, printer assembly CAD | 6 |
| Other independently grounded input | Wikidata variants, Dow index, OSM roads | 10 |
Table 10. Sources of workload inputs across the 100 accepted tasks.
Reliability. All 99 wall-clock tasks have a final unmodified-project CV below 10%; the remaining task, metamorpho-reallocate, measures transaction gas, the deterministic execution cost assigned by the Ethereum Virtual Machine to a fixed smart-contract transaction, and returns exactly 874,508 gas in three fresh containers. Table 11(a) shows that 84 of the 99 wall-clock tasks have a final CV below 6%.
| (a) Final measurement CVs | |
| CV range | Tasks |
| 20 | |
| 64 | |
| 8 | |
| 3 | |
| 4 | |
| 0 | |
| (b) Red-team tests | |
| Tests per task | Tasks |
| 1 | 60 |
| 2 | 20 |
| 3 | 12 |
| 4 | 6 |
| 7 | 1 |
| 8 | 1 |
Table 11. Reliability and integrity evidence: final CVs for the 99 wall-clock tasks and red-team tests for all 100 tasks, with one test assigned conservatively when an exact count cannot be recovered.
Integrity. We validate Integrity across six attack classes: empty or no-op solutions, skipped work, hard-coded outputs, precomputed or cached outputs, modification of protected inputs or tests, and access to verifier-only data. All 100 accepted tasks completed an explicit red-team pass before Integrity approval. Table 11(b) shows that 40 tasks were tested against at least two red-team cases, including two tasks with seven or eight cases. For example, SCC is a source-code statistics tool whose task scans the Linux v6.19 tree and emits JSON statistics; during red-team testing, the agent replaced the scanner with a program that returned the precomputed Linux result instead of scanning the source tree, which passed the workload available to the agent but failed when the verifier asked to scan a separate pinned checkout of SCC’s own source code.
Evaluation Protocol
For each submission, the Judge snapshots the agent’s current container and starts a temporary shadow container from that snapshot. The Judge creates the snapshot with docker compose commit, launches the shadow container under a separate Compose project, and removes its container, local image, and volumes after verification. The agent container remains paused during verification and resumes from the same workspace state after the shadow container is removed.
Inside the shadow container, the verifier runs through tests/test.sh and emits the correctness verdict and performance metric as JSON. The verifier code, workload copy, held-out cases, and expected outputs are absent from the agent container and available only inside the shadow container. Each task declares its warmups, repetition count, aggregation rule, and required correctness pass rate in task.toml, and its verifier returns a performance metric only when that pass rate is met.
The task clock continues while the shadow is created, executed, and removed, so every submission trades part of budget for additional correctness and performance feedback. After verification, the Judge returns the correctness verdict and performance metric to the agent as feedback for subsequent optimization. For a correct submission, the Judge computes the speedup score by dividing the unmodified baseline metric by the submission metric. Submissions that fail correctness, exceed the verifier timeout, or do not produce a valid performance metric cannot update the best-so-far trajectory .
Five minutes before the deadline, the agent receives a final delivery prompt asking it to leave the project in a coherent state before the Judge performs one final shadow verification of the current workspace. Figure 20 characterizes the time required for one complete Judge evaluation, showing that baseline Judge evaluation remains within the 20-minute limit for all 100 final tasks.
Figure 20. Distribution of complete baseline Judge evaluation time across the 100 final task packages, with a mean of 6.1 minutes, a median of 4.9 minutes, and a 20-minute evaluation limit.
Longer optimization budgets. To measure the benefit of additional optimization time, we run Terminus-KIRA with GPT-5.6-Luna at high reasoning effort three times on each of the 88 evaluated tasks, maintaining one persistent agent session across Judge submissions during every four-hour run.
The aggregate best-so-far correct speedup rises from after one hour to , , and after two, three, and four hours, respectively. Across the resulting 264 trajectories, Figure 21 shows that extending Terminus-KIRA’s budget from two to four hours improves the overall score by 4.2%, and the fourth hour alone adds 2.8%.
Figure 21. Performance improvements from additional optimization time, based on three four-hour runs for each of 88 PERFRACE tasks. Left: increase in the overall score at four hours relative to two and three hours; horizontal lines show 95% bootstrap intervals. Right: number of tasks in each improvement range.
The four-hour score is at least 5% higher on 24 tasks than at two hours and on 12 tasks than at three hours, showing that agents continue to find meaningful improvements late in the task. Among the 228 runs that improve over baseline, the best result is first reached at a median of 88.1% of the four-hour budget, and 51 of the 264 runs produce a new improvement of at least 5% during the final hour.
BoxBox: Method Details
Figure 22 summarizes the task and review loops. The following explains how harness revisions affect subsequent interactions and how review, hot reloading, and recovery preserve task continuity.

Figure 22. BoxBox pseudocode. Task execution and independent harness review form nested interaction loops, with kernel-managed hot reloading connecting them.
Effects on Subsequent Interactions
(1) describes two ways a harness revision can affect subsequent execution while the language model remains fixed. Changes to how the harness constructs model inputs can alter , influencing the agent’s next action. Changes to can alter how that action is carried out and how its result is returned as an observation. A revision can affect either or both, so its influence extends to the trajectory generated after activation.
The current harness also governs the review loop. Once a revision is activated, subsequent reviews use the updated implementation and can examine the experience accumulated under it. Harness changes can therefore affect both task execution and the process used to produce later revisions. Appendix A.4 illustrates concrete changes to guidance, measurement, command polling, and context handling.
Review
The task-solving agent requests review through boxbox() without having to propose a patch. The initial harness requires intervening task activity and a 30-minute cooldown after each completed review; once eligible, the agent decides whether and when to request another review. The kernel preserves the task conversation and launches a separate agent review session under the current harness. The reviewer can inspect task trajectories and Git history, but edits only the harness, treating project code as read-only. It may retain the current harness, add guidance, or modify executable behavior, committing any changes before ending review. The default review skill is provided in Appendix D.3.
Execution Context
Execution context includes both runtime-owned state and the existing task environment. The former includes the task conversation, review history, and records of which trajectory entries have already been reviewed. These are exported as a state snapshot. The terminal session and project workspace remain in place: shell state, ongoing processes, and project files are preserved through the existing environment connection.
When the agent calls boxbox(), the runtime yields control to the kernel, suspending the task interaction loop while review proceeds separately. This pauses the agent’s task decisions, not the entire container or its processes. Before importing state into a candidate runtime, the kernel makes a separate copy, preserving the original snapshot for recovery if the candidate fails. This separates changing the agent implementation from restarting its task environment.
Hot Reloading
Harness revisions are tracked as Git commits, while the kernel remains separate from the editable harness. Before switching, the kernel checks the committed candidate, loads its runtime, and verifies that it supports task execution and context transfer. The current runtime keeps its terminal connection until the candidate successfully imports the copied state. The kernel then transfers the same terminal session to the candidate and returns control to it, resuming task interaction from the preserved conversation in the same workspace. If activation fails, the kernel retains or restores the previous runtime and context. For agent-requested reviews, the reviewer can leave concise guidance on using new capabilities and continuing the task. This guidance is preserved across the update, and the resumed agent is prompted to read it before continuing. Task execution and review share the same budget and cost accounting. Table 12 summarizes how execution continues under each outcome.
| Outcome | Continuation |
| No harness change | Resume the current runtime with the preserved task context. |
| Successful update | Continue under the revised harness with transferred context, the existing terminal session, and the same project workspace. |
| Update fails | Retain or restore the previous runtime and its preserved context. |
| Runtime fault | Enter recovery and review using the retained execution records; restart the terminal while preserving project files. |
Table 12. Task continuation across review and runtime-update outcomes.
Recovery
The kernel retains a recovery point across harness updates. A newly activated runtime remains provisional until it reaches the next agent-requested review without a runtime fault. If it fails before that boundary, the kernel restores the previous confirmed harness and state and initiates a review using the fault evidence. Faults in a confirmed runtime also trigger review. Runtime-fault recovery starts a fresh terminal, so shell-local state is lost, but project files, harness history, and execution records remain available. The retained records let subsequent reviews examine failed revisions and help the task-solving agent determine how to continue interrupted work.
Prompts and Skills
This appendix presents the Filter prompt and Pair Verification skill used in benchmark construction, followed by the default BoxBox self-review skill.
Filter Prompt
The Filter assesses repository snapshots before Pair Verification.
Filter Prompt
You are the automated Filter for PerfRace project candidates. Decide
whether the supplied repository snapshot provides enough source-grounded
evidence to proceed to Pair Verification.
Evaluate four gates:
1. Accessible. The repository and revision are available, with an
explicit license permitting modification and redistribution. A fork must
have an independently maintained product identity.
2. Upstream reproducibility. Documentation, scripts, or manifests
support concrete setup/build and minimal smoke commands. Cite the
supporting path or manifest field for each command. Dependencies and
configuration must come from identifiable public sources.
3. Evaluation potential. Identify a finite project operation with fixed
inputs, clear start and end, observable output, a plausible efficiency
metric, and a correctness check. Establish a replayable real user or
downstream workload, or a self-contained computation with grounded input
semantics. Repository tests, benchmarks, demos, fixtures, and generated
inputs do not establish a portable workload. Flag reliance on
unavailable deployment state, private traces, physical devices, or an
unresolved workload source.
4. Resource feasibility. The operation must run in a Linux x86-64
container on commodity CPUs within ordinary resource limits. Reject
mandatory special hardware, unavailable private services, and thin
remote pass-throughs. Local services or fixed stubs are acceptable only
when meaningful project-controlled computation remains.
Evidence and optimization potential:
- Use only the supplied snapshot; cite metadata or repository paths for
every gate. Missing evidence fails the gate.
- Describe a preliminary process only when evaluation potential passes.
- Describe the local computation and its strongest optimization
opportunity. Rate potential as low for tiny or externally dominated
work, medium for a clear local hot path, and high for scale-sensitive
computation with multiple structural opportunities.
- Return concise evidence in the requested structured response.
Pair Verification Skill
The skill guides agent-led construction through human reviews of Realism, Reliability, and Integrity, followed by task publication.
Pair Verification Skill
# PerfRace Pair Verification
Turn one repository into a machine-evaluable task. The agent investigates and executes; the human reviews Realism, Reliability, and Integrity, then decides ACCEPT, REVISE, or REJECT.
Working agreement
Maintain a report linking claims to project sources, runnable probes, and raw results. At each review gate, present the evidence, uncertainty, and recommendation, allow discussion, and pause for the human decision. Reopen the affected reviews when later evidence invalidates an approved claim.
Stage 1: Explore & Design
Understand the project’s users and performance-sensitive workflows. Compare candidate workloads and select an independently grounded real user or downstream operation with reproducible inputs and scale. Upstream examples and benchmarks may guide discovery but must not become the scored workload.
Run the unmodified workload end to end and identify the project execution path. Estimate build cost, image size, verification time, and measurement variance before constructing the task. Propose the operation, input provenance and scale, visible workload, independent verifier copy, additional correctness cases, correctness utility, measurement boundary, and metric.
Human review: Realism. Explain why the operation, inputs, and scale represent meaningful project use, which alternatives were considered, and what remains uncertain. Obtain approval before building the canonical task.
Stage 2: Build & Calibrate
Package the pinned project, visible workload, instructions, container, and verifier. Keep only the project writable. Use an independent byte-identical scored input and additional held-out correctness cases. Correctness checks enforce the workload’s observable utility, allowing implementation changes that preserve it.
Keep candidate compilation and setup outside the timed user operation. Calibrate workload scale, warmups, repetitions, resources, and timeout; then freeze the measurement protocol. Complete verification must fit within 1,200 seconds, and the uncompressed image must not exceed 3 GiB. Target at least three seconds per timed execution; shorter operations require a clear signal and repeated CV at most 10%.
Check fresh-image correctness and flakiness. Diagnose variance before increasing workload size or resources. Enforce the same runtime policy for baseline and candidate. Run a baseline and one bounded candidate-agent attempt, retaining the trajectory and raw measurements. One correct, repeatable agent is sufficient; an incorrect or no-gain candidate does not reject the task if its trajectory demonstrates understanding and a reasonable optimization attempt.
Human review: Reliability. Present fresh-build correctness, raw measurements, dispersion, verification time, resources, and the candidate trajectory. Ask whether equivalent runs support the same verdict and measurement.
Stage 3: Validate Integrity
Audit agreement between instructions and verifier checks, including writable paths, workload copies, expected outputs, caches, and reward inputs. Run explicit red-team tests for leaked-data access, verifier modification, incomplete work, hard-coded answers, workload-specific precomputation, frozen runtime-policy overrides, and reward-path bypasses.
Count an attack only when a materially distinct invalid implementation reaches the final verifier. After hardening, replay the attack and record its verdict. Reopen Reliability if changes affect the workload, measurement, resources, or verifier execution.
Human review: Integrity. Present the boundary audit and red-team results. Ask whether an invalid or shortcut implementation can receive a valid score. Verifier ownership requires an independent immutable workload copy, not secrecy of the visible workload.
## Stage 4: Finalize
Summarize review decisions and limitations, remove construction artifacts from the release package, and run final checks. Confirm that the package contains all required inputs and no credentials, private evidence, or hidden answers.
Ask the human to ACCEPT AND PUBLISH, REVISE, or REJECT. Publish the pinned image and verified task package only after acceptance; preserve evidence for rejected candidates.
Self-Review Skill
The following listing presents the default review skill with performance-optimization-specific guidance omitted. “Target Agent” denotes the task-solving agent.
# Self-Review
Act as the macro-level controller of the Target Agent’s task-solving process. Review the trajectory to diagnose the task-solving process, then retain the current harness or select a focused-feedback or reusable-ability adaptation that improves subsequent Target progress within the remaining shared budget.
## Review the task-solving process
Treat each review as one iteration in a continuing self-review loop. At the start of every review, examine the accumulated trajectory comprehensively before selecting an intervention.
Use that broad diagnosis to choose a focused intervention that is supported by the evidence and likely to materially improve subsequent Target progress. Include only the feedback or reusable capabilities justified by the diagnosed problems in task execution and the evidence needed to evaluate their effect.
Never be satisfied with the current optimization state while task budget remains. Treat accepted improvements as intermediate evidence and compare the remaining opportunities by their expected contribution.
Diagnose the task-solving process
Treat the Target Agent’s framing, causal model, plans, explanations, and claims as hypotheses, not premises. Use recorded actions, observations, and measurements to identify all material unsupported links, distinguish evidence from inference, and identify plausible explanations not ruled out. Use the complete diagnosis to keep the intervention focused on the related gaps that most materially constrain progress. Treat claims that an experiment completed, improved performance, or excluded a direction as unverified until supported by the trajectory. Explain how each unsupported link affects task-execution cost, experiment selection, or progress.
Critique the Target Agent’s choice of actions, not just candidate outcomes. Match each conclusion to the scope the experiments actually tested. Do not generalize beyond the tested workload, configuration, mechanism variant, or measurement, and do not ignore what the evidence establishes within that scope. Distinguish direct findings from broader hypotheses, and identify unsupported assumptions or omitted dimensions that constrained the search.
After selecting the intervention focus, review two aspects of macro-level control as ways to advance it:
Task-execution cost. Examine repeated Target Agent work, which dependencies and evidence requirements have stabilized, and where intermediate model judgment remains valuable. Look for opportunities to reduce, fuse, automate, batch, or reuse work while preserving decision-relevant evidence.
Budget allocation. Evaluate exploration breadth and exploitation depth by comparing the causal support, remaining leverage, uncertainty, and expected contribution of validated mechanisms and untested hypotheses.
Decide the intervention
After completing the diagnosis, choose among retention, a focused-feedback adaptation, and a reusable-ability adaptation according to their expected contribution to target progress over the remaining budget. Retention means no new Target-facing guidance or other harness change.
Use focused feedback for a task-specific concern, unsupported judgment, omission, or unresolved evidence. Identify the concrete observation, the gap, and why it matters. Be careful about your own claims: an evidence gap does not establish an alternative explanation or direction; do not turn preferences into concerns or next-step menus. State only the concern and its evidentiary basis; do not require or recommend what the Target Agent should do next.
Activate a reusable ability when it directly supports the diagnosis and the trajectory supports repeated use. Also consider a harness adaptation when the Target Agent repeatedly makes the same consequential error and the evidence supports a general way to prevent it; otherwise use focused feedback. For a stable repeated action sequence, fuse or automate the mechanical work while preserving model judgment, decision-relevant evidence, meaningful failure boundaries, and a fallback to the unfused workflow. Validate the ability through its actual Target-facing path and use feedback to explain how to use it and its constraints.
Self-Review Skill (continued)
Condition each intervention on the current stage of task execution, inferred from the trajectory rather than elapsed time:
Early: examine plausible competing hypotheses.
Developing: reduce demonstrated repeated work and balance exploration with supported exploitation.
Saturated: identify which tested hypothesis families have low remaining value, and flag any claim that the broader search is exhausted unless the trajectory supports that scope.
Use the observed downstream effects of prior adaptations when deciding what the current stage needs.
References
- [1]Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457, 2025.arxiv.org/abs/2507.19457
- [2]Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. MLE-bench: Evaluating machine learning agents on machine learning engineering. In International Conference on Learning Representations, 2025.arxiv.org/abs/2410.07095
- [3]Yizhe Chi, Deyao Hong, Dapeng Jiang, Tianwei Luo, Kaisen Yang, Boshi Zhang, Zhe Cao, Xiaoyan Fan, Bingxiang He, Han Hao, et al. Frontier-Eng: Benchmarking self-evolving agents on real-world engineering tasks with generative optimization. arXiv preprint arXiv:2604.12290, 2026.arxiv.org/abs/2604.12290
- [4]Evan Chu, Rajan Agarwal, Abhishek Thangamuthu, Brendan Graham, and Justus Mattern. Frontier-SWE: Benchmarking coding agents at the limits of human abilities. https://www.frontierswe.com/blog/v1, 2026.
- [5]Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See-Kiong Ng. Mercury: A code efficiency benchmark for code large language models. In Advances in Neural Information Processing Systems, volume 37, pp. 16601–16622, 2024. doi: 10.52202/079017-0529.DOI
- [6]Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Gerard Moreno-Torres Bertran, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Vladislav Vorotilov, Mengjue Wang, Ian Yu, Amine Benhalloum, Grégoire Mialon, and Thomas Scialom. GAIA2: Benchmarking LLM agents on dynamic and asynchronous environments. arXiv preprint arXiv:2602.11964, 2026.arxiv.org/abs/2602.11964
- [7]Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Yin Zheng, Zejian Yuan, and Zejun Ma. SWE-Perf: Can language models optimize code performance on real-world repositories? In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Seoul, South Korea, 2026.arxiv.org/abs/2507.12415
- [8]Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint arXiv:2408.08435, 2024.arxiv.org/abs/2408.08435
- [9]Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M. Zhang. EffiBench: Benchmarking the efficiency of automatically generated code. In Advances in Neural Information Processing Systems, volume 37, pp. 11506–11544, 2024. doi: 10.52202/079017-0367.DOI
- [10]Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba. ALE-Bench: A benchmark for long-horizon objective-driven algorithm engineering. In Advances in Neural Information Processing Systems, volume 38, 2025.arxiv.org/abs/2506.09050
- [11]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.06770
- [12]Seth Karten, Joel Zhang, Tersoo Upaa Jr., Ruirong Feng, Wenzhe Li, Chengshuai Shi, Chi Jin, and Kiran Vodrahalli. Continual harness: Online adaptation for self-improving foundation agents. arXiv preprint arXiv:2605.09998, 2026.arxiv.org/abs/2605.09998
- [13]Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, et al. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. In International Conference on Learning Representations, volume 2024, pp. 54928–54958, 2024.
- [14]KRAFTON AI and Ludo Robotics. Terminus-KIRA: Boosting frontier model performance on Terminal-Bench with minimal harness, 2026. URL https://github.com/krafton-ai/kira.
- [15]Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu, Yixing Jiang, Chelsea Finn, Marco Pavone, Ion Stoica, and Azalia Mirhoseini. LLM-as-a-Verifier: A general-purpose verification framework. arXiv preprint arXiv:2607.05391, 2026.arxiv.org/abs/2607.05391
- [16]Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026.arxiv.org/abs/2603.28052
- [17]Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850, 2026.arxiv.org/abs/2604.25850
- [18]Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. Evaluating language models for efficient code generation. In First Conference on Language Modeling, 2024a.arxiv.org/abs/2408.06450
- [19]Shu Liu, Shubham Agarwal, Monishwaran Maheswaran, Mert Cemri, Zhifei Li, Qiuyang Mang, Ashwin Naren, Ethan Boneh, Audrey Cheng, Melissa Z. Pan, Alexander Du, Kurt Keutzer, Alvin Cheung, Alexandros G. Dimakis, Koushik Sen, Matei Zaharia, and Ion Stoica. EvoX: Meta-evolution for automated discovery. arXiv preprint arXiv:2602.23413, 2026.arxiv.org/abs/2602.23413
- [20]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations, 2024b.arxiv.org/abs/2308.03688
- [21]Bohan Lyu, Yucheng Yang, Siqiao Huang, Jiaru Zhang, Qixin Xu, Xinghan Li, Xinyang Han, Yicheng Zhang, Huaqing Zhang, Runhan Huang, et al. MLS-Bench: A holistic and rigorous assessment of AI systems on building better AI. arXiv preprint arXiv:2605.08678, 2026.arxiv.org/abs/2605.08678
- [22]Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, and Parthasarathy Ranganathan. SWE-efficiency: Can language models optimize real-world repositories on real workloads? In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Seoul, South Korea, 2026.arxiv.org/abs/2511.06090
- [24]Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026.arxiv.org/abs/2601.11868
- [25]Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for general AI assistants. arXiv preprint arXiv:2311.12983, 2023.arxiv.org/abs/2311.12983
- [26]Microsoft. Dadsv5 size series. Microsoft Learn, 2026. URL https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dadsv5-series. Accessed October 24, 2026.
- [27]Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Er-chao Zhao, Xiaoxi Jiang, and Guanjun Jiang. Trace2Skill: Distill trajectory-local lessons into transferable agent skills. arXiv preprint arXiv:2603.25158, 2026.arxiv.org/abs/2603.25158
- [28]Jun Nie, Yonggang Zhang, Jun Song, Qianshu Cai, Dahai Yu, Yike Guo, Xinmei Tian, and Bo Han. TTHE: Test-time harness evolution. arXiv preprint arXiv:2607.08124, 2026.arxiv.org/abs/2607.08124
- [29]Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. KernelBench: Can LLMs write efficient GPU kernels? arXiv preprint arXiv:2502.10517, 2025.arxiv.org/abs/2502.10517
- [30]Sungho Park, Wonjoong Kim, Rongyuan Tan, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. AutoSaddler: Automatic harness optimization with durable updates from agent execution traces. arXiv preprint arXiv:2608.23041, 2026.arxiv.org/abs/2608.23041
- [31]Ori Press, Brandon Amos, Haoyu Zhao, Yikai Wu, Samuel K. Ainsworth, Dominik Krupke, Patrick Kidger, Touqir Sajed, Bartolomeo Stellato, Jisun Park, Nathanael Bosch, Eli Meril, Albert Steppi, Arman Zharmagambetov, Fangzhao Zhang, David Pérez-Peñeiro, Alberto Mercurio, Ni Zhan, Talor Abramovich, Kilian Lieret, Hanlin Zhang, Shirley Huang, Matthias Bethge, and Ofir Press. AlgoTune: Can language models speed up general-purpose numerical programs? In Advances in Neural Information Processing Systems, volume 38, pp. 118982–119027, 2025. doi: 10.52202/0 85713-3585.arxiv.org/abs/2507.15887
- [32]Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7957–7968, 2023.arxiv.org/abs/2305.03495
- [33]Atharva Sehgal, James Hou, Akanksha Sarkar, Ishaan Mantripragada, Swarat Chaudhuri, Jennifer J. Sun, and Yisong Yue. FormulaCode: Evaluating agentic optimization on large codebases. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Seoul, South Korea, 2026.arxiv.org/abs/2603.16011
- [34]Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, and Ion Stoica. GSO: Challenging software optimization tasks for evaluating SWE-agents. In Advances in Neural Information Processing Systems, volume 38, 2025.
- [35]Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob R. Gardner, Yiming Yang, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. Learning performance-improving code edits. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=ix7rLVH yY.arxiv.org/abs/2302.07867
- [36]Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, Yuhan Cao, Yixiao Huang, Chris Durou, Haoyun Zhang, Jeffrey Lin, et al. Agents’ last exam. arXiv preprint arXiv:2606.05405, 2026.arxiv.org/abs/2606.05405
- [38]Siddhant Waghjale, Vishruth Veevrenadath, Zhiruo Wang, and Daniel Fried. ECCO: Can we improve model-generated code efficiency without sacrificing functional correctness? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15362–15376. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.emnlp-main.859.DOI
- [39]Hao Wang, Hanchen Li, Qiuyang Mang, Alvin Cheung, Koushik Sen, and Dawn Song. Do androids dream of breaking the game? systematically auditing AI agent benchmarks with BenchJack. arXiv preprint arXiv:2605.12673, 2026a.
- [40]Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, and Teng Xiao. Rethinking the evaluation of harness evolution for agents. arXiv preprint arXiv:2607.12227, 2026b.arxiv.org/abs/2607.12227
- [41]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972, 2024.arxiv.org/abs/2404.07972
- [42]Zhangchen Xu, Junda Chen, Yue Huang, Dongfu Jiang, Jiefeng Chen, Hang Hua, Zijian Wu, Zheyuan Liu, Zexue He, Lichi Li, Shizhe Diao, Jiaxin Pei, Jinsung Yoon, Hao Zhang, Mengdi Wang, Radha Poovendran, Misha Sra, Alex Pentland, and Zichen Chen. AutoLab: Can frontier models solve long-horizon auto research and engineering tasks? arXiv preprint arXiv:2606.05080, 2026.arxiv.org/abs/2606.05080
- [43]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2405.15793
- [44]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.03629
- [45]Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative ai by backpropagating language model feedback. Nature, 639(8055):609–616, 2025.DOI
- [46]Guibin Zhang, Leo Lu, Fangzhou Xie, Kang Zhu, Junhao Wang, Zhifei Xie, Zhaochен Yu, Zihang Liu, Zhongxiang Sun, Qian kun Li, Yue Liao, Heng Chang, Xiaobin Hu, Qibing Ren, Wangchunshu Zhou, Chuanrui Hu, Yafeng Deng, and Shuicheng Yan. JIT-Agent: Scaling harness intelligence via just-in-time harness evolution. arXiv preprint arXiv:2608.25593, 2026a.arxiv.org/abs/2608.25593
- [47]Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, and Shuyue Hu. Self-Harness: Harnesses that improve themselves. arXiv preprint arXiv:2606.09498, 2026b.arxiv.org/abs/2606.09498
- [48]Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762, 2024.arxiv.org/abs/2410.10762
- [49]Jiayi Zhang, Yongfeng Gu, Jianhao Ruan, Maojia Song, Yiran Peng, Zhiguang Han, Jinyu Xiang, Zhit ao Wang, Caiyin Yang, Yixi Ouyang, Bang Liu, Chenglin Wu, and Yuyu Luo. Harnessing agentic evolution. arXiv preprint arXiv:2605.13821, 2026c.arxiv.org/abs/2605.13821
- [50]Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models. arXiv preprint arXiv:2510.04618, 2025.arxiv.org/abs/2510.04618
- [51]Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19632–19642, 2024.arxiv.org/abs/2308.10144
- [52]Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In International Conference on Learning Representations, 2023.arxiv.org/abs/2211.01910
- [53]Deyao Zhu, Xin Zhou, Shengling Qin, Xuekai Zhu, et al. EdgeBench: Unveiling scaling laws of learning from real-world environments. arXiv preprint arXiv:2607.05155, 2026.arxiv.org/abs/2607.05155

















