WorksComputer Science
Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3
Abstract
A generally intelligent system acquires new skills efficiently without requiring humans to redesign its software scaffold for every new task. Yet language models rarely act alone. A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf. ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go. On it, human-steered systems already report ceiling scores across the public games, so we know effective designs exist. We ask whether frontier models can build and improve such a harness on their own. Through a fixed coding bootstrap, each model constructs a complete task harness, acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation. We measure task performance with Relative Human Action Efficiency (RHAE) and record dollar-denominated inference cost. Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]). Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance. No revision retains its parent, yet a post hoc rule that selects among existing snapshots by development score raises the held-out gain to 4.53 points ([0.81,8.68]), separating the generation of revisions from the decision to deploy them. Human-steered systems demonstrate further headroom. These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities.
Introduction
Modern language models can write programs, call tools, and act in unfamiliar environments, but only as part of a larger agentic system. In such a system, a symbolic program orchestrates calls to the model, shaping what it observes, what it retains, which actions are available, how it recovers from failure, and how much inference it spends. This surrounding system is commonly termed a harness [22, 26]. A system’s behavior therefore depends on both its harness and its underlying model or models.
This matters for the study of general intelligence, which requires the efficient acquisition of useful new skills in unfamiliar situations [8]. If each novel task requires a human engineer to inspect failures, redesign representations, add tools, and tune control flow, then what improves is not the system’s own capacity to adapt but the engineer’s understanding, transferred into the system by hand. The agent’s learning remains confined to a supplied mechanism whose improvement depends on the engineer. In contrast, humans routinely adapt to unfamiliar situations in everyday life.
AI systems can improve through changes to their underlying models or to the harnesses through which those models act. These routes are complementary: harness changes can provide access to new tools, maintain persistent memory, and coordinate multiple model calls, capabilities that changes to model weights alone do not supply. The harness route therefore raises a central question:
Can a model take on part of the engineering work that determines its own capabilities, using its execution experience to improve the software through which it acts?
ARC-AGI-3 aims to measure general intelligence through agents’ adaptation to novel games. Its main metric, Relative Human Action Efficiency (RHAE), combines level completion with action efficiency relative to human baselines. Harnesses developed under human steering already reach 100 RHAE on the public games with GPT-5.6 Sol and Opus 5 [23, 16, 6]. By contrast, Sol scores 38.3 on the public set in a simple benchmark harness with retained reasoning and compaction enabled [29]. An off-the-shelf Claude Code baseline scores 42.83 with Opus 4.8/Fable 5 (Zeng et al., 2026). Strong human-steered results make ARC-AGI-3 a revealing test of autonomous improvement: effective harness designs are known to exist for current models. The benchmark lets us examine how reliably models can discover and improve their own designs through experience in unfamiliar interactive environments.
Recent systems automate agent and harness design by searching over prompts, skills, workflows, tools, memory, and source code [17, 22, 26] (Zelikman et al. 2024; Zhang et al. 2025; Zhang et al. 2026a). Our replicated study examines how reliably revisions generalize across model conditions, evaluating every prescribed harness snapshot on held-out games and measuring gains, regressions, and cost. No single study settles whether models can improve the software through which they act. Our aim is one controlled, replicated measurement under an explicit protocol, complementing studies that use other tasks, seeds, and update rules.
We ask a direct question: Can a frontier reasoning model construct a complete interaction harness, learn from its execution record, and improve that same harness without human edits? We measure the outcome of updating with a fixed-lineage protocol. Through a fixed coding bootstrap, a model starts from a controlled minimal repository, builds an ARC-AGI-3 harness for the same model profile, receives feedback from five development games, and potentially revises the generated harness twice. We then evaluate the initial, revision-1, and revision-2 snapshots on twenty holdout games.
Our results expose a gap between autonomous harness engineering and reliable self-improvement. Models autonomously produce diverse harnesses spanning zero to near-ceiling performance. Yet their revisions do not consistently turn development experience into held-out gains: mean improvement on development games exceeds held-out improvement by 5.62 RHAE points, and cost reductions can accompany performance losses. The updaters never retain a parent, although a simple development-based choice among existing snapshots recovers part of the benefit. These findings show why the capacity to build an agent and the capacity to improve it need to be evaluated separately. We make three contributions:
We provide a general framework for autonomous harness adaptation and define the harness adaptation gap: the difference between improvement on development tasks and improvement on held-out tasks.
We introduce a replicated, all-snapshot protocol and conduct a controlled study across nine model–effort conditions: 27 independent harness lineages, 81 frozen snapshots, and 2,025 game evaluations, measuring task performance, deployment cost, and adaptation cost.
We perform an outcome-blind, reliability-audited analysis of the observable capabilities in 54 revision episodes, connecting development feedback to code changes and relating those changes to performance on games withheld from revision.
Harness Adaptation as a Capability
Let a deployed agent be , where is a pretrained model and is an executable harness. For a task , the inner loop maps an interaction history to actions under a fixed . We write its expected utility over task distribution as , with held fixed as the harness changes. The complete system is the unit of capability measurement [34, 18]. An outer-loop updater receives development evidence and produces

Figure 1. Controlled harness-adaptation workflow. At each stage, the same model profile uses the fixed coding bootstrap to propose a candidate. The controller checks that it can execute and interact with two development games. A failed candidate may be repaired; a passing candidate is frozen as . Five-game development evidence from and seeds the next revision. runs on the same games for evaluation only. All snapshots are evaluated on twenty held-out games after adaptation.
The same model profile appears in both loops. In the outer loop, it operates through a fixed general-purpose coding bootstrap , which supplies repository and development tools used to construct and revise . In the inner loop, it operates through the task-specific harness to interact with the environment. remains fixed while is the object of autonomous adaptation. Meta-learning methods such as MAML, RL, and AdA learn rapid adaptation within a supplied neural architecture and training procedure [13, 11, 3]. Here, pretrained weights remain fixed while the model constructs and revises the executable task harness.
We separate four capabilities: (i) constructing an executable harness, (ii) repairing interface and lifecycle failures, (iii) mapping execution evidence to a concrete update, and (iv) selecting updates that improve feedback-held-out tasks. The first three do not imply the fourth. A capable updater must also preserve working behavior and abstain when evidence is weak.
For revision , define the harness adaptation gap
A positive means that apparent adaptation on feedback-generating tasks exceeds held-out benefit. Let denote the expected execution cost per task and the cumulative cost of constructing and revising the harness through stage . We evaluate performance and both costs jointly:
This follows bounded-optimal and resource-rational accounts in treating computation as a costly action [33, 24, 25]. Our Pareto comparisons concern held-out performance and per-game deployment inference cost; construction and revision expenditure is reported separately.
In our ARC-AGI-3 instantiation, task utility is a game’s RHAE score, and we estimate and by mean RHAE over the five development and twenty held-out games, respectively. We record and as dollar-denominated inference costs for gameplay and harness development.
Related Work
LLM self-refinement and program optimization. Self-Refine, Reflexion, and CRITIC revise responses or task policies through self-feedback, retained verbal feedback, or tool-grounded critique [28, 36, 15]. Prompt and program optimizers learn reusable components from development data. OPRO and PromptBreeder search instruction space.
DSPy, MIPRO, TextGrad, and GEPA optimize modules inside declared language-model programs (Yang et al., 2024; Fernando et al., 2024; Khattab et al., 2024; Opsahl-Ong et al., 2024; Yuksekgonul et al., 2025; Agrawal et al., 2026). These methods normally retain a human-specified artifact schema or computation graph.
Automated agent and harness design. Prior work automates optimizer construction, code-defined agent design, and workflow search (Zelikman et al., 2024; Hu et al., 2025; Zhang et al., 2025), and extends autonomous revision to runtime logic and complete coding-agent repositories (Yin et al., 2025; Robeyns et al., 2025; Zhang et al., 2026a). Meta-Harness and Agentic Harness Engineering optimize executable harness source from scores and traces [22, 26]. HarnessCompass and RRSI study how such changes generalize beyond the tasks used for evolution (Zhang et al., 2026b; Xia et al., 2026). We study autonomous harness engineering as a capability of frontier models: how reliably can they construct the software through which they act and improve it from their own execution evidence? Our replicated comparison follows each independently constructed harness and both prescribed revisions, measuring initial capability alongside subsequent gains and regressions on games excluded from adaptation.
Evaluating autonomous improvement. SEAGym reports a schedule that completes 39 of 40 harness updates while test success falls by 14.5 percentage points (Zheng et al., 2026). On Terminal-Bench, validation-selected evolution yields little mean held-out gain, whereas on ARC-AGI-3 a harness that adapts within each evaluation game improves substantially (Wang et al., 2026); HarnessOpt-Bench finds that the best validation score seen during search is usually optimistic relative to the nominated harness’s test score (Ursekar et al., 2026). RRSI obtains transfer over 20–40 rounds with explicit proposal and acceptance regularization (Xia et al., 2026). Our construction-and-revision study extends this evidence to independently generated interactive-game harnesses, asking whether revisions based on some games transfer to others, and jointly tracking performance on development and held-out games and inference cost.
Harness design for ARC-AGI-3. EWM, OPINE-World, PRO-LONG, and Tycho demonstrate strong autonomous game solving within human-steered harnesses [32, 10, 14, 23]. Workspace Optimization lets agents revise executable world models and strategies during play while the reusable harness remains fixed [35]. Our study shifts harness design and revision to the model: it constructs the reusable task harness and revises it from its own execution evidence, without human edits or guidance between stages. We evaluate whether those revisions improve performance on games excluded from adaptation. Appendix C compares the protocol designs.
Experimental Design
ARC-AGI-3: An Interactive Reasoning Challenge
Unlike the static input–output transformations in earlier ARC tasks, ARC-AGI-3 presents an agent with sequential, unfamiliar grid worlds. At each turn, the agent observes one or more 64 × 64 color grids, together with the current game state, progress across levels, and available actions. Depending on the game, the interface offers directional inputs, a game-specific input, coordinate clicks, or undo. The agent is not told what the visible objects represent, how these inputs affect them, or what completes a level. It must discover the game’s rules through exploration and reasoning.
Each game contains multiple levels. Relative Human Action Efficiency (RHAE) assigns credit only to completed levels and discounts solutions that use more environment actions than hidden human baselines. Later levels receive greater weight. We also record actions, completed levels, terminal states, and dollar-denominated model inference cost.
Figure 2 shows two states from one such game, FT09.2 A harness can present these states to the runtime model as raw images, symbolic grids, object summaries, transition diffs, or combinations of them. It also controls context and memory, provides tools, organizes planning and reasoning, and allocates model calls. It can request one action or a sequence, use code to execute a discovered strategy, and reuse computations to reduce inference cost. These design choices are not specified by the benchmark.

Figure 2. Two observations from FT09 (Level 1 on the left, and Level 7 on the right), a public ARC-AGI-3 game. The agent must infer that the small patterned tiles specify which neighboring tiles must match or differ from a reference color. Clicking cycles tile colors, which need to be matched to target patterns. The later levels introduce controls that change several tiles together. These rules must be discovered through interaction.
Autonomous construction and revision
Two harnesses, one model. Each experimental condition pairs a model with a reasoning-effort setting and uses that profile for both harness development and gameplay. During construction and revision, the model operates through OpenCode, a general-purpose coding harness in the same category as Claude Code and Codex. OpenCode provides tools to inspect and edit files and run commands. It remains fixed throughout the campaign. During gameplay, the model is called through the generated task harness, the artifact being autonomously constructed and improved.
Each run receives a sandboxed workspace with Python, preinstalled scientific libraries, and shell tools. The repository includes documentation for interacting with the game, a client for calling the assigned model, and model and pricing specifications. The model must create run.sh, which launches its solver on a specified game without human input. The solver architecture is otherwise unspecified. The harness controls how much work is performed by deterministic code and when the model is consulted. The build model may create and revise solver code and run local checks, including mock-based tests.
Coding sessions do not expose the evaluator’s live game or runtime-model gateways. End-to-end trials occur through controller-run smoke qualification on two declared development games. If a candidate fails, the same model receives its failure report and execution logs and may repair it for at most ten cycles. Qualification requires the harness to run and interact with the game within fixed action and level limits, regardless of its RHAE score. Task performance is measured in the subsequent fixed-snapshot evaluations.
The qualified initial system runs on five development games. Revision 1 receives a fixed bundle containing its own source, traces, scores, actions, and inference costs. It may change any generated file or preserve the parent. The same procedure produces revision 2. No human edits or natural-language guidance are introduced between stages. Resource limits and infrastructure-recovery rules are specified in Appendix A.
To assess the outcomes of construction and revision, we compare the resulting harness versions under a common evaluation protocol.
Held-out evaluation. Every frozen harness version is evaluated on the five development games and twenty held-out public games. The held-out games provide no feedback for harness construction, revision, or selection. Our primary comparison is the change in mean held-out RHAE from the initial harness to revision 2 among runs completing all three stages. Construction failures are reported separately over all planned runs. We validate recorded gameplay by replaying the same actions locally and through the official ARC service, checking agreement in scores and game progress (Appendix G).
Conditions and uncertainty. The study covers nine model–effort conditions across seven model families, with three independent constructions per condition (Table 1). All 27 runs complete
construction and both revisions. All conditions follow the same construction-and-revision protocol. We estimate confidence intervals by bootstrapping whole lineages, preserving paired comparisons across versions. These intervals describe uncertainty for the observed study pool on the fixed game split. Comparisons between model–effort conditions are descriptive.
Quantitative Results
Held-out gains are small on average and vary substantially across lineages. Revision 2 raises mean development RHAE by 8.42 points and held-out RHAE by 2.80 points. The held-out 95% bootstrap interval is , compatible with no average benefit or a useful positive gain; with 27 lineages, the design detects mean changes of about 6.5 points with 80% power (Table 21). The interval’s upper end is small against the headroom that human-steered harnesses demonstrate for some of the same models (§ 7). At a descriptive five-point threshold, 9 lineages gain, 3 lose, and 15 change by less; strict signs give 15 gains and 12 losses (, two-sided sign test; Appendix § 15). These are observed suite outcomes, not estimates of how often revisions reduce expected performance.
| Target gain | Normal approx. | Noncentral | Balanced (per condition) |
| 3 | 124 | 126 | 126 (14) |
| 5 | 45 | 47 | 54 (6) |
| 10 | 12 | 14 | 18 (2) |
Table 21. Lineages needed for 80% power (two-sided ), assuming the observed paired-change SD of 11.92. The balanced column rounds the noncentral- total up to equal lineages in each of the nine conditions.
Each snapshot is evaluated once, so we measured execution variability separately. Ten evaluations of one frozen final harness on the same twenty held-out games yield a sample SD of 3.99 RHAE. All ten scores remain above this lineage’s initial-harness score, showing that its large observed improvement persists across repeated evaluations. The observed variation also shows why score changes of only a few points require cautious interpretation (Appendix § 15.§ 1). Paired repeats of three prespecified lineages reproduce the direction of both a large gain (+30.82, ) and a large regression (, ), an interval that reaches the five-point band, while a near-zero change remains unresolved (Table 26).
| Lineage | Original | Pair 1 | Pair 2 | Pair 3 | Mean [95%] | Direction | class |
| Fable high, rep. 1 | 16.05 13.77 | [, 7.33] | unresolved | unresolved | |||
| Sol high, rep. 1 | 20.38 53.56 | [9.97, 51.67] | gain | gain | |||
| Sol high, rep. 2 | 27.92 12.74 | [, ] | loss | unresolved |
Table 26. Paired repeated held-out evaluations. Each fresh difference is in mean held-out RHAE over the same 20 games; intervals are paired with df = 2. Direction is supported if the interval excludes zero; the class requires the whole interval beyond the band.
The pooled result also hides differences between games. Averaging across runs, held-out performance improves on 13 games and declines on seven. The largest mean gain is 20.44 RHAE points and the largest decline is 15.03. Aggregate improvement can therefore coexist with substantial regressions on particular games (Appendix § 15).
Development gains overstate transfer. The mean adaptation gap is 5.62 points (95% bootstrap interval ). Figure 3b shows several revisions that improve the five feedback-generating games while reducing held-out performance. Development improvement therefore overstates the average held-out gain under this update procedure. Before revision the two splits score alike (26.13 development and 24.54 held-out RHAE); the gap opens with revision. Its point estimate stays positive in every sensitivity analysis, but its size is uncertain: it ranges from 4.07 to 6.40 when one condition is omitted and from 3.02 to 7.84 when one development game is omitted from the development summary, and it is 4.21 ([, 8.87]) when games affected by execution recovery are excluded (Appendix § 15, Table 29). Without the three near-floor conditions, the held-out gain is 4.82 ([, 11.38]) and the gap 8.52 ([3.16, 14.11]; Table 24).
Figure 3. Construction strength varies, and development gains transfer unreliably. (a) Held-out condition means on a common scale. (b) Each condition has two points, averaging its three runs: initial-to-revision-1 (open) and initial-to-revision-2 (filled). Labels count strict-sign quadrants. Points below zero regress on held-out games. Triangles denote higher-effort conditions.
| Cohort | Lineages | Held-out | Dev. | Gap | Gains/losses |
| All conditions | 27 | +2.80 [−1.41, 7.41] | +8.42 | +5.62 [1.81, 9.75] | 15/12 |
| Excluding Terra, Luna, Sonnet | 18 | +4.82 [−1.44, 11.38] | +13.34 | +8.52 [3.16, 14.11] | 12/6 |
| development > 1 | 24 | +3.12 [−1.66, 8.29] | +9.44 | +6.32 [2.09, 10.83] | 13/11 |
| development > 5 | 21 | +3.64 [−1.79, 9.41] | +10.85 | +7.22 [2.45, 12.22] | 12/9 |
| development > 10 | 18 | +4.82 [−1.44, 11.38] | +13.34 | +8.52 [3.16, 14.11] | 12/6 |
Table 24.
| Excluded | Lineages | Pairs | Held-out gain | Dev. gain | Gap | Gains/losses |
| None | 27 | 540/135 | 2.80 | 8.42 | 5.62 | 15/12 |
| Incomplete-attempt slots | 27 | 514/128 | 2.54 | 6.77 | 4.23 | 15/12 |
| Fresh-restart slots | 27 | 526/135 | 2.86 | 8.42 | 5.56 | 14/13 |
| Union of both | 27 | 509/128 | 2.56 | 6.77 | 4.21 | 15/12 |
| Any multiple-attempt endpoint slot | 27 | 501/125 | 3.02 | 6.65 | 3.62 | 15/12 |
| Seven lineages with a union slot | 20 | 400/100 | 1.02 | 6.21 | 5.20 | 8/12 |
Table 29. Paired exclusion sensitivity. Rows remove the listed endpoint slots at both and of the same lineage and game; the last row removes whole lineages. Pairs count held-out/development game pairs. Intervals, where computed, are 95% percentiles of 20,000 whole-lineage resamples (seed 20261108), conditional on each row’s retained games.
Figure 4. Separate outcomes reveal which harnesses improve, save cost, or trade one for the other. Each row identifies a model–effort condition and run. Bars show magnitudes of observed changes in mean held-out RHAE and inference cost per game; headers specify the direction. Each metric uses a common linear scale across panels. Categories use strict signs of unrounded changes; Table 9 gives counts under practical thresholds.
Choosing what to deploy recovers part of the benefit. A post hoc rule selecting the snapshot with highest development RHAE yields a mean held-out gain of 4.53 points over the initial harness (95% interval [0.81, 8.68]), compared with 2.80 for always deploying . It retains in nine lineages and reduces observed regressions from 12 to 5, at a mean deployment cost of $9.15 rather than $7.57 per game. The rule was specified after held-out results were available. Its gain over is stable when any one development game is omitted from selection (+3.95 to +4.16), whereas its advantage over always deploying is smaller and less certain (Table 23); the rule remains to be evaluated prospectively.
| Omitted game | Gain vs. | + / − / = | Gain vs. | |
| None (all five) | 9/6/12 | +4.53 [0.80, 8.71] | 13/5/9 | +1.72 [0.38, 3.68] |
| tr87 | 8/6/13 | +3.95 [−0.19, 8.40] | 13/6/8 | +1.15 [0.36, 2.14] |
| vc33 | 8/6/13 | +4.13 [0.41, 8.36] | 13/6/8 | +1.33 [−0.21, 3.39] |
| r11l | 12/7/8 | +4.09 [0.35, 8.33] | 10/5/12 | +1.29 [−0.51, 3.43] |
| bp35 | 9/7/11 | +4.09 [0.68, 7.79] | 13/5/9 | +1.28 [−0.49, 3.41] |
| ft09 | 7/7/13 | +4.16 [0.16, 8.52] | 14/6/7 | +1.36 [−0.10, 3.38] |
Table 23.
Lower cost does not indicate improvement. For each immutable snapshot, solving cost is the mean logical inference cost over the 20 held-out games, including replayed responses. Cost per game falls in 18 of the 27 lineages. At the five-point threshold, five of these 18 also gain held-out RHAE, three lose, and ten change by less. By strict sign, nine revision-2 descendants improve both held-out RHAE and cost per game, six gain RHAE at higher cost, nine become cheaper while losing RHAE, and three worsen on both dimensions; the paired repeats support no strict score–cost classification, so these are observed outcomes (Table 26). Mean cost falls by $2.72 per game, but the median change is only −$0.54. The lower average expenditure thus combines revisions that achieve more at lower cost and revisions that spend less while also solving less effectively.
The experimental program corresponds to at least $18,129.27 under the frozen publication price schedule, including construction, revision, repair, qualification, and fresh selected-game inference. Appendix § 15 provides the cost breakdown, complete replicate-level trajectories (Table 3), and checkpoint and game-sensitivity analyses.
| OpenAI | Anthropic | ||||||||
| RHAE / cost (USD/game) | RHAE / cost (USD/game) | ||||||||
| Condition | Rep. | Initial | Revision 1 | Revision 2 | Condition | Rep. | Initial | Revision 1 | Revision 2 |
| GPT-6 Astra high | 1 | 97.14/11.90 | 93.77/10.86 | 97.55/10.24 | Claude Opus 5 xhigh | 1 | 27.85/9.18 | 28.79/8.18 | 36.39/12.48 |
| 2 | 58.03/8.58 | 98.76/12.53 | 92.62/11.96 | 2 | 46.63/16.62 | 46.01/12.76 | 46.82/11.44 | ||
| 3 | 93.59/18.77 | 93.23/11.87 | 92.94/12.92 | 3 | 22.29/10.77 | 36.84/8.65 | 36.63/7.75 | ||
| GPT-5.6 Sol xhigh | 1 | 43.95/31.39 | 18.49/4.89 | 21.99/17.18 | Claude Opus 5 high | 1 | 6.57/1.12 | 3.79/2.71 | 2.52/2.46 |
| 2 | 37.64/12.62 | 56.37/13.09 | 49.86/17.69 | 2 | 16.46/3.93 | 21.47/7.16 | 26.48/9.45 | ||
| 3 | 44.36/37.89 | 42.43/23.73 | 53.96/22.36 | 3 | 12.09/6.78 | 20.28/4.30 | 17.28/4.56 | ||
| GPT-5.6 Sol high | 1 | 20.38/12.87 | 29.24/8.19 | 53.56/8.15 | Claude Fable 5 high | 1 | 16.05/10.63 | 18.37/11.26 | 13.77/10.12 |
| 2 | 27.92/8.84 | 6.82/2.90 | 12.74/5.61 | 2 | 28.35/17.29 | 41.41/15.99 | 31.55/16.98 | ||
| 3 | 30.15/25.39 | 26.17/6.26 | 19.45/5.89 | 3 | 18.92/13.81 | 26.26/14.15 | 26.04/12.33 | ||
| GPT-5.6 Terra high | 1 | 4.77/4.68 | 0.94/4.70 | 0.49/4.24 | Claude Sonnet 5 high | 1 | 0.00/0.27 | 0.00/0.33 | 0.11/1.24 |
| 2 | 4.39/9.88 | 1.72/1.55 | 1.04/1.70 | 2 | 0.12/0.88 | 0.21/1.48 | 0.81/1.70 | ||
| 3 | 3.33/3.89 | 0.60/2.49 | 0.55/3.34 | 3 | 0.14/0.49 | 0.52/1.24 | 0.14/1.36 | ||
| GPT-5.6 Luna high | 1 | 1.52/0.46 | 1.71/0.57 | 1.90/0.40 | |||||
| 2 | 1.75/0.14 | 1.87/0.44 | 0.75/0.17 | ||||||
| 3 | 1.09/2.00 | 0.33/0.85 | 0.26/0.77 |
Table 3. All completed trajectories reported in the paper. Conditions are grouped by provider, with matched higher-effort profiles first. Logical costs use the publication price schedule and selected per-game token ledgers; three trajectories are recovered directly from those ledgers because final aggregate report assembly failed.
What Models Build and How They Revise
What harness designs emerge, and how do they change?
The generated systems span markedly different interaction strategies. Compact direct model controllers coexist with heuristic-first systems, strategist–executor hybrids, persistent manuals, explicit world models, replay systems, deterministic planners, and inference-budget managers. Architectures also vary across replicates of the same condition.
Across all 27 lineages, we audit all 54 parent-to-descendant transitions. This post hoc descriptive audit labels a revision as Preserve when it repairs or tunes the current policy, Extend when it adds a capability inside the same decision loop, Reorient when it materially changes control priority, context continuity, replay, or action/inference scheduling, and Replace only when it changes the harness family. These labels describe observable source and runtime structure; they are separate from the outcome-blind capability coding below.
Strategy change is common. Twenty-two of 27 first revisions reorient the harness, four extend it, and one primarily preserves or restores the incumbent. Second revisions include 12 reorientations, 12 extensions, and three preserves. No lineage replaces its harness family; revisions repeatedly reorganize memory, perception, batching, replay, and resource allocation. Appendix § 16 reports every replicate and Figure 5 summarizes size and code churn.
Figure 5. Harnesses grow and undergo substantial revision without reliable gains. Generated Python counts model-produced source and tests and excludes the supplied llm_client.py. (a) All 27 lineages on a log scale, colored by initial-to-revision-2 held-out outcome; the black path is the stage median. (b) Each point is one selected parent-to-descendant Python diff. Churn is added plus removed lines, and points above zero improve held-out RHAE.
One Sol-high revision addressed decision-limit failures and ineffective memory use by changing action budgeting, memory management, and observations. Held-out RHAE rose from 29.24 to 53.56 at essentially unchanged inference cost (Appendix § 16). In a Sol-xhigh lineage, revisions replaced response chaining with independent calls and bounded memory. Held-out RHAE fell from 43.95 to 21.99 as inference cost fell from $31.39 to $7.18 per game (Appendix § 16).
Which revision behaviors accompany held-out improvement?
We examine how models respond to execution feedback and whether their revision behaviors accompany held-out gains.
Corpus and codebook. The coding corpus contains all 54 parent-to-descendant revision episodes across the nine conditions, comprising 70 updater sessions. Packets preserve the development feedback available to the updater, compact chronological messages and tool events, the exact selected-system diff, and proximal test and smoke results. They mask model and condition identity, trajectory identifiers, future stages, and every downstream development and held-out outcome. The codebook covers evidence use, planned and enacted changes, verification, and post-result adaptation (Appendix § 17).
Blinding and reliability. Each outcome-blind packet was independently coded twice using GPT-5.6 Sol xhigh under the same frozen codebook. Consensus labels were frozen before linkage to downstream outcomes. We report unadjudicated exact agreement, quadratic-weighted Cohen’s ([9]), and Krippendorff’s ([20]). Scope (), post-result adaptation (0.84), and planning (0.70) pass the exploratory reliability threshold. Updater verification is uniformly top-coded with exact agreement, so its chance-corrected reliability is undefined. The planning interval is absent from 27 packets, with missingness concentrated in several conditions and larger edits; we therefore exclude it from outcome interpretation. Diagnostic probing (0.66), diagnosis (0.35), incumbent protection (0.47), and resource metareasoning (0.29) fall below the preset 0.67 reliability threshold. Visible narration is triangulated against tools, code, tests, and runtime behavior because verbalized reasoning may be unfaithful ([38, 21, 7]).
Do broader edits accompany larger gains? All 54 revision episodes include inspection of item-specific development evidence, all final diffs target at least one eligible evidence-backed issue, and every selected descendant passes its proximal operational checks. Completing an evidence-guided revision does not ensure a better agent: all episodes satisfy these process checks, while held-out outcomes vary. Broad rewrites average +3.75 immediate held-out change versus −0.20 for contained changes, with a wide interval that includes zero; a deterministic rule-based re-derivation of scope agrees with the consensus labels in 51 of 54 episodes and reproduces this contrast (Table 27). Appendix § 17 reports the complete exploratory association table.
| Code | Labels | Agreement | Contrast [95% interval] | |
| SCOPE | LLM consensus (published) | – | – | −3.94 [−9.09, 1.22] |
| Script, role-group subsystems | 51/54 | .90 | −3.77 [−9.35, 1.62] | |
| Script, module subsystems | 52/54 | .91 | −3.60 [−9.15, 1.74] | |
| ADAPT | LLM consensus (published) | – | – | −3.24 [−8.54, 2.38] |
| Script, failure-triggered | 46/54 | .64 | +0.48 [−5.09, 7.05] | |
| Script, any post-edit result | 43/54 | .55 | −2.39 [−6.25, 2.72] | |
| Script, structural smoke repair | 48/54 | .74 | −0.17 [−5.23, 5.70] |
Table 27. Deterministic re-derivation versus the LLM consensus over 54 episodes. Agreement is exact; is quadratic-weighted for SCOPE and unweighted for binary ADAPT. Contrasts follow Table 17 (scoped minus broad; further action minus handoff) with the same resampling procedure and seeds. Script variants were fixed before the comparison.
How do reactive revision episodes differ? In 36 episodes, the updater links the final result to its change and hands off. Their mean immediate held-out change is +2.48. In 18 episodes, a post-edit result triggers another edit, revert, test change, scope change, or criterion-based stop. Their mean is −0.76. The −3.24-point contrast is uncertain: its interval includes zero. Episodes also differ in parent performance, repair effort, and the changes attempted. The comparison therefore establishes neither a dependable advantage nor a harmful causal effect of further repair. A rule-based re-derivation of this code agrees in 46 of 54 episodes, and under it the contrast disappears (+0.48), so this association depends on how post-result adaptation is operationalized (Table 27).
Discussion
Construction strength does not ensure consistent improvement. Across nine conditions, the study establishes substantial autonomous harness-building capability while exposing a gap between development gains and held-out benefit. The pooled interval remains compatible with useful average improvement, and several lineages achieve large gains. Reliability here concerns consistent held-out benefit and capability preservation; observed regressions do not establish each revision’s effect on expected performance. Evidence inspection, targeted edits, and operational validation are common, but their exploratory associations cannot establish why revisions succeed or fail. Strong initial harnesses can lose observed performance while weaker constructions improve substantially.
A gap to human-steered harnesses remains. At the same Opus 5 xhigh setting, Tycho reaches 100 RHAE on every public game, whereas our revision-2 harnesses average 39.95 on twenty of those games ([23]). Both studies designate the same five development games, although Tycho also uses the full public set for policy selection. Tycho reports 100 with Sol, while VISTA and AVO report 100 with Opus 5 ([16, 6]). Retrodict reports 99.86 with Sol at maximum effort ([4]). These results establish substantial headroom for autonomous harness engineering. The development and selection procedures differ. So does what the model sees during play: the reference systems keep the complete interaction record available (a typed history in Tycho, every frame in VISTA, a replayable log in Retrodict), whereas our Sol harnesses use stateless calls or short chains with bounded notes and our Opus harnesses rely mainly on build-time code, consulting the model 14–18 times per game. Tycho’s reported totals imply roughly 600–1,060 model calls per game, but spend alone does not account for the gap: Retrodict reports about $26 per game, less than two of our initial Sol-xhigh harnesses spent. Model strength matters as well: GPT-6 Astra, whose self-built harnesses reach 92.6–97.6, scores 99.9 on the official leaderboard with a simple harness that preserves reasoning state ([2]). The comparison therefore does not isolate an effect of human steering. Appendix § 15 gives the comparison scope.
Abstention is part of adaptation. Although retaining the parent is permitted, every scored trajectory changes its implementation: each of the 54 revisions adds or removes at least 84 lines of code (median 449), including the four labeled Preserve. Reliable self-improvement includes recognizing when the evidence favors keeping the current harness. The evidence for that decision is often available: selecting among existing snapshots by development score retains in nine lineages and raises the held-out gain from 2.80 to 4.53. Generating a revision and deciding whether to deploy it are therefore distinct parts of autonomous improvement, and the updaters we study exercise only the first. Candidate revisions should be compared with a preserved baseline and rejected or rolled back when they do not justify replacement. RRSI’s reported gains with edit budgets and noise- and cost-aware acceptance rules illustrate one way to make such decisions explicit ((Xia et al., 2026)).
Implications for autonomous improvement. Harness adaptation complements model training by changing the software through which fixed model weights act ([22, 26]). For AI risk analysis, current task performance is only part of the picture: a system may also be able to improve the software that determines its future performance ((Wijk et al., 2025)). Model and harness improvement could reinforce each other if better harnesses support AI research and better models become more capable harness developers ([5]). Evaluating this route requires tracking whether revisions generalize, preserve prior strengths, and justify their cost. Our experiment holds both the model and coding bootstrap fixed. Claims about sustained capability acceleration or its risks require longer runs, broader tasks, and updaters that can themselves change [5] (Wijk et al. 2025).
Limitations
We use one benchmark, one set of feedback games, and two revisions. The results therefore leave open how harness adaptation behaves on other tasks or over longer improvement sequences. Because the games are public, some models may have encountered them during training.
Three runs per condition leave uncertainty about small differences between models, and 27 lineages detect only mean changes of several points. Gameplay can also vary even with an unchanged harness, as our ten-run diagnostic shows. We did not repeat every evaluation, so individual gains or regressions may partly reflect this variation. These limits follow from cost: at the recorded prices, a further revision stage would cost about $5,200 and ten evaluations of every snapshot about $145,000 (Appendix § 19). Because the same model builds the harness and plays through it, near-floor results conflate construction and gameplay. In a crossed evaluation, Sol-built harnesses do not lift Terra or Luna as players, so gameplay limits their scores, and Sonnet-built harnesses stay near the floor when Sol plays them, so harness quality limits Sonnet’s (Table 25). The reverse cells, such as a Terra-built harness played by Sol, were not run. A direct contamination test would require non-public games.
| Harness | Player | Player effect | Player effect | Builder effect | Builder effect | Revision crossed |
| Sol high | Terra high | [, ] | [, ] | [, ] | [, ] | [, ] |
| Sol high | Luna high | [, ] | [, ] | [, ] | [, ] | [, ] |
| Sonnet high | Sol high | [, ] | [, ] | [, ] | [, ] | [, ] |
Table 25. Crossed builder/player evaluation on the 20 held-out games. Entries are mean differences in RHAE over three lineages, with ranges. Player effect: crossed cell minus the same harness played by its builder. Builder effect: crossed cell minus the player’s own harness of the same replicate.
Inference expenditure varies across harnesses and models. Reporting score and cost together makes this visible, but does not separate model capability, harness design, and spending into independent effects.
The revision-behavior annotations in Section 6.2 were produced by two LLM annotation passes. A rule-based re-derivation of the two interpreted codes agrees closely for scope and moderately for post-result adaptation; agreement with independent human annotations was not evaluated. The coders use summarized execution records, which can lose context or event order. We exclude unreliable or inadequately observed labels. Episodes differ in their starting harness and the problems they attempt to fix, so an association cannot show that a particular revision behavior caused the outcome.
Conclusion
Harness adaptation offers a route to self-improving AI in which a system develops the software through which it acts. Our ARC-AGI-3 study demonstrates substantial autonomous harness-building capability, but a less dependable ability to convert execution experience into held-out gains. Development progress can overstate improvement, and lower cost does not indicate it: of 18 cheaper final harnesses, only five gain more than five held-out points. A simple development-based choice among existing snapshots, specified post hoc, recovers part of the benefit, which suggests that deciding whether to deploy a revision deserves as much attention as producing it. Our evidence is strongest for the specified update-and-deploy protocol and speaks less directly to autonomous improvement in general. Reported human-steered systems show that stronger harnesses are possible for some of the same models. Strong performance from a model–harness system should be distinguished from its ability to autonomously improve the software through which it acts. Our results make this distinction concrete: with model weights fixed, development gains and lower inference costs can coexist with held-out regressions, making transfer and preservation of existing capabilities essential tests of harness adaptation. The central challenge is to make improvements accumulate: extending what the system can do across tasks while preserving what it already does well.
AI Use Statement
Language-model assistants supported experimental-infrastructure implementation, code and methodological review, literature search, interpretation checks, manuscript editing, and the outcome-blind qualitative coding described above. The authors had the initial idea, selected the protocol and data, verified the analyses and claims, authored the final text, and take responsibility for the work.
Reproducibility Statement
Appendices A and B specify the protocol and starting prompt; Appendix C compares related systems. Appendices D–F provide supplementary results, strategy audits, and the coding method. The supplementary artifact contains the seed repository, frozen prompts, profiles, and splits, the scoring and feedback-curation code, all selected source snapshots, analysis tables, deterministic reproduction scripts, and a checksum-backed manifest (Appendix G).
Full Experimental Protocol
The nine model–effort conditions each have three independent constructions and follow the same construction-and-revision protocol (Table 1). All conditions enter the pooled analysis.
| Model profile | Reasoning effort | Context | Max output |
| GPT-6 Astra | high | 1.05M | 64K |
| GPT-5.6 Sol | high | 1.05M | 120K |
| GPT-5.6 Terra | high | 1.05M | 120K |
| GPT-5.6 Luna | high | 1.05M | 120K |
| Claude Fable 5 | high | 1.00M | 120K |
| Claude Opus 5 | high | 1.00M | 120K |
| Claude Sonnet 5 | high | 1.00M | 120K |
| GPT-5.6 Sol | xhigh | 1.05M | 120K |
| Claude Opus 5 | xhigh | 1.00M | 120K |
Table 1. Study conditions. Builder and runtime profiles are matched within each row. Context is the advertised window; max output is the configured per-response gameplay cap. Coding caps are given below. Exact model- and pricing-profile hashes are retained in run protocols.
Repository boundary. The initial repository provides the task contract, local engine client, controlled model client, pricing documentation, and the entry-point contract. It does not provide a solver architecture, game-specific rules, worked examples, or a reusable task policy. Generated systems have no internet access, no cross-game state, and no access to other game instances. The fixed coding bootstrap is OpenCode; the artifact records its binary and adapter fingerprints. Codex is used separately for outcome-blind annotation.
Development split. The five development games, tr87, vc33, r11l, bp35, and ft09, are those designated at random by [23]; the split was fixed before the campaign. The other 20 public games were fixed as feedback-held-out. The two smoke games were chosen from the development split to exercise click- and keyboard-based interaction.
Stage lifecycle. Each stage has one selected candidate. Controller-owned smoke qualification checks interface validity and bounded interaction on two declared development games. Each game permits 50 charged actions and stops after two completed levels or a win. Passage requires a charged action, a clean exit, and a registered stopping boundary on each game; no score threshold is used. Failed candidates may be repaired by the same condition for at most ten cycles. The selected initial candidate runs the five-game development evaluation. Its frozen source and feedback bundle seed revision 1. The same procedure seeds revision 2. All attempts, including failed repair and report-finalization sessions, remain in the audit ledger.
Hosted replay. Generated systems interact only with the local engine. The controller freezes complete action–observation traces, replays them locally, then submits them to a short-lived hosted scorecard. Acceptance requires score agreement and matching compared observations, with one explicitly logged visual-noise exception described in Appendix § 18. Hosted service behavior therefore cannot influence the generated policy.
Construction replacement policy. A construction may be replaced only for controller, provider-transport, or hosted service failures that occur outside model-produced behavior. Syntax errors, invalid protocol use, exhausted repair budgets, and failure to produce a qualified harness remain model or harness outcomes. Every planned construction in the retained study cohort produced a qualified initial system.
Resource bounds. Coding and gameplay sessions each have a 24-hour wall-clock limit. A generated workspace or selected snapshot may occupy at most 1 GiB. Model calls use a 3,600-second retry budget and the output limits in Table 1. Coding output caps are 128K for GPT-5.6 and Claude profiles and 64K for GPT-6 Astra. Exact storage limits, model profiles, and recovery controls are included in the artifact. Gameplay inference has no common total-dollar or model-call cap: expenditure is measured, and the build prompt asks systems to use inference economically, while the stated wall-clock, action, output, and storage bounds remain enforced.
RHAE definition. For a game with levels , hidden human action baselines , and system actions , a completed level within the action budget receives and other levels receive zero. With level weights , the game score is
where is the completed prefix. Reported split scores average this quantity over games. The second term caps credit by weighted completion.
Minimal Starting Prompt
The build model receives the following instruction. The controller substitutes only the runtime model’s display name and configured reasoning effort. In particular, the prompt does not prescribe a solver architecture, visual representation, memory design, planning method, or rule for any game.
Build a system for solving ARC-AGI-3 games. Work autonomously to implement, test, and document the complete solution without asking for human input.
The system will be evaluated on the tradeoff between RHAE and inference cost. Aim primarily for high RHAE while using inference economically. Use additional inference only when it is reasonably expected to produce a meaningful improvement in RHAE. Inference cost is measured in US dollars according to docs/pricing.md. RHAE rewards action efficiency. When outcomes have equal RHAE, fewer game actions are preferred.
At evaluation, the system receives exactly one current game and must infer its mechanics from observations available during that run. It runs with Python 3.12 without internet access or access to other games. It may use the configured runtime model only through the supplied llm_client.py. Exact model capabilities and limits are in docs/model.md.
The local rules and interface are in docs/arc_rules.md. The supplied files are read-only. Expose a non-interactive ./run.sh --game <game-id> --out-dir <directory> entrypoint.
Related-System Comparison
Table 2 compares the evaluation unit, selection rule, and editable artifact. Our protocol evaluates each individual descendant of a frozen parent on held-out tasks that did not influence the update, with no performance-based selection among qualified descendants. HUIB separately evaluates models’ ability to produce useful updates and to use updated artifacts in an online task stream [27]; here, one model profile fills both roles.
| Work | Editable artifact and schedule | Final evidence | Main protocol difference |
| STOP | One optimizer function over 1–4 recursive depths | Fresh optimization tasks | Restricted to an optimizer-function artifact. |
| ADAS | Code-defined agent functions under archive-guided search | All discovered ARC agents on held-out test; five evaluations per agent | Searches agent functions; our study tracks prescribed revisions of complete interactive harnesses. |
| AFlow / GEPA | Workflows or prompts under budgeted search | Validation-selected candidate on test | Constrained artifact and selection over many candidates. |
| SICA / DGM | Complete coding-agent codebase with archive-guided evolution | Same-benchmark selection plus transfer | Whole-code evolution with variable lineages and update counts. |
| Meta-Harness / AHE | Executable harness source under multi-iteration score-and-trace search | Best or final frozen harness | Seeded candidate search with selected-harness outcomes. |
| RRSI (Xia et al., 2026) | Harness source; 20–40 rounds with regularized proposals and acceptance | Evolved harness on held-out tasks and OOD benchmarks | Tests a selection intervention under multi-candidate search, rather than every prescribed descendant. |
| HarnessCompass | Minimal Bash-only coding harness over five rounds | Final harness on 450 untouched tasks | Close minimal-seed strict test in a different domain. |
| HUIB / SEAGym / Rethinking | Skills, prompts, memory, or seeded harnesses | In-stream or strict-test evaluation | Studies artifact use, regressions, and allocation of rollout budgets. |
| HarnessOpt-Bench | Executable seed under trusted execution | Inaccessible strict test after one nomination | Candidate search and final nomination introduce selection. |
| Workspace Optimization | Structured ARC workspace with online repair within games | Public 25-game suite | Artifact repair occurs within individual games. |
| Ours | Complete ARC harness with exactly two revisions | Every snapshot on 5 development and 20 strict held-out games | Prescribed lineages; primary endpoint is the final revision. Joint RHAE, actions, and dollar cost. |
Table 2. Closest protocol precedents. Strict test means that test tasks do not influence proposals, acceptance, stopping, or candidate selection.
Additional Quantitative Results
Table 3 gives the complete held-out score–cost trajectory for each completed construction, with full model versions and reasoning-effort profiles.
Construction and qualification outcomes. All 27 planned runs in the retained study cohort produce a smoke-qualified initial system and complete all three evaluations. Paired revision analyses use all 27 complete lineages. Across their 81 scored stages, 46 qualify the raw candidate without repair and 35 select a smoke-repaired candidate. Initial held-out RHAE averages 24.54 and ranges from 0 to 97.14.
Uncertainty estimation. Percentile confidence intervals use 100,000 whole-lineage bootstrap resamples with seed 20260902 and the artifact’s fixed pseudonymous trajectory order. Each resampled lineage retains all stages and games, preserving paired comparisons across versions. Intervals condition on the observed study pool and fixed game split, not new tasks or feedback partitions.
Cost accounting. Logical solving cost sums priced usage for fresh and replayed responses in each suite-selected game attempt; the per-game meters remain auditable independently of final experiment-report assembly. For cross-model comparison, dollar values are recomputed from the immutable token ledgers using first-party standard API list prices frozen on September 13, 2026. Run-time pricing profiles remain in the artifact. These dollar figures cover model inference; they exclude infrastructure and human labor and do not measure latency or energy use.
Table 4: Descriptive held-out means for matched reasoning-effort conditions. Each row contains three independent trajectories.
| Condition | Initial | Revision 2 | Change |
| Sol high | 26.15 | 28.58 | +2.43 |
| Sol xhigh | 41.98 | 41.94 | -0.05 |
| Opus high | 11.71 | 15.43 | +3.72 |
| Opus xhigh | 31.27 | 39.95 | +8.68 |
Table 4. Descriptive held-out means for matched reasoning-effort conditions. Each row contains three independent trajectories.
Development-based checkpoint selection. As a post hoc diagnostic, selecting the preserved snapshot with highest five-game development RHAE chooses in 9/6/12 lineages and yields 29.06 mean held-out RHAE: +4.53 over (95% interval [0.81, 8.68]) and +1.72 over always deploying ([0.38, 3.67]). Mean deployment cost rises from $7.57 to $9.15 per game. The rule uses only development scores and could be automated; its post hoc evaluation does not establish the performance of a prospectively specified selection policy (Table 5).
Table 5: Post hoc checkpoint diagnostic. Development selection chooses the snapshot with highest five-game development RHAE, breaking ties toward the earlier snapshot. “+/−/ =” counts held-out improvement, regression, and no change relative to . Costs exclude outer-loop construction and selection expenditure. The held-out oracle is diagnostic only.
| Deployment policy | RHAE | Gain | USD/game | +/−/ = | |
| Retain | 24.54 | 0.00 | 10.30 | 0/0/27 | 27/0/0 |
| Deploy | 27.34 | +2.80 | 7.57 | 15/12/0 | 0/0/27 |
| Select by development RHAE | 29.06 | +4.53 | 9.15 | 13/5/9 | 9/6/12 |
| Held-out oracle | 30.66 | +6.13 | 9.80 | 18/0/9 | 9/8/10 |
Table 5. Post hoc checkpoint diagnostic. Development selection chooses the snapshot with highest five-game development RHAE, breaking ties toward the earlier snapshot. “+/−/ =” counts held-out improvement, regression, and no change relative to . Costs exclude outer-loop construction and selection expenditure. The held-out oracle is diagnostic only.
Table 6: Logical solving inference cost per held-out game across all 27 scored lineages. Logical cost includes replayed responses.
| Snapshot | Mean USD/game | Median USD/game |
| Initial | 10.30 | 9.18 |
| Revision 1 | 7.15 | 6.26 |
| Revision 2 | 7.57 | 7.18 |
Table 6. Logical solving inference cost per held-out game across all 27 scored lineages. Logical cost includes replayed responses.
Table 7: Joint held-out score–cost movements. The “cheaper/RHAE down” category records joint decreases in cost and RHAE.
| Transition | Pareto gain | Score↑/cost↑ | Cheaper/RHAE↓ | Worse both | Tie/cost↓ | Tie/cost↑ |
| 6 | 9 | 9 | 2 | 0 | 1 | |
| 6 | 6 | 8 | 7 | 0 | 0 | |
| 9 | 6 | 9 | 3 | 0 | 0 |
Table 7. Joint held-out score–cost movements. The “cheaper/RHAE down” category records joint decreases in cost and RHAE.
The 27 lineages ran over 29 days (17 August to 15 September 2026) across two machines; with pilot and replaced runs, the experimental program spanned at least five weeks. It corresponds to at least $18,129.27 under the frozen publication price schedule. Fresh selected-game attempts account for $16,130.09. Across the 27 scored lineages, coding-bootstrap inference for construction, revision, and repair totals at least $1,352.34. Smoke qualification adds $646.84 across all planned runs.
We report endpoint score and solving cost jointly: nine descendants are strict deployment score–cost Pareto gains, while nine apparent savings are performance contractions. Where aggregate report accounting is unavailable, logical cost is recovered from immutable suite selections and complete per-game meters. Unselected game-attempt recovery overhead is outside the reported program lower bound (Appendix § 18). Coding cost contains three unresolved requests, all in revision 1; qualification contains seven. We do not impute these requests, so Table 8 reports lower bounds. The same treatment applies to one further unresolved provider request outside these two cost components.
| Component | Initial | Revision 1 | Revision 2 | Total |
| Coding loop, 27 scored | 607.92 | 371.78 | 372.64 | 1,352.34 |
| Smoke qualification, 27 planned | 408.02 | 111.11 | 127.72 | 646.84 |
| Fresh selected-game attempts, 27 scored | 6,749.95 | 4,712.60 | 4,667.55 | 16,130.09 |
| List-price-equivalent total | 7,765.89 | 5,195.49 | 5,167.90 | 18,129.27 |
Table 8. Inference cost in USD under the frozen publication price schedule. Selected-game cost counts fresh model inference; logical replay charges used for score–cost comparison are excluded. Coding-bootstrap cost covers construction, revision, and repair; qualification is runtime inference used to select a smoke-qualified candidate.
Human-steered reference systems. Human steering refers to development decisions and is compatible with AI-written code. Tycho reports 100.00 RHAE over all 25 public games with GPT-5.6 Sol at maximum reasoning effort and with Opus 5 at xhigh ([23]). VISTA and AVO also report 100.00 with Opus 5 ([16, 6]); VISTA uses xhigh and reports 98.27 for its separate Sol-max run. Retrodict reports 99.86 with Sol at maximum effort ([4]). In our study, revision-2 held-out means are 28.58/41.94 for Sol at high/xhigh and 15.43/39.95 for Opus at high/xhigh (Table 4). The Opus comparison matches both model version and reasoning effort. Tycho designates the same five development games as our study (tr87, vc33, r111, bp35, ft09) and scores 100 on each of the twenty remaining games. The performance gap therefore persists on the same evaluation games. The Sol comparison uses different reported effort labels (max versus high/xhigh).
The development and selection procedures differ. Tycho selects among four policies using Opus 4.8 on all 25 public games, then evaluates the selected policy with Opus 5 and Sol; our twenty held-out games provide no feedback for construction, revision, or selection. Retrodict likewise reports earlier development attempts on all public games. Reported costs vary substantially across these systems and use different pricing dates and accounting conventions; we do not normalize their historical costs to our frozen publication price schedule. Inference allocation differs as well. Tycho reports about 15,100 model calls with Opus 5 and 26,500 with Sol over the 25 games, roughly 600 and 1,060 per game. Our revision-2 harnesses average 13.9 (Opus high), 18.1 (Opus xhigh), 78.7 (Sol high), and 114.2 (Sol xhigh) logical model responses per held-out game. The two counts are instrumented differently, so the contrast is one of order of magnitude. Call counts are not the only difference, and spend alone does not account for the gap: Retrodict reports about $26 per game at its list prices, less than two of our initial Sol-xhigh harnesses spent ($31–38 per game for about 44 RHAE; Table 3). The comparison establishes attainable performance with these models, but does not isolate the contribution of human steering or establish equivalent efficiency.
The coding-agent baselines cited in the introduction come from within-study comparisons. Schema reports 42.83 RHAE for its generic Claude Code baseline and 98.98 for its world-model harness using the same Opus 4.8/Fable 5 pairing ((Zeng et al., 2026)). Twin reports 61.1 for off-the-shelf Codex with Sol and 93.3 with its world-model harness, holding the model, file bridge, and sandbox fixed; the stronger harness also uses more inference compute ([37]).
Fixed-harness execution variance
The study evaluates each frozen snapshot once. To measure run-to-run execution variance separately from construction and revision, we reran one snapshot after the campaign: Sol-high replicate 1 at . The snapshot was selected post hoc after campaign outcomes were available, and GPT-5.6 was chosen for its operational speed; this is a diagnostic case rather than a random sample of harnesses. The original evaluation ran in August 2026 and the nine fresh evaluations in September 2026. Recorded harness, controller, model-profile, provider-configuration, and environment fingerprints are identical across evaluations, although unexposed provider-side changes cannot be excluded. Each evaluation uses the same twenty held-out games and starts new runtime-model inference. No construction, revision, repair, checkpoint selection, or hosted replay occurs in the added runs.
| Evaluation | RHAE | Actions | Levels completed | Complete games |
| 0 | 53.56 | 8,492 | 102 | 8 |
| 1 | 46.86 | 8,509 | 94 | 6 |
| 2 | 43.38 | 9,382 | 90 | 6 |
| 3 | 48.21 | 9,389 | 92 | 7 |
| 4 | 45.32 | 8,515 | 91 | 6 |
| 5 | 55.37 | 8,459 | 104 | 8 |
| 6 | 46.52 | 9,435 | 99 | 7 |
| 7 | 49.23 | 9,410 | 92 | 6 |
| 8 | 52.10 | 8,648 | 96 | 8 |
| 9 | 44.75 | 10,135 | 93 | 5 |
Table 10. Ten evaluations of one unchanged Sol-high harness. Evaluation 0 is the original campaign observation. “Complete” counts games for which all levels were solved; RHAE also gives partial credit within the remaining games.
Across all ten evaluations, mean RHAE is 48.53, the sample standard deviation is 3.99, and the range is 43.38–55.37. These are descriptive statistics conditional on the post hoc snapshot. The nine fresh evaluations alone have mean 47.97 and sample standard deviation 3.79, so the observed variability does not depend on including the older campaign observation. The original 53.56 observation is the second highest and lies 5.03 points above the ten-evaluation mean. The mean absolute difference between two evaluations is 4.73 points. Every evaluation remains above this trajectory’s 20.38 initial score, so its endpoint improvement classification is unchanged. Replacing only its original observation by the ten-run mean changes the campaign-wide endpoint mean from +2.80 to +2.62; this is a mechanical sensitivity check rather than a measurement-error correction for the other trajectory.
Variation is larger at game level: the median within-game sample standard deviation is 18.78 RHAE points, and the three largest are 30.76 on sb26, 28.33 on m0r0, and 27.13 on tn36. Averaging twenty games therefore substantially stabilizes the suite score, while single-trajectory differences of a few points remain difficult to interpret precisely. The observed 3.99-point suite standard deviation gives empirical scale to the sensitivity in Table 9; one harness is insufficient to define a universal noise threshold. These post hoc evaluations are separate from the construction-and-revision study, and their spend is excluded from Table 8. The supplement releases all 200 evaluation–game rows and a standalone reproduction command for these statistics.
| Improve | Regress | Indeterminate | Pareto gain | |
| 0 | 15 | 12 | 0 | 9 |
| 1 | 11 | 8 | 8 | 7 |
| 5 | 9 | 3 | 15 | 5 |
Table 9. Endpoint robustness to practical RHAE thresholds. Changes within are indeterminate; deployment Pareto gains also require lower solving cost.
Weighting and game sensitivity. The endpoint mean is +2.80 under both trajectory and condition weighting, since each condition has three complete lineages, and +2.55 when the seven model families receive equal weight. The condition-stratified bootstrap interval is [−1.12, 7.01] under either weighting and remains inconclusive. Leaving out one condition at a time yields means from +1.72 to +3.59 and adaptation gaps from 4.07 to 6.40. Across held-out games, 13 have a positive mean endpoint change and seven a negative one; the largest gain is +20.44 on lp85, while the largest decline is −15.03 on sb26. The small pooled gain is distributed across conditions and concentrated in opposing game-level effects. These intervals condition on the fixed games and do not quantify uncertainty over new development partitions.
Omitting r11l only from the observed development summary reduces the mean development–heldout gap from 5.62 to 3.02 points. The corresponding gaps after omitting each of the other development games range from 4.68 to 7.84. This is a descriptive influence check, not a counterfactual experiment: all harnesses were still revised using the original five-game feedback. Tables 11 and 12 give completion, action, and per-game context. Higher level completion accompanies the pooled score gain, but aggregate action counts alone do not distinguish improved efficiency from earlier stopping. The supplement reproduces these summaries from the same published game rows.
| Stage | RHAE | Complete games | Levels completed | Actions |
| Initial | 24.54 | 73/540 (13.5%) | 34.5% | 259.6 |
| Revision 1 | 25.87 | 83/540 (15.4%) | 36.4% | 292.0 |
| Revision 2 | 27.34 | 94/540 (17.4%) | 37.1% | 273.6 |
Table 11. Held-out completion and action context across the same 540 evaluations per stage (27 lineages 20 games). Level completion is the mean within-game fraction of completed levels. Actions are mean charged actions; lower action counts need not imply better efficiency because systems may stop without completing a game. These summaries do not isolate the components of RHAE.
| Game | Split | RHAE | Complete | Levels | Actions |
| bp35 | dev | +3.43 | +3.7 | +5.3 | +46.5 |
| ft09 | dev | -0.45 | +11.1 | -2.5 | -2.1 |
| r11l | dev | +18.83 | +14.8 | +17.9 | +22.1 |
| tr87 | dev | +8.11 | +7.4 | +11.1 | +22.8 |
| vc33 | dev | +12.19 | +11.1 | +13.2 | +76.1 |
| ar25 | heldout | -1.29 | +3.7 | -4.2 | -80.7 |
| cd82 | heldout | +3.11 | +3.7 | +1.9 | -1.0 |
| cn04 | heldout | +1.74 | +3.7 | +0.0 | -39.9 |
| dc22 | heldout | -2.39 | +3.7 | -4.3 | +21.2 |
| g50t | heldout | -1.42 | +0.0 | -0.5 | -4.5 |
| ka59 | heldout | -1.46 | +7.4 | -1.6 | -13.0 |
| lf52 | heldout | -3.75 | -3.7 | -4.4 | +41.1 |
| lp85 | heldout | +20.44 | +29.6 | +13.0 | -24.1 |
| ls20 | heldout | +4.96 | +3.7 | +7.4 | +87.0 |
| m0r0 | heldout | -1.49 | -3.7 | +0.0 | -49.6 |
| re86 | heldout | +1.38 | +3.7 | -0.5 | -3.0 |
| s5i5 | heldout | +8.25 | +7.4 | +8.3 | -33.6 |
| sb26 | heldout | -15.03 | -18.5 | -13.0 | +5.0 |
| sc25 | heldout | +1.64 | +0.0 | +2.5 | +46.6 |
| sk48 | heldout | +4.69 | +3.7 | +5.1 | +72.4 |
| sp80 | heldout | +10.01 | +11.1 | +11.7 | +74.6 |
| su15 | heldout | +4.82 | +0.0 | +7.4 | +37.8 |
| tn36 | heldout | +4.49 | +7.4 | +5.3 | +44.1 |
| tu93 | heldout | +15.66 | +14.8 | +16.9 | +59.7 |
| wa30 | heldout | +1.71 | +0.0 | +1.2 | +39.9 |
Table 12. Per-game initial-to-revision-2 changes, averaging the same 27 paired lineages. Completion columns are percentage-point changes in complete games and mean fraction of levels completed. Rows describe observed outcomes; no alternative feedback split was used to generate the harnesses.
Harness and Strategy Audit
We inspect every selected snapshot, exact diff, updater record, and relevant smoke-repair artifact in all 27 trajectories. The strategy labels are post hoc descriptions derived with outcome visibility. Preserve (P) means repair or tuning without changing the operating strategy. Extend (E) adds a material capability within the same decision loop. Reorient (R) changes control priority, context continuity, replay, or action/inference scheduling while retaining the broader harness family. Replace would change that family; no transition does so.
Every lineage grows from the initial to the second-revision snapshot. Median size rises from 2,161 to 2,604 to 2,912 generated Python lines, and the mean endpoint increase is 1,052 lines. Both large and small diffs lie on each side of zero held-out change. Edit volume establishes substantive revision; held-out change determines whether that revision improved performance.
| Condition | Initial harness family | P/E/R | P/E/R |
| Astra high | Persistent Python workers with policy reuse | 0/0/3 | 0/1/2 |
| Sol high | Direct multimodal batched planner | 0/0/3 | 0/3/0 |
| Terra high | Compact visual/text planner | 0/0/3 | 0/2/1 |
| Luna high | Direct model controller | 0/2/1 | 0/2/1 |
| Fable high | Strategist, executor, and replay system | 1/0/2 | 1/0/2 |
| Opus high | Layered world-model/planner hybrids | 0/0/3 | 0/0/3 |
| Sonnet high | Heuristic/model hybrid | 0/1/2 | 1/2/0 |
| Sol xhigh | Multimodal batched planner | 0/1/2 | 1/2/0 |
| Opus xhigh | World-model and replay hybrids | 0/0/3 | 0/0/3 |
| All conditions | 1/4/22 | 3/12/12 |
Table 13. Strategy-transition counts across all conditions. Entries give Preserve/Extend/Reorient.
First revisions reorganize; second revisions diversify. First revisions most often change the operating strategy itself: models alter context persistence, replay flow, model-consultation cadence, action batching, or budget allocation. Second revisions split equally between extending capabilities and reorienting the strategy. Even changes described as incremental by their builders can change when the runtime model is consulted or how scarce actions and dollars are scheduled.
| Condition | Replicate | ||
| Astra high | 1 | R | R |
| Astra high | 2 | R | R |
| Astra high | 3 | R | E |
| Sol high | 1 | R | E |
| Sol high | 2 | R | E |
| Sol high | 3 | R | E |
| Terra high | 1 | R | E |
| Terra high | 2 | R | E |
| Terra high | 3 | R | R |
| Luna high | 1 | E | E |
| Luna high | 2 | E | R |
| Luna high | 3 | R | E |
| Fable high | 1 | R | P |
| Fable high | 2 | P | R |
| Fable high | 3 | R | R |
| Opus high | 1 | R | R |
| Opus high | 2 | R | R |
| Opus high | 3 | R | R |
| Sonnet high | 1 | R | E |
| Sonnet high | 2 | E | P |
| Sonnet high | 3 | R | E |
| Sol xhigh | 1 | R | E |
| Sol xhigh | 2 | E | P |
| Sol xhigh | 3 | R | E |
| Opus xhigh | 1 | R | R |
| Opus xhigh | 2 | R | R |
| Opus xhigh | 3 | R | R |
Table 14. Strategy transitions for all 27 replicates. P, E, and R follow the definitions above; Table 3 gives the corresponding score–cost trajectories.
Replicates disagree within the same condition. The strategy paths are identical across all Sol-high replicates but their held-out outcomes range from a 33.18-point gain to a 15.18-point loss. Fable, Luna, Sonnet, and Sol-xhigh also mix preserve, extend, and reorient transitions within one condition. Opus-high and Opus-xhigh reorient at every stage, yet their outcomes include both gains and regressions. Strategy change is therefore common, and its presence or coarse type does not certify improvement.
An evidence-linked improvement. Sol-high replicate 1 (T001), selected post hoc for illustration, connects feedback to a substantial gain. Three development games exhausted ’s self-imposed 100-decision limit; its meters recorded repeated cache writes with few reads. Revision 2 raised this internal limit to 120, removed explicit cache writes, and added animation-aware observations, object geometry, and completed-level memory. The pre-edit plan names these changes; the transcript records 17 passing local tests, and the candidate qualified without repair. Held-out RHAE rose from 29.24 to 53.56 while inference cost per game changed from $8.19 to $8.15. Several components changed together, so the case illustrates evidence-grounded adaptation with observed transfer without isolating a causal mechanism. The supplement’s case-study/ directory links feedback excerpts, the plan, source, tests, and outcomes.
Cost reduction with performance loss. In Sol-xhigh replicate 1 (T007), the first revision replaced response-ID chaining and history compaction with independent calls carrying a bounded notebook and recent outcome. Across the two revisions, held-out RHAE fell from 43.95 to 21.99 and inference cost from $31.39 to $7.18 per game; development RHAE also fell, from 37.11 to 31.73. Revision 2 partially recovered both scores. Several components changed together, so any attribution of the regression to lost contextual information remains untested. The supplement’s case-study/T007.md links the source changes and per-stage outcomes.
Capability Codebook and Sampling
The process-coding corpus covers all 54 parent-to-descendant revision episodes from the 27 study lineages. Each packet receives two independent annotations under the same frozen codebook, with model identity and downstream outcomes masked.
To relate observable revision behavior to held-out outcomes, we analyze visible model messages, tool calls and results, generated source, exact diffs, tests, fixed feedback packets, smoke records, runtime traces, actions, scores, and costs. We do not treat provider-hidden chain-of-thought as evidence of cognition.
Evidence boundary and units. Direct trace evidence is a logged event or immutable artifact. Operational relations are reproducible links, such as a feedback item followed by a targeted edit and relevant test. Claims about latent beliefs or understanding are interpretations and are excluded from quantitative coding. Complete trajectories are the independent units; events, games, and stages are nested observations.
Codebook organization. The codebook measures four stages of the update process:
feedback-evidence access, diagnostic specificity, and discriminating probes.
plan enactment, feedback coverage, edit alignment, and change scope.
incumbent protection, updater verification, and proximal result.
post-result adaptation and resource allocation.
| Code group | Central question | Observable indicators |
| Evidence access | Does the updater inspect item-specific support before editing? | Feedback summaries, traces, logs, score breakdowns, errors, and reproducers (EVID). |
| Diagnosis | Does it connect cited evidence to a mechanism and prediction? | Pre-edit symptoms, localized mechanisms, predictions, counterfactuals, and boundary conditions (DIAG). |
| Diagnostic probing | Does evidence discriminate among hypotheses? | Targeted inspection, stated alternatives, refutable expectations, and result-governed narrowing (PROBE). |
| Plan and alignment | Are targets, actions, criteria, and retained edits linked? | Enacted plans, eligible-feedback coverage, and final-edit alignment (PLAN, COV, ALIGN). |
| Change control | Does it contain risk and protect the incumbent? | Executable scope, fallbacks, baseline checks, comparison, narrowing, rollback, and no-change decisions (SCOPE, PROTECT). |
| Verification | Does it test the retained change and remain runnable? | Reproducers, targeted and regression tests, smoke repair, requalification, and proximal results (VERIFY, RESULT). |
| Post-result adaptation | Does a post-edit result govern the next action? | Interpretation followed by handoff, edit, revert, changed test, narrowed scope, or criterion-based stop (ADAPT). |
| Resource adaptation | Does evidence govern computation allocation? | Calls, context, batching, caching, tests, stopping, numeric triggers, and expected-value comparisons (META). |
Table 15. Episode codebook used for observable harness revision.
Codebook refinement and consensus. An initial blind calibration exposed temporal ambiguities. We tightened only the pre-edit and post-result anchors; every packet was then coded under frozen version 2 before its outcome linkage was opened. Consensus labels were frozen by a conservative rule: exact agreement stands, ordinal disagreement receives the lower evidenced capability level, and any missing interval remains missing.
Ordinal episode coding. The twelve variables are feedback-evidence access (EVID), diagnostic specificity (DIAG), discriminating probe strength (PROBE), plan enactment (PLAN), eligible-feedback coverage (COV), final-edit alignment (ALIGN), scope containment (SCOPE), incumbent protection (PROTECT), updater verification (VERIFY), proximal operational result (RESULT), post-edit result-grounded adaptation (ADAPT), and resource adaptation (META). Sequential codes credit only evidence visible before the relevant action. Final summaries can corroborate but cannot create a prior diagnosis, prediction, probe, plan, or resource criterion. Mandatory final smoke passage contributes to RESULT, but raises VERIFY only when a failure causes repair and requalification.
Outcome linkage. Coder files and conservative consensus labels were hashed before opening the private episode key. Associations use immediate parent-to-child held-out change. Confidence intervals resample trajectories. Permutation tests shuffle complete two-stage outcome profiles across trajectories while preserving stage. Our exploratory inclusion rule requires ; codes below this cutoff and invariant high-agreement codes are reported descriptively. The artifact provides aggregate reliability, prevalence, outcome, and association tables and fixes 10,000 resamples and permutations per process association with deterministic seeds. These tests are exploratory and unadjusted for multiple comparisons. Shuffling across conditions assumes exchangeable trajectory profiles, which condition-level differences may violate; the reported values do not remove confounding.
Table 16: Unadjudicated reliability under frozen codebook version 2. excludes variable-specific missingness. uses quadratic disagreement weights for ordered categories. uses nominal disagreement and exposes prevalence effects. Dashes denote undefined chance-corrected coefficients for invariant labels.
| Code | Exact | |||
| EVID | 54 | 1.00 | – | – |
| DIAG | 30 | .67 | .35 | .38 |
| PROBE | 53 | .85 | .66 | .67 |
| PLAN | 27 | .85 | .70 | .70 |
| COV | 54 | 1.00 | – | – |
| ALIGN | 54 | 1.00 | – | – |
| SCOPE | 54 | .93 | .96 | .88 |
| PROTECT | 54 | .74 | .47 | .47 |
| VERIFY | 54 | 1.00 | – | – |
| RESULT | 54 | 1.00 | – | – |
| ADAPT | 54 | .93 | .84 | .84 |
| META | 35 | .71 | .29 | .17 |
Table 16.
Table 17: Exploratory process associations for labels with adequate inter-session reliability and outcome coverage. Contrasts are immediate held-out RHAE change at the highest minus lowest shown code. Intervals resample complete trajectories; values permute trajectory profiles. Associations are descriptive.
| Code | Contrast/counts | Interpretation | ||
| Scope containment | 54 | .96 | Scoped minus broad, interval , . | |
| Post-result adaptation | 54 | .84 | Further-action minus handoff, interval , . |
Table 17.
Planning-label missingness. The planning interval is observable in 27 packets and absent in 27. Missing packets average 9.22 executable files changed and 1,147 lines added, compared with 3.74 files and 414 lines when planning is observed. Missingness is also condition-dependent: all Opus-high, Opus-xhigh, and Sonnet packets are missing, whereas all Astra, Luna, and Terra packets are observed. It is almost balanced between revision stages (13 missing at revision 1 and 14 at revision 2). Consequently, the reliable agreement coefficient for the observed subset does not justify a planning–outcome conclusion.
Reactive-episode context. Episodes with further result-driven action start from higher-scoring parents on average (32.94 versus 21.33 held-out RHAE) and use more updater sessions (1.67 versus 1.11). Within the ten trajectories whose two revisions receive different adaptation labels, the reactive revision has lower change in eight. These descriptive differences do not isolate the effect of repair from the problems that prompted it.
Campaign Integrity and Recovery Audit
The experimental protocol and coding bootstrap remained fixed throughout the campaign. A post-collection source audit confirms unchanged prompt, feedback, qualification, game-interface, scoring, and resource-control implementations across the recorded versions. Operational recovery and provenance maintenance account for changes in execution-support code; the artifact supplies a concise source change log. Retained-record checks cover all 27 complete submitted lineages. Table 18 summarizes these checks; the supplementary artifact provides pseudonymous audit exports and an offline consistency checker. These checks establish preservation and consistency of recorded controls, not an independent experimental rerun.
| Control | Verified result |
| Frozen source | All 81 original snapshot fingerprints and file counts match, covering 1,985 files. |
| Evaluated identity | All 2,025 selected evaluations match the corresponding snapshot, runtime model-profile fingerprint, environment fingerprint, and configured output cap. |
| Prompts and bounds | Each rendered prompt is constant across replicates within each of nine conditions; all 27 lineages share one resource-limit profile. |
| Hosted validation | All 81 validations are complete, cover 25 games, agree with recorded scores, match snapshot identity, and use the same trace identity as local validation. |
| Coding preservation | All 54 blinded packets match their manifests; their labels and rationales are unchanged from the hash-verified frozen coding passes and consensus. |
Table 18. Post-collection integrity checks against retained original records.
Execution recovery. Construction replacement and operational recovery are distinct. Recovery finishes an evaluation of the same frozen harness, using compatible journals for resumption when available. The controller also permits a fresh execution without a replay parent; such a restart can draw new model responses and is not deterministic resumption.
The selected-suite manifests record 2,130 attempts for 2,025 game–snapshot slots across 162 suites. Sixty-three slots have multiple attempts. Each slot contains exactly one clean attempt, which is selected; the records show no comparison of multiple clean completed scores to select a higher one. Of the 105 other attempts, 49 are classified as provider infrastructure failures, 11 as blocked, and 45 as incomplete without an outcome reason. Twenty-one slots contain a fresh restart; seventeen selected attempts are those fresh restarts. No resource-stop record was found among these 2,130 attempts. These classifications do not fully establish the cause of every interruption. The released audit contains the attempt histories and detailed categories. It covers manifest-listed histories and does not estimate a causal effect of recovery on scores.
Recovery costs. Endpoint solving cost includes fresh and replayed inference in the selected attempt. The program lower bound sums publication-priced fresh selected-game inference, observed construction/revision/repair inference, and smoke qualification. It excludes unselected game-attempt recovery overhead; unresolved terminal requests are not imputed. Operational recovery meters have not been fully reconciled to the publication pricing schedule. The reported lower bound should therefore not be interpreted as an exhaustive bill for the campaign.
Bounded frame exception. The validator accepts a narrowly specified frame-hash discrepancy caused by unseeded visual noise in 1f52, while requiring agreement on state, level completion, and available actions. The exact predicate and expected and observed frame hashes are retained in the audit. Hosted validations contain 213 such frame events, all on 1f52; local validation records the same event count. All 81 hosted validations report score agreement. This exception concerns visual observations, not a relaxed score criterion.
Auditable process examples. The supplement includes the frozen per-code rationales for both coding passes and consensus across all 54 episodes. Three excerpts were selected by fixed rules using packet IDs and labels, without downstream outcomes, to illustrate agreement, disagreement, and missing evidence. Each includes ordered evidence anchors, labels, rationales, and the consensus rule. Selection rules and compression markers are preserved in the artifact. These examples support inspection of the coding method; they are not complete transcripts or human validation.
Artifact availability. The supplementary artifact contains the seed repository, frozen prompts and splits, condition, model, and pricing profiles, the scoring and feedback-curation code, all 81 anonymized source snapshots with original and review-copy fingerprints, trajectory and game tables, coder and consensus labels with their rationales, three chronological coding examples, two revision case studies, integrity and recovery audits, the capability manual, deterministic analyses, and a checksum manifest. Offline checks verify configuration, source fingerprints, audit consistency, labels, and example selection. The release supports reproducing the reported analyses, inspecting harness changes, and testing new methods for autonomous improvement. The artifact is provided as harness-adaptation-review-artifact.zip in the ancillary files accompanying this arXiv version.
Additional Analyses
This appendix collects additional analyses. New bootstrap intervals use seed 20261108 and may differ from the main-text intervals by a few hundredths.
Scale of the development signal
The five development games are multilevel interactive environments (Table 19). A failed level ends in GAME OVER and can be reset any number of times, but each reset is charged to that level’s budget; there is no fixed number of lives and no whole-game action cap. The 54 feedback bundles that seed the revisions summarize 270 parent-game evaluations ( for revision 1, for revision 2), with 883 observed and 666 completed level instances and about 42,700 scored actions, from parent runs that made 12,347 model calls; the bundles, including traces, frames, and costs, total 352 MiB. Level instances repeat levels across harnesses, and not every harness reached every level. The 20 held-out games contain 149 levels and receive 1,620 evaluations, none of which informs construction, revision, or selection.
| Game | Levels | Baseline per level | budget per level | Max. charged per evaluation |
| tr87 | 6 | 40–146 | 200–730 | 2,070 |
| vc33 | 7 | 7–152 | 35–760 | 2,235 |
| r11l | 6 | 22–52 | 110–260 | 1,165 |
| bp35 | 9 | 21–163 | 105–815 | 3,255 |
| ft09 | 6 | 12–65 | 60–325 | 1,040 |
| All five | 34 | 7–163 | 35–815 | 9,765 |
Table 19. Development games. Baselines are the hidden human action counts per level; the controller charges at most actions to level . The last column is the most one game evaluation can charge if every level exhausts its budget; the total row gives the five-game suite.
Model versions and run dates
Table 20 lists the model version, reasoning effort, and run dates of each condition. Within a condition, the same model and effort serve construction, revision, and gameplay, and all three lineages share one model profile.
| Condition | Model version | Effort | Run dates (2026, UTC) |
| GPT-6 Astra | gpt-6-astra | high | 13–15 Sep |
| GPT-5.6 Sol | gpt-5.6-sol | high | 17–21 Aug |
| GPT-5.6 Terra | gpt-5.6-terra | high | 17–21 Aug |
| GPT-5.6 Luna | gpt-5.6-luna | high | 17–21 Aug |
| Claude Fable 5 | claude-fable-5 | high | 17–22 Aug |
| Claude Opus 5 | claude-opus-5 | high | 17–23 Aug |
| Claude Sonnet 5 | claude-sonnet-5 | high | 17–20 Aug |
| GPT-5.6 Sol | gpt-5.6-sol | xhigh | 18–28 Aug |
| Claude Opus 5 | claude-opus-5 | xhigh | 21–28 Aug |
Table 20. Model versions and run dates. Dates span the first construction start to the last recorded controller transition among a condition’s three lineages. All models were accessed through a hosted inference API that exposed no dated model versions, so the dates give the access window; provider-side updates within it cannot be excluded. The controller fixes the model and reasoning effort on every coding and gameplay request; neither the controller nor any released harness sets a sampling temperature.
Statistical power
For the primary paired comparison, the relevant variation is the spread of the 27 lineage changes , whose sample standard deviation is 11.92 (standard error 2.29). Assuming this SD for future independent lineages, a two-sided paired test at reaches 80% power for a mean change of about 6.4 (normal approximation) to 6.7 (noncentral ) RHAE points at 27 lineages; Table 21 gives the lineages needed for other effects. The 3.99 SD of Appendix § 15.§ 1 is the run-to-run spread of one harness’s suite score, not the uncertainty of the pooled mean. Repeats alone may not close this gap: if every snapshot had that execution SD, with independent errors, the remaining between-lineage variation would still leave a detectable effect of about 5.9 at 27 lineages with unlimited repeats per endpoint. This rests on an untested assumption; with an endpoint SD of 8, unlimited repeats would reduce it to about 2.1. All of these are planning approximations.
Cost of larger designs. The recorded list-price costs (Table 8) indicate what larger designs would require. A further revision stage for all 27 lineages would cost about $5,200 if it cost as much as revision 2 (675 game evaluations plus coding and qualification). Ten evaluations of every snapshot instead of one would add about $145,000 (nine further passes at $16,130), and three paired held-out repeats of and for every lineage about $28,000. The 126 lineages of Table 21 would cost about $85,000 at the observed mean of $671 per lineage. These extrapolations assume unchanged per-stage costs and condition mix. The paired repeats and the crossed evaluation reported in this appendix (810 game evaluations) are excluded from Table 8.
Table 22 decomposes the endpoint by stage. The gap is 4.13 after one revision and 5.62 after two; the increase, +1.49 [−1.33, 4.58], is uncertain, and we do not extrapolate it to further revisions.
| Transition | Development change | Held-out change | Gap | + / − / = |
| +5.46 [0.73, 10.55] | +1.33 [−2.47, 4.96] | +4.13 [0.60, 7.66] | 15/11/1 | |
| +2.96 [−0.30, 6.31] | +1.47 [−0.84, 4.13] | +1.49 [−1.33, 4.58] | 12/5/0 | |
| +8.42 [2.81, 14.76] | +2.80 [−1.41, 7.41] | +5.62 [1.81, 9.75] | 15/12/0 |
Table 22. Paired stage changes over all 27 lineages. The gap is development minus held-out change. Intervals are 95% percentiles of 100,000 whole-lineage resamples under a separate seed (20261108); differences of a few hundredths from intervals in the main text are Monte Carlo error. “+ / − / =” counts held-out gains, losses, and ties. The two transitions of a lineage are dependent.
Selection-rule provenance and sensitivity
The development-based rule of Table 5 was specified post hoc. The 24 lineages of the eight conditions other than GPT-6 Astra ran 17–28 August 2026, and their held-out scores were tabulated on 31 August. The rule first appears in our version history on 3 September (local time, UTC+2), already labeled a post hoc diagnostic, in a commit that descends from the one containing the held-out table. No analysis plan was preregistered: the campaign manifest of 17 August fixes replacements, prompts, and the replacement policy, but no endpoint or selection rule. Our code and notes contain no alternative development-based rule, and the rule has not changed since. Only the three GPT-6 Astra lineages, run 13–15 September, postdate it; on them it gains +0.08 over deploying , whose mean is 94.37, so they provide no validation.
Table 23 recomputes the selection after omitting one development game at a time, with all candidates fixed. Averaging over all subsets of a given size within each lineage, gains over are +4.08 [0.40, 8.19], +3.77 [0.12, 7.81], +3.49 [−0.03, 7.41], and +2.64 [−0.16, 6.75] for four, three, two, and one selection games; gains over are +1.28 [0.04, 2.95], +0.97 [−0.31, 2.54], +0.69 [−0.70, 2.22], and −0.17 [−2.13, 1.72]. Evidence for a positive gain over is thus more stable than for an advantage over always deploying . All candidates were revised using feedback from all five games, so an omitted game remains in the adaptation history: this checks the selector’s sensitivity, not adaptation, and is not a prospective validation.
Table 23: Leave-one-development-game-out selection. Each row selects among the fixed , , and by mean development RHAE on the remaining games, breaking ties toward the earlier snapshot. “+ / − / =” counts held-out improvement, regression, and no change relative to . Intervals use 100,000 whole-lineage resamples under a separate seed (20261108), so the all-five row differs from Table 5 by Monte Carlo error.
Floor conditions and builder/player coupling
Table 24 removes near-floor conditions while keeping all three lineages of every retained condition. The thresholds use only condition-mean development RHAE; they are post hoc baseline subsets that change the target population rather than correct the primary estimate.
Table 24: Floor-condition sensitivity. Thresholds apply to condition-mean development RHAE: > 1 drops Sonnet, > 5 also Luna, and > 10 also Terra, which coincides with excluding all three. The gap is development minus held-out change. Intervals are 95% percentiles of 100,000 whole-lineage resamples (seed 20261108).
Within a lineage the runtime model is fixed, so changes reflect revision plus execution noise. The near-floor harnesses run: all 675 Terra, Luna, and Sonnet games take charged actions and pass controller checks, 4.3% end in a nonzero harness exit not caused by an action cap (6.7% in the other six conditions), and Terra and Luna complete at least one level in 48% and 44% of their games (Sonnet 10%). With the same Terra player, held-out level completions fall from 64 at to 28 at and 22 at , lower in all three lineages; weak play also thins the feedback, as five of Sonnet’s six development bundles contain no completed level. Floor compression shrinks both terms of the gap; the three conditions together contribute −0.06 of its 5.62 points.
Crossed builder/player evaluation. To separate construction from gameplay, we re-evaluated frozen and snapshots with a different runtime model, changing only the controller-written model profile; generated code is byte-identical. All three Sol-high lineages were played by Terra-high and by Luna-high, and all three Sonnet-high lineages by Sol-high, each once on the 20 held-out games (36 suites including the prescribed development suites). Table 25 compares each crossed cell with the same harness played by its builder (player effect) and with the player’s own harness (builder effect). For Terra and Luna, gameplay limits the score: Sol-built harnesses lose 19–26 points when they play them, yet gain them only 1–6 points over their own harnesses. For Sonnet, the harness does: its harnesses run without errors, but Sol gains only 1–2 points when playing them, whereas Sol’s own harnesses score about 25 points higher. The reverse cells (a Terra- or Luna-built harness played by Sol, a Sol-built harness played by Sonnet) were not run, so these models’ other role is not assessed separately. Weak players also compress revision effects: under Terra or Luna, the Sol lineages’ changes (+33.18, −15.18, −10.70 with Sol) lie between −5.51 and +8.20. With one evaluation per cell, these comparisons are descriptive.
Paired repeated evaluations
To test whether individual endpoint classifications survive execution noise, we re-evaluate and of three lineages three times each on all 20 held-out games (360 game evaluations). The lineages were fixed from historical outcomes before any new inference. The near-zero case has the smallest absolute change among lineages with both endpoint means in [5, 90] and an absolute change below 5 (Fable-high replicate 1, 16.05 13.77); the gain and loss cases are the largest gain and largest loss among the Sol-high lineages (replicate 1, 20.38 53.56; replicate 2, 27.92 12.74). For the first two lineages, and run concurrently in each of three sequential blocks; the loss lineage’s six suites run concurrently afterwards, so its three pairs are not separated in time. Concurrency exceeds that of the original campaign. Harness, controller, model-profile, provider-configuration, and environment fingerprints are verified against the original evaluations, and no construction, revision, repair, hosted replay, or selection occurs.
Because provider-side changes since August cannot be excluded, fresh is compared with fresh , not with the original scores. Each pair’s difference averages the 20 paired game differences; we report the mean of the three differences with a paired interval (df = 2), call a direction supported only if that interval excludes zero, and use the -point band as a second classification. If both endpoints have an SD near 4 and are independent, the interval half-width is about 14 points, and because the gain and loss cases are extremes, some shrinkage toward zero is expected. Games whose failure carries an explicit provider-infrastructure classification are resumed by the controller’s standard mechanism, as in the original campaign; unexplained incomplete attempts and blocked validations stop the campaign. This policy was adopted after one such failure in the first block, before that suite had a computed score; the means of the other three first-block suites had already been computed.
All 18 suites completed. Fifteen of the 360 games (4.2%) hit an explicit provider-infrastructure failure (hard deadline or HTTP 504) and completed cleanly after one resume. Table 26 reports the results. Both large observed changes reproduce: the gain lineage improves in all three fresh pairs (+30.82, interval [9.97, 51.67]) and the loss lineage declines in all three (, [, ]). The loss interval’s upper end lies just inside the band, so its direction is supported while its practical class is unresolved under the preset rule. The near-zero lineage remains unresolved (, [, 7.33]). No case supports a strict score–cost classification: every cost-per-game interval includes zero.
Rule-based check of interpreted codes
The two codes interpreted in Section 6.2, SCOPE and binary ADAPT (further result-driven action versus handoff), are defined by file counts, line counts, and event order. We re-derived them with a deterministic script that uses no language model; its rules and implementation hash were frozen before any per-episode label, rationale, or outcome was opened. It computes SCOPE from the released parent and child snapshots and ADAPT from the packets’ chronological tool events, the evidence the coders saw (Table 27). SCOPE agrees closely and reproduces the published contrast; its three disagreements depend on how the codebook’s line-count denominator or subsystem granularity is read. ADAPT agrees less (, below the .67 inclusion threshold), and its contrast does not hold: the published negative mean rests on seven episodes that only the consensus codes as reactive (mean change −5.38), while the 11 episodes that both code as reactive average +2.18. Under script labels, reactive and handoff episodes also start from similar parents (24.65 versus 25.36 held-out RHAE). The ADAPT contrast therefore depends on how post-result adaptation is operationalized. The script is a second operationalization with its own errors, not a human validation.
Recovery sensitivity
All 2,025 selected scores, recomputed from the original local scorecards with the released scorer, match. Of the 105 additional attempts, 49 carry an explicit provider-infrastructure reason (42 HTTP 504 responses, six hard deadlines at 1,830 seconds, and one upstream timeout at 3,540 seconds), and 11 are blocked by validation checks (five model-response replay divergences, three game-replay divergences, and three missing local scorecards). The other 45 are incomplete: each retains one unresolved model request and no exit status or outcome reason, so we leave their cause unresolved. None is classified as a model-software failure; harness error exits occur within clean selected attempts and are scored as outcomes. The 45 incomplete attempts occupy 38 slots, 11 of which also hold one of the 17 selected fresh restarts, so 44 slots in eight lineages are affected (Table 28); all fresh restarts are on held-out games.
| Snapshot | Split | Incomplete attempts (slots) | Fresh restarts | Overlap | Union |
| Development | 7 (4) | 0 | 0 | 4 | |
| Held-out | 9 (7) | 6 | 3 | 10 | |
| Development | 0 (0) | 0 | 0 | 0 | |
| Held-out | 0 (0) | 1 | 0 | 1 | |
| Development | 3 (3) | 0 | 0 | 3 | |
| Held-out | 26 (24) | 10 | 8 | 26 | |
| All | 45 (38) | 17 | 11 | 44 |
Table 28. Recovery events by snapshot and split. Incomplete attempts lack an outcome reason; parentheses give their distinct slots. Fresh restarts are selected attempts without a replay parent. The union counts each affected slot once.
Table 29 removes affected games at both endpoints and averages paired changes with equal lineage weights. Under the union exclusion, the held-out gain moves from 2.80 to 2.56 and the gap from 5.62 to 4.21 with unchanged sign counts, but the gap interval, [−0.05, 8.87], then includes zero. This exclusion retains all 27 lineages but leaves as few as 11 held-out games and, for one lineage, a single development game. Excluding the seven lineages with an affected slot entirely gives a gap of 5.20 but a held-out gain of only 1.02; all seven had positive original changes, so this exclusion is not missing at random. Replacing each affected endpoint score in the union by any value in [0, 100] bounds the held-out gain to [−0.21, 6.46], the gap to [−1.38, 10.47], and losses to 12–15; these are deterministic bounds, not confidence intervals. The gap’s point estimate thus stays positive under every exclusion, while its interval, and its sign under worst-case bounds, depend on how recovery is treated.
References
- [2]ARC Prize Foundation. ARC-AGI-3 leaderboard. Official leaderboard, 2026. URL https://arcprize.org/leaderboard. Accessed 2026-11-13.
- [3]Jakob Bauer, Kate Baumli, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Satinder Singh, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei M. Zhang. Human-timescale adaptation in an open-ended task space. In Proceedings of the 4th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 1887–1935, 2023. URL https://proceedings.mlr.press/v202/bauer23a.html.arxiv.org/abs/2301.07608
- [4]Ryan Brown. Retrodict: An agent for ARC-AGI-3. Software repository and experiment report, 2026. URL https://github.com/ryanbbrown/Retrodict.
- [5]Mingguang Chen, Licheng Wang, and Bo Qu. Recursive self-improvement in AI: From bounded self-refinement to autonomous research loops. arXiv preprint arXiv:2607.07663, 2026a. URL https://arxiv.org/abs/2607.07663.
- [6]Terry Chen, Yeyin Zhu, Zhifan Ye, Jean-François Puget, and Humphrey Shi. NVIDIA AVO reaches 100% on ARC-AGI-3, demonstrating a frontier-level general-purpose architecture for long-horizon autonomous agents. NVIDIA Technical Blog, 2026b. URL https://developer.nvidia.com/blog/nvidia-avo-reaches-100-on-arc-agi-3-demonstrating-a-frontier-level-general-purpose-architecture-for-long-horizon-autonomous-agents/.
- [7]Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, et al. Reasoning models don’t always say what they think. arXiv preprint arXiv:2505.05410, 2025. URL https://arxiv.org/abs/2505.05410.
- [8]François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. URL https://arxiv.org/abs/1911.01547.
- [9]Jacob Cohen. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213–220, 1968. doi: 10.1037/h0026256.
- [10]David Courtis, Wenhao Li, and Scott Sanner. OPINE-World: Programmatic world modeling with ontology-error-prioritized interactive exploration for ARC-AGI-3. arXiv preprint arXiv:2607.01531, 2026.arxiv.org/abs/2607.01531
- [11]Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL²: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv preprint arXiv:1611.02779, 2016. URL https://arxiv.org/abs/1611.02779.
- [13]Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1126–1135, 2017. URL https://proceedings.mlr.press/v70/finn17a.html.arxiv.org/abs/1703.03400
- [14]Alexis Fox, Junlin Wang, Paul Rosu, and Bhuwan Dhingra. PRO-LONG: Programmatic memory enables long-horizon reasoning. arXiv preprint arXiv:2607.20064, 2026.arxiv.org/abs/2607.20064
- [15]Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. In International Conference on Learning Representations, pp. 57734–57811, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/fef126561bbf9d4467dbb8d27334b8fe-Abstract-Conference.html.arxiv.org/abs/2305.11738
- [16]Qiushi Han, Keya Hu, Linlu Qiu, Cathy Wu, and Kaiming He. VISTA: A visual harness for reasoning in an interactive world. Project release, 2026. URL https://vista-research.github.io/.arxiv.org/abs/2610.02200
- [17]Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/36b7acf6f6010652b3f2a433774a66fe-Abstract-Conference.html.arxiv.org/abs/2408.08435
- [18]Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI agents that matter. Transactions on Machine Learning Research, 2025. URL https://openreview.net/forum?id=ZyU4uFzMviZ.arxiv.org/abs/2407.01502
- [20]Klaus Krippendorff. Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3):411–433, 2004. doi: 10.1111/j.1468-2958.2004.tb00738.x.DOI
- [21]Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023. URL https://arxiv.org/abs/2307.13702.
- [22]Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026. URL https://arxiv.org/abs/2603.28052.
- [23]Jens Lehmann, Andrei Aioanei, and Sahar Vahdati. Tycho: Active abstraction with programmatic world models for ARC-AGI-3. arXiv preprint arXiv:2607.28287, 2026. URL https://arxiv.org/abs/2607.28287.
- [24]Falk Lieder and Thomas L. Griffiths. Strategy selection as rational metareasoning. Psychological Review, 124(6):762–794, 2017. doi: 10.1037/rev0000075.DOI
- [25]Falk Lieder and Thomas L. Griffiths. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and Brain Sciences, 43:e1, 2020. doi: 10.1017/S0140525X1900061X.DOI
- [26]Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850, 2026a. URL https://arxiv.org/abs/2604.25850.
- [27]Minhua Lin, Juncheng Wu, Zijun Wang, Zhan Shi, Yisi Sang, Bing He, Zewen Liu, Tianxin Wei, Zongyu Wu, Zhiwei Zhang, Dakuo Wang, Xiang Zhang, Benoit Dumoulin, Cihang Xie, Yuyin Zhou, Suhang Wang, and Hanqing Lu. Harness updating is not harness benefit: Disentangling evolution capabilities in self-evolving LLM agents. arXiv preprint arXiv:2605.30621, 2026b. URL https://arxiv.org/abs/2605.30621.
- [28]Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, volume 36, pp. 46534–46594. Curran Associates, Inc., 2023. doi: 10.52202/075280-2019.DOI
- [29]OpenAI. How enabling two settings tripled our scores on the ARC-AGI-3 benchmark. Official research report, 2026. URL https://openai.com/index/how-two-settingstripled-our-arc-agi-3-scores/. Accessed 2026-09-25.
- [32]Sergey Rodionov. Executable world models for ARC-AGI-3 in the era of coding agents. In Artificial General Intelligence, volume 16855 of Lecture Notes in Computer Science, pp. 198–210. Springer Nature Switzerland, 2026. doi: 10.1007/978-3-032-33195-3_15.DOI
- [33]Stuart Russell and Eric Wefald. Principles of metareasoning. Artificial Intelligence, 49(1–3):361–395, 1991. doi: 10.1016/0004-3702(91)90015-C.DOI
- [34]Stuart J. Russell and Devika Subramanian. Provably bounded-optimal agents. Journal of Artificial Intelligence Research, 2:575–609, 1995. doi: 10.1613/jair.133.DOI
- [35]Elad Sarafian, Gal Kaplun, Ron Banner, Daniel Soudry, and Boris Ginsburg. Workspace optimization: How to train your agent. arXiv preprint arXiv:2605.09650, 2026. URL https://arxiv.org/abs/2605.09650.
- [36]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pp. 8634–8652. Curran Associates, Inc., 2023. doi: 10.52202/075280-0377.DOI
- [37]Alexy Skoutney, Kirill Acharya, Gaston Longhitano, Madeleine Udell, Kevin Ellis, and Iddo Drori. Twin: Playing an unknown game with a test-time digital twin. arXiv preprint arXiv:2608.14490, 2026. URL https://arxiv.org/abs/2608.14490.
- [38]Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems, volume 36, pp. 74952–74965. Curran Associates, Inc., 2023. doi: 10.52202/075280-3275.DOI


