CoEvo-RSI: When Harnesses Evolve, Which Models Should Follow?

GitHub https://github.com/SuJQ0722/CoEvo-RSI

Hugging Face https://huggingface.co/collections/nullptr04/coevo-rsi

Introduction

An agent’s capabilities depend jointly on its brain—the language model that reasons and selects actions—and its body—the harness that defines its tool interfaces, action space, memory, and execution workflow ([7] (Lou et al. 2026)). Accordingly, progress has followed two complementary directions. Model training improves the agent’s learned policy through supervised learning, reinforcement learning, and self-distillation ((Xia et al., 2026; Yang et al., 2026b; Zhang et al., 2026b}). Harness optimization improves the surrounding system with the model fixed, through iterative modification of prompts, tools, and control flow guided by execution feedback ((Lou et al., 2026; Lin et al., 2026)), or population-based evolution that generates, evaluates, and recombines competing harness designs ((Lee et al., 2026c; Zhang et al., 2026h)). These directions meet in model–harness co-evolution, where model training and harness optimization jointly adapt an agent to its task domain ([7] (Luo et al. 2026a)). Recent systems demonstrate the promise of this approach across tool use, mathematical reasoning, search-based question answering, and chess ((Chen et al., 2026b,d; Kim et al., 2026)).

A co-evolution round couples harness evolution with model adaptation. Starting from a model–harness pair $(H_t, \theta_t)$, execution feedback $F_t$ guides changes to the harness, whose requirements in turn shape the model’s next adaptation ([7]). As either component evolves, the harness’s execution requirements and the model’s learned capabilities can drift apart. In this setting, we first identify and formalize the brain–body mismatch problem:

In plain terms, the model may struggle to operate its harness, and the harness may underuse its model’s abilities (Figure 2). Consider two concrete examples. ❶ Interface mismatch. The model emits XML tool calls, while the evolved harness expects JSON, causing otherwise correct calls to be rejected. ❷ Capability mismatch. The harness introduces an active context management tool for deciding what information to retain, summarize, or discard. A model unfamiliar with these operations may use it poorly; a historical checkpoint $\theta_i$ already possesses the relevant capabilities and provides a suitable starting point for adaptation. On the harness side, a historical design $H_j$ can supply an execution workflow suited to the model’s capabilities. These examples motivate selecting predecessors whose inherited behaviors and structures support further evolution.

To address this challenge, we propose CoEvo-RSI, a framework for model–harness co-evolution through bidirectional historical inheritance. Its evolutionary archive $A_{<t}$ records earlier harnesses, model checkpoints, their development histories, and execution outcomes. Each round begins with feedback from the current pair. A Curator organizes this evidence, and harness editors browse the archive, choose their own historical starting designs, and develop offspring harnesses. For each offspring, the framework selects or merges historical checkpoints to form candidate model–harness pairs. Evaluation identifies a promising pair for model adaptation under its harness, and validation determines which pair advances to the next round. The resulting designs, checkpoints, and outcomes extend the archive. This agent-driven inheritance connects the two sides of evolution: harness design draws on accumulated structural experience, while model initialization draws on capabilities suited to the emerging harness. Their coordination makes brain–body alignment the organizing principle of the evolutionary process. Our contributions are:

• Problem Formalization. We formalize brain–body mismatch in model–harness co-evolution and introduce the turn-level interface mismatch rate (TIMR) as an inexpensive diagnostic signal for observable interface violations.

• Method Design. We propose CoEvo-RSI, which combines an evolutionary archive with bidirectional historical inheritance. Harness editors select historical starting designs, and model inheritance selects or merges checkpoints to support adaptation under the evolved harness.

• Empirical Evaluation. We evaluate CoEvo-RSI across multi-agent failure attribution, home care, and SQL tasks, with co-evolved Qwen3.5-9B outperforming Qwen3.8-Max on Who&When Pro (+4.40 agent acc.) and DeepSeek-V4-Pro on TraceElephant (+14.16 agent acc.).

Related Work

Self-evolving agents improve through changes to external execution systems, learned model parameters, or both (Gao et al., 2026; Guo et al., 2026; Wang et al., 2026a). We organize the literature around these optimization targets and the reuse of historical components that connects them.

Recursive Harness Improvement. Early agent systems established how external organization can improve a model’s behavior: ReAct interleaves reasoning and actions, Reflexion retains verbal feedback across trials, and Voyager accumulates reusable executable skills (Yao et al., 2023; Shinn et al., 2023; Wang et al., 2023). Agent–computer interface design further shows how the operations exposed to a model shape its effectiveness (Yang et al., 2024). Subsequent work turns these hand-designed mechanisms into optimization targets, evolving context (Zhang et al., 2026g), memory (Zhang et al., 2025a, 2026e; Pan et al., 2026b), planning (Liu et al., 2026b), and retrieval (Lee et al., 2026b). At the system level, harness search jointly modifies prompts, tools, and control flow (Lee et al., 2026a,c; Lin et al., 2026; Lou et al., 2026; Zhang et al., 2026d,f). AutoHarness synthesizes executable support from environmental feedback, while Meta-Harness gives its proposer access to earlier source code, scores, and execution traces (Lou et al., 2026; Lee et al., 2026c). Such access preserves evidence that a single scalar score or compressed reflection can omit. Online adaptation extends this process to deployment streams and test-time feedback (Liu et al., 2026c; Nie et al., 2026b; Pan et al., 2026a), population search and reusable repositories retain alternative designs for subsequent exploration (Zhang et al., 2026h; Nie et al., 2026a; Luo et al., 2026b). The Darwin Gödel Machine similarly grows an archive of self-modifying agents, allowing earlier branches to become stepping stones for later improvements (Zhang et al., 2025c). Together, these approaches expand the search target from isolated modules to complete execution systems. They also motivate preserving design diversity and execution history, which CoEvo-RSI connects to a corresponding archive of model checkpoints.

Recursive Model Improvement. Model self-improvement converts generated experience into persistent changes in the policy. Three learning loops are particularly relevant. (i) Curricula adapt the source and difficulty of training problems to the learner, including self-play, tool-oriented task generation, and search-agent training without externally supplied demonstrations (Zhao et al., 2025; Acikgoz et al., 2026; Sundaram et al., 2026; Yang et al., 2026a,b; Yue et al., 2026). The resulting feedback loop changes which experiences the model encounters as its competence develops. (ii) Supervision turns demonstrations, privileged feedback, or the model’s own improved behavior into self-distillation targets (Zhao et al., 2026a; Shenfeld et al., 2026; Hübotter et al., 2026; Zhang et al., 2026b; Lu et al., 2026b). On-policy variants address the learner’s current behavior, making the distribution of generated trajectories central to what can be learned. (iii) Experience reuse extracts reusable skills and transfers their guidance into model behavior (Xia et al., 2026; Zhang et al., 2026c; Lu et al., 2026c; Wu et al., 2026; Yang et al., 2026c). For example, SkillRL jointly evolves a hierarchical skill library and the agent policy, linking experience abstraction with subsequent reinforcement learning (Xia et al., 2026). These mechanisms improve the agent’s intrinsic capabilities, while the harness determines which tools, observations, and execution constraints appear in its training trajectories. This dependence motivates choosing model initialization and adaptation in relation to the harness that will actually execute the policy.

Model–Harness Co-Evolution.

Joint adaptation closes the loop between the environment presented to a policy and the behavior learned under that environment. One line couples (i) context and weights through prompt optimization, continual adaptation, and the transfer of contextual guidance into parameters (Lu et al., 2026a; Tiwari et al., 2026; Tu et al., 2026; Zhao et al., 2026b). A second line couples (ii) executable harnesses and policies, extending adaptation to the tools, memory, middleware, and control flow used during rollout (Chen et al., 2026b,c,d; Kim et al., 2026; Qi et al., 2026; Qu et al., 2026; Hebbar et al., 2026; Karten et al., 2026; Mao et al., 2026; TaoLive AIGC LLM Team et al., 2026; Yu et al., 2026b). HarnessForge combines fault-guided harness tailoring with harness-conditioned policy alignment [7]. Co-Harness alternates validated harness patches with fine-tuning on trajectories generated by the improved scaffold [7], while WHALE alternates rejection-sampling fine-tuning with executable harness search (Kim et al., 2026). A third line brings (iii) execution and editing into a shared learning process, allowing policies to participate in the evolution of their own support (Luo et al., 2026a; Shi et al., 2026). These methods establish the value of adapting both sides and expose a common dependency: a harness update changes the execution requirements, and a model update changes which harnesses are effective. CoEvo–RSI makes bidirectional historical inheritance an explicit decision within this loop. Harness editors choose historical starting designs, and model inheritance independently selects or merges archived checkpoints for the resulting requirements. This enables recovery of useful earlier capabilities without tying model initialization to the latest generation or the harness’s parent branch. SD compares the inheritance procedures of representative frameworks.

Method

CoEvo–RSI (Figure 3) realizes model–harness co-evolution through bidirectional historical inheritance. An autonomous meta-agent coordinates two complementary decisions: harness editors select and develop historical designs, and model inheritance selects or merges checkpoints for the resulting execution requirements. Evaluation, model adaptation, and promotion turn these inherited components into the next agent, whose outcomes enrich the evolutionary archive.

Model–Harness Co-Evolution

An executable agent is a pair $p = (\mathrm{H}, \theta)$, where the harness $\mathrm{H}$ specifies prompts, tools, state management, and control flow, and $\theta$ denotes the task model’s parameters. Constraints $\mathrm{C}$ fix the environment, verifier, and admissible interfaces. Let $\rho$ be the task distribution and $\mathrm{P}_{\theta,\mathrm{H}}(\cdot\mid x;\mathrm{C})$ the distribution of execution trajectories for task $x$. A trajectory $\tau$ contains model responses, tool interactions, and the final output; its task score $s(x,\tau)$ is assigned by the fixed evaluator. The objective is

where $\mathbb{E}$ denotes expectation. For a task set $\mathbf{D}$ and $n_{\mathrm{eval}}$ executions per task, let $\tau_{x,u}$ be the trajectory from execution $u$. We estimate this objective by

For $p=(H,\theta)$, we also write $\widehat{J}_{\mathbf{D}}(p)$ for this estimate.

Each generation develops and adapts model–harness pairs through two inheritance steps. A meta-agent with fixed parameters $\phi$ and role policies $\pi_{\phi}^{\mathrm{role}}$ operates on the incumbent $p_t=(H_t,\theta_t)$ and an archive $A_{<t}$ of earlier harnesses, checkpoints, and execution histories, including the seed pair. First, the incumbent’s execution feedback $\mathbf{F}_t$ guides the generation of $K$ offspring harnesses $\mathbf{H}_{t,1:K}$ and shared evidence $e_t$:

where $e_t$ links historical designs to their models and outcomes. Second, model inheritance pairs these offspring with selected or merged checkpoints:

where $\mathbf{Q}_{t}^{\mathrm{sel}}$ is the candidate pair set. Evaluation selects $\bar{p}_t\in\mathbf{Q}_{t}^{\mathrm{sel}}$, adaptation produces $p_t^{+}$, and validation chooses $p_{t+1}$ from $p_t$, $\bar{p}_t$, and $p_t^{+}$. Completed outcomes extend the archive to $A_{<t+1}$.

Diagnosing interface mismatch

We introduce the Turn-level Interface Mismatch Rate (TIMR) as a diagnostic signal for interface mismatch. TIMR captures observable interface violations; broader capability alignment calls for complementary evaluation, as discussed in §B. We do not claim that TIMR fully characterizes brain–body mismatch; it captures one aspect while serving as a cheap-to-verify and empirically useful metric. For a pair’s evaluated trajectories $\mathbf{Z}(H,\theta)$, let $n_{\mathrm{lm}}(\tau)$ count model turns and $v_{\tau,u} \in\{0,1\}$ flag calls to unavailable tools, interface-contract violations, or malformed outputs at turn $u$. We define

where each violating turn counts once, even if later recovered; lower TIMR indicates fewer interface violations per model turn.

Harness Inheritance

We instantiate Equation (3) through evidence curation and historical base selection, linking offspring design to recorded model–harness compatibility.

Evidence curation. To find designs that address the incumbent’s failures, we diagnose observations $O_t$ from $p_t$ on the validation set $\mathbf{V}$, obtaining $\mathbf{F}_t=\operatorname{diagnose}(p_t,O_t)$. The Curator uses this feedback to retrieve relevant history. At step $q$, it chooses action $u_{t,q}$ from its inspection history $r^{\mathrm{cur}}_{t,q}$:

where index exposes archive metadata. Each action and its result $\operatorname{read}(\mathbf{A}_{<t},u_{t,q})$ extend the history until the Curator stops or exhausts its retrieval budget. From the completed history $r^{\mathrm{cur}}_t$, it constructs evidence $e_t$ and K modification hypotheses $b_{t,1:K}$:

where $\mathbf{L}_{\mathrm{ctx}}$ is the context-length budget. Linking each design to its observed behavior gives editors evidence for judging when to reuse it.

Historical base selection. Different hypotheses can favor different starting designs. Each of K independent editors receives $b_{t,k}$ and browses $\mathbf{A}_{<t}$ to choose its own base. Its history $r^{\mathrm{edit}}_{t,k}$ combines $e_t$ and $b_{t,k}$ with its inspection results. With $\mathbf{I}_{<t}$ indexing archived harnesses, editor $k$ selects

where $i_{t,k}\in\mathbf{I}_{<t}$ identifies the starting harness $\widetilde{\mathbf{H}}_{t,k}$. The editor incorporates relevant historical designs to produce

where $\mathbf{U}_{t,k}\subseteq\mathbf{I}_{<t}$ records the harnesses whose designs were adopted. Editors share a fixed archive; valid, distinct offspring proceed to evaluation. Each change can thus build on a suitable prior structure, with $(i_{t,k},\mathbf{U}_{t,k})$ preserving its ancestry.

Model Inheritance

We construct $\mathcal{Q}^{\mathrm{sel}}$ in Equation (4) through checkpoint selection and merging, guided by task performance and TIMR.

Model inheritance and selection. An offspring harness changes the behaviors its model must execute, making initialization a compatibility decision. Let $\mathbf{M}_{<t}$ index archived checkpoints with parameters $\theta_j$ and let $\boldsymbol{\alpha}=(\alpha_j)_{j\in\mathbf{M}_{<t}}$ denote their weights, with $\alpha_j\geq0$ and $\sum_j\alpha_j=1$. The meta-agent directly specifies these real-valued coefficients and decides how many merge candidates to evaluate in each round using historical evidence and evaluation feedback. A one-hot $\boldsymbol{\alpha}$ selects one checkpoint; multi-checkpoint candidates use TIES-Merging (Yadav et al. (2023)):

where TIES trims small task-vector entries, resolves sign conflicts, and merges sign-consistent updates using the configuration chosen by the meta-agent. At decision $q$, historical evidence $e_t$ and evaluation feedback guide the proposal

where $\mathbf{O}_{t,k}^{(q)}$ records task performance and TIMR under $\mathbf{H}_{t,k}$, with both components fixed during evaluation. The agent reads checkpoint provenance from the archive and interprets past results in their original harness context. Preliminary evaluations refine proposals for shortlisted harnesses, producing $\mathcal{Q}_t^{\mathrm{sel}}$. The validation set $\mathbf{V}$ then determines

This favors initializations that can already operate the offspring harness before adaptation.

Pair-specific adaptation and promotion. To reinforce successful behavior under the selected harness, $\bar{p}_t = (\overline{\mathbf{H}}_t, \bar{\theta}_t)$ generates its own trajectories on the training set $\mathbf{D}_{\mathrm{tr}}$. Rejection sampling retains its verifier-accepted executions for supervised fine-tuning of $\bar{\theta}_t$ with $\overline{\mathbf{H}}_t$ fixed, yielding $p_t^+ = (\overline{\mathbf{H}}_t, \theta_t^+)$. Promotion compares all three stages:

where $\succ_{\mathrm{prom}}$ ranks eligible offspring above $p_t$ by score on the same validation set $\mathbf{V}$. Eligibility requires a valid, completed evaluation with a gain over $p_t$ greater than the margin $\delta_{\mathrm{prom}}$; $p_t$ continues when none qualifies. Thus $p_{t+1} = (\mathbf{H}_{t+1}, \theta_{t+1})$ preserves the strongest validated outcome of inheritance and adaptation. All completed outcomes, including unsuccessful modifications, enter $\mathbf{A}_{<t+1}$ with harness ancestry and model provenance (checkpoint coefficients $\boldsymbol{\alpha}$ and training history) to inform later inheritance.

Experiment

Experimental Setup

Evaluation benchmarks. To test whether co-evolution generalizes beyond a single harness design, we evaluate CoEvo-RSI on three domains that demand different harness capabilities: long-trajectory reasoning, policy-constrained tool use, and structured device control. ❶ Failure attribution identifies the responsible agent (Who) and the decisive error step (When) in multi-agent trajectories. We hold out 200 TracerTraj (Zhang et al., 2025b) instances for validation during evolution, leaving 3,400 single-error training instances drawn from the remaining TracerTraj data, CORRECT-Error (Yu et al., 2026a), AFTraj (Zhang et al., 2026a), and self-constructed trajectories. The Hand and Algo. subsets of Who&When (Zhang et al., 2025d), Who&When Pro (Liu et al., 2026a), and TraceElephant [7] are reserved for out-of-distribution (OOD) testing. ❷ Role-aware text-to-SQL requires generating SQL only when the user’s role authorizes every referenced table and column, and refusing otherwise. We use the Spider subset of the RBAC benchmark (Fei et al., 2026) with its official split and its BIRD subset as an OOD test without any training. ❸ Smart-home instruction following maps instructions to device operations and rejects infeasible ones. We use the valid (VM) and invalid (IM) multi-device subsets of HomeBench (Li et al., 2025) with official splits.

Scope: affordable domain specialization

We deliberately choose specialized benchmarks because CoEvo-RSI targets compact agents for vertical applications. Examples include lightweight failure-attribution evaluators for large-scale RL training, local smart-home controllers, and enterprise website assistants operating under strict database-access permissions. Our scope is to rapidly optimize reliable, affordable model–harness pairs that deliver domain-specific performance competitive with much larger models.

Baselines. We compare against four families. ■ Direct LLMs: DeepSeek-V4.1-Flash, DeepSeek-V4-Pro-0813, GLM-5.3, MiniMax-M3, and Qwen3.8-Max, all with high thinking effort. ■ Harness-enhanced systems: for failure attribution, CHIEF (Wang et al., 2026b) and FALAT (Rafi et al., 2026) with MiniMax-2.5, AgentScope Bi et al., [6] with GPT-5.1, and DCFA (Wang et al., 2026c) with GPT-5, with results taken from their papers; for text-to-SQL, OpenCode Anomaly, [2], mini-SWE-agent (Yang et al., 2024), and Pi (Zechner, 2025) paired with frontier LLMs. ■ Harness evolution: Meta-Harness (Lee et al., 2026c) and Self-Harness (Zhang et al., 2026d), sharing the task model, meta-agent LLM, and number of rounds with CoEvo-RSI. ■ Model training: GRPO (Shao et al., 2024) and GSPO (Zheng et al., 2025) applied under the initial harness. ■ Model-harness Co-Evolution: HarnessForge Chen et al., [7].

Implementation. We evolve Qwen3.5-9B (Qwen Team, 2026) from a hand-written initial harness, using DeepSeek-V4.1-Flash (high reasoning effort) in Codex (OpenAI, 2025) as the meta-agent. Each run has ten rounds with K=8 harness candidates per round. The selected Qwen3.5-9B checkpoint samples up to three trajectories per training task under its paired harness at temperature 1, stopping at the first verifier-accepted one. The accepted trajectories are used for one epoch of full-parameter SFT (learning rate 5e-6, batch size 8) on four H100 GPUs. Full data, hyperparameter, baseline, and prompt details are in Sections A and E.

Main Results

Failure attribution. We first examine whether co-evolving a small model with its harness can outperform much larger models and single-sided evolution. Table 1 compares the co-evolved 9B agent with all four baseline families. On Who&When Hand, CoEvo-RSI reaches 68.39% Who and 29.31% When accuracy, ahead of the strongest harness-enhanced system FALAT + MiniMax-2.5 (66.30% and 25.90%). On Who&When Algo., it obtains 62.17% Who and 44.71% When accuracy, compared with the strongest baseline scores of 68.25% and 47.62%, respectively. On the other OOD benchmarks, CoEvo-RSI attains 69.43% Who accuracy on Who&When Pro, above Qwen3.8-Max (65.03%), and 70.78% on TraceElephant, 14.16 points above DeepSeek-V4-Pro-0813 (56.62%). Single-sided evolution falls short: Self-Harness trails CoEvo-RSI by 11.49 Who points on Who&When Hand, and on TraceElephant both harness evolution baselines drop below 10% When accuracy while CoEvo-RSI retains 28.92%; the RL baselines trained under the initial harness likewise trail CoEvo-RSI in Who accuracy on every benchmark, although they attain higher When accuracy on Who&When Pro and TraceElephant. Figure 4 shows how these gains accumulate: over ten rounds, AA improves by 11.11 points and SA by 5.55 points while TIMR stays at or below 0.22%. The strongest gains appear on Who&When Hand and in agent attribution on Who&When Pro and TraceElephant, showing the task-dependent benefits of model–harness alignment.

Attribution System	Who&When Hand		Who&When Algo.		Who&When Pro		TraceElephant
	Who↑	When↑	Who↑	When↑	Who↑	When↑	Who↑	When↑
Direct LLM attribution
DeepSeek-V4.1-Flash	60.34	12.07	68.25	18.25	47.89	61.77	35.16	23.29
DeepSeek-V4-Pro-0813	60.34	10.34	65.87	22.22	61.77	70.40	56.62	31.96
GLM-5.3	46.55	3.45	45.24	11.11	40.29	52.74	49.32	26.03
MiniMax-M3	53.45	10.34	63.49	17.46	51.26	59.83	52.05	27.85
Qwen3.8-Max	50.00	12.07	64.29	11.11	65.03	72.40	47.03	32.42
Harness-enhanced attribution
CHIEF + MiniMax-2.5	51.00	17.00	54.80	37.30	–	–	–	–
FALAT + MiniMax-2.5	66.30	25.90	68.00	46.00	–	–	–	–
AgentScope + GPT-5.1	–	22.41	–	25.40	–	–	–	–
DCFA + GPT-5	–	24.14	–	47.62	–	–	–	–
Harness evolution baselines
Meta-Harness	46.55	15.52	57.94	45.24	40.46	55.26	51.60	9.13
Self-Harness	56.90	24.14	56.35	38.89	47.94	54.11	49.77	7.76
Model training baselines
GRPO	37.93	17.24	57.14	37.30	62.51	63.94	44.29	35.16
GSPO	44.83	20.69	57.94	42.86	62.00	63.43	47.03	38.36
Harness-model co-evolution baselines
HarnessForge	32.76	22.41	58.73	42.06	63.03	52.97	50.23	8.68
Qwen3.5-9B	37.93	15.52	50.00	31.75	57.20	49.66	25.57	4.11
CoEvo-RSI (Ours)	68.39	29.31	62.17	44.71	69.43	57.39	70.78	28.92

Role-aware text-to-SQL. We next examine access control and query execution under co-evolution. Table 2 compares CoEvo-RSI with direct LLMs and general-purpose coding harnesses. On Spider, CoEvo-RSI achieves the best Safe-EX (80.85%), Safe-Deny (96.64%), and violation rate (1.57%), surpassing Pi + MiniMax-M3 (80.26%, 96.33%, and 1.72%). Figure 5 traces these gains: the harness successively acquires evidence checking, reference-level authorization, and verification with repair, raising Safe-EX from 77.09% to 80.85% and cutting the violation rate from 2.00% to 1.57%, a 21.5% relative reduction. On the OOD BIRD subset, its Safe-EX is close to GLM-5.3 (60.15% vs. 60.90%), alongside the highest Safe-Deny (91.26%), the lowest violation rate (5.65%), and the second-best AC-F1 (90.24%). These results indicate promising cross-database transfer for a 9B model. The safety–utility balance may partly reflect more conservative refusal, which can improve policy compliance while limiting successful execution of authorized queries.

RBAC-Text2SQL System	Spider				BIRD (OOD)
	Safe-EX↑	AC-F1↑	Safe-Deny↑	Viol.↓	Safe-EX↑	AC-F1↑	Safe-Deny↑	Viol.↓
Direct LLM generation
DeepSeek-V4.1-Flash	77.33	94.46	93.80	1.69	62.85	85.97	86.47	8.75
DeepSeek-V4-Pro-0813	78.33	94.90	95.65	2.04	64.16	90.39	90.35	7.00
GLM-5.3	77.41	89.73	79.77	2.89	60.90	82.13	78.29	6.40
MiniMax-M3	76.08	91.01	88.93	5.18	65.29	87.47	86.18	8.93
Harness-enhanced generation
OpenCode + DeepSeek-V4.1-Flash	79.58	95.52	94.85	2.41	74.86	83.06	84.04	5.90
mini-SWE-agent + DeepSeek-V4-Pro-0813	76.08	93.45	93.86	2.87	71.10	84.84	86.70	8.70
Pi + MiniMax-M3	80.26	95.85	96.33	1.72	70.81	86.80	88.60	8.70
Qwen3.5-9B	71.90	89.71	87.96	5.72	53.52	82.00	80.54	12.64
CoEvo-RSI (Ours)	80.85	95.51	96.64	1.57	60.15	90.24	91.26	5.65

Smart-home instruction following. Finally, we examine whether co-evolution transfers to structured device control. Table 3 compares CoEvo-RSI with direct LLMs on two multi-device subsets. On valid instructions (VM), CoEvo-RSI achieves the best success rate (75.51%) and F1 (89.23%), ahead of

Home Assistant System	VM Succ.↑	VM F1↑	IM Succ.↑	IM F1↑
Direct LLM generation
DeepSeek-V4.1-Flash	68.98	86.11	84.54	91.28
DeepSeek-V4-Pro-0813	75.10	89.18	83.51	90.83
GLM-5.3	70.61	86.65	64.95	81.42
MiniMax-M3	54.69	77.79	65.98	84.11
Qwen3.8-Max	53.88	79.26	84.54	91.24
Qwen3.5-9B	10.91	40.15	63.89	78.71
CoEvo-RSI (Ours)	75.51	89.23	84.54	91.80

DeepSeek-V4-Pro-0813 (75.10% and 89.18%). On invalid instructions (IM), it ties for the best success rate (84.54%) and obtains the highest F1 (91.80%). Thus, the co-evolved agent both executes feasible operations and rejects infeasible ones at the level of much larger models.

RSI or “overfit”?

Table 1 to Table 3 show that mismatch-aware co-evolution brings a compact agent close to frontier models on several domain-specific metrics. We view this as a practical form of RSI: an agent directs data generation, model selection, and training to produce stronger models. The evolved model–harness pair is nevertheless tailored to its task family: strong performance on held-out same-domain data, kept strictly separate from evolution, does not imply transfer to unrelated domains. As discussed before Our scope is to maximize a compact agent’s performance within a chosen domain, a practically valuable goal distinct from the broader RSI ambition of autonomous improvement across all domains.

Ablation Study

We compare full CoEvo-RSI with three variants over ten rounds on Who&When Algo. (Figure 4): Harness-only evolves the harness with a frozen model; Model-only trains under the initial harness; and Latest-only co-evolves both using only the latest generation. ❶ Joint evolution. Harness-only has lower SA than full CoEvo-RSI in every round and TIMR above 1% in eight of ten rounds, versus at most 0.22% for full CoEvo-RSI. Model-only maintains near-zero TIMR with SA at most 35%, indicating that interface compatibility alone is insufficient for task success. ❷ Historical reuse. Latest-only regresses at round 4 and finishes with the lowest SA. Full CoEvo-RSI recovers from its round-4 dip, reusing round-4 weights with new harnesses at rounds 6 and 8 to reach 61.11% and 61.90% AA. Together, these results support complementary roles for bidirectional inheritance and historical reuse in task performance, interface compatibility, and recovery.

Framework Analysis

Robustness and Stability Analysis. We assess sensitivity to the search budget by varying the number of harness candidates per round $K \in\{4, 6, 8, 10\}$ with all other settings fixed (Figure 6(a–Figure 6c)). AA stays between 45.11% and 52.17%, SA between 28.80% and 37.50%, and TIMR at or below 0.12% for every $K$. These results support the robustness of CoEvo-RSI across the tested candidate budgets. We also examine inference cost–accuracy trade-offs in Figure 7.

Cost Analysis. We analyze training cost and inference cost separately. ❶ Training cost. Across the three-round co-evolution runs in Figure 6(d–Figure 6f), API expenditure ranges from $2.06 to $3.69. All runs finish within a day (17.48–20.71 hours), with evaluation taking 4.54–6.11 hours. Our default K=8 achieves the shortest run span of 17.48 hours at an API cost of $2.72. We also compare training rollout budgets (Table 8): across ten rounds, CoEvo-RSI costs fewer rollout budgets than GRPO/GSPO and achieves higher agent accuracy on all four failure-attribution benchmarks (Table 1). ❷ **Inference cost. Figure 7 plots Who accuracy on Who&When Hand and Safe-EX on RBAC Spider against the API cost of a full evaluation pass, which for Who&When covers both the Hand and Algo. subsets. On Who&When Hand, CoEvo-RSI reaches 68.39% at $0.16, compared with 60.34% at $3.55 for DeepSeek-V4.1-Flash, a gain of 8.05 points at 22× lower cost. On RBAC Spider, it exceeds the strongest

Method	Sampled	Trained on
GRPO / GSPO (each)	81,600	81,600
CoEvo-RSI (10 rounds)	74,639	39,395

baseline, Pi + MiniMax-M3, by 0.59 Safe-EX points at 38× lower cost.

Inheritance Analysis. Figure 8 pairs every selected harness with every checkpoint from the RBAC run. Only the lower triangle reflects real choices, since round $i$ can pair $H_i$ only with checkpoints up to $M_i$. ❶ A growing archive improves interface compatibility. As candidates grow from one to ten checkpoints, the selected pairs move toward low TIMR in the bottom-right corner (mean normalized TIMR from about 0.72 in rounds 1–3 to 0.13 in rounds 6–10), and in seven of ten rounds the selected checkpoint is the lowest-TIMR one available. ❷ Inheritance reaches back into the archive. Five rounds re-select existing weights, and every later derived checkpoint merges with the round-3 checkpoint. Round 10 reuses that checkpoint: with high TIMR under harnesses 4–9 (0.54–1.00), it has the lowest TIMR under the final harness (0.02) and attains the best Safe-EX of the run (80.85%). Interface compatibility is pair-specific; latest-only inheritance misses such pairs.

Conclusion

We presented CoEvo-RSI, a bi-level framework that addresses the brain–body mismatch problem in model–harness co-evolution through agent-driven, bidirectional historical inheritance. A shared evolutionary archive and execution feedback guide harness development from historical designs, model checkpoint selection or merging, and subsequent adaptation under the selected harness. Using only SFT on its own rejection-sampled trajectories, CoEvo-RSI makes a compact task model competitive with larger models across multi-agent failure attribution, role-aware text-to-SQL, and smart-home instruction following. Together, these findings highlight brain–body alignment as a central consideration for self-improving agents and bidirectional historical inheritance as a practical route to evolving execution scaffolds (body) and intrinsic capabilities (brain) together.

Experimental Details

This section details the benchmarks, data, initial harnesses, and hyperparameters behind § 4.1. In every domain, training, validation, and test sets are disjoint at the level of sample IDs, tasks, and trajectories. The same validation set $V$ guides harness and checkpoint selection and promotion during evolution; test sets are reserved for performance reporting. For failure attribution, all 200 validation instances are held out from TracerTraj and removed from the training pool (§A.2).

Common Setup

Evolution. All domains run ten rounds of co-evolution with $K=8$ harness candidates per round. The candidate-sensitivity study in § 4.5 is the exception and uses three rounds. The meta-agent is DeepSeek-V4.1-Flash with high reasoning effort, running inside Codex. In each round, the meta-agent receives validation rollouts of the incumbent model–harness pair. Each editor browses the historical archive, selects an inheritance base, and develops a copy of that harness using the execution evidence and reusable historical designs. The prompt it receives is listed in §E. Each candidate is then evaluated on the full validation set. Candidates are compared using the domain-specific objectives in Table 4.

	Attribution	RBAC	HomeBench
Rollout
Max new tokens	8,192	4,096	8,192
Context length	65,536	32,768	65,536
Max agent turns	20	10	20
SFT
Max seq. length	65,536	8,192	65,536
Sequence parallel	2	1	2
Global batch size	8	8	8
Selection
Metrics	AA, SA (3:7)	Safe-EX, AC-F1	Success, F1

Task model and rollout. The task model is Qwen3.5-9B ((Qwen Team, 2026)) with thinking enabled at high reasoning effort. It is served with vLLM and sampled with temperature 1.0, top-$p$ 0.95, and top-$k$ 20. Per-domain generation budgets, context lengths, and turn limits appear in Table 4.

Model training. For each training task, the selected model $\bar{\theta}_t$ samples up to three trajectories under its paired harness $\bar{H}_t$, stopping at the first verifier-accepted one. These accepted trajectories, together with selected high-quality accepted trajectories replayed from earlier rounds, are used for one epoch of full-parameter SFT of the same model with ms-swift ((Zhao et al., 2024)). Training uses AdamW with learning rate $5 \times10^{-6}$, a cosine schedule with 3% warmup, a per-device batch size of 1, and 4 gradient-accumulation steps. It runs on four H100 GPUs with DeepSpeed ZeRO-3, bf16, and gradient checkpointing. The global batch size is 8 in all domains, and long-context domains additionally use sequence parallelism of size 2.

Failure Attribution

Task. Given the full log of a failed multi-agent run, the system must name the agent responsible for the failure (Who) and the decisive step at which the failure becomes unrecoverable (When).

Benchmarks. Table 5 lists all splits; all four test benchmarks are held out from training and evolutionary selection. Who&When ((Zhang et al., 2025d)) contains 58 hand-crafted (Hand) and 126 algorithm-generated (Algo.) trajectories. Who&When Pro (Liu et al., 2026a) is evaluated on its 1,750 multi-agent, text-only trajectories; we exclude single-agent trajectories, on which attribution is nearly saturated. For TraceElephant [7], we drop records whose decisive-step annotation is missing or invalid, which leaves 219 evaluable instances.

Split	Composition	#Inst.
Train	TracerTraj 1,618; CORRECT-Error 1,230; AFTraj 430; self-constructed 122	3,400
Validation	TracerTraj (held out)	200
Test (OOD)	Who&When Hand	58
Test (OOD)	Who&When Algo.	126
Test (OOD)	Who&When Pro (multi-agent, text-only)	1,750
Test (OOD)	TraceElephant (evaluable)	219

Training and validation data. Training instances come from TracerTraj (Zhang et al., 2025b), CORRECT-Error (Yu et al., 2026a), AFTraj (Zhang et al., 2026a), and a small set of trajectories we construct ourselves. We keep only trajectories that contain a single error, use dense one-based step indices, and have a responsible agent consistent with the decisive step. Of the 1,818 filtered TracerTraj instances, 200 are held out for validation and the remaining 1,618 are retained for training. Together with 1,230 CORRECT-Error, 430 AFTraj, and 122 self-constructed instances, this gives 3,400 training instances. The 200-instance TracerTraj validation set guides candidate and checkpoint selection and promotion. The source studies explicitly document evaluation-data isolation in their respective protocols (Zhang et al., 2025b; Yu et al., 2026a; Zhang et al., 2026a). We further rigorously checked our training data against the held-out TracerTraj validation set, Who&When Hand, Who&When Algo., Who&When Pro, and TraceElephant. These checks confirmed separation at the sample-ID and task levels and found no identical or highly similar trajectories across training and evaluation.

Metrics. Agent accuracy (AA, reported as Who) is the fraction of instances whose predicted responsible agent matches the annotation. Step accuracy (SA, reported as When) is the fraction whose predicted decisive step matches the annotated step exactly. The harness requires the model to answer through a structured final_answer tool, so predictions are parsed and scored by exact match against the ground truth without an LLM judge. Results are reported separately for each subset.

Initial harness. The hand-written harness exposes the trajectory summary-first: the model initially sees only per-step metadata such as step IDs and agent names. It must call inspect to read the full content of a step, and it submits the agent and step through final_answer. Ground-truth labels are never exposed, and the episode is limited to 20 turns.

Role-Aware Text-to-SQL

Task. Each instance pairs a natural-language question over a database with a user role whose column-and operation-level access policy is given. A request is authorized only if every table, column, and operation it needs is covered by the role’s policy (Fei et al., 2026). For authorized requests, the system must return SQL that executes to the reference result. Otherwise, it must refuse.

Benchmarks and data. Table 6 lists all splits. Training and evolution use only the Spider subset. The 8,000 training instances and 2,000 validation instances are disjoint samples from the official training split. Test results are reported on the full Spider test set and on the full BIRD subset, which is never used for training or evolution and therefore measures pure generalization to unseen databases.

Split	Source	#Inst.	#DB
Train	Spider (train)	8,000	–
Validation	Spider (train, held out)	2,000	–
Test (ID)	Spider (test)	6,926	20
Test (OOD)	BIRD	10,175	11

Metrics. Following Fei et al. (2026), each prediction falls into one of six outcomes, determined by the ground-truth decision and the system’s response:

	Answered with SQL		Refused
Ground truth	exec. correct	exec. wrong
Allow ($\mathrm{N}_{+}$)	$\mathrm{S}_{c}$	$\mathrm{S}_{w}$	$\mathrm{S}_{or}$
Deny ($\mathrm{N}_{-}$)	$\mathrm{S}_{vc}$	$\mathrm{S}_{vw}$	$\mathrm{S}_{pr}$

The sets $\mathrm{S}_{c}$, $\mathrm{S}_{w}$, $\mathrm{S}_{or}$, $\mathrm{S}_{vc}$, $\mathrm{S}_{vw}$, and $\mathrm{S}_{pr}$ contain correct, wrong, over-refused, violation-correct, violation-wrong, and properly refused predictions, respectively. An allowed request answered with policy-violating SQL also belongs to $\mathrm{S}_{vc}$ or $\mathrm{S}_{vw}$. SQL correctness is judged by execution equivalence on the sample database. Let $\mathrm{N}_{+}$ and $\mathrm{N}_{-}$ count allowed and denied requests, with total $\mathrm{N} = \mathrm{N}_{+} + \mathrm{N}_{-}$. Define $n_{\mathrm{ans}} = |\mathrm{S}_{c}| + |\mathrm{S}_{w}|$, $n_{\mathrm{viol}} = |\mathrm{S}_{vc}| + |\mathrm{S}_{vw}|$, and $n_{\mathrm{ref}} = |\mathrm{S}_{or}|$. The metrics are

For selection, we additionally track the LLM rate, which is the fraction of instances that reach the LLM, together with the coverage and accuracy of rule-based decisions.

Initial harness. The hand-written harness first applies deterministic rule routing, which links the question to schema columns and checks them against the role policy. When the rules are not confident, it falls back to a multi-turn LLM agent with two tools: test_sql executes a query on the database and returns rows or an error, and final_answer submits the access decision together with SQL when access is granted. The agent is limited to 10 turns.

Smart-Home Instruction Following

Task. Each HomeBench (Li et al. (2025)) query bundles 2 to 10 sub-instructions for devices in a simulated home. The system must emit one device command per feasible sub-instruction and an error for each infeasible one, such as a missing device or an unsupported operation. We use the VM and IM subsets, which contain only valid and only invalid multi-device queries, respectively.

Data. Table 7 (PDF p. 24) lists the splits. The 10,000 training instances are sampled from the official training split, and 2,000 further training-split instances are held out as the validation set. The official multi-device test set is used only for the final report.

Split	#Inst.
Train	10,000
Validation	2,000
Test	342

Metrics. Success is the exact match between the predicted and reference command multisets, including error outputs. F1 is the micro-averaged command-level F1 under the same official counting. Both are computed with the official scorer.

Initial harness. The hand-written harness gives the model a compact index of rooms and devices together with four official in-context examples, and the meta-agent may also edit these examples. Detailed device states and method signatures are available only through the read-only tools inspect_home and list_methods, and the model submits its commands through final_answer. The episode is limited to 20 turns.

Baseline Configurations

Harness evolution. We run Meta-Harness (Lee et al., 2026c) and Self-Harness (Zhang et al., 2026d) from their official implementations and add a failure attribution domain adapter for each. Both start from the same task model (Qwen3.5-9B), the same hand-written initial harness, and the same 200 held-out TracerTraj validation instances as CoEvo-RSI. Each runs for ten rounds. The task-model weights remain frozen throughout harness evolution. Both baselines use the same meta-agent LLM and settings as CoEvo-RSI, namely DeepSeek-V4.1-Flash with high reasoning effort, and each keeps its officially recommended proposer workflow. Meta-Harness drives the LLM through Codex and proposes eight candidates per round. Self-Harness uses its native multi-proposer workflow, which calls the LLM directly without a coding agent, and explores eight candidates per round. Each candidate in both baselines is evaluated twice on the validation set.

Model training. GRPO (Shao et al., 2024) and GSPO (Zheng et al., 2025) train Qwen3.5-9B under the initial harness on the same 3,400 training instances with verl (Sheng et al., 2024). The two runs share all settings except the policy loss (Table 9). Both use group-normalized advantages and no KL penalty.

	GRPO	GSPO
Shared
Learning rate	$1 \times10^{-6}$
Rollouts per prompt ($n$)	8
Prompts per step / mini-batch	1 / 1
Epochs (optimization steps)	3 (10,200)
Max prompt / response length	8,192 / 4,096
Rollout temperature	1.0
KL penalty / entropy coef.	none / 0
GPUs (rollout TP / actor SP)	4 (2 / 4)
Policy loss
Loss type	token-level	sequence-level
Loss aggregation	token-mean	seq-mean-token-mean
Clip ratio (low / high)	0.2 / 0.2	$3 \times10^{-4}$ / $4 \times10^{-4}$
Dual-clip constant	–	10.0

Training-sample budget. Table 8 compares the trajectories each method samples and trains on. A single RL run samples 3,400 prompts $\times$ 8 rollouts $\times$ 3 epochs and updates on every rollout. Across all ten rounds, CoEvo-RSI samples 8.5% fewer trajectories than one RL run and fine-tunes on 39,395 verifier-accepted trajectories, including replayed ones, which is 48.3% of the RL training samples, while also evolving the harness.

Scope and Interpretation of TIMR

Diagnostic scope. TIMR (Equation (5)) measures the frequency of observable interface violations within a model–harness pair: calls to unavailable tools, interface-contract violations, and malformed outputs. These events expose a concrete aspect of brain–body mismatch at the execution boundary and can be checked directly from traces. Their low verification cost makes TIMR useful for diagnosing candidate pairs during evolution. The broader problem definition also covers learned capabilities that a harness underuses or constrains. Such cases can produce valid tool calls throughout an unsuccessful trajectory; assessing them requires task outcomes and examination of the information and actions the harness makes available. TIMR therefore provides a focused operational view of interface compatibility within this broader evaluation. Raw TIMR lies in $[0, 1]$ (0–100%); Figure 8 uses global min–max scaling across all harness–checkpoint pairs to $[0, 1]$.

Relationship to task performance. Task success depends jointly on interface use, reasoning, access to relevant evidence, and the execution workflow. The ablations in Figure 4 illustrate these distinct factors: Model-only maintains near-zero TIMR with step accuracy at most 35%, whereas Harness-only has TIMR above 1% in eight of ten rounds and lower step accuracy than full CoEvo-RSI in every round. Full CoEvo-RSI keeps TIMR at or below 0.22% while improving task performance. These observations support using TIMR together with task scores to distinguish execution friction from other performance bottlenecks. They provide diagnostic evidence; attributing a performance gain specifically to lower TIMR would require controlled interventions that isolate interface behavior from changes in reasoning, information access, and execution budget.

Role in co-evolution. CoEvo-RSI reads TIMR alongside task performance when proposing checkpoint selections or merges, and uses validation task scores to select and promote model–harness pairs (Equations (12) and (13)). A low TIMR identifies a pair whose observed interactions largely satisfy the harness’s interface requirements; task evaluation establishes how effectively that pair solves the intended problems. This division gives TIMR a practical role as an inexpensive diagnostic signal, with the value of an evolutionary update assessed through downstream performance.

Interpreting Comparisons with Frontier Models

Our evaluation asks how far co-evolution can bring a compact task model toward frontier-level performance in a given domain. The compared models start from markedly different capability levels: on the Artificial Analysis Intelligence Index v4.3.2, Qwen3.5-9B (Reasoning) scores 11, compared with 36 for DeepSeek-V4-Pro-0813 (Max) and 45 for GLM-5.3 (Max) ([3 b,c]). These independent measurements contextualize the capability gap behind our task comparisons. Against this background, approaching or matching frontier systems on several demanding tasks is a substantive outcome for a 9B model. For example, CoEvo-RSI achieves 95.51% Spider AC-F1 against 95.85% for the strongest system on this metric, Pi + MiniMax-M3 (Table 2), and 75.51% HomeBench VM success against 75.10% for DeepSeek-V4-Pro (Max) (Table 3). Such proximity remains meaningful when individual metric rankings vary across benchmarks.

All SFT supervision comes from Qwen3.5-9B’s own verifier-accepted trajectories, generated by the selected checkpoint or, through replay, by earlier checkpoints and retained through rejection sampling (§A.1). The meta-agent guides harness evolution and checkpoint inheritance; model training consolidates the 9B task model’s successful execution experience. We view the resulting competitiveness as evidence of how much useful capability can be elicited and consolidated from a compact model through coordinated model–harness evolution.

Positioning: affordable domain specialization. As frontier models advance, a complementary deployment need is to make compact models effective within a well-defined domain. We position CoEvo-RSI as a tool for this need, particularly for frequent agent calls where cost and response-time budgets shape system design. Potential applications include device manufacturers integrating smart-home control and RL systems requiring failure-attribution feedback on repeated agent rollouts. These needs motivate our three benchmark domains: smart-home instruction following for device execution, multi-agent failure attribution for automated diagnosis, and role-aware text-to-SQL for policy-constrained data access. Each tests task-specific behavior under concrete operational requirements. The results support this practical motivation: CoEvo-RSI achieves 68.39% agent accuracy on Who&When Hand versus 60.34% for DeepSeek-V4.1-Flash. Under the API pricing in Figure 7, their full evaluation passes over Hand and Algo. cost $0.16 and $3.55, respectively, approximately 22× lower for CoEvo-RSI. Together with the close Spider and HomeBench results above, this demonstrates the value of specialization for delivering competitive task performance at low recurring cost. Small differences on individual metrics can be acceptable when the resulting agent meets an application’s quality requirements and operating budget. Our aim is to give practitioners an affordable model–harness pair optimized for the tasks they repeatedly need to solve.

Comparison with Related Evolution Frameworks

Table 10 compares six model–harness co-evolution methods and two harness-only methods through the sources of their inherited structure and capabilities. Harness history supports several forms of reuse, including archive-guided refinement, search over previous programs, and persistent component updates. On the model side, the compared co-evolution procedures continue the current checkpoint or inherit the policy associated with a parent branch. CoEvo-RSI makes both sources explicit decisions over the evolutionary archive: harness editors choose historical starting designs, and model inheritance independently selects or merges historical checkpoints for the resulting harnesses. This flexibility allows structure and capabilities from different generations to form new candidate pairs, whose execution feedback guides selection and subsequent adaptation. Bidirectional historical inheritance expands the candidate space for further co-evolution.

Method	Harness update / inheritance	Model initialization	Model update	Historical weights
				Select	Merge
Model–harness co-evolution
HarnessForge ([7])	Parent harness with archived design exemplars	Parent-branch policy	LoRA SFT	No	No
HarnessX ([7])	Current harness edits; variant branching and routing	Current checkpoint	Cross-harness GRPO	No	No
Co-Harness ([7])	Validated local patches to the current harness	Current checkpoint	SFT	No	No
HASE ((Luo et al., 2026a))	Persistent harness with phase-reviewed edits	Current checkpoint	GRPO	No	No
WHALE ((Kim et al., 2026))	Search and reuse of archived harness programs	Current checkpoint	Rejection-sampling SFT	No	No
SafeEvolve ((Mao et al., 2026))	Versioned prompt / skill updates with skill retrieval	Current policy; SFT cold start	SFT + RL	No	No
CoEvo-RSI (Ours)	Agent-selected historical bases and design reuse	Agent-selected or merged archived checkpoints	Rejection-sampling SFT	Yes	Yes
Harness-only evolution
Meta-Harness ((Lee et al., 2026c))	Agent-directed reuse of prior code and execution traces	Fixed model	–	–	–
Self-Harness ((Zhang et al., 2026d))	Current harness with accepted edits combined	Fixed model	–	–	–

Meta-Agent Prompts

The prompt given to the meta-agent in each round has two parts. The first is a shared skeleton that is identical across domains (Prompt A) and fixes the evidence discipline, workflow, and permission boundary. The second is a domain direction (Prompts B–D) that is inserted at the {domain direction} slot and states the domain’s metrics, interface contract, and leakage constraints. Placeholders in braces are filled with run-specific values. The prompts specify the shared evolution workflow and domain-specific instructions, with file paths represented by placeholders.

reas ♀♀♀♀♀♀દાવાદ

bounded validation, and leave a concrete evolution report. Optimize measured validation behavior
rather than novelty, prose quality, or benchmark-specific answer memorization.

Incumbent harness: {incumbent harness}
Historical archive: {harnesses, provenance, and outcomes}
Evidence: {validation rollouts of the incumbent}
Objective: {round instruction}

Domain direction:
{domain direction, see Prompts B–D}

Evidence discipline:

- Aggregate metadata locally and inspect only a bounded representative sample.
- Never dump the complete evidence or complete rollout messages into context.
- Use per-sample evaluation metrics to compare successful controls with failures.
- Classify failures as harness-addressable, model-capability-limited, or external/data.

Required workflow:

1. Audit the evidence schema, metric distribution, errors, and operational failures.
2. Produce a 3–6 item failure taxonomy with counts and representative sample IDs.
3. Before editing, print HARNESS DIAGNOSIS with baseline metrics, addressable failure mass,
   proposed files/mechanisms, expected metric effects, and risks.
4. Browse archived harnesses, inspect their code and execution histories, and select the
   historical harness that will serve as the candidate’s inheritance base.
5. Develop a copy of the selected base using the execution evidence and applicable historical
   designs. Record the editing parent and each adopted historical contribution.
6. Preserve the domain contract stated above and inferred from the parent code.
7. Run bounded syntax/import/interface checks and report unresolved risks.

Permission and completion contract:

- Only the candidate may change; do not add binaries or dependencies.
- Do not import sibling generated harnesses or encode benchmark answers or sample IDs.

• Do not activate the child; evaluation and selection happen outside this workspace.

• The final report must name changed files, the evidence-backed mechanism, expected metric effect, validation performed, assumptions, and risks. Write it to the candidate’s evolution report before replying.

• Do not stop after diagnosis or research: implement and validate the candidate.

shared across all domains

Prompt B: Failure Attribution

Domain: multi-agent failure attribution.

Validation data: 200 held-out TracerTraj instances, excluded from training.

Validation admission metrics: agent_accuracy, step_accuracy.

Ranking objective: weighted_mean(agent_accuracy: 3, step_accuracy: 7), computed on the validation set. Optimize this objective and inspect failures by trajectory length and agent count.

Diagnosis. Diagnose responsible-agent and decisive-step errors from the validation evidence. Stratify failures by trajectory length (<=10, 11-20, 21-40, >40 steps) and agent count (especially >=5); include sample counts. Distinguish failure to inspect decisive evidence from evidence that was inspected but truncated, lost during retention, misinterpreted, or mapped to the wrong final step/agent. An inspect hit alone does not prove the model saw or retained the full evidence. Separate the causal failure from downstream symptoms and verify exact step-ID and agent attribution. Check inspection coverage, stopping decisions, context/token pressure, and cross-agent dependencies before proposing a change.

Training evidence. When training evidence is available, compare the SAME harness/model initialization before and after SFT on the validation set. Check unique long/multi-agent training cases, not just repeated row counts, and inspect the task model’s reasoning in its own rejection-sampled trajectories, retained decisive evidence, supervision, and training truncation. Do not assume data curation or more repetition fixes a regression. Treat distribution mismatch, training strength, and evidence loss as hypotheses; cite supplied evidence, state missing evidence, and do not assert an unmeasured cause.

Change policy. Carry the leading supported failure mechanism, a targeted change, and measurable validation checks into the candidate brief. Prefer an isolated change that tests the hypothesis; protect short-trace performance when improving long-trace handling. Never hardcode validation answers or sample-specific rules. Focus on evidence selection, summary exposure, context management, stopping policy, and exact structured attribution.

Contract. Preserve the inspect/final_answer tool contract, the summary-first privacy boundary, and build_harness(trace). Never expose labels or full private step text before inspect is called.

Attribution only

Prompt C: Role-Aware Text-to-SQL

Domain: column-level Text2SQL RBAC with rule routing and agentic LLM fallback.

Headline metrics (in selection priority order):

• safe_ex: correct SQL on allowed samples / all allowed samples (primary gate)

• ac_f1: access-control F1 (primary gate)

• safe_deny: correct refusal rate on denied samples

• violation_rate: security breach rate (lower is better)

• over_refusal_rate: unnecessary refusal rate (lower is better)

• llm_rate: fraction of samples that invoke the LLM (lower is better)

• rule_rate / rule_accuracy: coverage and correctness of rule decisions

Selection: a child must not regress safe_ex or ac_f1 (tolerance 0.01), then must improve at least one of safe_ex, ac_f1, or llm_rate.

Agent loop. The LLM receives two tools: test_sql(sql) executes SQL against the sample database and returns rows or an error; final_answer(access_granted, sql) submits the authorization decision with optional SQL. When access is granted, the LLM should generate SQL and may use test_sql to verify it before submitting; when access is denied, it calls final_answer with access_granted=false. The LLM may iterate over multiple turns to refine its SQL.

Evolution space. Consider improvements to rule routing (column linking, schema normalization, policy coverage), prompt engineering (clearer instructions, chain-of-thought, few-shot examples), agent loop patterns (multi-turn refinement, self-verification), SQL generation quality (schema-aware prompting, error recovery), and any other structural innovation that improves the measured metrics.

Prevent false filtering. Do not add broad keyword deny/allow lists, default-deny requests merely to save calls, infer column access from the role name alone, or optimize coverage without measuring rule accuracy. Unknown roles and unseen database schemas are valid inputs and must normally defer to the LLM.

Contract. Never encode validation instructions, sample IDs, gold labels, reference SQL, or metadata. Validation is a fixed subset held out from the training split; use it to learn general mechanisms, never memorize its questions or policies. Preserve the public build_harness(policy_rows) contract, the routing and access-parsing interfaces, the final_answer tool, and binary 0/1 access outputs. Only training policy rows may be compiled into rule or index state.

RBAC only

Prompt D: Smart-Home Instruction Following

Domain. HomeBench multi-operation household instruction generation (2-10 sub-instructions per query; VM/IM). Improve exact-match success while retaining micro command F1.

Interface. Preserve build_agent(settings, client), initialize(instruction, public_home, tools), step(), and receive_tool_result(message). Each step makes exactly one model request and returns its actual message with one tool call. final_answer.commands must contain brace-enclosed instructions. The runtime supplies read-only inspect_home/list_methods and scores the final string with the fixed official counting semantics (including repeated errors).

Evolution space. Improve context selection, state grounding for relative changes, room/device resolution, compound-request coverage, bound handling, and error classification. Keep tool observations authentic and preserve the user request.

Initial harness. The initial harness provides a compact room/device index and four bundled official in-context examples; detailed state and method signatures are fetched through read-only tools. Examples illustrate format under other household states, not current-task facts. You may edit the bundled examples, but never retrieve examples or answers from other dataset samples. Ground current commands in the supplied current home.

Isolation. Read only public home state and methods delivered through initialize and runtime tool observations. Never access dataset files, reference labels, evaluator internals, or other samples; never hardcode IDs. Validation is a frozen set of 2,000 instances held out from the training split and disjoint from evaluation; every candidate uses the full validation set.

HomeBench only