WorksComputer Science
JoMoHa: Joint Model–Harness Optimization with Reinforcement Learning
Abstract
The performance of language agents depends jointly on model weights and the harness. Optimizing either alone can be constrained by the other component, limiting overall performance gains. Our analysis of naive alternating training reveals progressive incompatibility between the model and harness, which increasingly interferes with task execution and may also undermine model training, leading to performance degradation. To coordinate the evolution of models and harnesses, we introduce JoMoHa, to our knowledge the first framework to jointly train an executable harness generation policy and a task execution policy through reinforcement learning. JoMoHa outperforms the evaluated weight-only and harness-only baselines and achieves the highest success rates among the evaluated methods on ALFWorld, WebShop, and DBBench, averaging 70.3% and reaching 97.0% on ALFWorld. By learning both task execution and runtime improvement from its own generated trajectories, JoMoHa performs recursive self-improvement (RSI).
Introduction
A language agent consists of a model and a harness. The model weights determine its learned reasoning and decision-making capabilities, while the harness organizes how these capabilities are used. Improving an agent therefore admits two complementary paths: training its model weights and improving its harness.
Existing methods have made progress along both paths. Reinforcement learning methods train model weights from task feedback, improving the policy that makes decisions [15] (Yu et al. 2022). Alternatively, harness optimization changes the runtime surrounding the model [10, 14]. However, agent performance depends jointly on both components, and optimizing
either alone can leave overall gains constrained by the other. The comparison with weight-only and harness-only training in Figure 1 further motivates learning model weights and harnesses together.

Figure 1. Learning the runtime writer alongside the Solver on ALFWorld. The joint system reaches 97.0%, versus 90.5% for Frozen Engineer, 89.5% for Weight-only, and 80.5% for Harness-only. Weight-only does not use harness. Frozen Engineer keeps the initial code-writing model fixed but generates harnesses while training the Solver.
However, naively alternating updates to model weights and the harness does not ensure their compatibility. As the model changes, an existing harness may provide outdated or incorrect guidance that interferes with task execution and limits performance gains. Such guidance also shapes the experience used for training and may steer subsequent model updates toward less effective behavior, potentially worsening performance over time. We diagnose these issues through controlled comparisons and failure trajectory analysis in § 5.
WHALE mitigates this issue through small alternating updates to model weights and harnesses. Its published instantiation updates model weights through online rejection-sampling fine-tuning (RSFT) rather than RL, and uses Meta-Harness search driven by a fixed external proprietary frontier model [8]. However, harness optimization continues to rely on a fixed external proprietary frontier model to propose improvements. Execution feedback guides harness selection, but does not update the proposer itself. The capability to improve the harness therefore remains delegated to an external model rather than trained within the system. This motivates a different approach: making harness generation a trainable policy and using the same task execution outcome to train both the harness generator and the task performer.
Figure 2. From one-sided learning to joint model–harness RL. Weight-only trains the Solver under the fixed base executor, without a learned harness. Harness-only trains the Engineer against a frozen Solver. JoMoHa trains both policies from the same executed task outcome. Solid policy boxes denote updated parameters; dashed boxes denote fixed components. Dashed arrows carry training feedback, not gradients through the generated program. SAO denotes the single-rollout actor–critic backbone (Hou et al., [4]); Section 3 introduces the learning terms and notation.
We instantiate recursive self-improvement (RSI) by learning both task execution and harness construction from the system’s own generated programs, trajectories, and execution feedback. Harness-R1 supports the potential of this direction: its trained harness Engineer outperforms the evaluated fixed frontier editors, including proprietary models (Shao et al., [14]). However, it keeps the task-performing model frozen during Engineer training. This leaves an open question:
How can both policies improve through reinforcement learning from their shared task executions?
A natural starting point is to alternate updates to the two policies using task outcomes. Yet alternation alone does not ensure that their improvements remain compatible. We investigate this problem through controlled diagnosis of a long-phase alternating run in which performance initially improves and subsequently deteriorates (§ 5.1). By evaluating saved model checkpoints with and without the same fixed harness, we distinguish two sources of decline: the task-performing model loses ability, and the unchanged harness becomes more harmful as the model changes. In one task, a later model succeeds in 14 turns without a harness but reaches the 50-turn limit under the old harness (Table 3b). These findings motivate coordinating the two learners throughout training, rather than relying on alternation alone.
We introduce JoMoHa, to our knowledge the first framework to jointly train a harness-generation policy and a task-execution policy through reinforcement learning. We call the two policies the Engineer and the Solver. The Engineer generates an executable harness, and the Solver interacts with the environment under that harness. The generated program and its multi-turn execution form a paired training example (Figure 3). The shared task outcome trains both policies, each on its own generated tokens. Therefore, JoMoHa implements RSI: the improvement policy itself learns from execution feedback, alongside the task-performing policy. We release our code at https://github.com/xudongwu-0/coevolve.

Figure 3. Joint reinforcement learning of the Engineer and Solver. The Engineer writes a runtime program under which the Solver interacts with the task. The resulting shared reward trains both policies, each on its own generated tokens.
We evaluate JoMoHa on ALFWorld, WebShop, and D BBench, where it achieves the highest success rates among the evaluated methods: 97.0%, 45.8%, and 68.0%, respectively (§ 10.1, Table 4).
| Method | Pick | Look | Clean | Heat | Cool | Pick2 | All | Score | Succ. | Succ. | Avg. |
| Qwen3.5-9B (with default harness) | 75.8 | 66.7 | 16.9 | 28.8 | 9.2 | 47.5 | 40.6 | 66.0 | 31.2 | 61.0 | 44.3 |
| Prompt-based Agentic Methods | |||||||||||
| ReAct | 79.8 | 54.8 | 27.3 | 27.4 | 10.3 | 53.3 | 43.4 | 66.8 | 37.4 | 61.7 | 47.5 |
| Self-Refine | 70.7 | 45.2 | 11.7 | 28.8 | 6.9 | 57.4 | 39.0 | 61.7 | 29.0 | 57.3 | 41.8 |
| Reflection† | 91.9 | 90.5 | 24.7 | 56.2 | 19.5 | 73.8 | 59.2 | 61.2 | 43.6 | 64.7 | 55.8 |
| Fixed Frontier Harness Editors | |||||||||||
| Qwen3.5-397B | 76.8 | 66.7 | 14.3 | 31.5 | 11.5 | 47.5 | 41.2 | 66.4 | 32.8 | 63.3 | 45.8 |
| GLM-5.2 | 77.8 | 78.6 | 18.2 | 27.4 | 16.1 | 54.9 | 45.0 | 68.4 | 36.0 | 65.3 | 48.8 |
| Kimi-K2.6 | 73.7 | 71.4 | 14.3 | 30.1 | 12.6 | 49.2 | 41.4 | 63.7 | 31.8 | 62.7 | 45.3 |
| DeepSeek-V4-Pro | 72.7 | 69.0 | 13.0 | 30.1 | 16.1 | 47.5 | 41.0 | 65.1 | 32.4 | 64.3 | 45.9 |
| Gemini-3.5-Flash | 67.7 | 69.0 | 14.3 | 24.7 | 10.3 | 35.2 | 35.4 | 63.2 | 33.6 | 64.0 | 44.3 |
| GPT-5.5 | 80.8 | 78.6 | 31.2 | 28.8 | 17.2 | 35.2 | 43.2 | 61.4 | 36.6 | 64.0 | 47.9 |
| Harness-Adapted Agents | |||||||||||
| Agent SFT + Harness-R1 | 93.9 | 100.0 | 81.8 | 74.0 | 72.4 | 86.1 | 84.0 | 68.7 | 43.0 | 65.7 | 64.2 |
| WHALE | 97.7 | 92.9 | 55.2 | 79.3 | 66.7 | 87.8 | 80.5 | 71.63 | 43.4 | 65.3 | 63.1 |
| JoMoHa (ours) | 100.0 | 100.0 | 100.0 | 96.6 | 97.2 | 91.8 | 97.0 | 73.9 | 45.8 | 68.0 | 70.3 |
Table 4. Table 4. Benchmark comparison across agent and harness-optimization methods. Frontier models serve as fixed harness editors. Reflection uses success@2 (†). Avg. is the mean of ALFWorld All, WebShop success, and DBBench success; it excludes WebShop score. Best and second-best distinct values are bold on red and underlined on blue, including ties.
To coordinate the Engineer and Solver, JoMoHa combines collaboration-aware value estimation, role-specific return normalization, partner-trajectory weighting, and asymmetric update schedules. We examine these design choices through component studies in § 10.2.
Our contributions are threefold:
Diagnosing failure in alternating training. Fixed-component tests distinguish declining Solver capability from worsening harness compatibility. Trajectory analysis reveals how incorrect harness state disrupts otherwise successful execution. Together, these findings highlight risks to both task execution and continued learning, motivating compatibility-aware joint training.
Joint model–harness reinforcement learning. We introduce, to our knowledge, the first framework to jointly train an executable-harness generation policy and a task-execution policy through RL. The Engineer
and Solver learn from the same task outcome, making both harness construction and task execution trainable within a shared learning process.
Coordination adaptations and empirical evaluation
We adapt cooperative-learning mechanisms to the Engineer–Solver setting and evaluate JoMoHa across three environments. Frozen-Engineer controls, fixed-Solver harness comparisons, and component studies assess the benefits of training the Engineer, improving the harness, and coordinating the two policies.
Related Work
Model-weight optimization. Reinforcement learning improves language-model reasoning and decision-making from task feedback [15]. Single-Rollout Asynchronous Optimization (SAO) uses one rollout per prompt, a learned value model, and token-level calibration to stabilize asynchronous agentic RL [4]. For tool-using agents, Search-R1 learns when and how to retrieve information [6], while ReTool learns to interleave reasoning with code execution [1]. RAGEN studies the training dynamics of multi-turn agents [20]. These methods improve the task-performing policy under a prescribed runtime, rather than jointly learning a separate policy that constructs that runtime. JoMoHa trains the Solver alongside an Engineer whose generated harness shapes the Solver’s execution.
Harness optimization. A complementary line improves the system surrounding the model. ReAct organizes reasoning and action, while Self-Refine and Reflexion use textual feedback to revise behavior [23, 12, 16]. OPRO and DSPy support instruction and pipeline optimization [21, 7]. ADAS and AFlow search over agent architectures and executable workflows [25] (Hu et al. 2024), and Meta-Harness searches executable harnesses using prior code, scores, and execution traces [10]. These approaches motivate treating the runtime as an optimization object, but improving a program does not necessarily train its generator. Harness-R1 makes runtime editing a learned capability by training an Engineer through RL against a frozen target agent [14]. JoMoHa further trains the Solver alongside the Engineer, allowing both policies to adapt to their changing partner.
Joint learning and coordination. WHALE jointly improves model weights and harnesses through alternating updates [8]. Its published instantiation combines online RSFT with Meta-Harness search driven by a fixed external proprietary frontier model. Compared with WHALE, JoMoHa learns the harness-generating policy itself and trains both policies through RL from the same executed program–rollout pair (Table 1).
| Method | Harness adaptation | Engineer | Solver |
| Search-R1 / ReTool | Prescribed | – | RL |
| Meta-Harness | Code search | External | Fixed |
| Harness-R1 | Learned edits | RL | Fixed |
| WHALE | Code search | External | RSFT |
| JoMoHa | Learned generation | RL | RL |
Table 1. Comparison of model–harness learning procedures. Engineer and Solver columns describe parameter updates. A fixed Engineer can still search for new harnesses. For Harness-R1, the Solver is frozen during Engineer training. A dash denotes no separate harness Engineer.
Cooperative RL provides foundations for learning from team outcomes [2, 9, 3, 19] (Yu et al. 2022). These classical methods were primarily developed and evaluated in non-language environments. Adapting these ideas to LLM training, particularly our program-mediated setting, is nontrivial: the Engineer and Solver have separate token histories and different decision horizons, while the Engineer’s program shapes the Solver’s entire multi-turn rollout. Their execution conditions also change as both policies learn. Our adaptations address critic information, role-specific return scales, partner-trajectory weighting, and update frequency. Component studies examine their contributions in this setting, and the complete coordination design improves performance over the Joint-only baseline (§ 10.2).
Background
Reinforcement learning (RL) improves a policy, a distribution over actions, through interaction and reward [18]. A Markov decision process (MDP) describes this interaction using states, actions, transition dynamics, and rewards: the next state depends on the current state and action. An episode is one task attempt. Language agents observe only part of the environment, so their policies use the available history. An actor is the trainable policy; a critic predicts its future return and provides a baseline for advantages—whether an outcome is better or worse than expected.
For a task , the Engineer uses evidence to sample a proposal trajectory . Validation and fallback map it to the executed harness . The Solver acts under , producing trajectory and terminal task reward . For role , and are actor and critic parameters, with actor policy . Indices , , and denote generated-token positions, paired episodes, and saved checkpoints, respectively; and identify saved weights and code. The no-harness condition retains the base executor.
SAO learns from one rollout per prompt using a critic [4]. Both roles share the task reward, while their returns also include role-specific KL penalties that restrain divergence from fixed reference policies.
Why Learn Weights and Harnesses Together?
We examine an alternating-training baseline that updates the Solver and Engineer in separate phases. Such training may lead to over-adaptation between the Solver and its harness: each component adapts to a counterpart that subsequently changes, potentially making their learned behavior less compatible. On ALFWorld, this baseline initially improves from 83.0% to 90.5%, but then deteriorates across successive stages to 31.5% (Table 2). In comparison, JoMoHa reaches 97.0%. We analyze the alternating run to investigate how harness mismatch can affect both task execution and Solver learning.
| Training scheme | Checkpoint | Success (%) |
| Alternating training | 83.0 | |
| Alternating training | 90.5 | |
| Alternating training | 78.5 | |
| Alternating training | 48.5 | |
| Alternating training | 31.5 | |
| JoMoHa | 144 rounds | 97.0 |
Table 2. Performance degradation during alternating training on ALFWorld. Each alternating stage uses its corresponding Solver–harness pair . The highest alternating-stage success and the result of JoMoHa are bolded. The fixed-code diagnosis is reported in Table 3.
Diagnosing Model–Harness Compatibility
The stage-level results alone cannot distinguish declining Solver ability from worsening harness compatibility, because both components change across stages. We therefore hold one harness fixed and evaluate it with different Solver checkpoints.
The percentage success evaluates fixed Solver weights under a saved harness. The no-harness condition removes the learned harness while retaining the base executor. The harness contribution to a fixed Solver is
This comparison separates Solver performance without the learned harness from the harness’s immediate effect on execution.
Specifically, we select earlier Solver weights , later weights , and the saved harness from the alternating run. For each Solver, we evaluate the same fixed harness and the no-harness condition on the same ALFWorld evaluation tasks. The following aggregate comparisons and trajectory analyses examine both execution interference and Solver degradation.
Separating execution interference from Solver degradation
Table 3 reveals two distinct changes. Across each row, adding the same harness reduces success by 4.0 points for but 17.0 points for : its negative effect increases by 13.0 points. Down the no-harness column, success falls from 82.5% to 48.0%: the Solver loses 34.5 points even when the learned harness is removed. Together, these account for the decline under the fixed harness:
The first comparison shows that an unchanged harness can increasingly interfere with execution as the Solver changes. The additional loss cannot be explained by the Engineer generating a worse new harness. At , adding changes success to failure on 18.0% of evaluation tasks and failure to success on 1.0%.
The second comparison shows that removing the harness does not restore the later Solver’s earlier performance. The decline therefore extends beyond interference during the current execution. It is consistent with the concern that training under mismatched harnesses may steer the Solver toward less effective behavior. These findings motivate maintaining Solver–harness compatibility throughout training, rather than treating the harness only as an inference-time aid.
Case 1: Harness-induced failure on ALFWorld. We examine evaluation trajectories produced by the earlier and later Solver checkpoints from the alternating training run analyzed above. The task requires cleaning a sponge and placing it on a cart. The later Solver succeeds in 14 turns without the learned harness, but reaches the 50-turn limit under the old harness (Table 3b). The earlier Solver succeeds in both conditions, taking 16 and 14 turns, respectively. The later Solver therefore retains the behavior needed to complete this task; the failure appears when it is combined with the old harness.
The trajectory analysis reveals the following failure sequence:
The environment advances. Cleaning has completed, so the remaining work is to finish navigation and placement.
The harness fails to update its state. The executed action is stored in a nested field, but the old lookup uses a literal dotted key. The correct access is on the first line below; the old lookup is on the second:
ctx["action"]["final_action"]
ctx.get("action.final_action")
A dotted string is a different dictionary key, not a nested lookup, so the completed action is missed. A second state update can also overwrite earlier inventory evidence when the current observation omits the word indicating cleaning.
Stale guidance sends execution backward. Although the environment has confirmed cleaning, the harness continues directing the Solver to find and clean the object. The episode loops through navigation and reaches the turn limit.
Case 2: Solver failure without the learned harness. A contrasting ALFWorld case from the same training run exposes a failure in the no-harness condition. After cleaning a potato and navigating to the destination, the later Solver omits the final placement action and eventually reaches the turn limit. Because no learned harness is active, this failure cannot be attributed to interference from . It illustrates why removing an incompatible harness alone cannot address all failures of the trained Solver.
From Compatibility to Coordinated Learning
Taken together, these findings suggest that a mismatched harness can impair current task execution and may also distort the experience from which the Solver learns. This creates a risk of a negative feedback loop in which ineffective harness guidance and declining Solver capability reinforce one another. Maintaining a compatible Solver–harness pair throughout learning is therefore important.
Our unit of learning is a complete task attempt: the generated harness, the actions taken under it, and the final outcome. The Engineer learns to generate effective harnesses, while the Solver learns to complete tasks under them. To mitigate this risk, our method uses their shared task outcome to coordinate value estimation and policy updates, supporting the joint adaptation of both policies.
JoMoHa
JoMoHa jointly trains two LLM policies: an Engineer that generates a harness and a Solver that performs tasks under it. Both learn from the outcome of the same task attempt (Figure 3). To coordinate their learning, we use collaboration-aware value estimation, role-specific return normalization, partner-trajectory weighting, and asymmetric update schedules.
| Solver | No harness | Fixed | Harness effect |
| Earlier | 82.5 | 78.5 | |
| Later | 48.0 | 31.0 | |
| Change |
Table 3a. Aggregate success rates; harness effect is the difference within each row in percentage points.
| Solver | No harness | Fixed |
| Earlier | Success 16 turns | Success 14 turns |
| Later | Success 14 turns | Timeout 50 turns |
Table 3b. Case 1: Clean sponge and place on cart.
Learning from a Shared Task Outcome
For a task , the Engineer receives historical feedback and samples a proposal trajectory . Validation and fallback map that proposal to the executed harness . The Solver then interacts with the environment under , producing trajectory and task reward . An invalid proposal maps to , retaining the base executor.
The two policies share the objective
where is the validation/fallback result and is collected by the Solver under that harness. The task reward is binary on ALFWorld.
The same reward trains both policies: the Engineer learns which harnesses support successful execution, while the Solver learns how to complete tasks under those harnesses. Each policy is updated on its own generated outputs, without differentiating through harness execution. We use a KL-regularized SAO-based actor–critic backbone (Hou et al. [4]), with a fixed reference policy for each role. The local realization retains behavior–proximal token-ratio decoupling.
Value Estimation for Two Different Roles
Each policy has an independent critic that estimates its expected return, including task rewards and its own KL penalty. The critics provide baselines for computing advantages: whether an outcome is better or worse than expected.
Context: account for collaboration conditions. The usefulness of an Engineer proposal depends on the Solver, while the Solver’s performance depends on the harness. A critic predicts the role’s expected remaining regularized return from its current history and a short collaboration summary . The supplemental fields describe the task identifier, training version, and role-specific execution conditions. The task field is a path or identifier, not a full environment observation. Context is provided only to the critics and does not change the actors’ inputs.
ValueNorm: account for different return scales. The Engineer generates a harness once, whereas the Solver acts over multiple turns. Their different output lengths and KL penalties can produce different return scales despite the shared task reward. Writing the raw, unweighted critic target as , we normalize critic targets separately for each role:
where and are running return statistics. Values are restored to their original scale before advantages are computed. Hats denote normalized targets and predictions. This critic-target normalization is separate from standardizing actor advantages.
Coordinating Policy Updates
We compute advantages using generalized advantage estimation (Schulman et al. [13]) and standardize them separately for the two policies. We introduce two additional mechanisms to account for the changing partner and the different adaptation needs of the two roles.
Peer: weight experience using the partner’s trajectory. As both policies change, a recorded task attempt may become more or less representative of their current collaboration. Peer adjusts its influence using the paired partner trajectory:
where is the role-standardized advantage. Each partner weight compares the likelihood of the recorded trajectory under the policy snapshot used for the current update with its likelihood when collected. We bound the weight to , treat it as a constant during optimization, and leave critic targets unchanged.
Schedule: give the Solver additional adaptation. Changing the Engineer changes the harnesses under which the Solver learns. To give the Solver additional adaptation without another Engineer update, the coordination study repeats joint, joint, Solver-only rounds. The resulting update ratio is Engineer:Solver = .
Solver-only rounds freeze the Engineer’s parameters but still sample fresh harnesses. They use unit partner weights. Thus, the Solver continues learning under sampled harnesses while their generating policy remains fixed.
Algorithm 1 summarizes the procedure. At deployment, we select a saved harness and evaluate it with a frozen Solver checkpoint.
Algorithm 1 Joint learning of the Engineer and Solver
Require: Engineer and Solver policies, independent critics, reference policies, task/evidence stream
for each training round do
Choose active roles according to Schedule.
Engineer samples a code proposal from historical feedback.
Validate it to obtain the executed harness and run the Solver.
Record both outputs and their shared task reward.
for each active role do
Estimate values using Context and ValueNorm.
Compute and standardize advantages.
Apply Peer weights and update the actor through the SAO-based path.
Update the critic on normalized return targets.
end for
end for
Select a saved harness and Solver checkpoint for evaluation.
Experiments
Experimental setup. We evaluate JoMoHa on ALFWorld, WebShop, and DBBench, which test household task completion, goal-directed shopping, and database interaction, respectively [17, 22, 11]. We report success rates in all three environments, a separate WebShop score, and the equally weighted mean of the three success rates. The ALFWorld coordination study uses separate Qwen3.5-9B-class Engineer and Solver actors with independent critics, initialized from Agent SFT, and collects eight paired episodes per round for 144 rounds.
Baselines. Except for locally evaluated JoMoHa and WHALE, scores are taken from Harness-R1 [14]; we use its training and test sets, with checkpoint and harness selection on a separate validation set. The task model is Qwen3.5-9B; frontier model names denote harness editors, not replacement Solvers. Agent SFT + Harness-R1 additionally fine-tunes the task model, while WHALE uses its RSFT-plus-search recipe. WHALE alternates after eight environment interactions (16 training steps), uses the same harness-search settings as JoMoHa, and matches its total environment-interaction budget. § 10.2 examines one-sided learning, frozen-Engineer training, program swaps, and coordination components.
Main Results
JoMoHa achieves the highest success rates among the evaluated methods on all three benchmarks: 97.0% on ALFWorld, 45.8% on WebShop, and 68.0% on DBBench (Table 4). Its average success is 70.3%, exceeding Agent SFT + Harness-R1 by 6.1 percentage points and WHALE by 7.2 points. The gains over WHALE are 16.5, 2.4, and 2.7 points across the three environments, respectively.
The ALFWorld breakdown locates the largest gains on tasks that transform an object’s state before placement. Against the strongest competing result in each category, JoMoHa gains 18.2 points on Clean, 17.3 on Heat, and 24.8 on Cool, compared with 0.0–4.0 points on the remaining categories. This pattern extends beyond differences in ceiling effects: on Heat and Cool, the gains remove approximately 83.6% and 89.9% of the respective best competitor’s errors . These tasks require execution to distinguish a completed transformation from the remaining navigation and placement, making the results consistent with the state-tracking issue diagnosed in § 5.1. Pick2 remains the weakest category at 91.8%, indicating remaining room for improvement when multiple objects must be collected and placed.
Ablation Studies
We examine three sources of improvement: training the Engineer rather than only resampling its outputs, improving the harness at fixed Solver weights, and coordinating the two policies beyond the Joint-only baseline.
Training the Engineer beyond fresh harness sampling. Frozen Engineer fixes the initial Engineer parameters while continuing to sample fresh harnesses and train the Solver. It reaches 90.5% at step 144, versus 97.0% for JoMoHa (Figure 1); even its best reported checkpoint, 91.5% at step 108, remains 5.5 points below the complete method. Thus, the gap persists when the control can both resample programs and use its best reported checkpoint. Weight-only and Harness-only reach 89.5% and 80.5%, respectively. Together, these controls favor training both policies over the evaluated one-sided recipes, while the component study below examines the additional benefit of coordination.
Harness utility at unchanged Solver weights. Learning curves alone cannot separate improvements in Solver ability from improvements in the generated runtime. We therefore cross-evaluate saved programs and Solver checkpoints (Table 5). At the later Solver, replacing the initial program with the later program raises success from 92.5% to 96.5%, without changing model weights. Relative to no harness, the later program gains 5.0 points, with no observed success-to-failure regressions on this evaluation set. The earlier and later learned programs also both improve the earlier Solver from 91.5% under the initial program to 95.0%. These results show that the saved programs contribute useful behavior beyond the initial runtime, including at a different Solver checkpoint. The equal 95.0% scores on the earlier Solver, but 95.0% versus 96.5% on the later Solver, further show that program value depends on the Solver with which it is paired.
| Harness | Earlier Solver | Later Solver |
| No harness | 90.0 | 91.5 |
| Initial program | 91.5 | 92.5 |
| Earlier program | 95.0 | 95.0 |
| Later program | 95.0 | 96.5 |
Table 5. Table 5. Harness utility at fixed Solver weights. Fixed-program cross-evaluation: columns fix Solvers and rows fix programs. Earlier and later denote programs and Solvers saved at checkpoints 48 and 144, respectively. The initial program comes from the initial Engineer. Colors follow Table 4.
Coordinating the two learners. Joint-only trains both policies from paired task outcomes without Context, ValueNorm, Peer, or Schedule. Adding each mechanism separately raises its final success from 88.5% to 91.5–92.5%; combining all four reaches 97.0%, an 8.5-point improvement over Joint-only (Table 6). Context yields the highest single-addition point estimate, 92.5%, consistent with the value of informing each critic about its collaboration conditions. Schedule and Peer both achieve 92.0%, while ValueNorm achieves 91.5%.
| Variant | Success (%) |
| Joint-only | 88.5 |
| + Schedule | 92.0 |
| + Context | 92.5 |
| + Peer | 92.0 |
| + ValueNorm | 91.5 |
| JoMoHa | 97.0 |
Table 6. Table 6. Coordination-component additions. Success uses the saved harness and Solver. Each addition row changes one coordination component relative to Joint-only; the full method combines all four. Best and second best follow Table 4.
Discussion and Conclusion
One model learns to write the runtime; another learns to act under it. JoMoHa trains both from the same task result while keeping their token histories and value estimates separate. Fixed-component tests measure the two sides of this interaction. At fixed Solver weights, saved programs improve on the initial code. Learning the writer outperforms freezing it while still sampling new code, and coordinating the two learners improves on joint updates alone. The complete system has the highest reported success on all three environments in the benchmark table, averaging 70.3%. This is RSI through execution feedback: both the task policy and the policy that constructs its runtime are learned within the system.
References
- [1]Feng, J., Huang, S., Qu, X., Zhang, G., Qin, Y., Zhong, B., Jiang, C., Chi, J., and Zhong, W. ReTool: Reinforcement learning for strategic tool use in LLMs, 2025. URL https://arxiv.org/abs/2504.11536.
- [2]Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients, 2017a. URL https://arxiv.org/abs/1705.08926.
- [3]Foerster, J., Nardelli, N., Farquhar, G., Afouras, T., Torr, P. H. S., Kohli, P., and Whiteson, S. Stabilising experience replay for deep multi-agent reinforcement learning, 2017b. URL https://arxiv.org/abs/1702.08887.
- [4]Hou, Z., Li, Y., Tang, J., and Dong, Y. Single-rollout asynchronous optimization for agentic reinforcement learning, 2026. URL https://arxiv.org/abs/2607.07508.
- [6]Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., and Han, J. Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning, 2025. URL https://arxiv.org/abs/2503.09516.
- [7]Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Hq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., Zaharia, M., and Potts, C. DSPy: Compiling declarative language model calls into self-improving pipelines, 2023. URL https://arxiv.org/abs/2310.03714.
- [8]Kim, H., Lee, Y., Lee, G., Finn, C., and Lee, K. WHALE: A simple recipe for joint harness-weight optimization, 2026. URL https://arxiv.org/abs/2609.00196.
- [9]Kuba, J. G., Chen, R., Wen, M., Wen, Y., Sun, F., Wang, J., and Yang, Y. Trust region policy optimisation in multi-agent reinforcement learning, 2021. URL https://arxiv.org/abs/2109.11251.
- [10]Lee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O., and Finn, C. Meta-Harness: End-to-end optimization of model harnesses, 2026. URL https://arxiv.org/abs/2603.28052.
- [11]Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., Zhang, S., Deng, X., Zeng, A., Du, Z., Zhang, C., Shen, S., Zhang, T., Su, Y., Sun, H., Huang, M., Dong, Y., and Tang, J. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2308.03688.
- [12]Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-Refine: Iterative refinement with self-feedback, 2023. URL https://arxiv.org/abs/2303.17651.
- [13]Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016. URL https://arxiv.org/abs/1506.02438.
- [14]Shao, S., Zhang, K., Li, Q., Wang, S., Wang, H., Jiao, W., Lu, Y., Guo, Y., Liu, W., and Zhang, W. Harness-R1: Learning to edit executable runtime harnesses from agent failure trajectories, 2026. URL https://arxiv.org/abs/2608.02276.
- [15]Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300.
- [16]Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv.org/abs/2303.11366.
- [17]Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. URL https://arxiv.org/abs/2010.03768.
- [18]Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. MIT Press, 2 edition, 2018. URL https://mitpress.mit.edu/9780262039246/reinforcement-learning/.
- [19]van Hasselt, H., Guez, A., Hessel, M., Mnih, V., and Silver, D. Learning values across many orders of magnitude, 2016. URL https://arxiv.org/abs/1602.07714.
- [20]Wang, Z., Wang, K., Wang, Q., Zhang, P., Li, L., Yang, Z., Jin, X., Yu, K., Nguyen, M. N., Liu, L., Gottlieb, E., Lu, Y., Cho, K., Wu, J., Li, F.-F., Wang, L., Choi, Y., and Li, M. RAGEN: Understanding self-evolution in LLM agents via multi-turn reinforcement learning, 2025. URL https://arxiv.org/abs/2504.20073.
- [21]Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers, 2023. URL https://arxiv.org/abs/2309.03409.
- [22]Yao, S., Chen, H., Yang, J., and Narasimhan, K. Web-Shop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, volume 35, 2022. URL https://arxiv.org/abs/2207.01206.
- [23]Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2210.03629.
- [25]Zhang, J., Xiang, J., Yu, Z., Teng, F., Chen, X., Chen, J., Zhuge, M., Cheng, X., Hong, S., Wang, J., Zheng, B., Liu, B., Luo, Y., and Wu, C. AFlow: Automating agentic workflow generation, 2024. URL https://arxiv.org/abs/2410.10762.
