betaFeedback

WorksComputer Science

Operon-S1: a fast decision controller for long-horizon biomedical agents

Abstract

Language-model agents now carry out long scientific work in biology, but their judgements about conventions, skills, errors, evidence and stopping are made inside the reasoning that does the work, where they cannot be measured, replaced or improved on their own. Here we present Operon-S1, a fast decision system, a System 1 for the agent, that separates judging from doing. A decision harness poses the agent's recurring judgements as closed questions at 13 decision points and acts only through fixed messages that never contain an answer, so any model that answers the questions can serve as the judge. Across seven life-science benchmarks, a local decision model took over these judgements in 0.6 s at the median for an in-loop request on CPU at no request cost, with no measurable change in the agent's score on five of six benchmarks, whereas the general language model that drives the agent, asked the same questions, accounted for 8.0% to 35.2% of task cost. Every decision is recorded with its consequence. Retraining on labels from task construction, verifiers or execution raised decision accuracy on every question in each of three trained versions, and a new version enters service only through a promotion gate. Task-level results in this study use the public checkpoint. The same closed decisions serve wet-laboratory protocol troubleshooting and the design of a germline target enrichment panel. Like the regulator of an operon, which switches genes on and off without making their products, Operon-S1 steers the agent without doing its work, and makes the agent's judgements a layer that can be measured, replaced and trained.

Introduction

Language-model agents now retrieve data, write and run analysis code, and translate scientific intent into executable laboratory procedures, in biology and biomedicine[1, 2, 3, 4, 5, 6], where a general-purpose biomedical agent now carries out research tasks autonomously[2], and in chemistry[7]. As these tasks lengthen, the limit on an agent shifts from what it can compute to how it judges its own work. A single analysis spans tens of steps, and along the way the agent meets questions that no tool answers: which convention a loosely worded task implies, which skill or file applies, what a failed command means, whether a value it has computed already answers the question, whether a slow computation should continue or yield its best result and, in laboratory work, which step of a protocol caused a failure. In such agents these judgements share one model’s reasoning with the computation, so none of them is posed, measured or corrected on its own.

These judgements carry scientific weight. The independent-filtering threshold of a differential-expression analysis changes which genes are reported[8], a skill whose convention differs from the task’s can turn a correct analysis into a wrong answer, a misattributed protocol step sends the repeat experiment in the wrong direction, and a probe design judged by its design coverage alone can leave targets uncaptured. Once agents drive instruments, a misjudgement costs reagents and instrument time as well as compute. Reasoning agents can also favour extended internal reasoning over interacting with their environment, a failure termed overthinking[9], and reasoning models keep deliberating after they have reached a correct answer[10]. In both cases more computation does not supply the missing judgement.

Cognitive science describes reasoning as the interplay of a fast, intuitive process and a slow, deliberate one[11, 12], and it separates metacognitive monitoring, the assessment of one’s own progress, from metacognitive control, the decision to continue, change strategy or stop[13]. Fast decision models, sometimes called System-One models, map a short state and a closed question to a distribution over a fixed set of options in a single forward pass[14, 15, 16]. Related approaches route queries between cheap and expensive models[17, 18, 19], learn when to defer to a downstream decision maker[20] or score intermediate steps with learned verifiers[21, 22] and process reward models[23, 24]. Language models also act as judges of other models’ outputs[25, 26] and as critics that flag errors in code[27], and their verdicts are free text written by a model. Selective prediction, abstaining when uncertain, is the classical way to contain a fast model’s errors[28, 29], and conformal prediction gives such abstention distribution-free guarantees[30, 31]. In agent pipelines, System-One models have reduced calls to large models without loss of task success[32], supported scientific decisions over prepared evidence[33], scored the recurring decisions of search agents[34] and answered typed questions over clinical records[35].

The same work also marks the limits of this approach. For a decision model fine-tuned on a browser agent’s model-labelled decisions, held-out decision agreement rose from 19% to 75% without a corresponding gain in task success[36], savings shrank once the cost of the decision model was counted[16], and an open-weight local decision model changed 30% of its answers when the option order was reversed[16]. In a cascade between a fast verifier and a large model, a matched random-escalation control showed that the savings came from how much was escalated rather than from the fast model’s probabilities[37]. Scientific agents raise three further problems. The decisions that matter are scientific: conventions, analysis routes and whether the evidence suffices. A decision can sit tens of steps before the score it affects, because only the final answer is scored. And a fast model meant to improve as the agent works needs evidence of improvement from outside its own training loop.

An agent can hold its judgements in three ways, distinguished by two criteria: whether each judgement is posed on its own, and whether the model that answers it is trained and replaced against labels that no model produces. In the first, the judgements stay inside the reasoning of the model that does the work, and they change only when that model, its prompt or its skills change. In the second, judgements are lifted out to a fast model, one at a time[17, 20, 21] or several at once[34, 35], and the model is judged by cost, by task success or against labels from people or language models. In the third, every recurring judgement of the run is posed as a closed question with a verifiable label, and the model that answers is replaced only under a gate. Operon-S1 takes the third route: it separates judging from doing.

Self-improving agents have refined their behaviour through verbal reflection on failures[38] and iterative self-feedback[39], accumulated libraries of reusable skills[40], rewritten their own code[41, 42] and evolved programs under automated evaluators[43]. In these systems a change is kept when it improves the signal that drives the search. A controller for scientific judgements needs a stricter test, because its labels are imbalanced and a candidate can score well by learning only the most frequent answer. Operon-S1 therefore keeps a new version only if it beats its parent and three baselines on labels that no model produces. Statistical adoption gates can control false adoptions in adaptive model search[44], and no automated gate can in general both refuse every bad update and accept every good one[45], so the gate here gives its guarantees only on a bounded, labelled split.

Here we introduce Operon-S1, a fast decision controller for long-horizon biomedical agents, paired with the ProtoPilot agent[1] (Figure 1). As an external model, it takes over the two functions that metacognition research separates, monitoring and control[13]. The name refers to the operon[46], in which a regulatory protein switches a cluster of genes on or off without making their products, as the controller regulates the agent’s work without doing it, and “S1” to the fast process, System 1, of dual-process accounts of reasoning[11]. ProtoPilot coordinates protocol, computational and coding agents, verifies each stage against expert-grounded rubrics, and has generated standard operating procedures and instrument code that pass device-level validity checks across four laboratory automation platforms[1].

Operon-S1 separates judging from doing in a biomedical agent

Figure 1. Operon-S1 separates judging from doing in a biomedical agent. a, The slow system, the ProtoPilot agent, does all computation, from data and code to the answer. The decision harness poses the agent’s recurring judgements at 13 decision points from the start of the task to its submission (Table 1). At each point the fast system, the decision layer, receives a state rendered from the facts of the run, answers a closed question with option probabilities and acts only through fixed messages, and each decision is kept in a provenance record. Right, a convention judgement in a differential-expression task: which independent-filtering threshold the task implies (Figure 2b). The message names the convention and never the value; option lengths are illustrative. b, Six interchangeable controllers answer the same states, questions and messages and differ only in their judgements (Figure 3; Extended Data Table 2). c, Gated retraining with a data-synthesis pipeline. Each round builds labelled decisions from an executable curriculum and trains a candidate, which enters service only through the promotion gate (Figure 4a). Every trained version and its verdict are kept in an append-only registry (Table 3; Figure 4). d, Domains enter through profiles of the same harness: computational analysis, wet-laboratory protocol troubleshooting (protocol profile) and probe-panel design (Figure 5), each with one of its judgements. Schematic; no data are plotted.

Operon-S1 adds a decision layer with three parts to the agent, and ProtoPilot is the agent with which it is tested here. The decision harness is preset for each domain and triggers autonomously. It builds each state deterministically from the facts of the run, so the agent never assembles the context and every decision can be replayed, and it poses closed questions at 13 decision points from the start of a task to the submission of its answer (Table 1). The fast decision model answers each question with a probability for each option and acts through four fixed actions: brief, remind, end the tool loop or request one recheck. The local decision model, LAYA, runs on CPU. A hosted model, JEV, serves as a comparator, and a second comparator places a general language model in the same harness. LAYA is retrained in gated rounds on decisions from a data-synthesis pipeline. With the harness, the questions and the messages fixed, the system is designed to change only the decision model during retraining, so each change in behaviour can be attributed to it.

PointJudgementQuestion (type)ActionRound-1 label
S1analysis familyten classes (choice)scopes S6construction
S2skillswhich skills may help, with a no-skill option (choice); relevance of the top three (yes/no)brief: up to three skill names, never their textconstruction
S3skill needis a skill needed (yes/no)recorded only–
S4environmentwhich environment, when there is more than one (choice)brief: environment path–
S5filesis each input file relevant (yes/no)brief: relevant files–
S6conventionsconvention per slot that the task does not state (choice)brief: “the task appears to require . . . ; verify”–
S7answer typeexpected answer type (choice)brief; scopes F1construction
F1evidencedoes the evidence answer every part; is a step missing (yes/no ×2)end run (file-answered tasks: remind)–
P7long computationcontinue / one more round / switch to cheaper / stop with best (score)end run or remind–
P3error triageerror class, ten classes (choice)class-specific reminderexecution
P4convention in codedoes the code conform to the convention (yes/no)remind–
P8workspace boundsout of bounds (yes/no)remind–
D20answer checkdoes the answer conform (yes/no per slot)one recheck naming what to verify (text-answered tasks only)construction, verifier

Table 1. The 13 decision points of the decision harness, from the start of a task to the submission of its answer. The harness layer triggers each point autonomously from the facts of the run, poses its closed questions and abstains when the controller is not confident. The last column gives the source of the training labels in round 1. Points marked – are recorded without training labels in round 1.

Each design choice follows from a property of the science. Because a judgement can decide a score without changing the computation, a message may name a convention, a file or a quantity to verify but never an answer value. Because conventions, evidence, error classes and answer formats can be checked by construction, verifiers or execution, each judgement is a closed question whose answer can be compared with such a label.

We find that this layer can be measured, replaced and traced at little cost. Six controllers, from rules to a general language model, ran on the same states, questions and messages, and in the 2,361 runs whose answers can be matched as text no injected message contained the gold answer. The local decision model answered single in-loop requests in under a second at the median and accounted for 0.8% to 5.2% of wall-clock time at no request cost. With its public checkpoint as the judge, the decision layer lowered cost on SeqQA by 5.8% and agent steps on protocolqa2 by 10.3%, with no measurable score change on five of six benchmarks. On seqqa2, every acting controller lowered the score, the local model least. Training on verifiable labels raised accuracy over the public checkpoint on every decision question in each of three trained versions, analysis family from 0.26 to 0.99 in the first. The gains carried over to the agent’s own decision states on held-out benchmark items, where analysis-family accuracy rose from 0.056 to 1.000 and execution-labelled error-class accuracy from 0.174 to 0.408. The same closed decisions extend to laboratory-protocol troubleshooting, where the errors are of diagnosis rather than of routing, and to probe design, where a fall in panel coverage from 88% to 16% traces to three causes, each answered by one closed design action.

Results

Operon-S1 lifts the agent’s judgements into a separate decision layer

Operon-S1 separates judging from doing. The judgements of the agent go to a separate fast decision model through the decision harness, and all doing stays with the agent (Figure 1a). The slow system is the ProtoPilot agent[1], here driven by DeepSeek-V4.1-Flash at high reasoning effort, which plans, writes and runs code in a sandbox and composes the answer. The harness layer triggers autonomously during the run, reads its facts and poses closed questions. These cover the analysis family, the expected answer type, relevant files and candidate skills, and conventions that the task implies but does not state. They also ask whether the evidence already answers the task, whether a long computation should stop with its best result, which class an error belongs to, whether the code follows the required convention, and whether the answer meets its format and conventions (Table 1). In the terms of Nelson and Narens[13], the answers to these questions are the controller’s monitoring of the agent’s progress, and the actions they trigger are its control.

The controller answers each question with a probability distribution over its options and acts only when it is confident, abstaining otherwise as in selective prediction[28]. Its actions are to brief the agent, remind it, end the tool loop with a final-answer instruction or request one recheck, each delivered through a message written in advance that may name a convention, a file or a quantity to verify. All controllers share this policy and these messages and differ only in the distributions they return, with one exception for the rule-based controller (Methods), so a comparison between controllers is a comparison between their judgements.

Throughout, “controller” denotes any policy in the controller slot, including the no-op and rule-based controls, and “fast decision model” denotes the learned or hosted model behind it.

A no-op controller renders and records every state but always abstains, and leaves the agent’s model requests unchanged (Methods). The effect of a controller ϕ\phi on a task outcome YY is the contrast

Δϕ=E[Y(ϕ)]−E[Y(ϕ0)],(1)\Delta_{\phi} = \mathbb{E}[Y(\phi)] - \mathbb{E}[Y(\phi_0)], \tag*{(1)}

where ϕ0\phi_0 is the no-op controller and the expectation is over tasks and runs; the controller’s own latency is charged against each task’s time limit. The evaluation has seven arms. The baseline is the original agent without the harness. The other six run the same harness with different controllers (Figure 1b):

On tasks with a protocol document, the protocol profile replaces S1 by a choice among ten wet-lab domains, adds the answer type protocol fix, adds a fixed reminder after a failed read of the protocol document, replaces D20 by three yes/no questions on specificity, consistency with the extracted protocol text and whether the change explains the symptom, and silences F1, P7, P3, P4 and P8 (Methods).

  • the no-op control;

  • a rule-based controller that uses lexical matching, error-message patterns and budget fractions;

  • a self-verification control that requests a recheck whenever the answer check would pose a question;

  • JEV, a hosted System-One model[14], used as a comparator;

  • LAYA, a local 322-million-parameter encoder[47] served on CPU, whose public checkpoint is version 0 of the trained controller;

  • a general-LLM controller, in which DeepSeek-V4.1-Flash answers the same closed questions in the same harness, with the same states, policy and messages.

The general-LLM arm measures what a fast decision model adds over asking a general language model the same questions. In every batch of the main table, JEV, LAYA and the general-LLM controller have equal time budgets for each request (Methods).

For every decision we keep a provenance record of the distribution, the action, whether the agent adopted it and the task outcome (Figure 1a), which links a decision to a score assigned only to the final answer.

Judgements rather than computation decided most of the original agent’s lost items and surplus time

The judgements that the layer poses are where the original agent lost items and time. Most wrong answers of the original ProtoPilot agent on BixBench-Verified-50 arose from a judgement after the computation. The agent answered 12 of the 49 gradable items wrongly (Methods). On 5, the correct value was already in its output under another convention. On 1 it misread the direction of an effect, and on 2 it broke the required answer format. On 3 it lacked an offline resource, and on 1 none of its computed variants matched the reference. The first three groups, 8 of 12, turned on a choice made after the computation, where a controller can act. A further judgement, when to stop, costs mainly time, measured directly as the wall-clock time after the answer first appeared (Figure 2a). For each task with a locatable answer we recorded the answer time ti∗t_i^*, the first moment at which the value the agent finally submitted appeared in its own tool output, and measured the post-answer fraction of wall-clock time

Three-panel figure showing stopping after the answer, a convention mismatch, and skills and tool errors

Figure 2. Judgements, not the computation, decided most of the original agent’s lost items and time. Trajectories of the original agent under two earlier skill releases. a, Post-answer time: ω\omega (equation (2)) on the 15 BixBench-Verified-50 tasks with a locatable answer, and the post-answer share of all wall-clock time on SciAgentArena and seqqa2, a lower bound on ω\omega. b, A convention mismatch in which the reference value appeared in the agent’s own output. c, Items lost when a matching skill was skipped or loaded (210 judged trajectories); the annotation gives the range of unaided recovery from tool-error episodes on three benchmarks.

ω=∑i∈I(Ti−ti∗)∑i∈ITi,(2)\omega= \frac{\sum_{i \in I}(T_i-t_i^*)}{\sum_{i \in I}T_i}, \tag*{(2)}

where TiT_i is the wall-clock time of task ii and II the set of tasks with a locatable answer (Methods). On the 15 BixBench-Verified-50 tasks with a locatable answer, ω=0.47\omega= 0.47: 1,798 of 3,860 s elapsed after the answer was already in the agent’s tool output. On SciAgentArena, where t∗t^* is taken as the first successful run of the solution script, 30% of all wall-clock time on the benchmark followed t∗t^* (14,709 of 48,852 s), and on seqqa2 16%; because tasks without a located t∗t^* add only to the denominator, these shares are lower bounds on ω\omega (Methods). Because timed-out tasks score zero, continuing can also cost the whole item. On seqqa2, four of five timed-out codon-optimization tasks had produced a valid candidate 69–353 s into a run that then continued to the limit.

The convention errors share a pattern. On each of the 5 items, the reference value or the reference convention had already appeared in the agent’s own output, but the agent submitted a value computed under another convention (Figure 2b). In one case it ran a differential-expression analysis with the default independent-filtering threshold[48], which reproduced the reference, and then reported the result of a different threshold. The difference between a right and a wrong answer was the choice between the default threshold of 0.1 and a threshold of 0.05.

Skill selection and error triage cost fewer items than conventions. Across 210 judged trajectories, loading a matching skill cost an item three times, because the skill text was wrong or its convention differed from the benchmark’s, whereas skipping one cost an item once (Figure 2c). On BixBench-Verified-50, SciAgentArena and BiomniBench-DA, the agent recovered unaided from 82–94% of tool-error episodes, and the median error run lasted one call. Headroom therefore differs from one judgement to the next, which is why the layer poses, measures and trains each judgement on its own rather than repairing one failure class at a time.

The same analysis was applied to the baseline rounds of the formal evaluation, which used later skill releases and a higher reasoning effort (Methods; Extended Data Figure 1). Of 121 wrong answers on the four benchmarks scored per item (SeqQA, seqqa2, BixBench-Verified-50 and SciAgentArena), a rule codebook assigned 9 to conventions, which it detects only where an answer matcher exists, 7 to stopping at the time limit and 1 to answer format. On 10 of 27 wrong answers with an automatic gold matcher, the correct value had already appeared in the agent’s own output. On tasks with a locatable answer, the post-answer fraction ω\omega was 0.33 on BixBench-Verified-50, 0.35 on SciAgentArena and 0.49 on seqqa2 (Extended Data Figure 1).

Three-panel plots showing failure classes, answer correctness and post-answer share of wall-clock time

Extended Data Figure 1. Errors after the computation in the formal evaluation. a, Failure classes of wrong baseline answers by rule codebook, per benchmark (wrong/total: SeqQA 11/600, seqqa2 23/400, BixBench-Verified-50 6/50, SciAgentArena 81/189, BiomniBench-DA 38/50, protocolqa2 32/125, OmicsBench 2/3). Convention errors are detectable only on benchmarks with answer matchers (SeqQA, seqqa2 and BixBench-Verified-50). b, Share of wrong answers with an answer matcher whose correct value had appeared in the agent’s own tool output, per arm,

The local decision model costs little beside the work it judges

A separate judge pays off only if it costs little beside the work it judges and leaves the agent unchanged when it abstains. We evaluated on seven benchmarks spanning sequence reasoning, laboratory-protocol troubleshooting, end-to-end data analysis and multi-step omics studies, 1,417 tasks in total (Table 2): LAB-Bench SeqQA[49], LAB-Bench 2 seqqa2 and protocolqa2[50], BixBench-Verified-50[51, 52], SciAgentArena[53], BiomniBench-DA[54, 55] and the public split of OmicsBench[56]. Each benchmark ran under its official prompt, network policy and primary metric, with every task in a fresh sandbox. The original agent was scheduled for five rounds to estimate between-round variation, and Extended Data Table 2 gives the number of rounds that completed on each benchmark (Methods). Paired changes in time, steps and accuracy against the baseline are shown in Figure 3a,Figure 3b and Extended Data Table 2. The hypotheses are that the controller reduces cost at non-inferior accuracy (H1) and that its gains exceed those of an extra recheck (H2). H1 is judged on five benchmarks, 1,289 tasks: SeqQA, seqqa2, BixBench-Verified-50, SciAgentArena and BiomniBench-DA. H2 is judged on SeqQA and seqqa2. OmicsBench, on which the controllers are inert, is run in the baseline arm only. protocolqa2 was added to the evaluation after the hypotheses were written, so its results are reported as exploratory.

BenchmarkTasksMetricNetworkJudgePer-task limits
LAB-Bench SeqQA600accuracydatabase allowlistnone1 CPU, 2 GiB, 1,200 s
LAB-Bench 2 seqqa2400accuracy (official validators)database allowlistnone1 CPU, 2 GiB, 1,200 s, last 60 s reserved
LAB-Bench 2 protocolqa2125accuracy (official judge prompt)offlineLLM (substitute model)2 CPU, 4 GiB, 2,400 s
BixBench-Verified-5050open-answer accuracydatabase allowlistLLM for range answers (substitute model)2 CPU, 16 GiB, 3,600 s
SciAgentArena189mean correctnessofflinenone4 CPU, 8 GiB, 3,600 s
BiomniBench-DA50rubric score, 0–100web via logged proxyLLM rubric (substitute model)2 CPU, 16 GiB, 7,200 s
OmicsBench (public)3mean node scoreweb via logged proxyofficial4 CPU, 12 GiB, 43,200 s

Table 2. Benchmarks and settings. Per-task time limits are twice the official or default limit (Methods).

Efficiency, accuracy and decision provenance of six controllers in one harness

Figure 3. Efficiency, accuracy and decision provenance of six controllers in one harness. a, Paired relative change in wall-clock time and agent steps against the baseline (ratio of paired sums minus one) per controller and benchmark. Tasks: SeqQA 600, seqqa2 400, BixBench-Verified-50 50, SciAgentArena 189, BiomniBench-DA 50 (common set at the data cutoff) and protocolqa2 125. Hollow markers, BiomniBench-DA changes of JEV and LAYA, shown for reference (Methods). protocolqa2 is exploratory. b, Paired change in the official score in points (accuracy ×\times 100; BiomniBench-DA rubric points, 0–100), with the non-inferiority margin (dashed). SciAgentArena has a single reference round and no margin. Arrows mark intervals beyond the axis. c, For each decision point, pooled over the six benchmarks: firing rate per task, action rate as a share of firings, adoption rate as a share of actions whose adoption is defined, and the outcome difference between adopted and non-adopted cases, which is descriptive because adoption is not randomized. The difference is weighted by tasks over benchmarks scored 0–1 with at least five cases in each group and is drawn only when at least ten cases are pooled. d, Post-answer fraction ω\omega (equation (2)) on tasks with a locatable answer, per arm and benchmark. Error bars, 95% cluster-bootstrap intervals.

BenchmarkArmTasksScoreΔ score [95% CI]Δ wall (%)Δ steps (%)Δ cost (%)Non-inferior
SeqQABaseline (R = 2)6000.982 ± 0.001–––––
SeqQANo-op6000.985+0.004 [−0.005, +0.013]−2.3 [−8.2, +3.9]−0.8 [−5.2, +4.0]−1.3 [−5.9, +3.5]no
SeqQARule-based6000.990+0.009 [+0.001, +0.018]−1.1 [−6.8, +5.1]−1.4 [−5.8, +3.2]−3.3 [−8.1, +1.8]yes
SeqQASelf-verification6000.980−0.001 [−0.011, +0.008]−1.1 [−6.9, +5.2]−2.8 [−7.5, +1.9]−0.6 [−5.9, +4.9]no
SeqQAJEV6000.982+0.001 [−0.009, +0.011]+1.7 [−4.3, +8.1]−0.4 [−5.1, +4.6]+5.9 [+0.0, +12.0]no
SeqQALAYA v06000.985+0.004 [−0.005, +0.013]−4.7 [−10.3, +1.5]−1.3 [−5.7, +3.3]−5.8 [−10.8, −0.6]yes
SeqQAGeneral LLM6000.987+0.006 [−0.003, +0.013]−2.6 [−8.2, +3.3]−1.2 [−5.6, +3.4]+53.6 [+46.4, +61.5]no
seqqa2Baseline (R = 3)4000.943 ± 0.007–––––
seqqa2No-op4000.943+0.004 [−0.009, +0.017]−1.8 [−11.5, +8.1]−4.5 [−9.1, +0.2]−3.2 [−10.1, +4.1]yes
seqqa2Rule-based4000.920−0.018 [−0.035, −0.003]−2.3 [−10.5, +6.1]−5.5 [−9.8, −0.9]−7.1 [−12.8, −1.4]no
seqqa2Self-verification4000.920−0.018 [−0.036, −0.002]−6.0 [−14.2, +2.7]−2.1 [−6.3, +2.2]+0.0 [−6.7, +6.8]no
seqqa2JEV4000.900−0.038 [−0.060, −0.018]+6.9 [−2.2, +16.4]+2.2 [−2.8, +7.5]+10.3 [+3.0, +18.1]no
seqqa2LAYA v04000.925−0.013 [−0.028, −0.001]+2.6 [−5.3, +10.8]+2.1 [−3.0, +7.6]+6.7 [−1.1, +15.1]no
seqqa2General LLM4000.890−0.048 [−0.072, −0.025]+8.8 [+0.8, +18.0]−3.7 [−8.4, +1.3]+49.8 [+40.5, +60.1]no
BixBench-Verified-50Baseline (R = 2)500.880 ± 0.014–––––
BixBench-Verified-50No-op500.840−0.030 [−0.092, +0.023]+2.5 [−10.2, +15.9]+9.2 [−1.0, +20.0]+5.0 [−4.3, +16.4]no
BixBench-Verified-50Rule-based500.880+0.010 [−0.056, +0.064]−1.0 [−12.8, +9.3]−5.8 [−15.7, +5.3]+3.1 [−10.3, +19.5]no
BixBench-Verified-50Self-verification500.900+0.030…………

Extended Data Table 2. Main results. Data cutoff 29 October 2026, 18:30 (UTC+8), on the tasks graded in the baseline and in every arm. Official metric of the baseline (mean over the RR baseline rounds that entered the reference, ± s.d. across rounds where R>1R > 1) and, for each controller arm, the paired change against it in score, wall-clock time, agent steps and cost including the controller. Non-inferior: lower bound of the score change above εb\varepsilon_b (equation (16)). H1 is judged on SeqQA, seqqa2, BixBench-Verified-50, SciAgentArena and BiomniBench-DA, and H2 on SeqQA and seqqa2. protocolqa2 is exploratory, and OmicsBench runs the baseline only.

The no-op control checks that the harness itself leaves the agent unchanged. Against the baseline, the no-op arm changed the official score by −0.030 to +0.014 and wall-clock time by −6.9% to +40.5%, and every 95% interval of its score, time and step changes included zero on the six benchmarks with paired tasks except the wall-clock change on BiomniBench-DA (+40.5% [+18.0, +66.4]), where its batches ran partly at a different concurrency (Methods) (Extended Data Table 3). An audit of the 16,042 messages injected in 5,743 task runs found no message that contained the gold answer in any of the 2,361 runs whose answers can be matched as text. The remaining 3,382 runs have free-text, file or sequence-design answers that cannot be matched automatically.

Relative to the baseline, the decision layer with LAYA version 0 as judge lowered cost on SeqQA by 5.8% [0.6, 10.8] at no request cost of its own; per task, JEV requests cost 0.19 to 0.48 thousandths of a dollar and general-LLM requests 2.1 to 9.8. On SciAgentArena the steps and cost of this arm changed by −6.8% [−18.6, +3.8] and −4.7% [−17.2, +7.6]. Its wall-clock change ranged from −14.0% to +2.6% across the hypothesis benchmarks, and its only score change with an interval excluding zero was −0.013 [−0.028, −0.001] on seqqa2 (Figure 3a,Figure 3b; Extended Data Table 2). JEV changed wall-clock time by −9.8% to +6.9% and steps by −6.2% to +4.0%, and lowered the seqqa2 score by 0.038 [0.018, 0.060]. The general-LLM controller changed wall-clock time by −5.2% to +16.2%, changed cost by −4.4% to +61.5%, and lowered the seqqa2 score by 0.048 [0.025, 0.072]. The rule-based controller reduced steps on SciAgentArena by 9.8% [1.2, 19.3] and on seqqa2 by 5.5% [0.9, 9.8], where its score fell by 0.018 [0.003, 0.035]. H1 requires an efficiency gain on at least four of the five hypothesis benchmarks and non-inferiority on every benchmark with a verdict (SciAgentArena, with one reference round, has none), and at the data cutoff it holds for no arm: the efficiency condition held on at most two benchmarks. Non-inferiority is limited by precision on two benchmarks. On BixBench-Verified-50 the score interval of the no-op control itself (−0.030 [−0.092, +0.023]) extended below the margin of 0.028, because 50 tasks in 33 clusters give an interval wider than the margin, and on SeqQA the margin is 0.0023, because two baseline rounds that agree closely give a between-round standard deviation of 0.001 (Methods).

On protocolqa2, an exploratory benchmark, LAYA version 0 and the general-LLM controller reduced steps by 10.3% [4.9, 15.7] and 10.9% [4.8, 16.8] with score changes of −0.008 and 0.000, and JEV changed the score by +0.032 [−0.024, +0.088]. On BiomniBench-DA, with all 50 tasks paired, JEV, LAYA version 0 and the general-LLM controller reduced steps by 16.6% [9.1, 23.2], 13.9% [5.9, 21.2] and 16.1% [9.5, 22.3] with no score change outside its interval, and the changes of JEV and LAYA are for reference only, because their batches ran at a different concurrency (Methods).

These figures include the controller’s own requests, latency and cost. In offline replay of 8,265 gate rows, the local controller answered in 266 ms at the median and 581 ms at the 95th percentile on CPU. During the evaluation, the median error-triage request (P3) took 576 ms for LAYA, 222 ms for JEV and 1,462 ms for the general-LLM controller, against 8 to 34 s for a slow-system step across benchmarks. The brief issues two sequential requests over the longest states. It took 7.3 s at the median on the local model, against a median of 164 s for a whole SeqQA task. The brief carries 7 to 31 questions and its requests queue on four shared replicas, whereas single in-loop requests took 0.5 to 0.8 s at the median (workspace bounds 531 ms, error triage 576 ms, answer check 667 ms, convention in code 811 ms, long computation 839 ms). Efficiency is reported per benchmark. Across benchmarks, the controller accounted for 0.3% to 1.4% of wall-clock time and 0.4% to 2.2% of cost per task for JEV, and for 1.9% to 16.9% and 8.0% to 35.2% for the general-LLM controller (Extended Data Figure 2). LAYA version 0 accounted for 0.8% to 5.2% of wall-clock time and no request cost.

Speed and cost including the controller

Extended Data Figure 2. Speed and cost including the controller. a, Time per controller request at four decision points of the formal evaluation (Table 1), for LAYA version 0 on CPU, JEV and the general-LLM controller. Dots are medians and lines extend to the 95th percentile. Points with fewer than 30 requests are omitted. The shaded band is the range of the median slow-system step across benchmarks, computed as wall-clock time minus controller time, divided by agent steps, over the baseline rounds of the main table. b, Controller share of wall-clock time per benchmark for JEV, LAYA version 0 and the general-LLM controller, with 95% intervals. c, Controller share of cost per task for JEV and the general-LLM controller. The brief is not shown. LAYA runs locally with no per-request price.

Provenance records separate whether a judgement is right, whether the agent acts on it and whether the task succeeds

Monitoring and control are measured separately. Monitoring quality is the agreement of each judgement with its label, by per-question accuracy and calibration (Extended Data Figure 3; Extended Data Table 6). On benchmark states with execution- or benchmark-derived labels (error class, analysis family and answer type), per-question accuracy ranged from 0.865 to 1.000 for JEV and from 0.855 to 0.998 for the general-LLM controller, with expected calibration errors of 0.013 to 0.039 and 0.033 to 0.045. Version 0 of LAYA, the public checkpoint used zero-shot, was poorly matched to these states: 0.164 on error class (expected calibration error 0.360), 0.087 on analysis family and 0.663 on answer type, against 0.994 on answer format, where JEV scored 0.680 and the general-LLM controller 0.903 (Extended Data Table 6). Round 1 trains

QuestionnMajorityRuleLAYA v0JEVGeneral LLMHeadroom
Error class (P3)3,4340.3830.9530.164 [0.132, 0.194] / 0.360 (836)0.864 [0.841, 0.885] / 0.036 (868)0.846 [0.820, 0.871] / 0.026 (891)on 3 of 6 benchmarks
Analysis family (S1)4,5540.8780.5930.087 [0.052, 0.126] / 0.399 (1,139)0.974 [0.961, 0.984] / 0.013 (1,139)0.963 [0.944, 0.980] / 0.040 (941)yes
Answer type (S7)2,4001.0000.9330.603 [0.625, 0.702] / 0.267 (600)1.000 [1.000, 1.000] / 0.039 (600)0.998 [0.994, 1.000] / 0.046 (472)yes (one label value)
Answer format (D20)4810.9960.9960.994 [0.985, 1.000] / 0.053 (477)0.680 [0.638, 0.722] / 0.132 (485)0.903 [0.875, 0.928] / 0.054 (473)no

Extended Data Table 6. Decision accuracy on benchmark states. Main-table round 1 of the rule-based, LAYA version 0, JEV and general-LLM arms on SeqQA, seqqa2, protocolqa2, BixBench-Verified-50, SciAgentArena and BiomniBench-DA, original batches graded by 29 October 2026, 11:00 (UTC+8). Labels are execution-derived (error class, from the failing call’s output), benchmark-derived (analysis family, answer type) or a grader proxy (answer format: the grader can parse the answer). Rule accuracy counts an unanswered state as wrong, as the gate does. Model columns give accuracy [95% cluster-bootstrap interval] / expected calibration error (answered states). n, labelled states (answer format: states at which it was asked). Headroom: rule accuracy at most 0.95; for error class it holds on BixBench-Verified-50, SciAgentArena and BiomniBench-DA. The answer-type label exists only on SeqQA and has one value. Questions without a model-free label on benchmark states are not listed. General-LLM values use original batches only and differ by at most 0.004 from Extended Data Fig. 3, which includes resource-failure reruns.

Calibration and selective prediction

Extended Data Figure 3. Calibration and selective prediction. a, Reliability diagrams of JEV and the general-LLM controller on benchmark states, for the error class (labels from the failing call’s output), the analysis family and the answer type (labels from the benchmark). Bins of 0.1 in stated confidence with at least five decisions are shown, and each panel gives the expected calibration error. b, Accuracy of answered error-triage questions against coverage as the action threshold varies. Dots mark the threshold of 0.6 used in the evaluation.

these questions on curriculum states. At the action threshold of 0.6, on error triage, JEV answered 91.2% of questions with an accuracy of 0.911 on those answered, and the general-LLM controller answered 92.9% with an accuracy of 0.882 (Extended Data Figure 3). Control is measured by the contrast of equation (1), together with the post-answer fraction ω\omega of each arm (Figure 3d). In the baseline ω\omega was 0.49 on SeqQA, 0.49 on seqqa2, 0.33 on BixBench-Verified-50 and 0.35 on SciAgentArena. Three arms changed it with an interval excluding zero: the general-LLM controller raised it on seqqa2 by 0.081 [0.036, 0.127], and the rule-based and self-verification controls lowered it on SciAgentArena by 0.044 [0.001, 0.092] and 0.061 [0.016, 0.123].

The self-verification control isolates the value of looking again before the answer is submitted. It uses the same states, triggers and messages as the learned controllers but requests a recheck whenever the answer check would pose a question. On SeqQA and seqqa2, JEV differed from the self-verification control by +0.002 [−0.010, +0.013] and −0.020 [−0.043, +0.003] in accuracy, at 6.5% and 10.2% higher cost. H2 is therefore not supported for JEV. LAYA version 0 and the general-LLM controller differed from the self-verification control by +0.005 and +0.005, and by +0.007 and −0.030, in accuracy on SeqQA and seqqa2, with every interval including zero, an exploratory comparison.

Against the baseline, on SeqQA, seqqa2, BixBench-Verified-50 and SciAgentArena together, LAYA version 0 turned 13 wrong answers right and 21 right answers wrong. JEV did so for 15 and 38 answers, and the general-LLM controller for 16 and 40 (exact McNemar test [57] with Holm correction across benchmarks; Extended Data Table 4). Among the McNemar tests, after Holm correction the seqqa2 losses of the rule-based controller (0 gained, 12 lost, P = 0.002), the self-verification control (2 gained, 14 lost, P = 0.021), JEV (1 gained, 21 lost, P < 0.001), LAYA version 0 (0 gained, 10 lost, P = 0.010) and the general-LLM controller (1 gained, 25 lost, P < 0.001) remained significant, and no other test did. The seqqa2 losses of JEV and the general-LLM controller were concentrated in the synonymous-mutation and molecular-weight families, and changes fell on items of medium and high baseline difficulty (Extended Data Figure 4). Because the loss is visible decision by decision, it can be located. On the 23 development items on which JEV lost answers that the baseline got right, the agent loaded the convention skill in 13 runs under JEV against 16 in the baseline.

Paired score change against the baseline by difficulty stratum, trajectory length, and seqqa2 task family

Extended Data Figure 4. Where the controller helps. Paired score change against the baseline by difficulty stratum (pass rate over the other controller arms) (a), by tertile of trajectory length in agent steps (b) and per seqqa2 task family, 15–20 tasks per family (c). Strata with fewer than ten tasks are not shown. Error bars, 95% cluster-bootstrap intervals, clipped at the axis limits.

The provenance records locate where effects arise (Figure 3c). For each decision point we report how often it fired, how often the controller acted, how often the agent adopted the action and the outcome difference between adopted and non-adopted cases, a descriptive quantity because adoption is not randomized. The brief fired on every task, and the agent loaded a suggested skill after 38% of briefs under JEV and 7% under LAYA version 0. The answer check requested a recheck on 8.0% of its firings under JEV. The evidence check fired on at most one task per benchmark, and the long-computation check acted on 33% of its firings under the rule-based controller and 14% under the general-LLM controller on SciAgentArena. Workspace-bounds and convention-in-code reminders were adopted in 82% to 100% of cases (Figure 3c).

Gated retraining couples a data-synthesis pipeline with a promotion gate

LAYA is retrained in gated rounds (Figure 1c, Figure 4a). Each round runs the data-synthesis pipeline, which turns an executable curriculum into decisions with verifiable labels, mines hard examples from the parent version’s errors, trains a candidate, fits a temperature-scaling calibration[58] for it, and tests it against a promotion gate. A promoted version controls the agent and is the parent whose errors the next round mines, so each round starts from the last promoted version.

Gated retraining of the fast decision model

Figure 4. Gated retraining of the fast decision model. a, One round of retraining: the data-synthesis pipeline (verified synthesis from an executable curriculum), leakage firewall, hard-example mining with a random-subset control of equal budget, training and calibration, and the promotion gate, whose conditions are given in equations (3) and (14); promoted versions enter the registry and the next round mines their errors. b, Gate-split accuracy per question of the zero-shot public checkpoint (version 0), the majority class (the most frequent training label) and the rule-based controller. Skill choice is scored against the single constructed label, and skill relevance pools all skills. c, Training raised accuracy on every decision question. Left, gate-split accuracy per question of version 0 and the first trained version, rsi-v2, with the majority class and the rule-based controller, with skill choice scored against each state’s set of valid skills, as the gate does. Right, accuracy of version 0 and rsi-v6 on the agent’s own decision states from held-out benchmark items (analysis family on seqqa2; error class labelled by execution, pooled over the held-out halves of seqqa2 and protocolqa2), with paired 95% cluster-bootstrap intervals.

Round 1 draws its decisions from a synthetic curriculum of 1,517 executable bioinformatics tasks in 25 families, each with two independent implementations, metamorphic checks[59] and a verifier that accepts its reference answer and rejects constructed negatives. Four generators turn these tasks into decisions with the controller’s own state renderer and questions, so that training states have the form of deployment states (Extended Data Table 1). Labels come only from construction, a verifier or code execution. This follows programmatic supervision[60] but needs no label model, because these labels are exact. A fault example is kept only if running it fails. Round 1 builds every decision from templates. A leakage firewall admits only rows with these label sources, no lineage through the comparator and no overlap with the benchmarks’ task text[61, 62] (Methods).

QuestionGeneratorLabel sourceTrainCalGateOODMajorityVersion 0 (gate)Rule (gate)
Analysis familyG1construction8451154281100.920.260.79
Answer typeG1construction8451154281100.490.400.48
Skill choiceG2construction4281154281100.690.020.68
Skill relevanceG2construction5,8921,5505,7391,4300.870.130.05
Answer formatG3verifier44052212620.500.440.66
Error classG5execution4,8087291,47400.080.430.85

Extended Data Table 1. Round-1 synthetic decisions. Question rows per split before the firewall (rows that share a state are counted once per question), gate-split accuracy of the majority class (the most frequent training label; ties broken lexicographically), and gate-split accuracy of the zero-shot checkpoint (version 0) and the rule-based controller after the firewall. Question rows exceed row files because one state can carry several questions. Skill choice is scored against the single constructed label. Under the gate’s scoring against each state’s set of valid skills, version 0 scores 0.04 and the majority class 0.89 (Figure 4c). Row files before the firewall: 12,413 training, 2,561 calibration, 8,281 gate and 1,712 out-of-distribution; the firewall removed 16 calibration and 16 gate rows, leaving 2,545 and 8,265. Accuracies are on the 8,265 gate rows after the firewall (427 rows for analysis family, answer type and skill choice, and 5,725 skill-relevance questions); skill relevance pools all skills.

A loss-weighted variant of hard-example mining[63] samples rows within each question with a weight that increases with the parent version’s loss and doubles for confident errors. Each candidate is paired with a random-subset control trained on the same number of rows per question, which gives the gate a baseline of equal training budget. Let Q∗Q^{*} be the questions with headroom, on which the rule-based controller scores at most 0.95. A candidate θ′\theta^{\prime} replaces its parent θ\theta only if Q∗≠∅Q^{*} \ne\varnothing and, for every q∈Q∗q \in Q^{*},

Q0.025[Δ^q∗]>0,ECE⁡q(θ′)≤0.10,acc⁡q(θ′)>max⁡{acc⁡q(rule),acc⁡q(random),acc⁡q(majority)}.(3)\begin{aligned} Q_{0.025}\left[\widehat{\Delta}_{q}^{*}\right] &> 0,\qquad\operatorname{ECE}_{q}(\theta^{\prime}) \le0.10,\\ \operatorname{acc}_{q}(\theta^{\prime}) &> \max\left\{\operatorname{acc}_{q}(\mathrm{rule}),\operatorname{acc}_{q}(\mathrm{random}),\operatorname{acc}_{q}(\mathrm{majority})\right\}. \tag*{(3)} \end{aligned}

where Δ^q=acc⁡q(θ′)−acc⁡q(θ)\widehat{\Delta}_{q} = \operatorname{acc}_{q}(\theta^{\prime}) - \operatorname{acc}_{q}(\theta) is computed on identical states and Q0.025Q_{0.025} is the lower 2.5% quantile of its cluster-bootstrap replicates Δ^q∗\widehat{\Delta}_{q}^{*} over entities. ECE is the expected calibration error of the uncalibrated candidate, and “random” is the random-subset control. The full rule (equation (14)) adds a minimum of 200 labelled rows per question, a regression limit of one point and a refusal of candidates whose answers collapse onto one option. Release gates have been proposed for self-improving agent runtimes as an engineering safeguard[64], and here the gate is also part of the scientific claim. The gate releases a version only if it beats its parent, the rule-based controller, the majority class and a random-subset control trained on the same number of rows, and every trained version, its data, checkpoint and verdict are kept in an append-only registry. When a round learns from execution-labelled benchmark states, each benchmark task is answered only by a version whose lineage never trained on its group, an out-of-fold scheme analogous to cross-fitting[65] and stacked generalization[66].

The first round shows why the gate needs a majority-class baseline. Decision labels are imbalanced: on the gate split, 92% of analysis-family labels belong to a single class and 87% of skill-relevance labels are negative (Extended Data Table 1). Version 0, the public checkpoint used zero-shot, scored 0.26 on analysis family against 0.92 for the majority class and 0.79 for the rule-based controller, and 0.02 on skill choice against 0.69 and 0.68 (Fig. 4b). The skill-choice score is also below the chance level of about 1/16 for 16 options. On answer type and answer format the checkpoint scored 0.40 and 0.44, against majority-class rates of 0.49 and 0.50, and only on error class, with ten classes, did it exceed the majority class (0.43 against 0.08; rule-based controller 0.85). The rule-based controller is itself below the majority class on four of the six questions, so it cannot expose a candidate that learned only the label prior. Under such imbalance accuracy misleads, the accuracy paradox[67]: a candidate that always gave the most frequent training label would score 0.92 on analysis family and 0.87 on skill relevance, tens of points above version 0, and of the baselines in equation (3) only the majority class is certain to refuse it. Balanced accuracy[68] corrects for the imbalance, but the gate’s verdict rests on accuracy against the three baselines.

A judgement enters retraining as soon as it can be labelled without a model. Round 1 therefore trains the judgements that construction, verifiers or execution label on the curriculum: analysis family, answer type, skill choice and relevance, answer format and error class.

Training raised accuracy on every decision question, and the gains carried over to the agent’s own states

The first trained version, rsi-v2, was trained on 2,500 rows of curriculum decisions and raised accuracy over version 0 on all six questions of the gate split, with lower bounds above zero. Under the gate’s scoring, in which a skill choice is correct if it is among the state’s valid skills, analysis family rose from 0.26 to 0.99, skill choice from 0.04 to 0.86, skill relevance from 0.13 to 0.98, error class from 0.43 to 0.91, answer type

VersionTraining dataGate splitΔ\Delta v0Held-out states
Version 0public checkpoint, zero-shot——seqqa2 family 0.056; error class 0.174
rsi-v22,500 rows of curriculum decisions6 questionspositive on all 6 (family 0.26 → 0.99)—
rsi-v5synthetic protocol decisions, labels by construction4 questionspositive on all 4 (answer type 0.24 → 1.00)—
rsi-v61,000 new synthetic rows and 200 replay rows per arm, mix set by controllers’ errors on the development halves250 rows per question, 4 questions+0.452 to +0.868, lower bounds above 0seqqa2 family 1.000; error class 0.408

Table 3. Versions of the decision model. Every version was judged under gate-rules@2 (equations (3) and (14)). Δ\Delta v0, gain over version 0 on the gate split with paired 95% cluster-bootstrap intervals. Held-out states, accuracy on the agent’s own decision states in held-out benchmark runs.

from 0.40 to 0.93 and answer format from 0.44 to 0.85 (Figure 4c). Its calibration error was at most 0.08 on every question.

Further rounds raised accuracy in the same way (Table 3). rsi-v5, trained on synthetic protocol decisions, raised accuracy over version 0 on all four questions of its gate split, with lower bounds above zero: analysis family from 0.11 to 0.53, answer type from 0.24 to 1.00, skill choice from 0.04 to 0.46 and skill relevance from 0.53 to 0.68. rsi-v6 was trained on a curriculum shaped by the agent’s own trajectories: its training mix, the number of rows per decision type, was set by the decision errors that the controllers made on the development halves of seqqa2 and protocolqa2 in the main run. Its rows came from the curriculum generators and the verified synthetic protocols with labels by construction, so no development item became a training row, and each arm was trained on 1,200 rows, 1,000 drawn for it and 200 shared replay rows (Methods). On a held-out gate split of 250 generated decisions per question, it raised accuracy over version 0 on all four questions, with lower bounds above zero: analysis family by 0.868 [0.819, 0.912], answer type by 0.656 [0.605, 0.707], skill choice by 0.844 [0.799, 0.885] and skill relevance by 0.452 [0.365, 0.532]. Because decision labels are imbalanced, we also report balanced accuracy, the mean recall over classes: every trained version exceeded the majority class on every question (24 of 24 question–split pairs, +0.12 to +0.81).

These gains carry over from generated decisions to the agent’s own states. Because only the decision model changes, a trained version can be scored on the agent’s own decision states from the held-out halves, whose items training never drew on (Methods). On these states, rsi-v6 raised analysis-family accuracy on seqqa2 from 0.056 to 1.000 (+0.944 [+0.908, +0.974]; calibration error from 0.449 to 0.015; one label value) and execution-labelled error-class accuracy from 0.174 to 0.408 (item-level paired difference +0.186 [+0.045, +0.329]), the two judgements on which version 0 was weakest on benchmark states (Extended Data Table 6). Training also made the decision model robust to option order: re-asked with the options in another order, the public checkpoint changed 37% [34%, 40%] of its answers, in line with the sensitivity reported for open-weight decision models [16], and rsi-v6 changed 6% [4%, 8%] (750 choice questions on gate states). On the same states, the version trained on rows selected by the parent’s errors changed about half as many answers as a version trained on a random subset of the same size (5.6% [4.2%, 7.5%] against 11.3% [9.3%, 13.8%]), so error selection made the judgements more stable. The quantity that training targets and the gate certifies is decision accuracy. Whether a correct decision changes a task’s score depends further on whether the agent acts on it and on the headroom of the judgement (Figure 2; Figure 3c), which the main run measures with version 0 as its LAYA arm.

The seqqa2 baseline gain comes from a convention skill layer, measured apart from the decision layer

Skills and the decision layer can both shape a judgement, such as which convention to report, so we measured their contributions separately. SeqQA, seqqa2, BixBench-Verified-50 and protocolqa2 ran with skill release r04b. The other benchmarks ran with release r03. r04b adds a general sequence-computation layer and a separately named convention layer for LAB-Bench to r03, together with a patch to the protocol-audit skill. The conventions in that layer were taken from the published LAB-Bench 2 scoring code. Skills were developed only on the development half of seqqa2, split by a hash of the item identifier, and the held-out half was tested once per release. On those 196 held-out items, at high reasoning effort, accuracy was 90.8% with the convention layer and 77.4% without it (151 of 195 gradable items). The held-out comparison for r03 comes from a medium-effort run (76.0%, 149 of 196). In a paired ablation under the same conditions, high and medium effort differed within noise (Methods). On a random set of 60 SeqQA items, accuracy was 60 of 60 with the convention layer and 58 of 60 without it. On BixBench-Verified-50, r04b and r03 both answered 43 of 50 items correctly. The gain on seqqa2 therefore comes from the convention layer, and the seqqa2 baseline of the main run is reported with that provenance.

Between the earlier round and the main run, the JEV effect on seqqa2 changed from +0.008 to −0.041, a change of −0.048 [−0.081, −0.016], most of it in the synonymous-mutation family, and on SeqQA from +0.020 to +0.001 (Extended Data Figure 5). On protocolqa2, which ran under both releases, r04b changed the baseline score by +0.030 [−0.013, +0.072] relative to r03, and the JEV effect changed from −0.018 under r03 to +0.031 under r04b, a change of +0.048 [−0.031, +0.127]. The r04b baseline took 91% longer per task, and the two series also differ in run time and in the frozen code tree, so this comparison does not isolate the skill effect. Between the earlier round and the main run, both the reasoning effort and the skill release changed. That comparison is reported as their joint effect.

Skill release and controller effect

Extended Data Figure 5. Skill release and controller effect. a, Controller effect on score (arm minus baseline) in the earlier run (hollow) and in the main run (filled), per benchmark and arm (paired tasks: SeqQA 594, seqqa2 386, BixBench-Verified-50 49, SciAgentArena 158). The two runs differ in skill release and in the agent’s reasoning effort. For protocolqa2 (124 tasks) the comparison is skill release r03 against r04b at the same effort, and the two series also

A domain profile extends the same harness to laboratory-protocol troubleshooting

Laboratory-protocol troubleshooting needs different judgements from computational analysis. In protocolqa2 the agent reads a protocol and a reported failure, and it must name the step that caused the failure and the change that fixes it. The judgements are of a different kind. On the development half (62 items, about 17 graded runs each across the baseline rounds and controller arms), 780 of 1,060 graded answers were correct, and 14 items held 191 of the 280 wrong answers. Most wrong answers blamed a salient but standard parameter or step, located the fault at the wrong step, or used the protocol’s controls only in part; others were distracted by a second oddity, restored the wrong original value, missed an absent or swapped reagent or trusted an altered reference value in the document. These are diagnosis errors rather than routing errors. The default harness was built for computational analyses: its analysis families and answer types do not describe a protocol fix, and its skill shortlist ranked a general protocol-repository skill ahead of the matching wet-lab skills.

We therefore built a protocol profile of the harness. It applies only to tasks whose workspace contains a protocol document, and on other tasks the harness is unchanged. The profile ships with the skill release as a catalog of ten wet-lab domains and their skills, and it adds the answer type “protocol fix”. For the answer it asks three closed questions: whether the answer names one step, reagent or parameter with a concrete change, whether the cited step matches the protocol text the agent extracted, and whether the change would explain and fix the reported symptom. A failed attempt to read the protocol document triggers a fixed reminder that lists the text-extraction tools available in the sandbox. As with every message, no catalog entry, template or reminder contains an answer or item-specific content.

The profile was designed after the protocolqa2 results of the main run were seen, and it was tuned on a development half of protocolqa2. It is therefore exploratory and outside the hypotheses. On the 62 held-out items, tested once and held out from tuning, although their main-run outcomes under r03 had been seen, JEV with the protocol profile answered 0.823 correctly, against 0.758 for the baseline under the same development skill release (5 items gained, 1 lost, sign test P = 0.22). In the main run, three rounds of JEV and of the general-LLM controller with the protocol profile changed the score by +0.011 [−0.023, +0.045] and +0.035 [−0.002, +0.073] relative to the baseline, at 11.7% and 15.0% less wall-clock time, and by −0.021 and +0.035 relative to the same controllers under the default harness. With the local decision model as judge, the protocol profile raised the score by +0.060 [+0.014, +0.108] relative to the baseline at 7.9% fewer steps, and by +0.068 [+0.012, +0.128] relative to the same model under the default harness (two rounds; Extended Data Table 5).

On synthetic protocol probes, LAYA version 0 chose wet-lab domains correctly, but its answers to the three answer questions ran in the wrong direction. It rated a generic answer as more specific than a specific one, so the profile has LAYA abstain on those questions. To train them we built laya-proto from synthetic protocol data with labels by construction. Claude Opus 5.5 wrote 128 of the 154 protocols from domain and topic prompts written by the authors, without benchmark text. DeepSeek-V4-Pro, at maximum reasoning effort, wrote the remaining 26 from the same prompts. Each protocol came with four to six single-substring faults, each with an observation and its correction. Two verifiers checked every fault: a structural check, and an independent diff check that the faulted protocol differs from the clean one at exactly one line, at the stated step. Labels are assigned by construction from the teacher-written fault and its correction, which the diff verifier checks, and no model assigns a label. The domain and skill come from the generating topic, and the answer type from the scenario template. The three answer questions are labelled from candidate answers built to be correct, to cite a wrong step, to misquote the right step, or to stay generic. Splits are by topic group, so no protocol or topic crosses splits. A bench firewall, whose corpus was never sent to the teacher, rejected any protocol that shared a title or a word 8-gram with the protocolqa2 questions or reference answers (0 of 154), and any row whose state shared a word 8-gram with those texts or with the staged protocol files (535 training, 167 calibration and 247 gate rows). The gate judges domain, answer type, skill choice and skill relevance, and the three answer questions are kept with their held-out protocols. laya-proto was judged by the gate as rsi-v5 (above).

Probe design poses the same closed decisions over measured state

Probe design separates measuring from judging in the same way as analysis. A germline target enrichment panel uses hybridization probes to enrich selected genomic regions before sequencing. Its probes are useful only if they bind their targets and nothing else, remain thermodynamically stable without folding on themselves[69], and cover each target evenly enough to be read[70, 71, 72]. Germline panels test all of these conditions, because paralogous gene families contain near-identical exons[73], exons can be GC-extreme or shorter than a probe, and population alleles introduce mismatches at binding sites. In Operon-S1, probe design is divided along the same lines (Figure 5a). Tools measure every sequence property: tiling with MrBait[74], genome-wide alignment with BWA-MEM[75] and BLAST[76], nearest-neighbour melting temperature, hairpin and self-dimer checks, and coverage. A slow agent plans and handles exceptions over a shared panel state, in a pre-synthesis design loop and a post-experiment redesign loop. What remains are closed choices over the measured state among eight design actions: keep, densify, relax the GC or melting-temperature window, lengthen, shorten, tighten the specificity screen, escalate to another assay, or stop. The decision layer poses these choices as closed questions over the measured state.

Germline target enrichment panel design

Figure 5. Germline target enrichment panel design. a, Division of labour: tools measure, a slow agent plans and handles exceptions in a pre-synthesis design loop and a post-experiment redesign loop, and the remaining decisions are closed choices among eight design actions. b, Mean per-target design and post-screen coverage (equation (17)) for two designs (90 nt, 45-bp step; 70 nt, 35-bp step), over all 123 targets (82 of which belong to the C4A/C4B pair) and over the 41 targets outside that pair. c, Measured causes of non-coverage and the design actions they map onto: identical paralogous exons (escalate), an exon shorter than a probe (shorten) and a GC-rich interval (densify). d, Laboratory capture of the 69-gene panel: per-sample on-target fraction, fraction of target bases at ≥30×\ge30\times and uniformity in 73 samples from seven batches (B1–B7, chronological); lines, medians; uniformity, fraction of target bases at ≥0.2×\ge0.2\times the sample’s mean depth. e, Exon footprint coverage of the initial single-pass design and of the agent’s design that was synthesized and captured, for 69 genes; point size, exon length; orange and blue, change of more than 10 points up or down; grey, within 10 points.

The first of these decisions, per-probe quality control, was measured directly. On 150 held-out keep-or-redesign decisions (75 of each label; 25 target groups held out from training; labels from the genome-wide specificity screen), the hosted decision model answered 0.913 [0.857, 0.949] correctly, against 0.500 for the public LAYA checkpoint, the chance level of this balanced set (Methods). One rule over visible features, fitted to the labels, reproduces all 3,859 keep-or-redesign labels in-sample, so this decision serves to check the layer against a known answer. The decisions that remain open are the method and parameters for a region, the attribution of low coverage of unknown cause, the protocol for GC-extreme regions and the choice among candidate panels.

On a 123-target, 31,855-bp germline panel on GRCh37[77], a default design of 549 probes (90 nt, 45-bp step) reached a mean per-target coverage of 88.4% (Figure 5b). After genome-wide specificity screening, 207 probes passed, 24 targets kept at least one passing probe and 16 were fully covered, and mean per-target coverage by passing probes fell to 15.8% (Methods). Because 82 of the 123 targets belong to the C4A/C4B pair, the other 41 targets are also reported separately: their mean coverage fell from 71.2% to 47.5%, and all 16 fully covered targets are among them. A denser design (70 nt, 35-bp step) covered every target base before screening, yet only 21 of 123 targets were fully covered after it (mean 20.4%; 61.2% outside C4A/C4B), and a targeted rescreen of the gaps raised fully covered targets from 16 to 19. Design coverage, post-screen coverage and capture are therefore different quantities, and the design summary alone overstates what a panel covers after screening.

The remaining gaps had three causes, each calling for a different action (Figure 5c): sequence shared with another locus, exons shorter than a probe and GC-rich intervals. In the complement C4A/C4B pair, 35 of 41 exon pairs are identical in sequence. Shared probes gave a mean per-target design coverage of 96.9% over the pair’s targets, and none passed screening, because no probe can tell the two genes apart. At four boundary targets in two high-identity paralog pairs, none of 1,342 candidate units yielded a gene-specific passing probe, whereas 571 of 839 passed at a more divergent target of one of these pairs. At eight further zero-coverage targets the best off-target similarity was 0.967–1.0, and an expanded screen of 1,049 candidates of 60–120 nt across 17 targets added 58 passing probes and no newly covered target. At such loci the appropriate action is to escalate to another assay. A 72-bp TP53 exon, by contrast, was not covered by any passing probe of the 90-nt design. The 70-nt, 35-bp-step design, which also widens the GC and thermodynamic windows, fully covered 10 of 11 TP53 targets, against 9 of 11 in the default design. A GC-rich interval in another target gene (71.9% GC; 46 of 71 120-bp windows above 70%) is the candidate for densification. When the same interval was densified in the 69-gene panel described below, 2 of 23 added 120-nt probes passed every thermodynamic threshold, and in a single spike-in experiment with one control and one supplemented library from reference DNA (NA12878)[78], unnormalized mean depth over that interval was 2,586× and 3,090×, a single paired observation.

Laboratory capture shows what such a panel achieves. A 69-gene germline panel of 120-nt probes, designed by the probe-design agent, captured 73 human DNA samples in seven batches, with an on-target fraction of 78.3 ± 5.5% (mean ± s.d.), a median of 99.46% of target bases at ≥30× and 86 of 87 genotype-derived calls concordant with an orthogonal assay in 29 samples (three calls each; 98.9%; exact 95% CI 93.8–99.97% treating calls as independent) (Figure 5d). Median on-target fraction ranged from 0.67 to 0.85 across batches, and uniformity also differed between batches. Compared with an initial design written in a single pass by a language-model agent without the structured pipeline, the agent’s design of that panel changed exon footprint coverage by more than 10 percentage points in 34 of 69 genes, raising 15 and lowering 19 (Figure 5e). Its largest gain, C4A/C4B from 4.0% to 97.2%, came from shared probes that capture both paralogs and cannot tell them apart, the same limit that the current design meets. The 69-gene design overlapped 111 of the 123 targets of the current design (95 fully, 16 partially).

Discussion

In long-horizon biomedical agents the limit has moved from computation to judgement: most of the original agent’s lost items and surplus time came from judgements about conventions, skills, evidence and answer format rather than from the computation. Operon-S1 separates judging from doing and makes these judgements a layer of their own. Each judgement can be checked against a label, the decision model can be exchanged on fixed states, every judgement is traced to its consequence, and the local model answered in

under a second at no request cost. With the local model as judge, the layer’s efficiency effects were specific to benchmarks, lower cost on SeqQA and fewer steps on protocolqa2, and the rule-based controller also reduced steps on SciAgentArena and seqqa2, which suggests that part of the value lies in posing the right question at the right point. Training on verifiable labels raised the accuracy of every trained version on every decision question, and the gains carried over to the agent’s own decision states on held-out items.

The fast decision model here takes the monitoring and control roles that Nelson and Narens [13] describe. It monitors the state of slower deliberation and controls its course at every recurring judgement, from scoping the task to checking the answer, a role distinct from routing between cheap and expensive models[17, 32]. Monitoring is measured by how well each judgement matches its label. Control is measured by the interventional contrast of each arm against the no-op control (equation (1)). In the original agent’s trajectories, headroom differed between judgements (Figure 2), and the judgements with the most headroom are those that execution can label, which suits a fast, calibrated model that abstains when unsure and learns from execution-derived labels. Protocol troubleshooting has its own map of judgements, and it enters through its own profile of the same harness. Probe design shows the same pattern: per-probe calls reduce to a rule, whereas choices of route, parameters and escalation remain closed decisions over measured state.

The design separates three quantities: whether a decision is right, whether the agent acts on it and whether the task is scored correct. Earlier work found the first and the last to diverge[36]. The no-op controller anchors the comparison, the provenance record links each decision to its outcome, and the promotion gate admits a new version only on evidence that the training loop does not itself generate (equations (3) and (14)). A scientific agent that distils its trajectories into skills[79] gains reusable procedures, and the controller adds a measured decision layer.

The operon describes this division of labour: its regulator reads a signal and switches genes without making their products, and a different regulator can be swapped in without changing the genes. The decision layer reads the state of the run, acts only through fixed messages and can be exchanged without changing the agent. The layer is defined by its questions, actions and labels. Six controllers, from rules to a general language model, ran in the same harness on the same states, so the decision model can be exchanged without changing the agent, and a domain enters through a profile of the harness rather than a new architecture. It is likely to help when three conditions hold: (1) a task runs for tens of steps, so that judgements about conventions, routes, evidence and stopping recur; (2) those judgements can be posed as closed questions over a summary of the state; and (3) their correctness can be verified by execution or construction, so that the controller learns without model-generated labels. Computational analysis, laboratory-protocol troubleshooting and probe design all meet these conditions. On short single-step questions the controller’s fixed start-up cost dominates, and where a rule reproduces every decision, the rule suffices.

The evaluation covers one slow-system family, its computational work and protocol troubleshooting. The general-LLM arm poses the same closed questions to the model that drives the slow system, which bounds how much of the effect depends on a separate fast model. The protocol profile was tuned on the development half of protocolqa2 and tested once on the held-out half. Skill releases differ between benchmarks, so effects are compared within a benchmark (Methods). In the panel application the current design is computational, and its laboratory reference is the capture of the 69-gene panel.

These properties matter more in the life sciences than in most computational fields. A biological conclusion turns on conventions that a task rarely states, such as the genome build, the coordinate system, the normalization, the threshold and the direction of an effect[8], and on attributions such as which step of a protocol failed. A wrong computational judgement costs a rerun, whereas a wrong judgement about a protocol or a probe design costs reagents, instrument time and an experimental cycle. Laboratory practice already answers this with records that let another scientist check each decision, and the provenance record gives an agent’s judgements the same form.

Through ProtoPilot’s protocol-to-instrument layer1, the same closed decisions, such as whether to densify, shorten or escalate, can be posed during wet-laboratory execution, where their labels come from the experiment itself. As agents move from analysis to instruments, the judgements that decide cost and outcome become a larger share of the work, and a decision layer that judges without doing, and that can be measured, replaced and trained under a gate, is the part of the system that has to be trusted.

Methods

Slow system

The slow system is the ProtoPilot agent runtime1 (frozen commits 7d92ccb, 7417042 and cb88472, each recorded with its batches) driven by DeepSeek-V4.1-Flash (API model deepseek-flash, build 1017, system fingerprint bc19ba97. . . ) with reasoning effort set to high and a 786,432-token context window, beyond which the runtime compacts history. Language-model judges ran at maximum reasoning effort. All arms share one system prompt. Within a benchmark, all arms share one skill release. Release r03 (87 skills, manifest SHA-256 b02db3b6. . . ) is used for BiomniBench-DA, SciAgentArena and OmicsBench. Release r04b (92 skills, manifest SHA-256 589b01cd. . . ) is used for SeqQA, seqqa2, BixBench-Verified-50 and protocolqa2. r04b contains r03 and adds three things. The first is a general sequence-computation layer: rules for using only the sequences a task provides, six-frame scans of genomic windows and numeric precision, plus skills for melting temperature, molecular weight, sequence composition and complexity, and coding-sequence primer design. The second is a separately named convention layer for LAB-Bench, whose conventions were taken from the published LAB-Bench 2 scoring code and reproduce it on 3,000 synthetic sequences. The third is a patch to the protocol-audit skill. The held-out comparison used r04 and its ablation r04-noB, which removes the convention layer and every reference to it. They differ from r04b and r04b-noB only by the protocol-audit patch. Skills were developed only on the development half of seqqa2 (204 items, by parity of a SHA-256 hash of the item identifier), and the held-out half (196 items) was tested once per release. In a paired ablation on 61 seqqa2 items (41 that an earlier configuration answered correctly and r03 at medium effort did not, and 20 controls), r03 answered 33 correctly at high effort and 31 at medium effort (5 items right only at high effort, 3 only at medium, sign test P = 0.73). With the skills of the earlier configuration, which quoted benchmark wording and were therefore not used, it answered 58 and 54. No effect of the effort setting on seqqa2 was detectable at this sample size (5 against 3 discordant items), whereas the skills of the earlier configuration raised the counts from 33 and 31 to 58 and 54. No release contains item identifiers, sequences, answers or paraphrased items. A leak scan against all 400 seqqa2 items found no sequence, identifier or 25-mer hits, and one 8-gram of generic task wording. Skills are loaded on demand, agents may install packages into a per-task prefix and shared environments are read-only. Per-task time limits are twice the official limit where a benchmark publishes one (SciAgentArena, BiomniBench-DA) and twice the adapter’s default otherwise (600 s for SeqQA and seqqa2, 1,200 s for protocolqa2), identical for all arms (Table 2), and timed-out tasks score zero. For seqqa2 the final 60 s of the limit are reserved to collect a last answer, and items answered in this window are labelled as finalized after timeout. A forced final step sends the same explicit final-answer instruction in every arm; it is part of the unmodified agent and therefore of the no-op arm, and its text is identical in every batch of the main table.

Motivating trajectories

The trajectories of Figure 2 (BixBench-Verified-50, SciAgentArena, seqqa2 and BiomniBench-DA) were recorded with the original agent in earlier rounds, under earlier skill and environment versions, on items that overlap the formal evaluation, and were used only to motivate the design. The values of ω\omega are single-run, time-weighted descriptive values; for SciAgentArena and seqqa2 the post-answer time is divided by the wall-clock time of all tasks, which bounds ω\omega from below. The skill-routing counts come from an earlier, smaller skill release. Each wrong BixBench-Verified-50 answer was read in full and assigned to one class. In these trajectories one item, whose reference range has its bounds reversed, could not be graded, so 49 of the 50 items are gradable. In the formal evaluation that range is graded between its two end points in every arm (Benchmarks, deviation (x)).

Decision points and states

A decision point is a tuple

d=(trig⁡d,ρd,{(qd,j,Od,j,κd,j)}j∈Jd),(4)d = \left(\operatorname{trig}_d,\rho_d,\{(q_{d,j},O_{d,j},\kappa_{d,j})\}_{j\in J_d}\right), \tag*{(4)}

where trig⁡d(ht)∈{0,1}\operatorname{trig}_d(h_t) \in\{0,1\} decides whether the point fires on the history ht=(a1,o1,…,at,ot)h_t = (a_1,o_1,\ldots,a_t,o_t) of agent actions and tool observations, ρd\rho_d renders the history into a state s=ρd(ht)s = \rho_d(h_t), and each sub-question qd,jq_{d,j} has a finite option set Od,jO_{d,j} and a type κd,j∈{choice,score,yes/no}\kappa_{d,j} \in\{\text{choice},\text{score},\text{yes/no}\}, with Od,j={0,1}O_{d,j} = \{0,1\} for yes/no questions and four ordered levels for the score question of the long-computation check. Table 1 lists every decision point. States are rendered by pure functions of the task prompt, workspace listing, environments, skill shortlist and message history; they contain at most 900 tokens (state and questions together at most 1,024), with a fixed task excerpt (the first 1,200 and last 600 characters). Skill candidates are the top 15 BM2580 matches over skill descriptions plus a no-skill option. States of a running task contain the last tool call’s name, command summary, status, exit code and duration, the tails of standard error (2,000 characters) and standard output (1,000), the budget fraction, loaded skills, a count of repeated commands and the tail of accumulated code (8,000 characters). Runs of 20 or more nucleotide characters are replaced by length placeholders. Option order is shuffled81 with a seed derived from the item and decision point and recorded. Conventions come from a registry of slots grouped by analysis type; each slot has a question, options with their meanings, patterns that detect when the task already states the convention and patterns that detect the value the agent used. The registry may not contain an option that is supported only by a proven benchmark defect.

Control policy

A controller ϕ\phi returns, for each sub-question, a distribution pϕ(⋅∣s,qd,j)∈P(Od,j)p_\phi(\cdot\mid s,q_{d,j}) \in\mathcal{P}(O_{d,j}), the set of distributions over Od,jO_{d,j}; the rule-based and self-verification controllers return point masses and the no-op controller returns nothing. All controllers share the policy, thresholds and messages below and differ only in pϕp_\phi, except that the rule-based controller cannot end a run through P7 (see below). A fixed policy maps the distributions and two deterministic state facts, files(ss) and best(ss) (defined below), to an action and a message,

αd=πd({pϕ(⋅∣s,qd,j)}j∈Jd,s)∈{brief,remind,force-final,recheck,abstain},m=μ(αd,y^)∈M∪{∅},(5)\begin{aligned} \alpha_d &= \pi_d\left(\{p_\phi(\cdot\mid s,q_{d,j})\}_{j\in J_d},s\right) \in\{\text{brief},\text{remind},\text{force-final},\text{recheck},\text{abstain}\},\\ m &= \mu(\alpha_d,\hat{y}) \in\mathcal{M}\cup\{\varnothing\}, \tag*{(5)} \end{aligned}

where y^j=arg⁡max⁡ypϕ(y∣s,qd,j)\hat{y}_j = \arg\max_y p_\phi(y\mid s,q_{d,j}), M\mathcal{M} is a finite set of templates written in advance and ∅\varnothing denotes no message. Abstention sends no message. Templates may insert the names of files, environments or conventions, never an answer value. The action force-final closes the tool loop and sends the final-answer instruction. A choice question yields an action only if its top option is both probable and separated from the runner-up,

p(1)≥τactandp(1)−p(2)≥δ,(6)p_{(1)} \ge\tau_{\mathrm{act}}\quad\text{and}\quad p_{(1)}-p_{(2)} \ge\delta, \tag*{(6)}

with τact=0.6\tau_{\mathrm{act}} = 0.6 and δ=0.2\delta= 0.2. Two decision points can end the tool loop. The evidence check asks whether the evidence answers every part of the task (pdonep_{\mathrm{done}}) and whether a required step is missing (pmissp_{\mathrm{miss}}); the long-computation check rates four options and uses the probability of stopping with the best result so far (pstop)(p_{\mathrm{stop}}):

πF1={force-final,pdone≥τdone∧pmiss≤τmiss∧¬files⁡(s),remind,pdone≥τdone∧pmiss≤τmiss∧files⁡(s),abstain,otherwise,πP7={force-final,pstop≥τforce∧best⁡(s)∧ϕ≠rule⁡∧¬files⁡(s),remind,pstop≥τstop and the first case does not apply,abstain,otherwise,(7)\begin{aligned} \pi_{\mathrm{F1}} &= \begin{cases} \text{force-final}, & p_{\mathrm{done}} \ge\tau_{\mathrm{done}} \land p_{\mathrm{miss}} \le\tau_{\mathrm{miss}} \land\neg\operatorname{files}(s),\\ \text{remind}, & p_{\mathrm{done}} \ge\tau_{\mathrm{done}} \land p_{\mathrm{miss}} \le\tau_{\mathrm{miss}} \land\operatorname{files}(s),\\ \text{abstain}, & \text{otherwise,} \end{cases}\\ \pi_{\mathrm{P7}} &= \begin{cases} \text{force-final}, & p_{\mathrm{stop}} \ge\tau_{\mathrm{force}} \land\operatorname{best}(s) \land\phi\ne\operatorname{rule} \land\neg\operatorname{files}(s),\\ \text{remind}, & p_{\mathrm{stop}} \ge\tau_{\mathrm{stop}}\text{ and the first case does not apply,}\\ \text{abstain}, & \text{otherwise,} \end{cases} \tag*{(7)} \end{aligned}

where files⁡(s)\operatorname{files}(s) marks tasks answered through output files (BiomniBench-DA and SciAgentArena), for which ending the tool loop would prevent any answer from being written, and best⁡(s)\operatorname{best}(s) that the state records a best result so far. The rule-based controller never ends a run through P7, because its stopping signal is a budget fraction rather than a judgement of progress, and it reminds instead. Thresholds were fixed in the decision-layer configuration before any formal benchmark run and are shared by all controllers: τformat=0.3\tau_{\mathrm{format}} = 0.3 (the answer check requests a recheck when the probability that the answer meets its stated format falls below it), τact=0.6\tau_{\mathrm{act}} = 0.6, δ=0.2\delta= 0.2, τdone=0.85\tau_{\mathrm{done}} = 0.85, τmiss=0.15\tau_{\mathrm{miss}} = 0.15, τstop=0.8\tau_{\mathrm{stop}} = 0.8, τforce=0.9\tau_{\mathrm{force}} = 0.9. The evidence check is asked at most once per task, there is at most one recheck, which requires at least 60 s of remaining budget and is not used for file-answered tasks, and the controller makes at most 60 requests per task, answering every trigger that fires at the same time in a single request. OmicsBench is a multi-turn session benchmark on which the decision layer is inert; it is run in the baseline arm only, and the controller arms run on the other six benchmarks.

No-op equivalence and attribution

Let req⁡ϕ(ht)\operatorname{req}^{\phi}(h_t) be the model request that the agent issues after history hth_t under controller ϕ\phi, and req⁡(ht)\operatorname{req}(h_t) that of the unmodified agent. On every history in its test suite, a regression test enforces

req⁡ϕ0(ht)=req⁡(ht),(8)\operatorname{req}^{\phi_0}(h_t) = \operatorname{req}(h_t), \tag*{(8)}

so that, apart from latency, the agent under the no-op controller behaves as the original agent, and equation (1) contrasts interventions that differ only in the controller. Budget fractions in the state are computed from wall-clock time in every arm. The provenance record supports a diagnostic decomposition of Δϕ\Delta_\phi over decision points,

Δϕ=∑dPr⁡[trig⁡d]Pr⁡[αd≠abstain⁡∣trig⁡d]Pr⁡[Ad=1∣αd≠abstain⁡,trig⁡d]γd+ξ,\Delta_\phi= \sum_d \Pr[\operatorname{trig}_d] \Pr[\alpha_d \ne\operatorname{abstain} \mid\operatorname{trig}_d] \Pr[A_d = 1 \mid\alpha_d \ne\operatorname{abstain}, \operatorname{trig}_d] \gamma_d + \xi,
γd=E[Y(ϕ)−Y(ϕ0)∣Ad=1,trig⁡d],(9)\gamma_d = \mathbb{E}[Y(\phi) - Y(\phi_0) \mid A_d = 1, \operatorname{trig}_d], \tag*{(9)}

where AdA_d indicates that the agent adopted the action. The remainder ξ\xi is zero if non-adopted actions have no effect, each decision point fires at most once and the effects of adopted actions at different points add. Because AdA_d and trig⁡d\operatorname{trig}_d are defined under ϕ\phi, γd\gamma_d is not identified from the data, and the adopted-versus-non-adopted contrast we report is descriptive.

Post-answer fraction

For equation (2), time is measured from the start of the task, ti∗t_i^* is the timestamp of the first tool observation whose output contains the submitted value after normalization of number formatting (numbers rounded to the submitted precision, sequences matched as substrings, trivial values not located), and II is the set of tasks for which such an observation exists; for SciAgentArena, ti∗t_i^* is the end of the first run of the solution script that exits successfully. ω\omega is a ratio of summed times.

Monitoring resolution

The evidence check is posed before the agent’s next model request, and the agent’s answer almost always arrives in its last message, so in the main run the check fired twice and no evidence-check state had a located answer time. The resolution measure below could therefore not be estimated. Monitoring is evaluated on the states recorded under the no-op controller, which never intervenes, by replaying them through each controller offline. Let S+S^{+} and S−S^{-} be evidence-check states in tasks that were scored correct, at times t≥ti∗t \ge t_i^{*} and t<ti∗t < t_i^{*} respectively. Both sets come from solved tasks and are separated by whether the answer is already in hand; because the evidence check fires at most once per task, the comparison is between tasks. The monitoring resolution of controller ϕ\phi, the area under the receiver operating characteristic curve 82 as used for metacognitive sensitivity 83, is the probability that it rates a state in which the answer is already in hand above one in which it is not,

AUROC⁡ϕ=Pr⁡[pdoneϕ(s+)>pdoneϕ(s−)]+12Pr⁡[pdoneϕ(s+)=pdoneϕ(s−)],(10)\operatorname{AUROC}_{\phi} = \Pr\left[p_{\mathrm{done}}^{\phi}(s^{+}) > p_{\mathrm{done}}^{\phi}(s^{-})\right] + \frac{1}{2}\Pr\left[p_{\mathrm{done}}^{\phi}(s^{+}) = p_{\mathrm{done}}^{\phi}(s^{-})\right], \tag*{(10)}

with s+∈S+s^{+} \in S^{+} and s−∈S−s^{-} \in S^{-} drawn independently; it is defined when both sets are non-empty. It is 0.5 for a controller that carries no information about the agent’s progress. Its calibration on the same states is reported as ECE. Replay uses the deployed per-request timeouts. Intervals are cluster-bootstrap intervals over tasks.

Controllers and serving

All controllers share one client, which sends the state and questions, does not follow redirects, applies per-request time budgets, retries once on rate-limit or server errors and validates the response schema. JEV, LAYA and the general-LLM controller have the same budgets: 30 s for each request, 45 s for each request of the brief, and 92 s for the brief in total. The general-LLM controller sends the same states and questions to DeepSeek-V4.1-Flash at high reasoning effort. A timed-out or failed request leads to abstention and never fails a task. Keys exist only in the orchestrator process and never in the sandbox. The hosted controller calls JEV [14] (served version jev-1.14.0 on every request). The time budgets of JEV and LAYA were raised to equal those of the general-LLM controller after the main run had started (Deviations during the main run). The local controller calls a server bound to loopback on dedicated CPU cores (four replicas of eight cores behind a local proxy), which agents cannot reach because the egress proxy refuses loopback and private addresses. LAYA version 0 is the public multilingual checkpoint [47] (revision e4e9ddf); trained versions are registered by name and served by the same process.

Synthetic curriculum

The curriculum has 25 task families covering sequence, alignment, primer, cloning, statistics and omics computations. After removal of four training tasks flagged by a contamination check, it contains 1,517 tasks (1,373 parent tasks and 144 perturbation siblings). Sequences come from public European Nucleotide Archive 84 references, and sequences that occur inside benchmark items are blacklisted. Each task has two independent implementations and metamorphic checks; before the removal, the verifier accepted every reference answer and rejected all 7,681 constructed negatives, and two independent recomputations, of 100 and of 200 tasks, found no errors (exact two-sided 95% upper bounds 85 3.6% and 1.8%; 1.5% one-sided for the 200 tasks). Splits are grouped by entity, with no entity in more than one split: 972 training, 201 development, 232 test and 112 out-of-distribution tasks. For retraining, about 10% of training entities, chosen by entity hash, form a calibration split and the rest the training pool; development and test together form the gate split (433 tasks); the out-of-distribution split is read by no selection step. Contamination checks against all benchmarks (8-gram overlap, canary strings, identifiers) found no further hits.

Decision synthesis

Four generators (G1–G3, G5) call the controller’s renderer and question functions. G1 labels analysis family and answer type from frozen maps. G2 labels skill relevance from a frozen family-to-skill map, with no-skill rows capped at 25%. G3 builds compliant and non-compliant final answers in a 1:1 ratio, and compliant answers must pass the task verifier. A fifth generator, G4, builds convention mismatches where a task’s constructed negatives map onto a registered slot; it is not used in round 1. G5 injects faults of ten classes into code run in a sandbox (unprivileged user, no network, 20-s timeout, address-space limit) and labels each case with the injected class, kept only when execution fails with that class; in round 1 it ran 10,000 template cases (1,000 per class), discarding cases that exited successfully or duplicated an existing state. Round-1 prose and fault code come from templates. In teacher variants, a language model writes prose around answer and convention placeholders that must be preserved verbatim and writes fault-inducing code; it never sees benchmark text and never produces a label, and its calls are cached by prompt hash and capped in cost. Round 1 produced 12,413 training, 2,561 calibration, 8,281 gate and 1,712 out-of-distribution rows before the firewall (Extended Data Table 1).

Leakage firewall

Every row rr with state ss carries a label-source tag λ(r)\lambda(r), a source benchmark, a generator and a lineage that names the batches and arms it derives from. A row enters training only if

admit⁡(r)=1[schema⁡(r)]1[¬comp⁡(r)]1[lineage⁡(r) declared]1[λ(r)∈Λ]×1[maxrun⁡(s)<20]1[h(s)∉H]1[G8(exc⁡(s))∩(G8(B)∖G8bp)=∅]=1,(11)\begin{aligned} \operatorname{admit}(r)={}&\mathbf{1}[\operatorname{schema}(r)]\mathbf{1}[\neg\operatorname{comp}(r)]\mathbf{1}[\operatorname{lineage}(r)\ \text{declared}]\mathbf{1}[\lambda(r)\in\Lambda]\\ &\times\mathbf{1}[\operatorname{maxrun}(s)<20]\mathbf{1}[h(s)\notin H]\mathbf{1}\left[G_8(\operatorname{exc}(s))\cap\left(G_8(\mathcal{B})\setminus G_8^{\mathrm{bp}}\right)=\varnothing\right]=1, \tag*{(11)} \end{aligned}

where schema requires every field to be present, comp marks lineage through the hosted comparator directly or through any batch that ran a comparator arm, Λ={construction,verifier,execution}\Lambda=\{\text{construction},\text{verifier},\text{execution}\} (no human-labelled rows exist), maxrun is the longest run of nucleotide characters, HH is the set of hashes of evaluation states, exc⁡(s)\operatorname{exc}(s) is the task excerpt of the state, G8(B)G_8(\mathcal{B}) is the set of word 8-grams in the task prompts of the six benchmarks evaluated when the round-1 data were built (protocolqa2, added later, is checked against the same rules) and G8bpG_8^{\mathrm{bp}} the boilerplate grams of the benchmark prompts that occur in at least max⁡(5,⌈0.02N⌉)\max(5,\lceil0.02N\rceil) of the NN prompts. In round 1, with no benchmark states in training, HH and B\mathcal{B} cover all benchmark items. When benchmark states enter training under cross-fitting, HH and B\mathcal{B} are restricted to the items of the held-out group. Duplicate state–question pairs are refused at selection. Counts of each rejection class are given in Extended Data Table 1.

Hard-example mining

Within each question, rows are sampled without replacement by taking the largest keys ui1/wiu_i^{1/w_i} with ui∼U⁡(0,1)u_i\sim\operatorname{U}(0,1) 86, with weight

wi=(12+12min⁡(ℓi,5)5)(1+1[y^i≠yi∧p(1),i≥τact]),(12)w_i=\left(\frac{1}{2}+\frac{1}{2}\frac{\min(\ell_i,5)}{5}\right)\left(1+\mathbf{1}[\hat{y}_i\ne y_i\land p_{(1),i}\ge\tau_{\mathrm{act}}]\right), \tag*{(12)}

where ℓi\ell_i is the parent version’s negative log-likelihood of the label (probabilities floored at 10−610^{-6}; ℓi=5\ell_i=5 if the parent gave no answer), y^i\hat{y}_i its top option and p(1),ip_{(1),i} that option’s probability (max(pp, 1−p1-p) for yes/no questions). In round 1 the parent is version 0. A total budget of 2,500 rows is allocated across questions by largest remainder. The random-subset control uses the same per-question quotas, sampled uniformly from an independent stream seeded by SHA-256(“13|random”).

Training and calibration

Models were fine-tuned from the public checkpoint with the vendor training tool [47] on CPU, using soft cross-entropy, shuffled options, maximum length 1,024 tokens (matching the served state budget), seed 13, an effective batch of 64 (micro-batch 1, gradient accumulation 64, 8 threads, identical for candidate and control) and 3 epochs, as an unprivileged user without network access and without swap. Candidate and control were trained under a 12-GiB memory cap with identical selections. Runs with fewer than 100 optimizer updates are refused and training logs are scanned for collapse warnings. A checkpoint’s identity is the SHA-256 of its sorted file hashes. Calibration uses temperature scaling[58]: each question qq receives a temperature

βq=arg⁡min⁡β∈G 1nq∑i∈Dcal,q−log⁡softmax⁡(log⁡pi/β)yi,G={0.25⋅16j/63}j=063,(13)\beta_q = \underset{\beta\in\mathcal{G}}{\arg\min}\ \frac{1}{n_q}\sum_{i \in D_{\mathrm{cal},q}}-\log\operatorname{softmax}(\log p_i/\beta)_{y_i}, \qquad\mathcal{G}=\{0.25\cdot16^{j/63}\}_{j=0}^{63}, \tag*{(13)}

with βq=1\beta_q=1 if nq<30n_q<30 or if the improvement in negative log-likelihood is at most 10−610^{-6}; thresholds are not refitted. The frozen benchmark code serves raw checkpoint probabilities, so each candidate is gated both uncalibrated and calibrated, and only an uncalibrated pass can promote a version into the benchmark arm.

Promotion gate

The gate judges each question kind q∈Qq\in\mathcal{Q} (relevance questions for different skills are pooled; convention fit is excluded in round 1; rules gate-rules@2, SHA-256 70459d6f. . .), with headroom set Q∗={q:acc⁡q(rule)≤0.95}\mathcal{Q}^{*}=\{q:\operatorname{acc}_q(\mathrm{rule})\le0.95\}. With Δ^q\widehat{\Delta}_q as in equation (3), computed on identical state hashes,

promote⁡(θ′)⟺Q∗≠∅∧ ∀q∈Q:nq≥200∧Δ^q≥−0.01∧mode⁡q(θ′)−mode⁡q(y)≤0.20∧ ∀q∈Q∗: conditions (3)∧ acc⁡q(random)≥acc⁡q(majority)−0.05,(14)\begin{aligned} \operatorname{promote}(\theta') &\Longleftrightarrow\mathcal{Q}^{*}\ne\varnothing\\ &\wedge\ \forall q\in\mathcal{Q}: n_q\ge200\wedge\widehat{\Delta}_q\ge-0.01\wedge\operatorname{mode}_q(\theta')-\operatorname{mode}_q(y)\le0.20\\ &\wedge\ \forall q\in\mathcal{Q}^{*}:\ \text{conditions (3)}\\ &\wedge\ \operatorname{acc}_q(\mathrm{random})\ge\operatorname{acc}_q(\mathrm{majority})-0.05, \tag*{(14)} \end{aligned}

where a question that the rule-based controller does not answer counts as incorrect for it, and mode⁡q\operatorname{mode}_q is the share of the most frequent answer and the majority control answers the modal training label. When the modal label share is at least 0.80 the collapse rule cannot bind, and such questions are protected only by the requirement to beat the majority class. Calibration error87 is

ECE⁡q=∑k=110∣Bk∣nq∣acc⁡(Bk)−conf⁡(Bk)∣\operatorname{ECE}_q=\sum_{k=1}^{10}\frac{\lvert B_k\rvert}{n_q}\left\lvert\operatorname{acc}(B_k)-\operatorname{conf}(B_k)\right\rvert

over ten equal-width bins of top-option probability; nqn_q counts the answered gate rows, empty bins contribute zero, and for yes/no questions only bins from 0.5 upwards can be occupied. Q0.025Q_{0.025} in equation (3) is the 2.5th percentile of a percentile cluster bootstrap88 (10,000 resamples, seed 13) over entities, or over fault templates for error classes. Verdicts other than promotion are distinguished: insufficient evidence (nq<200n_q<200), insufficient headroom (Q∗=∅Q^{*}=\varnothing) and reject. A random-subset control more than 0.05 below the majority class is unhealthy, and the verdict is then insufficient evidence. Control accuracy is computed only on rows that ask the question. A constructed candidate that always answers the most frequent training label, with one-hot or calibrated probabilities, fails the gate in every configuration (20 of 20). An audit found that the first version of the gate could pass a candidate that learned only the label prior, before that version judged any candidate. The revised rules (gate-rules@2, SHA-256 70459d6f. . .) judged every candidate reported here. Training rows of three curriculum families whose prompts contained their own answers were removed before training. For rsi-v6, decision errors of the non-comparator controllers were mined from the graded records of the development halves only (held-out records were skipped before they were opened). The counts per decision type set how many rows each type received, and the rows themselves were new curriculum tasks from fresh generator seeds and re-rendered verified protocols, with labels by construction. The candidate and its random-subset control each drew 1,000 rows from the same pool and shared 200 replay rows (1,200 training rows per arm), the gate split was held out from that pool, and the leakage firewall was applied before either arm was drawn. Offline gate replays use the same 30-s timeout for every controller. Option-order sensitivity was measured by replaying 1,000 gate states with every choice question asked again under a different option order and counting changed answers, with Wilson 95% intervals. Only the public checkpoint may serve as a root of the registry.

Cross-fitting

For the promoted arm, each benchmark task xx is assigned to a group g(x)=SHA-256⁡(key⁡(x)) mod Kg(x) = \operatorname{SHA\text{-}256}(\operatorname{key}(x)) \bmod K with K=5K = 5; the key is the data capsule for BixBench-Verified-50, the data directory for SciAgentArena and the subtask and item for the others. The promoted arm answers xx with

ψ(x)=the latest promoted version θ with g(x)∉trainedOn⁡(θ),(15)\psi(x) = \text{the latest promoted version }\theta\text{ with }g(x) \notin\operatorname{trainedOn}(\theta), \tag*{(15)}

where trainedOn⁡(θ)\operatorname{trainedOn}(\theta) is the union of the groups on which θ\theta and all its ancestors were trained; for version 0 it is empty, so ψ(x)\psi(x) always exists. The arm is a mixture of versions. The routing manifest is frozen and hashed before the arm runs. In the main run every task is answered by version 0. Benchmark states entered training only through a parity split of seqqa2 and protocolqa2, the even and odd values of SHA-256 of the item identifier: errors were mined only from the development halves, and in-loop evaluations use only the held-out halves (196 and 62 items).

Protocol profile and protocol training data

The protocol profile is selected per arm and applies per task only when the workspace holds a protocol document (a staged text, PDF or word-processor file whose name or prompt refers to a protocol). Without it, every request and every injected message is byte-identical to the default harness, which a regression test checks. On protocol tasks the evidence, long-computation, error, convention and out-of-bounds checks do not fire. The brief asks at most six questions, among them the domain, the skill, the answer type and the relevance of up to two candidate skills. The answer check requests one recheck when the probability that the answer is specific, or that the change explains the symptom, falls below 0.25. Whether the cited step matches the extracted text is recorded only, because on development transcripts it did not separate correct from incorrect answers. The answer check reads the protocol lines only from the agent’s own tool output, and the harness never reads task files itself. The profile arms ran JEV and the general-LLM controller for three rounds each, and LAYA version 0 for two rounds, on protocolqa2 under r04b. For laya-proto, the teacher was Claude Opus 5.5, called through its command-line interface without tools or project context at medium reasoning effort. DeepSeek-V4-Pro, at maximum reasoning effort, wrote the 26 protocols for which Opus returned no structured output. It wrote protocols, faults and facts, and it never produced a label. Each of the eleven domain classes (the ten wet-lab domains and a general class) has seven topic groups with two variants each, 154 protocols in all, and a SHA-256 hash of the group assigns four groups to training, one to calibration and two to the gate. The answer questions of gate and calibration protocols are written to separate held-out files and never enter the gate or calibration rows. Every row carries the lineage of its teacher call and no lineage through the comparator.

Comparator

JEV is used only as an inference-time comparator. Under the provider’s terms its outputs are never used as labels, rewards, curricula or selection signals, which the builder, gate and registry enforce through lineage checks. Conclusions about the local controller and its retraining do not depend on it.

Benchmarks and deviations from official settings

SeqQA and seqqa2 report accuracy; BixBench-Verified-50 uses the official open-answer protocol (first answer block, with a language-model judge for range answers); SciAgentArena reports mean correctness, OmicsBench the mean node score and BiomniBench-DA the mean rubric score (0–100). Timeouts score zero. Network policy follows each benchmark: BiomniBench-DA and OmicsBench had web access through a logged proxy that refused benchmark-data hosts, SciAgentArena and protocolqa2 were offline, and SeqQA, seqqa2 and BixBench-Verified-50 used an allowlist of scientific databases that excluded answer sources. Deviations: (i) BiomniBench-DA used the 50 tasks of its public release and ran natively in our sandbox rather than in its container harness, with 2 CPUs and 16 GiB per task; all 50 task images only copy the data directory, which we confirmed. (ii) Its judge script and rubrics were unchanged, but the judge model was DeepSeek-V4-Pro instead of Gemini 3.5 Flash, with maximum output raised from 8,192 to 32,768 tokens because the judge’s reasoning otherwise truncated its JSON. (iii) BixBench-Verified-50 range answers were judged by DeepSeek-V4-Pro. (iv) OmicsBench’s private test set is unavailable, so its three public tasks were run. (v) SciAgentArena used its 189 publicly reproducible tasks. (vi) BixBench-Verified-50 sets no memory limit; we used 16 GiB and 2 CPUs per task. (vii) Every batch runs on a frozen commit recorded with it. The main table combines two frozen commits that differ only in the time budgets of JEV and LAYA, and rounds started under earlier configurations are excluded. (viii) Time limits are doubled as stated under Slow system. (ix) One R annotation package that would otherwise download gene sets at run time was given a sealed, read-only local cache, identical for all arms. (x) BixBench-Verified-50 item a4d8a810 gives its reference range as (−0.064, −0.084), and we grade it on (−0.084, −0.064). (xi) protocolqa2 publishes no per-task limit, so its limit is twice the adapter’s default of 1,200 s, and its open answers are graded with the official labbench2 evaluation prompt by DeepSeek-V4-Pro. Its protocol files are staged from a local copy checked against SHA-256 digests. (xii) When the BiomniBench-DA judge ended a response without any text, the identical request was sent again, at most three times in total and within the verifier time limit. Every request is counted in the judge cost and recorded per task.

Deviations during the main run

The main run started at 01:48 (UTC+8) on 29 October 2026. It departed from its plan in the following ways. Each departure is recorded with the batches it affects, and the analysis rules were fixed before the affected results were computed. All times are UTC+8.

(a) Disk full. Between 01:58 and 02:40 the data volume filled twice. Two BiomniBench-DA tasks staged a full 17.9-GB copy of one count matrix per attempt, and 22 such copies accumulated. The volume was enlarged, and the interrupted batches were resumed under the same identities, configurations and commit. Graded records that showed a disk-full error (313 records, plus 18 that showed it only in the transcript) were not used. Each was replaced by a rerun of the same task under the same configuration, unless its whole batch had already been superseded by a later rerun. No affected task was left without a clean graded record. In a sensitivity analysis, every task with an attempt that started in the 01:58–02:40 window is removed from all arms of its benchmark.

(b) Isolation audit of protocolqa2. The frozen list of answer hosts had no entry for protocolqa2, so its batches were audited with the protocolqa2 hosts added (Evaluation integrity).

(c) Time budgets of JEV and LAYA. At 03:32 the JEV and LAYA batches on BiomniBench-DA and SciAgentArena were stopped at about 20% completion. They were rerun on a second frozen commit in which JEV and LAYA have the same time budgets as the general-LLM controller, 30 s for each request and 45 s for each request of the brief, instead of the shorter budgets of the first commit. In the main table, JEV and LAYA come from this commit and the other arms from the first. The two commits differ only in these budgets. The stopped batches are kept as evidence for the shorter budgets and are not in the main table.

(d) Concurrency on BiomniBench-DA. After the resume in (a), BiomniBench-DA ran at a concurrency of 3 instead of 6. The baseline, no-op, rule-based, self-verification and general-LLM batches ran partly at 6 and partly at 3, and the JEV and LAYA batches at 3 throughout. Because concurrency changes wall-clock time, the wall-clock and step changes of JEV and LAYA on BiomniBench-DA are shown for reference and do not count towards H1. Their score changes count towards H1.

(e) Host oversubscription and the seqqa2 relaunch. At about 05:30 the host load reached 845 on 96 cores. In the seqqa2 batches running at that point, 185 items had been graded, 36% to 62% of them per batch had reached the time limit, and the median wall-clock time per batch was 750 to 1,029 s, against 75 s in the relaunched baseline round. All seqqa2 batches were stopped and relaunched at lower concurrency under new batch identities, with the same configuration, skill release and commit. Only the relaunched batches enter the analysis, and the stopped batches are kept as evidence of the overload. A second SeqQA baseline round stopped at 13 of 600 items when the service manager timed out creating task slices. Its replacement was paused at 137 of 600 items and enters only σb\sigma_b.

(f) Paused baseline rounds. From 05:40, baseline rounds 2–5 on SciAgentArena, BiomniBench-DA and protocolqa2 were paused so that the first round of every arm could finish, and they were resumed later under the same identities. A baseline round enters the reference only if it covers at least 90% of the items of the most complete round. Extended Data Table 2 reports the number of rounds RR that entered on each benchmark. Where R<5R < 5, the between-round standard deviation and the margin use the actual RR, and where R=1R = 1 the margin is undefined and non-inferiority is not judged.

(g) Skill releases. Benchmarks use different releases (Slow system). Effects are reported per benchmark and never pooled across releases. r04b was developed on the seqqa2 development half, so the seqqa2 baseline of the main run may be optimistic. The protocolqa2 main results come from a rerun under r04b that started at 04:06. The first protocolqa2 run, under r03, is reported only as a skill ablation, which also differs in run time and commit.

(h) Protocol profile. The protocol profile of the harness was designed after the first protocolqa2 results were seen and was tuned on a development half of protocolqa2. Its arms, JEV and the general-LLM controller for three rounds each and LAYA version 0 for two rounds, ran on a third frozen commit that adds only the profile. They are reported as exploratory, outside Extended Data Table 2 and outside H1 and H2.

(i) Baseline and no-op. Baseline rounds run the original agent without the harness, and the no-op arm runs in round 1 only. The empirical check of equation (8) is therefore the no-op round against the mean of the baseline rounds.

(j) Earlier round. An earlier round at medium reasoning effort is used only for the motivating analyses and the effort comparison. During that round, test suites run on the same host killed tool processes of about 140 live tasks in three windows. These tasks are excluded from every arm of the earlier round. The main run is a new run at high effort, on code in which tests stop only their own units. In the same earlier round, the long-computation check never fired because its timing function rejected the decoded time stamps. Its controller effects therefore come from the other decision points. The main run uses the corrected code.

Evaluation integrity

Each task ran in new network, mount and process namespaces with a private process table, on a minimal root assembled from read-only bind mounts (system directories, the benchmark’s environments, reference data and the task’s skill snapshot) plus one writable workspace, as an unprivileged user with no capabilities and no privilege escalation. Home, temporary and cache directories were per task, and tasks run offline had no network socket. File tools resolve real paths and refuse writes through symbolic links, and graders that execute agent code run as an unprivileged user in their own process namespace. Before each round, a pre-flight probe had to fail every attempt to read other tasks, earlier rounds, gold answers, grader environments or host files; after each batch, an audit checked for requests that reached an answer host, file-tool paths outside the workspace, the skill snapshot and the agent’s own toolchain, new writes to host directories and private sentinels in transcripts and workspaces, and any of these voided the batch. Over the 96 batches of the main table (71 cells and 25 reruns after the disk-full incident; 11,880 records), none met a voiding condition: no request reached an answer host (the proxy blocked 3), and no host write or sentinel hit occurred. Rounds, arms and controller versions carry separate batch identities and cannot be merged or resumed across configurations. Because BiomniBench-DA permits web access, each run was audited after the fact against a table of source studies covering 14 of its 50 tasks; scores are reported over all and over unflagged tasks, and flagged counts are lower bounds. We checked that each language-model judge separates answers of different quality by submitting a strong, a deliberately weakened and an empty answer on three tasks per benchmark. The BiomniBench-DA judge scored them 100/44/0, 100/59/0 and 93/53/0. The BixBench-Verified-50 judge scored 1/0/0 on all three tasks in one run; in another it rejected one strong answer, a verdict it gave once in five gradings of that answer, and we report this variance with its scores. The check exposed a bug in our BiomniBench-DA bridge, which returned an error where the official script scores empty output as zero; it was fixed before these results. Every grader was audited with reproducing scripts. The official seqqa2 validators contain defects, among them water subtracted twice in molecular weight, primer checks on one strand, substring matching of gene names and a unit error in KmK_{\mathrm{m}}, and the BiomniBench-DA rubric parser drops negative levels. Main tables report official scores unchanged. Every completed batch in the main table, including the resource-failure reruns, passed the isolation audit, with no network request to an answer source and no file access outside the workspace, the skill snapshot and the agent’s toolchain.

Statistics

Each arm is paired with the baseline on the same tasks, with the per-task mean of the baseline rounds as reference (R=5R = 5 scheduled; Extended Data Table 2 gives the rounds that entered). 95% confidence intervals are cluster-bootstrap intervals over group keys (10,000 resamples, seed 13). Infrastructure failures are excluded and counted. Timeouts keep the official score, which is zero on BiomniBench-DA. Tasks missing from any baseline round or from the arm are excluded and counted. The non-inferiority margin 89 for benchmark bb is

εb=2.8 σbR,(16)\varepsilon_b = \frac{2.8\,\sigma_b}{\sqrt{R}}, \tag*{(16)}

where σb\sigma_b is the standard deviation of the per-round mean headline score of the baseline over the paired tasks, computed from every baseline round with at least 30 graded tasks, and RR is the number of reference rounds (Extended Data Table 2). A benchmark with R=1R = 1 receives no non-inferiority verdict. The constant 2.8 (z0.975+z0.80z_{0.975} + z_{0.80}, for α=0.05\alpha= 0.05 and power 0.8) is a power-calculation quantity, and we use it as a margin rather than as a detectable difference. For R=5R = 5 the margin is 1.25 σb1.25\,\sigma_b. The margin does not include the run-to-run variation of a single-round arm. A wider margin, 2.8 σb1+1/R2.8\,\sigma_b\sqrt{1 + 1/R}, which includes it, is reported alongside as a sensitivity analysis and is not used for any verdict. H1 holds if the paired relative change in wall-clock time or in steps has an upper bound below zero on at least four of the five hypothesis benchmarks, every hypothesis benchmark with a verdict is non-inferior (lower bound above −εb-\varepsilon_b) and controller costs are included. H2 holds if, on SeqQA and seqqa2, JEV beats the self-verification control with a lower bound above zero in score or an upper bound below zero in efficiency. Because H1 and H2 are decision rules over confidence intervals rather than tests with a single statistic, each is judged by its interval conditions. Exact McNemar P values are corrected across benchmarks within each arm by Holm’s method90. A hypothesis whose cells are incomplete at the data cutoff is reported as undetermined. Differences are reported in each benchmark’s own metric. H3 (which decision point carries the effect) is exploratory, as are all results on protocola2 and OmicsBench and all comparisons not named in the hypotheses. A single-round arm whose primary lower bound lies between zero and the margin receives a second round, and the two rounds are judged together, an adaptive design. Intervals resample tasks, and between-round variation of the baseline enters through σb\sigma_b. On SeqQA and seqqa2, later baseline rounds ran at a lower host load than the arms, so wall-clock changes there are paired with baseline round 1, which ran in the same period; scores and steps use all reference rounds. At the data cutoff, arms are paired on the tasks graded in the baseline reference and in every arm of the benchmark, a benchmark enters H1 only if these cover at least max⁡(30,N/2)\max(30, \mathrm{N}/2) of its N tasks, and σb\sigma_b may use the completed part of a paused baseline round with at least 30 tasks. The motivating trajectories used agent commit 1da1a4a and two earlier skill releases (45 and 83 skills).

Panel coverage

For a set T\mathcal{T} of targets gg and a probe set PP with footprints fp⁡(p)\operatorname{fp}(p), mean per-target design and post-screen coverage and the fully covered fraction are

\begin{aligned} C_{\mathrm{design}} &= \frac{1}{|\mathcal{T}|}\sum_{g\in\mathcal{T}}\frac{\left|\operatorname{bases}(g)\cap\bigcup_{p\in P}\operatorname{fp}(p)\right|}{|\operatorname{bases}(g)|},\\ C_{\mathrm{screen}} &= \frac{1}{|\mathcal{T}|}\sum_{g\in\mathcal{T}}\frac{\left|\operatorname{bases}(g)\cap\bigcup_{p\in P_{\mathrm{pass}}\operatorname{fp}(p)\right|}{|\operatorname{bases}(g)|},\\ F_{\mathrm{screen}} &= \frac{\left|\left\{g\in\mathcal{T}:\operatorname{bases}(g)\subseteq\bigcup_{p\in P_{\mathrm{pass}}}\operatorname{fp}(p)\right\}\right|}{|\mathcal{T}|}. \end{aligned}} \tag*{(17)}\end{aligned}

where Ppass⊆PP_{\mathrm{pass}}\subseteq P are the probes that pass genome-wide specificity screening. For the default design Cdesign=0.884C_{\mathrm{design}}=0.884, Cscreen=0.158C_{\mathrm{screen}}=0.158 and Fscreen=16/123F_{\mathrm{screen}}=16/123; for the 70-nt design Cdesign=1.0C_{\mathrm{design}}=1.0, Cscreen=0.204C_{\mathrm{screen}}=0.204 and Fscreen=21/123F_{\mathrm{screen}}=21/123. Restricted to the 41 targets outside C4A/C4B, Cdesign=0.712C_{\mathrm{design}}=0.712 and Cscreen=0.475C_{\mathrm{screen}}=0.475 (default) and 1.0 and 0.612 (70 nt). The 70-nt design also widens the GC window from 0.30–0.70 to 0.20–0.85 and relaxes the hairpin, dimer and background-hit limits.

Probe-design tools

Probes were tiled with MrBait v1.2.7. Specificity was screened genome-wide on GRCh37 with BWA-MEM v0.7.19 and BLAST 2.16.0 under fixed specificity thresholds, and coverage computed with bedtools v2.31.191 and mosdepth92. Capture libraries were processed with fastp93, BWA-MEM (v0.7.17), Picard MarkDuplicates94 and bamdst95 against targets padded by 60 bp; uniformity is the fraction of target bases with depth at least 0.2 times the sample’s mean depth. Melting temperature, hairpin and self-dimer checks used nearest-neighbour thermodynamics[69]. Per-probe keep-or-redesign decisions were split by target group into training, development and test sets of 2,818, 827 and 591 records, and the held-out evaluation set is a labelled slice of the test groups with 150 binary decisions (25 targets, 75 of each label). The label is the outcome of the genome-wide specificity screen for the probe. The hosted decision model answered each decision zero-shot as a closed two-option question, and the public LAYA checkpoint was scored on the same set.

Compute

All runs used CPUs only, on a shared 96-core, 182-GB host without GPUs, inside a dedicated control group capped at 90 cores and 160 GB without swap, with separate allocations for benchmark items, controller serving and training. Per-task core budgets were 1 (SeqQA, seqqa2), 2 (BixBench-Verified-50, BiomniBench-DA, protocolqa2) and 4 (SciAgentArena, OmicsBench). Baseline rounds and controller arms ran concurrently on the same host, and absolute wall-clock times depend on its load. Slow-system and judge costs were taken from provider usage and a fixed price table.

Data availability

The synthetic curriculum, the decision datasets, the protocol training data, gate reports, decision records and per-task run records are available from Genoria AI. The benchmark items are available from their original releases[49, 50, 51, 52, 53, 54, 55, 56].

Code availability

The decision harness, controller clients, curriculum and decision-synthesis code, training, calibration and gate code, the evaluation runtime and the analysis code are available from Genoria AI.

Methods references

Acknowledgements

We thank the developers of the benchmarks and open-source tools used in this work.

Author contributions

Genoria AI Team designed the study, built the system, ran the experiments, analysed the results and wrote the paper.

Competing interests

The authors are employees of Genoria AI.

Extended Data

SeqQA, seqqa2, protocolqa2, and open-answer accuracy (SeqQA, seqqa2, protocolqa2), and mean correctness (SciAgentArena, BiomniBench-DA) are judged on the reference set. Non-inferiority is the score change above −ε_b (equation (16)). Changes in wall-clock time, steps and cost are ratios of paired sums minus one. On SeqQA and seqqa2, wall-clock changes are paired with baseline round 1, which ran in the same period as the arms (Methods). Cost excludes agent tokens and controller requests and excludes grading. On BiomniBench-DA, the wall-clock and step changes of JEV and LAYA (ψ) are for reference only, because their batches ran at a different concurrency (Methods). ε_b uses the baseline rounds with at least 30 graded tasks (Methods); SciAgentArena has one reference round (R = 1), and its non-inferiority is not judged (–).

Extended Data Table 3 | No-op control against the baseline. Paired change of the no-op arm against the baseline in official score and wall-clock time, the between-round standard deviation of the baseline, the share of tasks whose outcome changed between baseline rounds, and the audit of injected messages for gold answers, per benchmark. No injected message contained the gold answer in the 2,361 runs whose answers can be matched as text.

BenchmarkTasksΔ score [95% CI]Δ wall (%)Δ steps (%)Baseline σ\sigma (rounds used for σ\sigma)Tasks changed between baseline rounds
SeqQA600+0.004 [−0.005, +0.013]−2.3 [−8.2, +3.9]−0.8 [−5.2, +4.0]0.001 (2)2.2%
seqqa2400+0.004 [−0.009, +0.017]−1.8 [−11.5, +8.1]−4.5 [−9.1, +0.2]0.007 (3)3.0%
protocolqa2125−0.000 [−0.061, +0.061]−2.2 [−8.9, +4.8]−3.6 [−8.4, +1.6]0.021 (5)13.1%
BixBench-Verified-5050−0.030 [−0.092, +0.023]+2.5 [−10.2, +15.9]+9.2 [−1.0, +20.0]0.014 (2)10.0%
SciAgentArena189+0.014 [−0.027, +0.049]−6.9 [−20.2, +5.9]+0.5 [−10.0, +9.6]0.010 (5)8.5%
BiomniBench-DA50+0.770 [−2.270, +3.840]+40.5 [+18.0, +66.4]−1.3 [−8.5, +6.5]0.979 (3)0.0%

Table 3 (PDF p. 36).

Extended Data Table 4 | Tasks gained and lost against the baseline. For each controller arm and benchmark, tasks right in both, gained (wrong to right), lost (right to wrong) and wrong in both, with exact McNemar P values, Holm-corrected across benchmarks. BiomniBench-DA has no binary correctness and is not listed.

ArmBenchmarkTasksRight in bothGainedLostWrong in bothPP (Holm)
No-opSeqQA6005847540.7741.000
No-opseqqa240037525180.4531.000
No-opprotocolqa2125821012210.8321.000
No-opBixBench-Verified-5050420260.5001.000
No-opSciAgentArena18910187731.0001.000
Rule-basedSeqQA6005868330.2270.906
Rule-basedseqqa2400368012200.0000.002
Rule-basedprotocolqa212582812230.5031.000
Rule-basedBixBench-Verified-5050431151.0001.000
Rule-basedSciAgentArena18999129690.6641.000
Self-verificationSeqQA6005826751.0001.000
Self-verificationseqqa2400366214180.0040.021
Self-verificationprotocolqa212583911220.8241.000
Self-verificationBixBench-Verified-5050423231.0001.000
Self-verificationSciAgentArena18910385730.5811.000
JEVSeqQA6005836651.0001.000
JEVseqqa2400359121190.0000.000
JEVprotocolqa212585119200.8241.000
JEVBixBench-Verified-5050411350.6251.000
JEVSciAgentArena18910078741.0001.000
LAYA v0SeqQA6005865360.7271.000
LAYA v0seqqa2400370010200.0020.010
LAYA v0protocolqa21258925290.4531.000
LAYA v0BixBench-Verified-5050421251.0001.000
LAYA v0SciAgentArena18910276741.0001.000
General LLMSeqQA6005866350.5081.000
General LLMseqqa2400355125190.0000.000
General LLMprotocolqa212584810230.8151.000
General LLMBixBench-Verified-5050422241.0001.000
General LLMSciAgentArena18998710740.6291.000

Table 4.

ComparisonTasksΔ score [95% CI]Δ wall (%)Δ steps (%)
JEV + protocol profile − baseline125+0.011 [−0.023, +0.045]−11.7 [−17.8, −5.0]−4.2 [−9.2, +1.0]
General LLM + protocol profile − baseline125+0.035 [−0.002, +0.073]−15.0 [−21.3, −8.4]−9.7 [−14.1, −5.4]
JEV + protocol profile − JEV (default harness)125−0.021 [−0.077, +0.035]−9.8 [−18.7, −0.2]+0.9 [−5.8, +7.8]
General LLM + protocol profile − general LLM (default harness)125+0.035 [−0.024, +0.093]−13.3 [−21.3, −4.6]+1.4 [−5.5, +8.5]
LAYA version 0 + protocol profile − baseline125+0.060 [+0.014, +0.108]–−7.9 [−12.3, −3.3]
LAYA version 0 + protocol profile − baseline, held-out half62+0.055 [−0.011, +0.129]–−8.9 [−15.5, −1.3]
LAYA version 0 + protocol profile − LAYA version 0 (default harness)125+0.068 [+0.012, +0.128]–+2.6 [−3.7, +9.4]

Extended Data Table 5. Protocol profile on protocolqa2 (exploratory). JEV, the general-LLM controller and LAYA version 0 with the protocol profile against the baseline and against the same controllers under the default harness, in score, wall-clock time and steps. Protocol-profile arms are averaged per item over their rounds: three for JEV and the general-LLM controller, two for LAYA version 0, whose wall-clock change is omitted because its batches ran in a different load period.

with Wilson 95% intervals. c, Post-answer share of all wall-clock time, a lower bound on ω\omega, per arm and benchmark, with 95% cluster-bootstrap intervals.

differ in frozen code and run time. The general-LLM arm did not run in the earlier run, and BiomniBench-DA is omitted (16 paired tasks). b, The same comparison per seqqa2 validator family (15–20 tasks per family).

References

  1. [1]Jiang, Y., Tang, W., Sun, H., Tang, Z., Hou, Y. et al. A self-evolving agentic system for automated generation and execution of biological protocols. Preprint at https://arxiv.org/abs/2606.31763 (2026).
  2. [2]Huang, K. et al. Autonomous biomedical research with an artificial intelligence agent. Science 393, eadz4351 (2026).DOI
  3. [3]Gottweis, J. et al. Accelerating scientific discovery with Co-Scientist. Nature 655, 487–496 (2026).DOI
  4. [4]Ghareeb, A. E. et al. A multi-agent system for automating scientific discovery. Nature 655, 497–505 (2026).
  5. [5]Li, O., Agarwal, V., Zhou, S., Gopinath, A. & Kassis, T. K-Dense Analyst: towards fully automated scientific analysis. Preprint at https://arxiv.org/abs/2508.07043 (2025).
  6. [6]Swanson, K., Wu, W., Bulaong, N. L., Pak, J. E. & Zou, J. The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies. Nature 646, 716–723 (2025).
  7. [7]Boiko, D. A., MacKnight, R., Kline, B. & Gomes, G. Autonomous chemical research with large language models. Nature 624, 570–578 (2023).DOI
  8. [8]Bourgon, R., Gentleman, R. & Huber, W. Independent filtering increases detection power for high-throughput experiments. Proc. Natl Acad. Sci. USA 107, 9546–9551 (2010).DOI
  9. [9]Cuadron, A. et al. The danger of overthinking: examining the reasoning-action dilemma in agentic tasks. Preprint at https://arxiv.org/abs/2502.08235 (2025).
  10. [10]Chen, X. et al. Do NOT think that much for 2+3=? On the overthinking of o1-like LLMs. Preprint at https://arxiv.org/abs/2412.21187 (2024).
  11. [11]Kahneman, D. Thinking, Fast and Slow (Farrar, Straus and Giroux, 2011).
  12. [12]Evans, J. St B. T. & Stanovich, K. E. Dual-process theories of higher cognition: advancing the debate. Perspect. Psychol. Sci. 8, 223–241 (2013).DOI
  13. [13]Nelson, T. O. & Narens, L. Metamemory: a theoretical framework and new findings. Psychol. Learn. Motiv. 26, 125–173 (1990).DOI
  14. [14]Almeida, D. Introducing System One models & Jev. TypeSafe AI Blog https://typesafe.ai/blog/introducing-system-one-models-and-jev (2026).
  15. [15]Deka, P. From retrieval to typed decisions: calibrated System One models from biomedical sentence encoders. Preprint at https://arxiv.org/abs/2610.02486 (2026).
  16. [16]Li, J. Fast models, slow evidence: a paired and self-audited evaluation of System-1 decision models for LLM agent harnesses. Preprint at https://arxiv.org/abs/2610.02267 (2026).
  17. [17]Chen, L., Zaharia, M. & Zou, J. FrugalGPT: how to use large language models while reducing cost and improving performance. Preprint at https://arxiv.org/abs/2305.05176 (2023).
  18. [18]Ong, I. et al. RouteLLM: learning to route LLMs with preference data. Preprint at https://arxiv.org/abs/2406.18665 (2024).
  19. [19]Ding, D. et al. Hybrid LLM: cost-efficient and quality-aware query routing. In Proc. 12th Int. Conf. Learning Representations (2024).arxiv.org/abs/2404.14618
  20. [20]Madras, D., Pitassi, T. & Zemel, R. Predict responsibly: improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems 31 (2018).
  21. [21]Lightman, H. et al. Let’s verify step by step. In Proc. 12th Int. Conf. Learning Representations (2024).arxiv.org/abs/2305.20050
  22. [22]Cobbe, K. et al. Training verifiers to solve math word problems. Preprint at https://arxiv.org/abs/2110.14168 (2021).
  23. [23]Uesato, J. et al. Solving math word problems with process- and outcome-based feedback. Preprint at https://arxiv.org/abs/2211.14275 (2022).
  24. [24]Wang, P. et al. Math-Shepherd: verify and reinforce LLMs step-by-step without human annotations. Preprint at https://arxiv.org/abs/2312.08935 (2023).
  25. [25]Zheng, L. et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (2023).arxiv.org/abs/2306.05685
  26. [26]Bai, Y. et al. Constitutional AI: harmlessness from AI feedback. Preprint at https://arxiv.org/abs/2212.08073 (2022).
  27. [27]McAleese, N. et al. LLM critics help catch LLM bugs. Preprint at https://arxiv.org/abs/2407.00215 (2024).
  28. [28]Geifman, Y. & El-Yaniv, R. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems 30 (2017).arxiv.org/abs/1705.08500
  29. [29]Chow, C. K. On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theory 16, 41–46 (1970).
  30. [30]Angelopoulos, A. N. & Bates, S. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. Preprint at https://arxiv.org/abs/2107.07511 (2021).
  31. [31]Quach, V. et al. Conformal language modeling. In Proc. 12th Int. Conf. Learning Representations (2024).arxiv.org/abs/2306.10193
  32. [32]Wu, T. & Lim, W. Y. B. REFLEX with Jev for efficient selective control in LLM agents. Preprint at https://arxiv.org/abs/2609.26532 (2026).
  33. [33]Deng, B., Fan, S., Zhang, H. & Xie, X. Jev for scientific decisions: evaluating semantic choices and their consequences. Preprint at https://arxiv.org/abs/2609.24965 (2026).
  34. [34]Cao, C. et al. SearchJev: a calibrated System-1 model for search agents. Preprint at https://arxiv.org/abs/2610.05107 (2026).
  35. [35]Zhang, A. et al. A general-purpose System One decision model for clinical records. Preprint at https://arxiv.org/abs/2610.27099 (2026).
  36. [36]AIMultiple. Laya typed-decisions, fine-tuned on browser decisions. Model card at https://huggingface.co/AIMultiple/laya-typed-decisions-browser-ft (2026).
  37. [37]Do Rosario, A. & Atef, K. Jev-as-a-Judge for RAG claim verification: confidence-gated cascading between a non-generative verifier and a frontier LLM. Preprint at https://arxiv.org/abs/2610.20634 (2026).
  38. [38]Shinn, N. et al. Reflexion: language agents with verbal reinforcement learning. Preprint at https://arxiv.org/abs/2303.11366 (2023).
  39. [39]Madaan, A. et al. Self-Refine: iterative refinement with self-feedback. Preprint at https://arxiv.org/abs/2303.17651 (2023).
  40. [40]Wang, G. et al. Voyager: an open-ended embodied agent with large language models. Preprint at https://arxiv.org/abs/2305.16291 (2023).
  41. [41]Zhang, J., Hu, S., Lu, C., Lange, R. & Clune, J. Darwin Gödel Machine: open-ended evolution of self-improving agents. Preprint at https://arxiv.org/abs/2305.22954 (2025).
  42. [42]Zelikman, E., Lorch, E., Mackey, L. & Kalai, A. T. Self-taught optimizer (STOP): recursively self-improving code generation. In Proc. 1st Conf. Language Modeling (2024).arxiv.org/abs/2310.02304
  43. [43]Novikov, A. et al. AlphaEvolve: a coding agent for scientific and algorithmic discovery. Preprint at https://arxiv.org/abs/2506.13131 (2025).
  44. [44]Guo, Y. Evidence-gated research: statistically controlled model adoption in adaptive search. Preprint at https://arxiv.org/abs/2610.01751 (2026).
  45. [45]Moser, B. B., Frolov, S., Nauen, T. C., Raue, F. & Dengel, A. The self-improvement problem. Preprint at https://arxiv.org/abs/2610.21646 (2026).
  46. [46]Jacob, F. & Monod, J. Genetic regulatory mechanisms in the synthesis of proteins. J. Mol. Biol. 3, 318–356 (1961).
  47. [47]Convai Innovations. Laya Multilingual (convaiinnovations/laya-multilingual) and laya software v0.4.15. https://huggingface.co/convaiinnovations/laya-multilingual (2026).
  48. [48]Love, M. I., Huber, W. & Anders, S. Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2. Genome Biol. 15, 550 (2014).DOI
  49. [49]Laurent, J. M. et al. LAB-Bench: measuring capabilities of language models for biology research. Preprint at https://arxiv.org/abs/2407.10362 (2024).
  50. [50]Laurent, J. M., Bou, A., Pieler, M., Igoe, C. et al. LABBench2: an improved benchmark for AI systems performing biology research. Preprint at https://arxiv.org/abs/2604.09554 (2026).
  51. [51]Mitchener, L. et al. BixBench: a comprehensive benchmark for LLM-based agents in computational biology. Preprint at https://arxiv.org/abs/2503.00906 (2025).
  52. [52]Phylo. BixBench-Verified-50. Dataset at https://huggingface.co/datasets/phylobio/BixBench-Verified-50 (2026).
  53. [53]Liu, T., Wang, A. X., Panescu, A., Chen, L. X. et al. Benchmarking AI agents for addressing scientific challenges across scales. Preprint at https://arxiv.org/abs/2606.12736 (2026).
  54. [54]Qu, Y., Lu, Y., Tu, X. et al. BiomniBench: process-level evaluation of LLM agents for real-world biomedical research. Preprint at bioRxiv https://doi.org/10.64898/2026.05.12.724604 (2026).
  55. [55]Phylo. BiomniBench-DA (public release). Dataset at https://huggingface.co/datasets/phylobio/BiomniBench-DA (2026).
  56. [56]Shaked, G. & Nagisetty, S. OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study-scale bioinformatics. Preprint at https://arxiv.org/abs/2610.21074 (2026).
  57. [57]McNemar, Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12, 153–157 (1947).pubmed.ncbi.nlm.nih.gov/20254758
  58. [58]Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. On calibration of modern neural networks. In Proc. 34th Int. Conf. Machine Learning (PMLR 70) 1321–1330 (2017).arxiv.org/abs/1706.04599
  59. [59]Segura, S., Fraser, G., Sanchez, A. B. & Ruiz-Cortés, A. A survey on metamorphic testing. IEEE Trans. Softw. Eng. 42, 805–824 (2016).DOI
  60. [60]Ratner, A. et al. Snorkel: rapid training data creation with weak supervision. Proc. VLDB Endow. 11, 269–282 (2017).
  61. [61]Kaufman, S., Rosset, S., Perlich, C. & Stillman, O. Leakage in data mining: formulation, detection, and avoidance. ACM Trans. Knowl. Discov. Data 6, 15 (2012).
  62. [62]Kapoor, S. & Narayanan, A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4, 100804 (2023).DOI
  63. [63]Shrivastava, A., Gupta, A. & Girshick, R. Training region-based object detectors with online hard example mining. In Proc. IEEE Conf. Computer Vision and Pattern Recognition 761–769 (2016).arxiv.org/abs/1604.03540
  64. [64]Soni, D. Falsifiable release gates for self-improving systems: standing invariants at scale. Preprint at https://arxiv.org/abs/2607.13070 (2026).
  65. [65]Chernozhukov, V. et al. Double/debiased machine learning for treatment and structural parameters. Econom. J. 21, C1–C68 (2018).
  66. [66]Wolpert, D. H. Stacked generalization. Neural Netw. 5, 241–259 (1992).DOI
  67. [67]Valverde-Albacete, F. J. & Peláez-Moreno, C. 100% classification accuracy considered harmful: the normalized information transfer factor explains the accuracy paradox. PLoS ONE 9, e84217 (2014).
  68. [68]Brodersen, K. H., Ong, C. S., Stephan, K. E. & Buhmann, J. M. The balanced accuracy and its posterior distribution. In Proc. 20th Int. Conf. Pattern Recognition 3121–3124 (2010).DOI
  69. [69]SantaLucia, J. Jr & Hicks, D. The thermodynamics of DNA structural motifs. Annu. Rev. Biophys. Biomol. Struct. 33, 415–440 (2004).pubmed.ncbi.nlm.nih.gov/15139820
  70. [70]Mamanova, L. et al. Target-enrichment strategies for next-generation sequencing. Nat. Methods 7, 111–118 (2010).DOI
  71. [71]Gnirke, A. et al. Solution hybrid selection with ultra-long oligonucleotides for massively parallel targeted sequencing. Nat. Biotechnol. 27, 182–189 (2009).DOI
  72. [72]Samorodnitsky, E. et al. Evaluation of hybridization capture versus amplicon-based methods for whole-exome sequencing. Hum. Mutat. 36, 903–914 (2015).DOI
  73. [73]Mandelker, D. et al. Navigating highly homologous genes in a molecular diagnostic setting: a resource for clinical next-generation sequencing. Genet. Med. 18, 1282–1289 (2016).DOI
  74. [74]Chafin, T. K., Douglas, M. R. & Douglas, M. E. MrBait: universal identification and design of targeted-enrichment capture probes. Bioinformatics 34, 4293–4296 (2018).DOI
  75. [75]Li, H. Aligning sequence reads, clone sequences and assembly contigs with BWA-MEM. Preprint at https://arxiv.org/abs/1303.3997 (2013).
  76. [76]Camacho, C. et al. BLAST+: architecture and applications. BMC Bioinformatics 10, 421 (2009).DOI
  77. [77]Church, D. M. et al. Modernizing reference genome assemblies. PLoS Biol. 9, e1001091 (2011).DOI
  78. [78]Zook, J. M. et al. Integrating human sequence data sets provides a resource of benchmark SNP and indel genotype calls. Nat. Biotechnol. 32, 246–251 (2014).DOI
  79. [79]S1-NexusAgent Team. S1-NexusAgent: a self-evolving agent framework for multidisciplinary scientific research. Preprint at https://arxiv.org/abs/2602.01550 (2026).

Paper details

vlnc:W_8pd9gst5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents