betaFeedback

WorksComputer Science

Openhagen Worlds: Calibrated Environments for Measuring and Training Scientific Discovery Agents

Abstract

Agents that do science must choose informative experiments and reason from their results, but most discovery benchmarks cannot separate that skill from recall, unstated scoring rules or shortcut policies. We introduce Openhagen Worlds, a suite of 27 procedurally generated environments (some with finite truth spaces), from classic puzzles of experimental reasoning to simulator-grounded physics and chemistry, in which an agent runs experiments under a budget and commits an answer that a deterministic verifier checks against a hidden truth. Every observation states the complete scoring rule, and each of the 178 difficulty levels is calibrated: it is admitted only if, on 64 development instances, a scripted oracle succeeds on at least 0.75, the strongest plausible non-deductive policy on at most 0.25 and each gated shortcut on at most 0.35. The 92 core levels span all 27 families and are sized to train small models; the 86 frontier rungs extend the 15 discovery worlds with harder questions and with campaigns of three, five and eight or more committed stages on one hidden system. On the same 84 withheld frontier instances, one per family and rung, DeepSeek V4.1 Flash solves 32 at its default reasoning effort with 16,384 tokens per call and 57 at its maximum effort with up to 393,216 tokens per call (oracle: 79); the headroom that remains is largest in eight-stage campaigns. On-policy expert iteration gives a 9B model modest gains: its core-tier success on trained families rises from 0.090 to 0.148, partly through better action format, with no detectable loss on GPQA Diamond. A hosted service plays the core tier's withheld instances.

Introduction

Reinforcement learning with verifiable rewards has made programmatic checks a central training signal for reasoning models [13, 19], and agents are now asked to do science: they are expected to choose informative experiments, take the results into account and commit to a conclusion. For such agents an environment is only useful if its reward measures that skill. The answer must be checked exactly against something the agent cannot look up, the conditions for success must be stated, no policy that skips the reasoning may score well, and the levels must stay hard as models improve.

Existing discovery environments cover a wide range of science and have shown how far agents are from autonomous discovery (Table 1). To our knowledge, none of them admits a level only after showing that scripted policies which skip the reasoning stay below a fixed success rate on it, and none shows the agent the complete rule it is judged by: tolerances, partial credit and judge rubrics stay inside the evaluator. Most end each episode with a single answer, and, to our knowledge, none chains several irrevocably committed answers on one hidden system, with later stages depending on earlier ones, so none tests whether an agent can carry its own conclusions through a campaign. And on the easiest levels, frontier models can already approach the ceiling: on NewtonBench, reasoning models reach 70–100% symbolic accuracy in the easiest setting, while in the hardest only two of them exceed 6% [50].

EnvironmentPrimary emphasisExperimentsTrainingGated levelsCampaigns
INTERACTIVE SCIENCE ENVIRONMENTS
ScienceWorld [42] [2022]Elementary science tasks in a text world••◦◦
DiscoveryWorld [17] [2024]End-to-end discovery tasks in eight topics•◦◦◦
EXPERIMENTAL DESIGN AND LAW DISCOVERY
BoxingGym [12] [2025]Design scored by information gain•◦◦◦
NewtonBench [50] [2026]Laws under counterfactual shifts•◦◦◦
PhysGym [5] [2025]Physics discovery with controlled priors•◦◦◦
DiscoverPhysics [44] [2026]Altered laws of motion in N-body worlds•◦◦◦
ENVIRONMENTS FOR TRAINING
SciDisco [45] [2026]Data-analysis environments for turn-level RL◦•◦◦
Reasoning Gym [40] [2025]Procedural reasoning tasks◦•◦◦
Openhagen WorldsCalibrated discovery across 27 families, two tiers••••

Table 1. Comparison of representative discovery and reasoning environments. Rows are grouped by their primary target. Experiments: the agent chooses its own experiments over several turns. Training: the work trains models on the environment. Gated levels: every level is admitted only after scripted non-deductive policies are shown to stay below a fixed success threshold on it. Campaigns: episodes that chain several separately and irrevocably committed sub-answers on one persistent hidden system under a pooled budget, later stages depending on earlier truths or commits. Entries for other works follow their published descriptions.

To fill this gap we introduce Openhagen Worlds, a suite of 27 procedurally generated environment families with 178 calibrated difficulty levels (§ 3). Each family hides a truth, publishes its full scoring rule in every observation and verifies a committed answer exactly or within a published tolerance. The families range from classic puzzles that isolate experimental reasoning, such as group testing and the counterfeit-coin problem, to simulator-grounded physics and chemistry. Each level is admitted by a calibration gate: on 64 development instances, a scripted oracle must succeed on at least 0.75 of them, while the strongest plausible non-deductive policy may succeed on at most 0.25 and every gated shortcut on at most 0.35; we call a level that passes the gate calibrated. The gate’s measurements also show what each family tests: with the oracle interpreting the readings, a fixed sequence of every allowed experiment solves 0.98–1.00 of core mechanism-discrimination development instances, where the difficulty lies in the analysis, but only 0.02–0.28 of rule-induction ones, where the experiments must be chosen (Table 15). The 92 levels of the core tier span all 27 families. The 86 rungs of the frontier tier extend the 15 discovery worlds with harder single questions (H1–H3) and with long-horizon campaigns of three, five and eight or more committed stages on one hidden system (L1–L3). Evaluation instances come from a key that is never distributed, and a hosted service plays them.

FamilyOracle d0Oracle d1Oracle d2Naive d0Naive d1Naive d2Shortcut d0Shortcut d1Shortcut d2Exhaustive d0Exhaustive d1Exhaustive d2
circuit fault0.980.980.910.110.120.050.020.000.00–
impedance topology0.940.970.980.110.110.050.000.000.00–
heat inclusion0.950.920.920.020.020.000.000.000.000.580.520.47
hidden law0.950.980.890.030.060.050.000.000.00–
control plant1.000.970.910.020.000.030.000.000.00–
reaction network0.950.910.910.000.030.020.020.090.030.560.450.50
mechanism discrimination†1.000.980.970.160.140.090.230.230.220.981.001.00
pathway fault1.001.001.000.060.030.060.110.060.02–
knockout network1.001.001.000.140.030.050.000.000.000.170.050.06
group testing1.001.001.000.190.080.030.020.030.03–
counterfeit coin†1.001.001.000.030.020.020.020.020.02–
rule induction1.001.000.890.020.000.000.000.000.000.280.250.02
causal intervention0.981.000.940.000.000.000.000.050.02–
trace debugging1.001.001.000.050.060.030.000.000.00–
automaton learning†1.001.000.970.000.020.000.000.000.00–

Table 15. Calibration of the discovery worlds’ core levels on 64 development instances per level (d0–d2). Shortcut: the strongest shortcut policy, including the answer prior. Exhaustive: every allowed experiment in a fixed order, then the oracle’s analysis, where the family defines it. † held out of all training.

Openhagen Worlds was built first as a training ground for smaller models. Its core tier is sized so that models of a few billion parameters find first successes and a learning signal, with the aim of shaping them into specialised scientific reasoners for narrow domains, where a small model trained for one kind of experiment can be cheaper and more reliable than a general one. That a frontier model approaches saturation on this tier is therefore expected: the pilot model solves all 45 of its core instances, and Flash solves 0.93 of the core episodes it finishes, though 0.37 of its episodes end without a legal action because its turns are cut at the per-call token limit. The frontier tier applies the same construction to harder rungs, admitted by the same gate. A pilot with a stronger model at xhigh reasoning effort, below its “max” setting, solves most of that tier, and so does Flash at its maximum reasoning effort, so difficulty has to be a ladder that keeps climbing. Probes on development instances show where it can climb: most longer campaigns admitted by the same gate hold the stronger model, while rungs that only add hidden structure do not (§ 7.1).

We evaluate scripted references, two locally run models built on open-weight Qwen models and a fast frontier model through its API on the withheld instances (§ 6). The core tier separates the local models from the oracle and approaches saturation for frontier models. On its discovery worlds, DeepSeek V4.1 Flash solves 0.58 of its episodes; most of the others end without a legal action because its turns are cut at the per-call token limit, and of the episodes it finishes it solves 0.93. On the frontier tier, Flash solves 0.27–0.49 of its episodes on each of the five rungs H1–H3, L1 and L2, and 0.11 on the eight-stage campaigns (L3), where only a window of recent records stays visible; the oracle solves 0.91–0.94 on every rung. Qwen3.8-27B reaches 0.04–0.16 there. Most failed episodes of both models end on a committed wrong answer rather than on malformed output; for Flash, whose thinking reaches its per-call limit of 16,384 tokens on 27% of frontier decisions, these endings do not separate reasoning errors from the budget, and episodes that end on a turn re-asked after the limit succeed on 0.18, against 0.69 for the others. Its frontier numbers therefore measure reasoning under that budget, and much of the difficulty is a budget effect: the o=0o = 0 frontier instances, one per family and rung. Flash at its maximum effort with up to 393,216 tokens per call solves 57 of 84, against 32 under the cap and 79 for the oracle (§ 6). A pilot with GPT-6.1 Sol via the Codex CLI’s “ultra” preset (server-reported reasoning effort xhigh), not the API protocol, on one final instance (o=0o = 0) per family and rung or level, succeeds on 0.85 of frontier episodes, +0.46 [+0.36, +0.57] above Flash capped at 16,384 tokens per call but +0.17 [+0.07, +0.26] above Flash at its maximum effort, and stays below the oracle, −0.10 [−0.18, −0.01], a gap that a bootstrap over families resolves only on the long-horizon rungs (Appendix § 19). The headroom that holds is therefore measured against the oracle, not against Flash’s capped rates: Flash at its maximum effort stays below the oracle over all rungs (−0.26 [−0.36, −0.18]), in the eight-stage campaigns the stronger model solves 8 of 14 (oracle 13; −0.36 [−0.64, −0.14]) and Flash at its maximum effort 7 of 14, and development-instance probes find more headroom in the longer campaigns of § 7.

Beyond aggregate scores, we analyse the episodes and train on the suite (§ 8 and § 9). Campaign accuracy falls stage by stage, and Flash runs about as many experiments as the 27B and succeeds far more often, under a larger per-call budget. On-policy expert iteration gives a 9B model modest gains on trained and held-out families, with no detectable regression in general reasoning on GPQA Diamond. In a separate study on two laboratory families outside the suite (§ 9.3), online reinforcement learning raises the same model’s success on held-out replication worlds from 0.221 to 0.721 in one training run, above Flash with the same tools at low and default reasoning effort (0.446 and 0.371) when Flash plays under the 9B’s per-turn and per-episode token caps; with up to 24,000 tokens per call, Flash stays well ahead on a broader benchmark of laboratory data analysis (0.849 against 0.605), and the registered checkpoint fails one of that benchmark’s registered guards. A run that splits its updates between replication and method-comparison worlds improves on both families. To summarise, our contributions are as follows:

  • We present Openhagen Worlds, 27 calibrated, procedurally generated discovery environments (with finite truth spaces in some families; Appendix § 17) with 178 difficulty levels in two tiers, each level admitted by a per-level calibration gate, with withheld evaluation instances, a hosted evaluation service and a public

sample.

  • We characterise the calibration and difficulty of both tiers, showing that every level separates deduction from every gated non-deductive policy, and that a stronger model at xhigh reasoning effort solves most of the frontier tier but stays well below the oracle on its longest campaigns.

  • We evaluate a fast frontier model and locally run models built on open-weight Qwen models on the withheld instances, with a stronger frontier model in a pilot, and report success, failure modes, decisions, cost per episode and the effect of the per-call token budget.

  • Through episode analysis and two training studies, we show how campaign errors compound, that on-policy expert iteration gives modest gains that partly reach held-out families, and that online reinforcement learning on laboratory worlds raises a 9B model’s success on held-out worlds of its training family well above its starting point and above a fast frontier model held to the 9B’s token caps, while that model stays well ahead at a larger budget on broader laboratory analysis.

Related Work

Discovery environments. Text and simulated worlds have long been used to evaluate agents that experiment. ScienceWorld poses elementary-school science tasks in an interactive text environment [42], and DiscoveryWorld asks agents to complete full discovery cycles across eight topics [17]. BoxingGym scores experimental design by expected information gain and model discovery by whether an explanation lets a second agent predict [12]. NewtonBench, PhysGym and DiscoverPhysics ask agents to recover physical laws from interactive experiments, often under shifted laws or withheld prior knowledge [5, 44, 50]. These benchmarks measure valuable skills, but to our knowledge none admits a level only after showing that non-deductive policies stay below a fixed success rate on it (ScienceWorld reports a random-action baseline and NewtonBench a test without observed data, neither used to admit levels), and none tests campaigns in which committed answers must be carried across stages.

Environments for training. Procedural generation separates training from evaluation instances [7], and Reasoning Gym applies it to verifiable reasoning tasks for reinforcement learning [40]. Xu et al. [45] build SciDisco, discovery environments with hidden evidence graphs, and train a 14B model with turn-level reinforcement learning; Kriz et al. [18] train the reasoning behind experiment choice with an information-gain reward; and Patel [26] finds that a forward computation distils into a student while a backtracking search procedure does not. Openhagen Worlds generates every instance, so the same families can be trained on and withheld, and it pairs held-out families with named trained relatives to test transfer.

Verifiable rewards and shortcuts. Benchmark builders remove or rewrite items that partial-input baselines or adversarial filters solve without the intended reasoning [14, 21, 49]. The calibration gate applies this idea per difficulty level, with scripted non-deductive policies in place of trained models. A policy optimised against a reward can also satisfy it in unintended ways [1, 38], so every episode in Openhagen Worlds records the experiments it ran, and a training filter can check the route to an answer as well as the answer. Canary strings let corpus builders exclude evaluation data [39]; because a generator distributed for training would otherwise also generate the test, we combine them with a private key for the evaluation instances.

Self-training. Expert iteration, which began as imitation of a search expert [2], and rejection-sampling fine-tuning train a model on its own verified or highest-reward samples [10, 27, 37, 47, 48]; on-policy training typically forgets less than fine-tuning on another model’s outputs [4, 35], and, in controlled card-game and navigation tasks, reinforcement learning generalised further out of distribution than supervised fine-tuning [6]. We apply this route to multi-turn agentic environments, with a capped share of teacher episodes rendered in the student’s own format. Group-relative policy optimisation replaces a learned value function with the mean reward of several attempts at the same problem [23, 34]; we use it for online reinforcement learning on laboratory worlds.

Benchmark Construction

Construction pipeline

Every family and level passes through the same four-stage pipeline (Figure 1). The stages are run by code, not by annotators: a family is a generator, a verifier, a set of scripted policies and a datasheet, and a level is admitted or rejected by the gate’s measurements alone.

Four-stage pipeline showing Design, Verify, Gate, and Freeze

Figure 1. The four-stage pipeline that admits every level of Openhagen Worlds. Stages 01 and 02 are built once per family; stages 03 and 04 run for every difficulty level, in both tiers.

Stage 1: Hidden truth and published rule. A family’s generator draws a hidden system (a resistor network with a fault, a kinetic scheme, a causal graph with a confounder, a program with a bug) from its level’s settings. The agent sees a public observation that is re-rendered every turn: the objective, the objects and their limits, the action grammar, the experiment budget and its use, the history of readings and the complete scoring rule, including every tolerance and every condition that ends the episode. Nothing that decides success is left unstated.

Stage 2: Exact verification and identifiability. The verifier scores a committed answer with reward 1 or 0. Discrete answers must match exactly; answers with many spellings are scored by semantic equivalence (a rule over number triples on all 1,000 triples of its domain, a program fix by its output on every input, an automaton by behavioural equivalence, a molecule by constitutional identity); continuous answers have published tolerances; and design answers, such as a controller, a chromatographic method or a process, are verified by simulating them on the true system. Identifiability filters reject instances whose required distinctions cannot be made with the allowed experiments, a failure Shetty and Braga-Neto [36] document in DiscoverPhysics [44].

Stage 3: The calibration gate. The gate runs scripted policies that read only the public observation. The oracle keeps the exact set of hypotheses consistent with the evidence, or a posterior over them, chooses the experiment that splits them best [3] and commits when one remains; it is an idealised reference, not a level models are expected to reach. The naive policy is the strongest plausible non-deductive policy for the family, such as probing in a fixed order or regressing on observational data. Shortcut policies are specific habits that could pass for skill, such as measuring at both safety limits, plus an answer prior that commits another instance’s correct answer. On the frontier tier the panel also includes the strategy that solves the next easier rung, a stage-by-stage policy that re-solves each stage of a campaign from scratch and, on the longest campaigns, an oracle without notes. A level is admitted only if, on 64 development instances, the oracle succeeds on at least 0.75, the naive policy on at most 0.25 and every shortcut on at most 0.35, and only if truths, oracle replays and instance digests are identical across processes. The exhaustive design, where a family defines one, runs every allowed experiment in a fixed order and then applies the oracle’s analysis; it is reported, not gated, and shows whether a family tests the choice of experiments or only the analysis of evidence. On the frontier tier, the other reported policies are likewise deductive: each applies the oracle’s exact analysis to a simpler design, such as the previous rung’s design, a design for the current stage only, or a fixed set of probes, and some succeed on more than 0.35 of development instances, which shows that the rung’s difficulty there lies in the analysis rather than in the design. One reported habit is not deductive: on control plant’s long-horizon rungs, tuning the outer loop as if the inner loop were perfect succeeds on 0.44–0.72 of development instances, because a fast inner loop often makes it good enough, and it is reported rather than gated.

Stage 4: Ladders and withheld instances. Admitted levels form a family’s ladder. An instance is a pure function of its family, split, level, index and key, so every episode can be replayed. Training and development instances use separate keys, and the final instances come from a third key that is never distributed; every model plays the same ones, and all comparisons are paired. Generators, level settings and verifiers are chosen on development instances; the final instances informed the design of the frontier tier only, which was tuned against a screen of one model on them (§ 3.2 and § 10). The hosted evaluation service plays exactly the core tier’s final instances and reveals only the observation, reward, termination and success.

Contamination controls. One concern runs through every stage: a generator distributed for training must not also generate the test. Beyond the private key, every distributed file carries a canary string and every observation an instance hash, famous instances (the twelve-coin puzzle, textbook rule-induction rules, named Bayesian networks and control plants) are excluded by construction, and a keyed filter rejects generated programs that match a protected program-execution benchmark (Appendix § 17).

Families, tiers and splits

Families. The 27 families form four groups (Figure 2; Table 13 in Appendix § 16, where Table 14 also defines the terms used throughout). The fifteen discovery worlds are the central families of the suite and the only ones with a frontier tier. Four of them are classic puzzles of experimental reasoning (group testing, counterfeit coin, rule induction and automaton learning), and the others idealise laboratory and reasoning problems, from fault finding in circuits and kinetic models to causal inference and program repair; we treat all fifteen as tests of experimental reasoning rather than of domain science. Two MicroWorld families test long exact computation and evidence auditing. The simulator-grounded science of the suite is in its last ten families: five PhysicsWorld families wrap simulators (rigid-body dynamics, a panel method, ray transfer, modal and hydraulic models), and five ChemistryWorld families model spectroscopy, equilibria, kinetics, chromatography and voltammetry. Each family has a datasheet that records what is hidden, the grammar and scoring rule, the difficulty settings, the calibration, the instance space, contamination notes and the simulator with its known limits. Two laboratory families outside the suite, used to train and evaluate a model by reinforcement learning, are described in § 9.3.

The 27 families of Openhagen Worlds in four groups, and the two tiers

Figure 2. The 27 families of Openhagen Worlds in four groups, and the two tiers. Every family has a core ladder; the 15 discovery worlds also carry the frontier tier of harder (H) and long-horizon (L) rungs. Table 13 lists what each family hides, its experiments and how its readings are computed. † held out of all training; ‡ two levels use an evaluation-only grammar.

Table 13. The 27 families. Recover: what the agent must deduce. Experiments: what it can do before committing. Backing: how readings are computed. Each group heading gives its number of core levels per family. † held out of all training; ‡ two levels use an evaluation-only grammar.

FamilyRecoverExperimentsBacking
Discovery worlds: twelve trained, three held out; three core levels each
circuit faultthe open, shorted or drifted resistor of a networkvoltage and current probesnodal analysis
impedance topologyan R/L/C network’s topology and element valuesimpedance at chosen frequenciescomplex impedance
heat inclusionposition, size and contrast of an insert in a heated platethermocouples, heated edgefinite volumes
hidden lawa power law’s exact exponents, then an extrapolationmeasurements in safe rangesclosed form
control planta plant with dead time, or a controller meeting a specstep and sine experimentsexact discretisation
reaction networka kinetic scheme among candidates and a rate constantbatch runsclosed form
mechanism discrimination†one of four nested mechanisms, then an extrapolationmeasurements in a safe rangeclosed form
pathway faultthe faulty step of a curated kinetic modeltime courses and boluseslibRoadRunner
knockout networkevery synthetic-lethal pair of a signalling networksingle and double knockoutsBoolean reachability
group testingthe contaminated sample, within ⌈log⁡2N⌉\lceil\log_2 N \rceil testspooled testsexact
counterfeit coin†the off-weight coin and its direction, at the boundbalance weighingsexact
rule inductiona hidden rule over number triples, by equivalencemembership testsexact
causal interventiondirect causes or a joint effect under confoundinginterventionslinear causal model
trace debuggingthe buggy line of a program and its fixruns and line tracesinterpreter
automaton learning†a hidden Mealy machine with an error statemembership, equivalence queriesexact
MicroWorld: six core levels each
certificate auditthe revoked root and corrupted feed of a certificate graphdirect verificationsexact
exact execution‡three exact values of a long generated programnone; one committwo interpreters
PhysicsWorld simulator families: four core levels each
rigid-body identificationpayload, damping, friction, offset and spring of a hingereleases and holdsrigid-body dynamics
airfoil inferencelift slope, zero-lift angle, Reynolds-dependent stall; designwind-tunnel runspanel method
optical bencha sealed lens train’s focal properties, then, a set-upimages at chosen positionsray transfer
coupled oscillatorsclamps, broken links, walls and stiffnesses of a cart chainimpact testsmodal model
fluid network†a burst, crack, draw or blockage, its site and sizereadings under head, valveshydraulic solver
ChemistryWorld: three core levels each
structure elucidationone constitutional isomer of a molecular formulaspectra under a credit budgetRDKit isomer bank
titration equilibriathe acid–base system, concentrations and pKapK_a valuespH, indicator endpointscharge balance
kinetics process designan Arrhenius network, then a process meeting a specisothermal batch runsclosed form
chromatography methodretention of a mixture, then a method separating every pairgradient runsretention model
electrochemical mechanisman E, EC, CE or ECE mechanism and its rate constantcyclic voltammetryintegral-equation CV

Table 13. The 27 families. Recover: what the agent must deduce. Experiments: what it can do before committing. Backing: how readings are computed. Each group heading gives its number of core levels per family. † held out of all training; ‡ two levels use an evaluation-only grammar.

TermMeaningSec.
family, level, d0d_0–d5d_5One environment type; one of its core difficulty settings, numbered from the easiest.3.2
core tier, frontier tierThe 92 core levels of all 27 families; the 86 harder and long-horizon rungs of the 15 discovery worlds.3.2
rung, H1–H3, L1–L3A frontier level: harder single questions (H) or campaigns of 3, 5 and 8+ stages (L).3.2
campaign, stage commitA long-horizon episode on one hidden system; a committed sub-answer that closes one stage.3.2
oracle, naive, shortcut policyThe gate’s scripted policies: a planning, deducing oracle; the strongest plausible non-deductive policy; specific non-reasoning habits, including an answer prior.3.1
exhaustive designEvery allowed experiment in a fixed order, then the oracle’s analysis; reported, not gated.3.1
development, final instancesInstances for calibration and design (64 per level and block) / withheld evaluation instances from a private key.3.1
invalid endingAn episode that ends on replies without a legal action.3.3
completed-episode successSuccess among episodes that do not end invalid.5.1
resolvedA paired difference whose 95% interval excludes zero.5.1
headroomOracle success minus model success on the same final instances.5.1
Concise-9B, Narrow-9B, Broad-9BThe 9B model before training, after the narrow phase and after the broad phase.9.1
Warm-9B, LabRL-9B, MixRL-9BConcise-9B after a laboratory warm-up, then after reinforcement learning on replication worlds, or on replication and method-comparison worlds.9.3
forced close, re-askAt the thinking cap, the thinking is closed and an answer requested (local models); at the API’s token limit, one thinking-off request for the action that carries the cut reasoning (API model).5

Table 14. Terms used in the paper, with the section that defines them.

Core tier. Core levels, numbered d0d_0, d1d_1, … from the easiest, scale the size of the hypothesis space, the noise, the slack in the budget, the depth of the deduction or the length of the computation. The discovery worlds have three core levels each, the MicroWorld families six, the PhysicsWorld families four and the ChemistryWorld families three: 92 levels in all. Easier entry levels below them give small models their first successes during training and are not evaluated.

Frontier tier. The frontier tier adds six rungs to each discovery world. Harder rungs (H1–H3) are harder, not noisier: no rung raises the noise or tightens a tolerance relative to the family’s hardest core level. Instead they enlarge or structure the hypothesis space or add an inference step the core levels never need, and each must defeat the strategy that solves the rung below it. Long-horizon rungs (L1–L3) are campaigns: three, five and eight or more stages on one persistent hidden system, each closed by a committed answer, under a pooled experiment budget that is too small to re-solve every stage from scratch. Later stages depend on earlier truths and in some families on the agent’s own earlier commits, such as its committed controller or patch, and an episode succeeds only if every stage is right. L3 campaigns show only a window of recent records and give the agent a notebook, so it must decide what to write down. Pooled over families, each ladder is graded, from an entry rung that a fast frontier model (DeepSeek V4.1 Flash) solves about half the time to upper rungs that it solves far less often. That grading was a design target: the tier was built after Flash’s results on the core tier, aiming for Flash success of about 0.10–0.40 at each family’s top rung, and the target was read from screens of Flash on final instances. Rung settings were chosen on development instances, and no rung setting, generator or verifier changed after any model played the final instances; after the first screen, however, the frontier families’ action parsers (a lenient second reading of a rejected reply), the part of the campaign wording, the budget display of one family and the re-ask after a cut turn were revised, and the same final instances were played again. Of the 90 possible rungs, 86 are built and admitted; pathway fault has only L1 and L3, and only its L1 is evaluated here.

Held-out families and splits. Three discovery worlds are held out of all training, each with a close trained relative: mechanism discrimination (reaction network, hidden law), counterfeit coin (group testing) and automaton learning (rule induction, trace debugging); fluid network is held out among the PhysicsWorld families, and two exact-execution levels use a construction that no training level contains. Final instances number 24 per family and level on both tiers (2,208 core instances).

Episodes and protocol

An agent receives the public observation and takes one action per turn (Figure 3): an experiment, a sandboxed Python analysis of the readings (in families that offer the tool), or a COMMIT, which submits the answer and, outside campaigns, ends the episode. Actions are accepted as bare commands, JSON objects or tool calls, and equivalent spellings of a command are accepted, so that interface conventions are not what is measured. A reply without a legal action is returned with the parser’s reason and the exact legal form and costs one turn, and two such replies in a row end the episode. An experiment outside the safe limits or beyond the budget ends the episode at zero, as the observation states. We call an episode that ends on replies without a legal action an invalid ending.

Group testing, core level d0 (illustrative episode)
Group testing, core level d0 (illustrative episode)

Observation (abridged; re-rendered in full every turn)
  samples:       S01, S02, ..., S32
  objective:     exactly one sample is contaminated; a pooled test of any set of samples
                 is positive if and only if the set contains the contaminated sample
  grammar:       TEST samples=<comma-separated sample labels, e.g. S01,S02>; at most 5
                 tests. Then: COMMIT sample=<label>.
  success rule:  the COMMIT succeeds only if it names the contaminated sample; an unknown
                 sample or a test beyond the budget ends the episode at zero
  budget:        5 tests, 0 used; 8 laboratory turns
Episode
  1  ACTION: TEST samples=S01,S02,...,S16   ->  negative
  2  ACTION: TEST samples=S17,S18,...,S24   ->  positive
  3  Next I will test S17 to S20.           ->  not executed: no legal action (legal forms:
                                               TEST samples=<labels>, COMMIT sample=<label>)
  4  ACTION: TEST samples=S17,S18,S19,S20  ->  negative
  5  ACTION: TEST samples=S21,S22           ->  positive
  6  ACTION: TEST samples=S21               ->  negative
  7  ACTION: COMMIT sample=S22
Verifier  correct sample; reward 1

Figure 3. The episode format. On the easiest core level of group testing, 32 samples and ⌈log⁡232⌉=5\lceil\log_2 32 \rceil= 5 pooled tests leave room only for a halving plan. Turn 3 shows how the protocol treats a reply without a legal action. The excerpt was constructed to show the format; Appendix § 14 shows recorded oracle episodes from the public sample.

Benchmark Analysis

Calibration

All 178 levels pass the gate (Figure 4). On the 92 core levels, the oracle succeeds on 0.80–1.00 of development instances, the naive policy on at most 0.19 and the strongest shortcut on at most 0.33; the ten PhysicsWorld and ChemistryWorld families also pass on a second independent block. On the 86 frontier rungs, every rung passes on two blocks: the oracle succeeds on 0.78–1.00, the naive policy on at most 0.14, every gated shortcut on at most 0.31, and copying the grammar’s example answer on none.

Figure 4. The calibration gate on both tiers. Each bar spans a policy’s success over a family’s levels, on 64 development instances per level: the core levels (a; first of two blocks for the PhysicsWorld and ChemistryWorld families) and the admitted frontier rungs over both development blocks (b; pathway fault has two rungs). Shaded bands mark the oracle gate (green, ≥ 0.75) and the naive gate (red, ≤ 0.25); the dashed line is the shortcut gate (0.35). Per-level values are in Appendix § 17.1 held out of all training.

The exhaustive design separates experiment choice from analysis. Where it fails, the evidence has to be chosen: knockout network (0.05–0.17), rule induction (0.02–0.28), airfoil inference (0.03–0.12), coupled oscillators (0.00–0.16) and the optical bench (0.16–0.27) cannot be solved by running experiments in a fixed order. Where it succeeds often, the bottleneck is the analysis: mechanism discrimination (0.98–1.00), chromatography method (0.75–0.92), electrochemical mechanism (0.69–1.00) and structure elucidation (0.64–0.91). Families in between, such as heat inclusion (0.47–0.58) and reaction network (0.45–0.56), need both, and each datasheet reports where its family lies, so that a study of experiment choice can weight the families where the exhaustive design fails.

Campaigns cannot be solved one stage at a time. On every long-horizon rung, the stage-by-stage policy, which runs the family’s single-question oracle afresh at each stage with the pooled budget split evenly, stays below the shortcut gate (at most 0.27), while the oracle that reuses earlier records clears 0.78. On L3 the oracle without notes also stays below the gate. The campaigns therefore reward carrying evidence forward, which no core level tests.

Difficulty and headroom

For the models in Table 2, the core tier and the two frontier ladders are well separated. On the core tier the oracle succeeds on 0.978 of the 360 instances that every system played, DeepSeek V4.1 Flash on 0.583 and Qwen3.8-27B on 0.358; on the harder and long-horizon rungs the oracle stays at 0.91–0.94 while Flash falls to the values in Table 2. Difficulty is not simply the number of decisions an episode takes. The examples in Table 3 show that the driver is the kind of work required: core levels need one deduction at the information limit, harder rungs need an inference step the core levels never ask for, and campaigns need evidence that was gathered, and in L3 written down, stages earlier. Group testing makes the point: Flash solves 0.96 of its five-stage outbreak campaigns (L2) and 0.06 of its eight-stage ones, once older results leave the window.

LadderRungsFamiliesLevelsnnOracleFlash27B9BWhat makes it hard
Cored0–d215453600.9780.5830.3580.094one hidden system, one commit
HarderH1–H314429380.9160.3670.083–larger or more structured hypotheses, deeper inference
Long-horizonL1–L315449480.9290.3190.090–3, 5 and 8+ committed stages on one system

Table 2. Core levels and the two frontier ladders on the discovery worlds. Flash: DeepSeek V4.1 Flash (snapshot V4.1-Flash-1017). Success on final instances, pooled over the levels or rungs of each row; the core row is the 360 instances (8 per family and level) that every system played. Concise-9B was evaluated on the core tier only. On the frontier rows, the 27B column covers the 8 instances per family and rung that it played (336 and 344). Per-rung values are in Table 5.

RungFamilyTask (simplified)FlashMain difficulty driver
d0group testing“Find the one contaminated sample among 32 with at most 5 pooled tests.”1.00a halving plan at the information limit
d2rule induction“Identify a hidden rule over number triples by membership tests; the rule is scored on all 1,000 triples.”0.00a large rule grammar; tests must split it
H1control plant“Identify a plant with dead time and a possible zero, then commit a controller that meets a specification a zero-free design misses.”0.58an inference step the core levels never need
H2counterfeit coin“Find up to two counterfeit coins among 19, each heavy or light, within the ternary weighing limit.”0.42deviations can cancel on one pan
L1control plant“Identify the actuator loop and commit its controller; then identify the outer process through that loop and tune it.”0.33later experiments run through the agent’s own commits
L2circuit fault“Five successive faults on one board; each repair restores one part, and the healthy parts keep their true values.”0.00early readings make later small drifts detectable
L3group testing“An outbreak on a sample rack over eight rounds; older pool results leave the visible history, and a notebook keeps what the agent writes.”0.06deciding what to record (L1: 1.00, L2: 0.96)

Table 3. Example tasks for each rung, with the success of DeepSeek V4.1 Flash (snapshot V4.1-Flash-1017) on that family’s final instances (8 per core level; 24 per frontier rung, 18 on L3). Tasks are paraphrased from the observations and simplified to fit the space. The last column names the main difficulty driver.

Experiment Setup

We evaluate three kinds of systems on the withheld final instances:

  • Scripted references: the gate’s oracle and naive policy, which read only the public observation and play the same instances as the models.

  • Local models: Concise-9B, which is Qwen3.5-9B [28] after a post-training stage that shortens its reasoning at matched accuracy [24], and Qwen3.8-27B [29], a reasoning model of the same family. Both run locally with thinking on, a 4,096-token thinking cap per decision with a forced close, and sampling at temperature 1.0, top-pp 0.95 and top-kk 20.

  • API model: DeepSeek V4.1 Flash [9], a fast frontier model, in its snapshot V4.1-Flash-1017, served under the deepseek–flash API name since 17 October 2026, with thinking on at the provider’s default

reasoning effort (“high”; 8) and a limit of 16,384 tokens per call; on the pilot’s frontier instances it also plays at its maximum effort with up to 393,216 tokens per call (§ 6).

  • Pilot: GPT-6.1 Sol via the Codex CLI’s “ultra” preset (server-reported reasoning effort xhigh), not the API protocol, on one final instance (o=0o = 0) per family and rung or level. Every call ran through Codex CLI 0.166.0-alpha.5.1 with its top preset, “ultra” (maximum reasoning with automatic task delegation), for which the provider’s server reports reasoning effort “xhigh” in trace probes of the same CLI build; the call logs record the requested preset, not the server-side effort. The model also accepts a higher single-agent effort, “max”, which the pilot did not use. Codex’s multi-agent features were turned off, but this CLI build still offered the model its collaboration (sub-agent) tools; no stored call record of the pilot or the probes shows them used. Each decision sends the same messages as the API protocol in one stateless call with every other tool disabled, and the model’s final message is the turn’s action; no per-call token cap of ours applies, so there is no re-ask (Appendix § 19).

Table 4 lists every comparator setting with its budget.

SystemReasoning effortSetting and budgetInstances
Scripted oracle, naive policy–Deterministic policies; no model budgetevery instance the models play
Concise-9B, Narrow-9B, Broad-9B, Qwen3.8-27Bthinking onLocal; at most 4,096 thinking tokens per decision, then a forced close; temperature 1.0, top-pp 0.95, top-kk 20; context 20,480 tokenscore tier, 24 per family and level; Qwen3.8-27B also 8 per family and frontier rung
DeepSeek V4.1 Flash (snapshot V4.1-Flash-1017)“high”, the provider’s defaultAPI; thinking on; at most 16,384 tokens per call. Core tier, PhysicsWorld and ChemistryWorld: a call cut at the budget returns no action. Frontier tier: one thinking-off re-ask that carries the cut reasoning. 27% of its frontier decisions are cut at the budgetcore tier, 8 per family and level; frontier tier, up to 24 per family and rung (19 on H3, 18 on L3); PhysicsWorld and ChemistryWorld (Table 24)
DeepSeek V4.1 Flash (snapshot V4.1-Flash-1017), maximum budget“max”API; thinking on; at most 393,216 tokens per call, no re-ask; no call reached the limito=0o = 0 of each family on every frontier rung, the pilot’s frontier instances (Table 6)
GPT-6.1 Sol (pilot and probes)Codex’s top preset, “ultra”; server-reported xhigh, below the single-agent “max”Codex CLI 0.166.0-alpha.5.1; multi-agent features off, collaboration tools still offered, no use found in the call records; no per-call token cap of ours, and no call truncated in 19,838 calls (largest 46,457 output tokens); every tool disabled; one stateless call per decision; not the API protocolone final instance per family and rung or level; 8 or 16 development instances per candidate rung
Laboratory worlds (§ 9.3; Appendix § 20)
Warm-9B, LabRL-9B, MixRL-9Bthinking onLocal; panel: at most 4,096 tokens per turn (8,192 in method comparison) and 16,384 per episode, temperature 1.0 without truncation; P+X and workflow: at most 12,000 tokens per call, temperature 0.6, top-pp 0.95the 38 panel worlds, 8 episodes each; P+X and workflow development items, one attempt each
DeepSeek V4.1 Flash“low” (reference) and “high” on the panel; “high” otherwiseAPI; thinking on; panel: the local models’ tools, prompts, verifier and per-turn caps, counted in their tokenizer; P+X and workflow: at most 24,000 tokens per callas for the local models

Table 4. Every comparator setting and its budget. Decision: one turn of an episode. The oracle and naive policy read only the public observation, as the models do.

Implementation details. Every system sees the same observations and plays under the protocol of § 3.3. On the frontier tier, a turn whose API call reaches the token limit gets one re-ask with thinking off that carries the cut turn’s reasoning and asks for the action line only, and the reply is the turn’s action. Like the local models’ forced close, the re-ask thus keeps the thinking so far in context. DeepSeek V4.1 Flash’s thinking reaches the limit on 23–39% of frontier-tier decisions per rung, so the per-call budget shapes these episodes (§ 6). A frontier episode may take as many decisions as its lab-action turns plus Python runs. Local models run at a context length of 20,480 tokens; on the frontier tier, the longest prompt in the gate’s oracle and naive transcripts is 5,580 tokens. Qwen3.8-27B plays every frontier rung; Concise-9B plays the core tier, where it is the starting point of the training study.

Evaluation metrics

Success. Let I\mathcal{I} be a set of final instances and ri∈{0,1}r_i \in\{0,1\} the verifier’s reward for a system’s episode on instance ii. A system’s success on I\mathcal{I} is

S(I)=1∣I∣∑i∈Iri,(1)S(\mathcal{I}) = \frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}} r_i, \tag*{(1)}

pooled over families with equal instance counts per family and level, so that no family dominates. On campaigns, ri=1r_i = 1 only if every stage commit and the final commit are right; per-stage verdicts are kept for analysis.

Completed-episode success. We also report success among episodes that do not end invalid, Sc=∑iri/∑iviS_c = \sum_i r_i/\sum_i v_i with vi=1v_i = 1 for a valid ending, which separates reasoning from failures to produce an answer.

Paired differences and intervals. Because every system plays the same instances, two systems AA and BB are compared by the paired difference

ΔA,B=1∣I∣∑i∈I(riA−riB),(2)\Delta_{A,B} = \frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}(r_i^A-r_i^B), \tag*{(2)}

with a 95% percentile interval from 10,000 bootstrap resamples of instances; single rates carry Wilson intervals. Intervals describe these families and instances, not new ones. A difference is resolved when its interval excludes zero, and we say a system beats another only then. Because episodes of one family may be correlated, Appendix § 19 repeats every headline comparison with a bootstrap over whole families, computed exactly by enumerating every resample (no Monte Carlo error); all stay resolved except Flash minus the 27B on L3 alone, the held-out training gains of both models, which rest on three families, and the pilot model’s gap to the oracle over all frontier rungs, which stays resolved on the long-horizon rungs. With 72 episodes per family, a single rate near 0.5 has an interval of about ±0.11.

Headroom and cost. Headroom is oracle success minus model success on the same instances. For the API model we report the mean cost per episode at the provider’s list prices, including cached-input discounts and the peak-hour rate for calls made in peak hours, under its capped budget of 16,384 tokens per call, and the mean number of decisions per episode.

Results

The core tier separates the local models and approaches saturation for frontier models. On the core tier’s 360 shared instances, DeepSeek V4.1 Flash succeeds on 0.583 and Qwen3.8-27B on 0.358, a resolved difference of +0.225 [+0.169, +0.278], against 0.978 for the oracle and 0.094 for Concise-9B (Table 5). Flash ends 0.37 of these episodes invalid, almost all on two turns in a row at its token limit; of the episodes it completes, it solves 0.925. These core-tier rates are confounded with Flash’s per-call budget and may understate it: 14.9% of its core decisions are cut at the cap, and episodes with a cut turn succeed on 0.24 against 0.96 without one; the same holds on PhysicsWorld (19.2% of decisions cut over all 350 finished episodes; 0.08 against 0.34) and ChemistryWorld (20.1%; 0.15 against 0.90). When it finishes, rule induction (0.43 of completed episodes) and pathway fault (0.67) hold it back most. On all 1,080 core instances of the discovery worlds, the 27B succeeds on 0.358 of the trained families’ episodes and 0.394 of the held-out families’, against an oracle at 0.964 and 0.995, a headroom of about 0.60; it ends fewer than 0.05 of them invalid, so that headroom is not a format effect (Appendix § 21), though the 27B’s thinking is near its 4,096-token cap on most decisions (a mean of 3,813–4,039 tokens per decision). In the simulator-grounded PhysicsWorld and ChemistryWorld families, the 27B succeeds on 0.03 and 0.18 of final instances, against 0.91 and 0.95 for the scripted oracle on the same instances (Table 24). Flash, in the core tier’s setting, is also far below the oracle on both groups. On the instances it played (320 PhysicsWorld episodes, the first 16 of each level’s 24, and all 360 ChemistryWorld episodes), it succeeds on 0.13 and 0.33, against 0.90 and 0.95 for the oracle, paired differences of −0.77 [−0.82, −0.72] and −0.63 [−0.68, −0.57]; it is above the 27B by +0.10 [+0.06, +0.14] and +0.15 [+0.10, +0.20] on the same instances. Most of its failures are invalid endings: 0.83 and 0.64 of its episodes end on replies without a legal action, 73% and 93% of those on two replies in a row with no content at all, as happens when a turn is cut at the per-call limit, and it solves 0.73 and 0.90 of the episodes it completes. The GPT-6.1 Sol pilot, by contrast, solves most of both groups at one instance per family and level (Table 7).

SystemCore d0–d2Harder H1Harder H2Harder H3Long-horizon L1Long-horizon L2Long-horizon L3Frontier pooledDecisionsCost ($) core / frontier
Scripted reference
Oracle0.9780.9260.9110.9100.9420.9230.9210.923––
Local models (thinking ≤ 4,096 tokens per decision)
Concise-9B0.094–––––––––
Qwen3.8-27B0.3580.1610.0360.0540.1170.1070.0450.08719.6–
API model (thinking on, ≤ 16,384 tokens per call)
DeepSeek V4.1 Flash0.5830.4910.3210.2670.4670.3180.1070.34319.70.037 / 0.132

Table 5. Success on the final instances of the discovery worlds, by tier and rung. API model: DeepSeek V4.1 Flash (snapshot V4.1-Flash-1017). The core column is the 360 instances that every system played (8 per family and level). Frontier rungs: DeepSeek V4.1 Flash and the oracle on 24 instances per family (the first 19 on H3 and 18 on L3, the rounds every family finished; 1,886 episodes), Qwen3.8-27B on 8 per family. Decisions: mean turns per frontier episode, for Flash over its 1,886 episodes and for the 27B over its 680; on the instances both played, Flash averages 20.5 (§ 8). Cost: mean cost per episode on the core / frontier tier at the provider’s list prices, including the peak-hour rate for calls made in peak hours, under the 16,384-token cap, the latter with every turn paid. The best model in each column is in bold. DeepSeek V4.1 Flash minus Qwen3.8-27B on the instances both played is resolved on the core tier, +0.225 [+0.169, +0.278], and pooled over the frontier tier, +0.249 [+0.213, +0.284] (680 instances).

Table 7. Pilot: GPT-6.1 Sol via the Codex CLI’s “ultra” preset (server-reported reasoning effort xhigh), not the API protocol, on one final instance (o=0o = 0) per family and rung or level. Successes over instances (rate) for each system on the same final instances; Sol: Wilson 95% interval for GPT-6.1 Sol. Flash: DeepSeek V4.1 Flash (snapshot V4.1-Flash-1017) under the API protocol of § 5, on PhysicsWorld and ChemistryWorld from its own final-instance run; 27B: Qwen3.8-27B. The oracle is shown on the frontier tier. One pathway-fault L1 instance that could not be reproduced on the pilot’s machine is excluded. Sol minus Flash (capped at 16,384 tokens per call, default effort) on the frontier tier: +0.46 [+0.36, +0.57]; minus the 27B: +0.76 [+0.67, +0.85]; minus the oracle: −0.10 [−0.18, −0.01] (family bootstrap: −0.10 [−0.19, 0.00]; Table 20).

CutnnSolSol, 95% CIFlash27BOracle
Frontier tier, all rungs8471/84 (0.85)[0.75, 0.91]32/84 (0.38)7/84 (0.08)79/84 (0.94)
H11412/14 (0.86)[0.60, 0.96]8/14 (0.57)4/14 (0.29)12/14 (0.86)
H21412/14 (0.86)[0.60, 0.96]7/14 (0.50)0/14 (0.00)13/14 (0.93)
H31413/14 (0.93)[0.69, 0.99]4/14 (0.29)1/14 (0.07)13/14 (0.93)
L11412/14 (0.86)[0.60, 0.96]6/14 (0.43)1/14 (0.07)14/14 (1.00)
L21414/14 (1.00)[0.78, 1.00]5/14 (0.36)1/14 (0.07)14/14 (1.00)
L3148/14 (0.57)[0.33, 0.79]2/14 (0.14)0/14 (0.00)13/14 (0.93)
Core levels, discovery worlds4545/45 (1.00)[0.92, 1.00]31/45 (0.69)18/45 (0.40)–
PhysicsWorld core levels2017/20 (0.85)[0.64, 0.95]1/20 (0.05)0/20 (0.00)–
ChemistryWorld core levels1515/15 (1.00)[0.80, 1.00]6/15 (0.40)5/15 (0.33)–

Table 7.

FamilynnConcise-9BNarrow-9BBroad-9BQwen3.8-27BnFn_FFlashOracle
rigid-body identification960.0100.0000.0100.021640.2030.906
airfoil inference960.0210.0000.0100.052640.2500.990
optical bench960.0100.0210.0100.021640.0780.802
coupled oscillators960.0000.0000.0000.000640.0310.958
fluid network†960.0100.0100.0000.062640.0780.885
structure elucidation720.0140.0280.0690.333720.8061.000
titration equilibria720.0690.0000.0280.125720.1670.931
kinetics process design720.0000.0140.0280.014720.0140.931
chromatography method720.0140.0140.0000.056720.2920.944
electrochemical mechanism720.0560.0690.1530.347720.3470.944
certificate audit1440.0490.1320.1880.764–––
exact execution‡1440.6180.5620.6110.743–––

Table 24. The core tier outside the discovery worlds, on all final instances (24 per family and level), except Flash. Flash: DeepSeek V4.1 Flash (snapshot V4.1-Flash-1017) in the core tier’s setting (thinking on, 16,384 tokens per call, no re-ask), on the instances in column nFn_F: the first 16 final instances of each PhysicsWorld level and all 24 of each ChemistryWorld level. Paired with the other systems on the same instances, Flash minus the oracle is −0.77 [−0.82, −0.72] on PhysicsWorld and −0.63 [−0.68, −0.57] on ChemistryWorld; minus Qwen3.8-27B, +0.10 [+0.06, +0.14] and +0.15 [+0.10, +0.20]; minus Concise-9B, +0.12 [+0.08, +0.16] and +0.29 [+0.24, +0.35]. Oracle: the gate’s scripted oracle on the same final instances of the PhysicsWorld and ChemistryWorld families, pooled over their levels (0.91 and 0.95 per group); it was not played on the MicroWorld final instances. † held out of all training; ‡ includes the two evaluation-only levels.

Flash on the frontier tier, under its capped budget. On up to 24 instances per family and rung (19 on H3, 18 on L3), Flash succeeds on 0.49, 0.32 and 0.27 of the harder rungs’ episodes and on 0.47, 0.32 and 0.11 of the campaigns’, while the oracle stays at 0.91–0.94 on the same instances (Figure 5a). Pooled over families, each ladder is graded: Flash solves about half of its entry rung’s episodes (0.49 on H1, 0.47 on L1), and pooled success falls with every rung above it, though not in every family (Table 25 and Table 26). Flash’s completed-episode success is within 0.007 of its success on every rung, because re-asks almost always return an action; the per-call budget still shapes these episodes, as the next paragraph shows. On the 680 instances that both models played (8 per family and rung), Flash beats the 27B by +0.249 [+0.213, +0.284], by +0.292 [+0.241, +0.342] on the harder and by +0.206 [+0.160, +0.256] on the long-horizon rungs, and the difference is resolved on every rung at the episode level, down to +0.071 [+0.009, +0.134] on L3; a bootstrap over families does not resolve L3 alone, +0.071 [−0.009, +0.152] (Appendix § 19).

Figure 5. Success by rung and how frontier episodes end. (a) Pooled success on the final instances of the discovery worlds, with 95% Wilson intervals; markers are offset slightly so that intervals do not overlap. (b) The share of each frontier rung’s episodes that succeed, commit a wrong answer, end on a stated budget or safety rule, or end on malformed replies or the turn cap.

FamilyH1 FH1 QH1 OH2 FH2 QH2 OH3 FH3 QH3 O
circuit fault0.170.000.960.170.000.880.210.000.84
impedance topology0.250.000.920.000.000.960.000.000.89
heat inclusion0.080.000.830.040.000.790.000.000.74
hidden law0.460.000.960.210.000.920.160.000.89
control plant0.580.000.710.710.000.880.470.001.00
reaction network0.710.380.830.460.000.960.840.501.00
mechanism discrimination†0.620.120.920.790.120.790.420.120.79
pathway fault–––––––––
knockout network1.000.251.000.920.001.000.840.001.00
group testing0.920.880.960.580.121.000.370.120.95
counterfeit coin†0.880.251.000.420.251.000.370.001.00
rule induction0.080.000.920.080.000.920.000.000.79
causal intervention0.960.381.000.000.000.920.000.001.00
trace debugging0.120.001.000.120.001.000.000.001.00
automaton learning†0.040.000.960.000.000.750.050.000.84
pooled0.490.160.930.320.040.910.270.050.91

Table 25. The harder rungs per family: success of DeepSeek V4.1 Flash (F; snapshot V4.1-Flash-1017), Qwen3.8-27B (Q) and the oracle (O) on the final instances: F and O on 24 per family (19 on H3), Q on 8. † held out of all training.

FamilyL1 FL1 QL1 OL2 FL2 QL2 OL3 FL3 QL3 O
circuit fault0.000.000.750.000.000.830.000.001.00
impedance topology0.210.000.920.000.000.710.220.000.83
heat inclusion0.040.000.960.000.000.830.000.000.89
hidden law0.920.120.920.540.000.920.170.120.89
control plant0.330.001.000.250.000.960.110.001.00
reaction network0.750.250.960.710.000.830.110.000.83
mechanism discrimination†0.790.380.790.710.250.960.560.250.67
pathway fault0.290.000.96––––––
knockout network0.120.001.000.000.001.000.000.001.00
group testing1.001.001.000.961.001.000.060.250.94
counterfeit coin†0.790.001.000.540.251.000.060.001.00
rule induction0.620.001.000.620.001.000.220.001.00
causal intervention0.670.000.920.080.000.960.000.000.94
trace debugging0.290.001.000.040.001.000.000.001.00
automaton learning†0.170.000.960.000.000.920.000.000.89
pooled0.470.120.940.320.110.920.110.040.92

Table 26. The long-horizon rungs per family, as in Table 25: DeepSeek V4.1 Flash (F; snapshot V4.1-Flash-1017), Qwen3.8-27B (Q) and the oracle (O); F and O on 24 per family (18 on L3), Q on 8.

Part of the difficulty may be the per-call budget. Flash’s thinking reaches the 16,384-token limit on 27% of frontier decisions, from 23% on L1 to 39% on H3. Of the 1886 frontier episodes with a complete decision log, 1290 end on a turn answered by the thinking-off re-ask; they succeed on 0.18 (233 of 1290), against 0.69 (413 of 596) for episodes whose final decision was a full thinking turn. Hard instances also cause long thinking, so the split is correlational and does not show how much a larger budget would recover. We therefore read Flash’s frontier results as reasoning under a fixed per-call budget; the next paragraph measures the budget effect on one instance per family and rung. The pilot model runs without a per-call cap of ours; over its 164 episodes the Codex CLI reported a mean of 1,872 output tokens per decision, reasoning included; token counts are not directly comparable across models, and the comparison with Flash is between a model without our cap and one under it.

At its maximum effort and a large per-call budget, Flash solves two thirds of the pilot’s frontier instances. On the pilot’s frontier instances, the o=0o = 0 instance of each family on every rung, DeepSeek V4.1 Flash at reasoning effort “max” with up to 393,216 tokens per call and no re-ask (Table 4) succeeds on 57 of 84 episodes (0.68, Wilson interval [0.57, 0.77]), against 32 at its default effort under the 16,384-token cap, a paired difference of +0.30 [+0.19, +0.40] (Table 6; over families, +0.30 [+0.15, +0.44], Table 20). The difference is +0.24 [+0.10, +0.38] on the harder and +0.36 [+0.19, +0.52] on the long-horizon rungs, and it is resolved on every rung except H1, where capped Flash already solves 8 of 14. None of its 1,796 calls reached the larger limit (the largest returned 170,299 completion tokens), and every episode ended on a committed answer. It stays below the oracle, −0.26 [−0.36, −0.18] over all rungs; the gap is resolved on H3, L2 and L3 and largest on L3, −0.43 [−0.71, −0.21]. The pilot model stays ahead, +0.17 [+0.07, +0.26] over all rungs (16 instances won, 2 lost), against +0.46 [+0.36, +0.57] for capped Flash, so at its maximum effort Flash closes about two thirds of its gap to the pilot model and about half of its gap to the oracle. Per rung, the pilot model’s lead is resolved on H3, +0.36 [+0.14, +0.64], and L2, +0.29 [+0.07, +0.50]; on the eight-stage campaigns the two solve 8 and 7 of 14, +0.07 [−0.14, +0.29]. At list prices the maximum setting costs from about $0.12 per episode on H1 to $0.53 on L3, with means of 198,000 to 837,000 completion tokens per episode. Flash’s capped frontier rates therefore understate it substantially, and much of the frontier tier’s difficulty for Flash is its budget; the headroom that holds is measured against the oracle. With one instance per family and rung, these 84 episodes measure the size of the budget effect on the pilot’s instances, not Flash’s tier-wide rates at its maximum effort, which we have not measured.

RungnnMaxDefaultSolOracleMax − defaultSol − maxMax − oracle
H1141081212+0.14 [−0.14, +0.43]+0.14 [0.00, +0.36]−0.14 [−0.36, 0.00]
H2141171213+0.29 [+0.07, +0.57]+0.07 [−0.14, +0.29]−0.14 [−0.36, 0.00]
H314841313+0.29 [+0.07, +0.50]+0.36 [+0.14, +0.64]−0.36 [−0.64, −0.14]
L1141161214+0.36 [+0.14, +0.64]+0.07 [0.00, +0.21]−0.21 [−0.43, 0.00]
L2141051414+0.36 [+0.07, +0.64]+0.29 [+0.07, +0.50]−0.29 [−0.50, −0.07]
L31472813+0.36 [+0.14, +0.64]+0.07 [−0.14, +0.29]−0.43 [−0.71, −0.21]
H1–H34229193738+0.24 [+0.10, +0.38]+0.19 [+0.05, +0.33]−0.21 [−0.36, −0.10]
L1–L34228133441+0.36 [+0.19, +0.52]+0.14 [+0.02, +0.26]−0.31 [−0.45, −0.17]
All8457327179+0.30 [+0.19, +0.40]+0.17 [+0.07, +0.26]−0.26 [−0.36, −0.18]

Table 6. Table 6. Flash at its maximum reasoning effort on the frontier tier. The o=0o = 0 final instance of each of the 14 families on every frontier rung, the frontier instances of the pilot. Max: DeepSeek V4.1 Flash (snapshot V4.1-Flash-1017) at reasoning effort “max”, at most 393,216 tokens per call, no re-ask. Default: DeepSeek V4.1 Flash at its default effort under the frontier protocol (16,384 tokens per call, re-ask after a cut turn), on the same instances. Sol: the pilot of Table 7 (GPT-6.1 Sol via the Codex CLI’s “ultra” preset (server-reported reasoning effort xhigh), not the API protocol). Successes over instances; differences paired over instances with 95% bootstrap intervals.

Table 20. Paired differences with episode-level and family-level bootstrap intervals (Flash: DeepSeek V4.1 Flash, snapshot V4.1-Flash-1017; Sol: GPT-6.1 Sol, the pilot model; 27B: Qwen3.8-27B). Episode bootstrap: 10,000 resamples of paired episodes; family bootstrap: whole families drawn with replacement, each with all of its paired episodes, computed exactly by enumerating every resample (no Monte Carlo error); both are 95% percentile intervals. Families: the number of families resampled; episodes: the paired episodes they contain. Pilot rows: the pilot’s frontier instances (Table 7), with DeepSeek V4.1 Flash at its default effort under the 16,384-token cap and at its maximum effort (Table 6).

ComparisonFamiliesEpisodesDifferenceEpisode bootstrapFamily bootstrap
Core tier, DeepSeek V4.1 Flash − Qwen3.8-27B15360+0.225[+0.169, +0.278][+0.111, +0.347]
Frontier tier, DeepSeek V4.1 Flash − Qwen3.8-27B15680+0.249[+0.213, +0.284][+0.163, +0.335]
harder rungs14336+0.292[+0.241, +0.342][+0.167, +0.432]
long-horizon rungs15344+0.206[+0.160, +0.256][+0.119, +0.296]
L3 only14112+0.071[+0.009, +0.134][−0.009, +0.152]
Trained families, Broad-9B − Narrow-9B12837+0.045[+0.019, +0.072][+0.017, +0.080]
Trained families, Broad-9B − Concise-9B12837+0.059[+0.031, +0.085][+0.019, +0.121]
Held-out families, Broad-9B − Concise-9B3216+0.083[+0.023, +0.144][+0.000, +0.153]
Held-out families, Broad-9B − Narrow-9B3216+0.102[+0.051, +0.157][+0.000, +0.167]
Held-out families, 27B after two rounds − base3216+0.120[+0.046, +0.194][+0.000, +0.306]
Invalid endings, Broad-9B − Narrow-9B272,181−0.043[−0.054, −0.032][−0.071, −0.020]
Invalid endings, Broad-9B − Concise-9B272,181−0.074[−0.087, −0.061][−0.119, −0.035]
Pilot, Flash at max − default effort1484+0.298[+0.191, +0.405][+0.155, +0.440]
Pilot, Flash at max effort − oracle1484−0.262[−0.357, −0.179][−0.417, −0.143]
Pilot, Sol − Flash at default effort1484+0.464[+0.357, +0.571][+0.310, +0.619]
Pilot, Sol − Flash at max effort1484+0.167[+0.071, +0.262][+0.060, +0.286]
Pilot, Sol − oracle1484−0.095[−0.179, −0.012][−0.190, +0.000]
long-horizon rungs1442−0.167[−0.286, −0.071][−0.286, −0.071]

Table 20.

A stronger model solves most of the frontier tier (pilot). In the pilot (GPT-6.1 Sol via the Codex CLI’s “ultra” preset (server-reported reasoning effort xhigh), not the API protocol, on one final instance (o=0o = 0) per family and rung or level; Table 7), GPT-6.1 Sol succeeds on 71 of 84 frontier episodes (0.85, Wilson interval [0.75, 0.91]), against 32 for Flash, 7 for the 27B and 79 for the oracle on the same instances. Against Flash capped at 16,384 tokens per call at its default effort, the difference is +0.46 [+0.36, +0.57], with 39 paired instances won and 0 lost, a gap that mixes model strength with Flash’s budget: against Flash at its maximum effort on the same instances it shrinks to +0.17 [+0.07, +0.26], about a third of it (§ 6). It exceeds the 27B by +0.76 [+0.67, +0.85] and stays below the oracle, −0.10 [−0.18, −0.01], a gap that a bootstrap over families, which allows for correlated rungs of one family, does not resolve (−0.10 [−0.19, 0.00]; Table 20). It solves 12–14 of the 14 instances on each rung from H1 to L2, and on the harder rungs together its difference from the oracle is not resolved, −0.02 [−0.14, +0.10]. On the long-horizon rungs it is below the oracle, −0.17 [−0.29, −0.07], and the bootstrap over families resolves that gap (−0.17 [−0.29, −0.07]); it comes mostly through L3, where it solves 8 of 14 campaigns against 13 for the oracle, −0.36 [−0.64, −0.14]. It also solves all 45 core-level instances of the discovery worlds, against 31 for Flash, and 17 of 20 PhysicsWorld and 15 of 15 ChemistryWorld instances, against 1 and 6 for Flash. A stronger model at xhigh reasoning effort therefore solves most of the frontier tier, and Flash at its maximum effort solves 0.68 of the same instances; the headroom that remains is in the eight-stage campaigns and in the gap to the oracle. With one instance per family and rung, these are pooled estimates with wide intervals, and we draw no per-family conclusion from them.

Long memory is the hardest axis. On L3, Flash succeeds on 27 of 252 episodes and the 27B on 5 of 112, against an oracle at 0.92. The L3 campaigns are not harder per stage than L2: they are longer, and older records leave the window. Several families that the models handle at L1 or L2 collapse at L3, including group testing (Flash 1.00 at L1, 0.96 at L2 and 0.06 at L3) and counterfeit coin (Flash 0.79 at L1, 0.06 at L3; Figure 6). L3 is also the rung where the pilot’s stronger model falls furthest below the oracle (Table 7).

Figure 6. Success per family and rung. Final instances: 8 per cell on the core levels; on the frontier rungs 24 per cell for DeepSeek V4.1 Flash (19 on H3, 18 on L3) and 8 for Qwen3.8-27B. A dash marks a rung that does not exist or was not evaluated: pathway fault has only L1 and L3, and its L3 is not evaluated here. Every cell’s oracle success is in Table 25 and Table 26. † held out of all training.

How frontier episodes end. Across the frontier tier, 83–90% of Flash’s failures per rung are committed wrong answers, and most of the rest end on a safety rule that the observation states (Figure 5b). Only 10 of its 1,886 frontier episodes, and 18 of the 27B’s 680, end on malformed replies or the turn cap. Because more than a quarter (27%) of Flash’s decisions are cut at its per-call budget (above), these endings describe its episodes under that budget and do not by themselves separate reasoning errors from truncation. Rule endings concentrate in automaton learning, where a query beyond the remaining symbol allowance ends the episode, as the observation says (91 of Flash’s 141 unsafe endings).

Flash runs about as many experiments as the 27B and succeeds far more often. On the instances both played, Flash and the 27B take almost the same number of decisions per frontier episode (20.5 and 19.6) and run similar numbers of experiments on most rungs (for example 6.3 and 5.8 on H1, 18.9 and 16.4 on L2), yet Flash succeeds 3.9 times as often. The difference lies mostly in what each model infers from comparable evidence, under different budgets: the 27B’s thinking averages 3,877–3,999 of its 4,096 tokens per decision on the frontier tier. Flash spends 91,000–363,000 completion tokens per frontier episode, at a mean cost of about $0.13 per episode at list prices (about $0.07 on H1 and $0.27 on L3), against $0.037 per core episode.

Families differ more than tiers. Per family, frontier difficulty ranges from rungs that Flash solves on more than nine in ten instances (causal intervention at H1; group testing at H1, L1 and L2; hidden law at L1; knockout network at H1 and H2) to three families where it succeeds on at most 12 of its 133 frontier episodes (automaton learning, circuit fault, heat inclusion; Figure 6). The 27B succeeds on no frontier episode in eight of the 15 families, including circuit fault and heat inclusion, which the oracle solves on most of the same instances (Table 25 and Table 26).

Long-Horizon Campaigns

The frontier tier shows that a stronger model at xhigh reasoning effort solves most single questions and short campaigns but falls well below the oracle on the eight-stage ones (8 of 14 against 13 of 14). This section probes longer campaigns built by the same construction and admitted by the same gate. Every result here is a probe on development instances with the pilot model (GPT-6.1 Sol via the Codex CLI’s “ultra” preset (server-reported reasoning effort xhigh), not the API protocol), eight or sixteen per rung, with the scripted oracle on the same instances, except two coupled rungs that were later also played on final instances (§ 7.3); the rungs are candidates, not part of the evaluated suite.

Which rungs hold a stronger model

Because the pilot model solves most of the frontier tier, we probed candidate rungs above it with the same model and protocol on development instances, eight per rung (sixteen for rungs near the band edge), with the scripted oracle on the same instances (Table 8). Every candidate first passed the calibration gate of § 3.1 on two blocks of 64 development instances. The protocol extends a rung at 3–5 of 8 by eight more instances and judges it on all sixteen. This was done for the rungs of Table 8 and for three long campaigns of Table 9; four later rungs at 3–5 of 8 (heat inclusion L5, drug pipeline C2 and C3, and instrument-in-the-loop C3) were not extended and are reported at eight instances, so their status is less certain. A rung holds the pilot model when it solves at most 0.4 of the probe instances, the target band; a rung solved on more than 0.6 is not admitted, and one between 0.4 and 0.6 holds it only partly. The protocol also prescribes a pre-specified harder setting and a new probe for a partly-holding rung; only hidden law L4 was rebuilt that way (its 14-law version in § 7.2), and the other partly-holding rungs are reported as probed.

Table 8. Candidate rungs above the frontier tier, probed on development instances (not final instances) with GPT-6.1 Sol via the Codex CLI’s “ultra” preset (server-reported reasoning effort xhigh), not the API protocol. Gate: the oracle’s success on the two calibration blocks of 64 development instances; every candidate is admitted by the gate. Sol and Oracle: successes on the same probe instances. Holds: the pilot model solves at most 0.4 of them (the target band); partly: between 0.4 and 0.6; not admitted: more than 0.6. Rule induction’s existing rungs were probed on two blocks of eight. Differences are paired over the probe instances with 95% bootstrap intervals.

FamilyRungGate (oracle)SolOracleSol − oracleStatus
Long campaigns (L4: 12–13 stages; L5: 16–20 stages)
counterfeit coinL41.00 / 1.002/88/8−0.75 [−1.00, −0.38]holds
counterfeit coinL51.00 / 1.001/88/8−0.88 [−1.00, −0.63]holds
hidden lawL40.92 / 0.978/1614/16−0.38 [−0.69, −0.06]partly
hidden lawL50.84 / 0.800/86/8−0.75 [−1.00, −0.38]holds
hidden lawH40.92 / 0.918/88/80.00 [0.00, 0.00]not admitted
hidden lawH50.92 / 0.808/88/80.00 [0.00, 0.00]not admitted
group testing, one wrong resultH41.00 / 0.987/88/8−0.13 [−0.38, 0.00]not admitted
group testing, one wrong resultH50.98 / 0.948/88/80.00 [0.00, 0.00]not admitted
group testing, two wrong resultsH41.00 / 1.007/88/8−0.13 [−0.38, 0.00]not admitted
group testing, two wrong resultsH50.91 / 0.957/87/80.00 [−0.38, +0.38]not admitted
Existing frontier rungs
rule inductionH10.91 / 0.928/1614/16−0.38 [−0.69, −0.06]partly
rule inductionH20.89 / 0.917/1614/16−0.44 [−0.69, −0.13]partly
rule inductionH30.84 / 0.914/1614/16−0.63 [−0.88, −0.38]holds

Table 8.

FamilyRungGate (oracle)SolOracleSol − oracleStatus
Long campaigns on discovery worlds
automaton learningL41.00 / 1.007/1616/16−0.56 [−0.81, −0.31]partly
automaton learningL50.98 / 1.007/1616/16−0.56 [−0.81, −0.31]partly
causal interventionL40.97 / 0.942/88/8−0.75 [−1.00, −0.38]holds
causal interventionL50.91 / 0.880/87/8−0.88 [−1.00, −0.63]holds
control plantL40.83 / 0.896/1614/16−0.50 [−0.75, −0.19]holds
control plantL50.89 / 0.881/87/8−0.75 [−1.00, −0.38]holds
group testingL41.00 / 1.002/88/8−0.75 [−1.00, −0.38]holds
group testingL51.00 / 1.001/88/8−0.88 [−1.00, −0.63]holds
heat inclusionL40.88 / 0.837/88/8−0.13 [−0.38, 0.00]not admitted
heat inclusionL50.84 / 0.844/86/8−0.25 [−0.63, 0.00]partly
hidden lawL4 (14 laws)0.89 / 0.890/87/8−0.88 [−1.00, −0.63]holds
impedance topologyL41.00 / 1.007/88/8−0.13 [−0.38, 0.00]not admitted
impedance topologyL51.00 / 0.988/88/80.00 [0.00, 0.00]not admitted
knockout networkL41.00 / 1.000/88/8−1.00 [−1.00, −1.00]holds
knockout networkL51.00 / 1.000/88/8−1.00 [−1.00, −1.00]holds
mechanism discriminationL40.94 / 0.978/88/80.00 [0.00, 0.00]not admitted
mechanism discriminationL50.97 / 0.986/87/8−0.13 [−0.50, +0.25]not admitted
rule inductionL41.00 / 1.000/88/8−1.00 [−1.00, −1.00]holds
rule inductionL51.00 / 1.000/88/8−1.00 [−1.00, −1.00]holds
trace debuggingL41.00 / 1.002/88/8−0.75 [−1.00, −0.38]holds
trace debuggingL51.00 / 1.000/88/8−1.00 [−1.00, −1.00]holds
Coupled multi-engine campaigns
chemical processC11.00 / 0.978/88/80.00 [0.00, 0.00]not admitted
chemical processC21.00 / 0.987/88/8−0.13 [−0.38, 0.00]not admitted
chemical processC30.97 / 1.006/88/8−0.25 [−0.63, 0.00]not admitted
chemical processC40.95 / 0.921/88/8−0.88 [−1.00, −0.63]holds
drug pipelineC11.00 / 0.987/88/8−0.13 [−0.38, 0.00]not admitted
drug pipelineC20.95 / 0.975/88/8−0.38 [−0.75, −0.13]not admitted
drug pipelineC30.78 / 0.913/86/8−0.38 [−0.88, +0.25]holds
instrument in the loopC31.00 / 1.004/88/8−0.50 [−0.88, −0.13]partly

Table 9. Long-horizon campaigns probed on development instances (not final instances) with GPT-6.1 Sol via the Codex CLI’s “ultra” preset (server-reported reasoning effort xhigh), not the API protocol, eight instances per rung (sixteen where a rung near the band edge was probed again; four rungs at 3–5 of 8, heat inclusion L5, drug pipeline C2 and C3 and instrument-in-the-loop C3, were not probed again, § 7.1). Gate: the oracle’s success on the two calibration blocks of 64 development instances; every rung is admitted by the gate. Sol and Oracle: successes on the same probe instances, pooled over two blocks of eight where a rung near the band edge was probed again; differences paired with 95% bootstrap intervals. Status as in Table 8.

Long campaigns hold the stronger model. Two long-horizon rungs of counterfeit coin (L4, 13 batches from 6 mints; L5, 20 batches from 10 mints) and an L5 campaign of hidden law (a 16-law cascade whose last answer needs constants of early laws) are admitted by the gate with the oracle at 0.80–1.00 per block, and they hold the pilot model at 0 to 2 of 8 probe instances, below the oracle on the same instances on each (counterfeit coin L4 −0.75 [−1.00, −0.38], L5 −0.88 [−1.00, −0.63]; hidden law L5 −0.75 [−1.00, −0.38]). They combine an exact pooled budget, a history window of about a stage and a half, a notebook just large enough for a consolidated plan, and late questions about early facts that no later stage can re-measure. Hidden law L4, with 12 laws, is solved on 8 of 16, above the target band; its harder version with 14 laws is in § 7.2.

Added hidden structure does not. Rungs that add hidden structure to a single question, such as factor combinations read through a drifting instrument (hidden law H4 and H5) or a stated number of faulty assay results (group testing H4 and H5, with one or two wrong results), pass the gate but are solved on 7 or 8 of 8 probe instances, so they are not admitted. With a stated model class and a Python tool, the pilot model runs the oracle’s own exhaustive structure search and sequential design; without the tool, it designs error-correcting pools unaided. Existing frontier rungs can be probed the same way: rule induction’s H1 and H2 hold the pilot model only partly and its H3 holds it (Table 8).

Failures are memory and planning, not format. An audit of all 35 failed probe episodes of the candidate rungs found no invalid ending; 2 malformed replies cost a turn and no budget. The audit includes a first version of the counterfeit-coin L5 rung, which the pilot model solved on 5 of 8 probe instances, above 0.6, so it was not admitted and was redesigned as the L5 rung of Table 8. The 32 failures in long campaigns all used the whole pooled budget, and 29 of them wrote fewer notes than the notebook holds; the other 3 filled it only from turn 51 on, after the readings they needed had left the window. They re-derived forgotten facts until the budget ran out or carried early constants too loosely to answer the late questions. The 3 failures on the structure rungs are ties: the pilot model’s own record left two explanations equally likely, and it chose the wrong one. Long campaigns that make memory and planning bind are therefore where frontier headroom remains; these probes are development-instance evidence about candidate rungs, not results on the evaluated suite.

Long campaigns for fourteen discovery worlds

Following that pattern, long-campaign rungs extend twelve more discovery worlds, and hidden law gains a harder L4 with 14 laws, thirteen families in all (L4 with 12 stages, 14 for hidden law, and L5 with 16, or four and five epochs where a family changes regime): one persistent hidden system, a pooled budget at the oracle’s exact need, a window of about one and a half stages, a notebook just large enough for a consolidated plan, and a final question about early facts that no later stage can re-measure, such as the original system before an announced change, a part no longer on the bench, or the first version of a program. All 25 rungs pass the calibration gate on both development blocks, with the oracle at 0.83–1.00 and every gated shortcut, including each family’s coupling shortcut, at most 0.05. Of the 21 rungs probed with the pilot model, 13 hold it (at most 0.4 of the probe instances), 3 hold it only partly (0.4–0.6) and 5 are not admitted (more than 0.6), so harder versions of those families are needed (Table 9); probes on the 4 rungs of circuit fault and reaction network are not reported here.

Coupled multi-engine campaigns

The coupled campaigns chain the suite’s simulators into one project. A chemical-process campaign carries a compound from titration and kinetics through chromatography to a charge plan, and a drug-pipeline campaign carries a candidate from binding and potency assays through pharmacokinetics to a dose, each stage computed by the same laboratory engine and each later stage depending on numbers committed or measured earlier. Their rungs C1–C4 pass the gate with the oracle at 0.78–1.00 per block and every gated shortcut at most 0.23, including a policy that uses only one engine and one that re-measures instead of remembering. The pilot model solves most of the shorter rungs; the longest drug-pipeline rung (C3) holds it at 3 of 8 instances, against 6 for the oracle, a difference that eight instances do not resolve (−0.38 [−0.88, +0.25]), and the four-lot chemical-process rung (C4) at 1 of 8, against 8 (−0.88 [−1.00, −0.63]). An instrument-in-the-loop campaign, in which an optics calibration feeds a spectrophotometric titration and kinetics, holds it only partly, at 4 of 8 (−0.50 [−0.88, −0.13]). These two rungs and drug pipeline C2 were probed on eight instances only (§ 7.1); drug pipeline C3 and the instrument-in-the-loop rung were later also played by the pilot model, with the same rung code, on 32 final instances each, outside the evaluated suite: it solves 6 of 32 drug-pipeline campaigns, against 26 for the oracle (−0.63 [−0.81, −0.44]), inside the target band, and 15 of 32 instrument-in-the-loop campaigns, against 32 (−0.53 [−0.72, −0.34]), in the partly-holding band, consistent with their probe status. All 147 failed probe episodes of the long and coupled campaigns in Table 9 ended on a committed answer, none on an invalid ending; reading 123 of them (every coupled-campaign failure and up to ten per long-campaign probe) found, as in § 7.1, values that left the window without being noted, or were noted without their numbers, and no format failure. Long campaigns in which memory and planning bind are therefore where the strongest model we tested still fails; these probes show the direction of the ladder, not results on the evaluated suite.

Episode Analysis

We analyse the frontier episodes of Flash and the 27B to see where campaigns go wrong.

Campaign accuracy falls stage by stage. Among campaigns that reach their final commit, Flash gets the first stage right in 0.81–0.85 of them on every long-horizon rung, but only 0.55 of third stages on L1, 0.49 of fifth stages on L2 and 0.38 of eighth stages on L3 (Figure 7). The 27B follows a similar curve, lower at every stage. Later stages are not harder in isolation; they depend on earlier truths, on records the agent must keep and, in some families, on the agent’s own earlier commits.

Figure 7. Campaign accuracy by stage. The share of campaigns, among those that reached their final commit, whose kk-th stage commit is right; dashed lines mark whole-campaign success. Campaigns reaching the final commit on L1, L2 and L3: 340, 310 and 224 for Flash (24 instances per family; 18 on L3); 110, 93 and 94 for the 27B (8 per family). L3 campaigns with a ninth stage (29 for Flash, 9 for the 27B) are not shown beyond stage eight.

Most failed campaigns go wrong before the last stage. Of Flash’s 310 L2 campaigns that reach the final commit, 107 are fully right; of the other 203, 103 first go wrong at stage 1 or 2 and 46 at the final stage. Campaigns whose only wrong stage is the last make up 0.15–0.19 of those that reach the final commit on each long-horizon rung. Errors therefore mostly start early and propagate, rather than accumulating only in the final integration step. Flash’s episodes are read under its per-call cap; a cut turn can make any stage’s commit wrong.

Failed episodes replay exactly. An audit of 40 failed Flash episodes in six family rungs without a success, from an earlier run of the frontier tier with the same families and verifiers, found no scoring, parser or simulator fault: replays reproduce every recorded reading, action and verdict, and each of the 32 committed answers re-scores as wrong, so these failures are not artefacts of scoring, parsing or simulation.

Training on the Worlds

Because the rewards are exact and gated, the suite can also be used for training. We train Concise-9B on the core tier by on-policy expert iteration and evaluate it on the core tier’s final instances, and then train the same model by online reinforcement learning on two laboratory families outside the suite (§ 9.3).

Method

In each round the policy plays training-split instances several times, keeps its successful episodes and fine-tunes on every decision of those episodes, its own thinking and action [27, 37, 48]; malformed turns inside rewarded episodes are never targets. Training has two phases. The narrow phase runs three rounds on the twelve trained discovery worlds with the policy’s own episodes only, ending an episode on its first reply without a legal action, and yields Narrow-9B. The broad phase starts from Narrow-9B, runs four rounds under the standard protocol on the 23 families that are not held out, and yields Broad-9B. It balances the kept episodes across families, samples every level including the entry levels, and adds a capped share of successful episodes from Qwen3.8-27B and DeepSeek V4.1 Flash, rendered in the student’s chat format; it also trains on a laboratory data-analysis family outside the suite, built on the same statistics engine as the laboratory families of § 9.3 but distinct from them and not evaluated here (the reinforcement learning of § 9.3 starts from Concise-9B, not from Broad-9B). Episode counts and optimisation settings are in Appendix § 20; the exact composition of the training mixture is withheld. Training comparisons use the final instances whose hidden system does not appear among the training draws. Four criteria for the broad phase were fixed before its evaluation, all relative to Narrow-9B except the last: fewer invalid endings; higher success on the trained discovery worlds, with a paired lower bound above zero; the same on the six families where Narrow-9B succeeded on fewer than 0.10 of episodes; and GPQA

Diamond [31] within eight questions of Concise-9B, with median length at least 70% of Concise-9B’s. All four pass; the comparisons with Concise-9B were not pre-specified.

Results

Expert iteration gives modest gains. On the twelve trained families (837 final instances), Broad-9B succeeds on 0.148, against 0.103 for Narrow-9B, +0.045 [+0.019, +0.072] (pre-specified), and 0.090 for Concise-9B, +0.059 [+0.031, +0.085] (Table 10 and Figure 8). On the three held-out families, it succeeds on 0.167, against 0.083 for Concise-9B, +0.083 [+0.023, +0.144], and 0.065 for Narrow-9B; the gain comes from mechanism discrimination (0.083 to 0.250 over Narrow-9B) and counterfeit coin (0.111 to 0.250), while automaton learning stays at zero, so a bootstrap over the three families does not resolve it (Appendix § 19). In absolute terms the gains are small: after training, the model still fails on more than four in five core-tier episodes of both the trained and the held-out families. Outside the discovery worlds, ChemistryWorld rose from 0.031 to 0.056 (not resolved) and the certificate audit from 0.049 to 0.188 (+0.139 [+0.069, +0.208] over Concise-9B); PhysicsWorld and exact execution did not move.

Table 10. Training results on the core tier’s final instances. Differences are paired by episode with 95% bootstrap intervals (exploratory per family); discovery-world rows sorted by Broad-9B − Narrow-9B. ChemistryWorld (360 episodes) and the four trained PhysicsWorld families (384) are pooled; exact execution and the certificate audit (144 each) and fluid network (96) are single families. Heat inclusion, knockout network and rule induction score 0.000 for every model and are omitted. On the trained families, Broad-9B − Narrow-9B is the pre-specified comparison. {sup}* Narrow-9B below 0.10. {sup}† held out of all training. {sup}‡ includes the two evaluation-only levels.

FamilyConcise-9BNarrow-9BBroad-9BBroad-9B − Narrow-9BBroad-9B − Concise-9B
group testing0.3260.5650.804+0.239 [+0.065, +0.413]+0.478 [+0.283, +0.674]
circuit fault0.0970.1110.208+0.097 [−0.014, +0.208]+0.111 [+0.000, +0.222]
control plant{sup}*0.0830.0000.097+0.097 [+0.042, +0.167]+0.014 [−0.069, +0.097]
hidden law0.1810.1110.194+0.083 [−0.042, +0.208]+0.014 [−0.097, +0.125]
trace debugging0.1670.1940.250+0.056 [−0.042, +0.153]+0.083 [−0.028, +0.194]
pathway fault{sup}*0.0970.0830.125+0.042 [−0.056, +0.125]+0.028 [−0.069, +0.125]
impedance topology{sup}*0.0000.0000.014+0.014 [+0.000, +0.042]+0.014 [+0.000, +0.042]
reaction network0.0970.1810.181+0.000 [−0.111, +0.111]+0.083 [−0.014, +0.181]
causal intervention0.1110.1530.139−0.014 [−0.111, +0.097]+0.028 [−0.083, +0.139]
twelve trained families0.0900.1030.148+0.045 [+0.019, +0.072]+0.059 [+0.031, +0.085]
mechanism discrimination{sup}†0.1530.0830.250+0.167 [+0.056, +0.278]+0.097 [−0.042, +0.236]
counterfeit coin{sup}†0.0970.1110.250+0.139 [+0.028, +0.264]+0.153 [+0.042, +0.278]
automaton learning{sup}†0.0000.0000.0000.0000.000
three held-out families0.0830.0650.167+0.102 [+0.051, +0.157]+0.083 [+0.023, +0.144]
ChemistryWorld0.0310.0250.056+0.031 [+0.008, +0.056]+0.025 [+0.000, +0.050]
PhysicsWorld, trained0.0100.0050.008+0.003 [−0.005, +0.010]−0.003 [−0.016, +0.010]
exact execution{sup}‡0.6180.5620.611+0.049 [−0.014, +0.111]−0.007 [−0.056, +0.035]
certificate audit0.0490.1320.188+0.056 [−0.021, +0.132]+0.139 [+0.069, +0.208]
fluid network{sup}†0.0100.0100.000−0.010 [−0.031, +0.000]−0.010 [−0.031, +0.000]
on the 1,053 compared instances
invalid endings0.0950.0490.013
experiments per episode2.692.783.06

Table 10.

Figure 8. Training results, from Table 10. (a) Pooled success on the final instances, with 95% Wilson intervals (unpaired; the paired differences are in Table 10). (b) Per-family paired difference Broad-9B minus Concise-9B with 95% bootstrap intervals for the fifteen discovery worlds, sorted by Concise-9B’s success (right column). Open markers and † mark held-out families.

Part of the gain is action format. Over the 2,181 final instances of the comparison, invalid endings fell from 0.090 for Concise-9B and 0.059 for Narrow-9B to 0.016 for Broad-9B, −0.043 [−0.054, −0.032] against Narrow-9B. The largest family gain, group testing (0.326 to 0.804 on its 46 final instances; +0.239 [+0.065, +0.413] over Narrow-9B and +0.478 [+0.283, +0.674] over Concise-9B), is mostly format: Concise-9B ends 0.59 of those episodes invalid. On the other eleven trained families, Broad-9B is still +0.034 [+0.009, +0.059] above Concise-9B, and every result is read with its invalid-ending rate.

Gains appear where the model already succeeds. All ten discovery worlds where Concise-9B succeeds at all have positive point estimates, from 0.014 to 0.478 (Figure 8b). The five where it never succeeds gain at most one episode, as success filtering predicts: such a family supplies none of the policy’s own episodes at its core levels. Headroom does not explain where the gains fall: pathway fault, with a headroom of 0.82 against the 27B, gains 0.028, and heat inclusion, with 0.92, gains nothing. In the last round, rule induction succeeded on three quarters of its entry-level episodes and on none at its core levels, so the hardest families still need a first success at the levels that are evaluated.

On-policy expert iteration at 27B (low-rank adapter, own episodes, two rounds). A lighter variant of the recipe, applied to Qwen3.8-27B, trains a low-rank adapter on the model’s own successful episodes only, without a narrow phase or a teacher share, and was read after two rounds. On the three held-out discovery worlds, success rises from 0.394 for the base model to 0.514 (216 paired final instances), +0.120 [+0.046, +0.194] at the episode level; the gain comes from counterfeit coin (24 to 46 of 72) and mechanism discrimination (47 to 51), while automaton learning stays at 14 of 72, so a bootstrap over the three families does not resolve it, +0.120 [+0.000, +0.306]. On the twelve trained families the difference is not resolved, +0.019 [−0.013, +0.052] (852), and GPQA Diamond stays at 179 of 198, as for the base model.

No detectable loss of general reasoning. On GPQA Diamond, Broad-9B answers 156 of 198 questions correctly, against 155 for Concise-9B and 157 for Narrow-9B, with a median generation length 98% of Concise-9B’s (Table 19). No GPQA outcome was used in training.

Concise-9BNarrow-9BBroad-9B
correct155157156
discordant with Concise-9B–14 / 129 / 8
median generated tokens11,34911,03511,094

Table 19. GPQA Diamond, all 198 questions, one response per question per model with identical prompts, options, question order and seeds, scored on the last boxed answer. Discordant: questions answered correctly by only this model / only Concise-9B.

Episode records support route checks. In causal intervention, success changed little (0.111 for Concise-9B, 0.139 for Broad-9B, of 72 final instances each), but the route to the joint-effect answer did: the share of joint-effect episodes answered from observational estimates, without an intervention, rose from 0.22 (5 of 23) to 0.79 (19 of 24). Such estimates land inside the published tolerance on 0.07–0.16 of development instances (about 140 per level), well below the shortcut gate but close to the policy’s own success on that question. Because every episode records its experiments, a training filter can also require the family’s target experiment, which we recommend where a policy’s success is low.

Reinforcement learning on laboratory worlds

Expert iteration learns only from successful episodes, and on the suite its gains are small. We therefore also train Concise-9B by online reinforcement learning, which learns from failed attempts as well as successful ones, on two laboratory families outside the suite, in which the agent analyses an experiment’s data with a statistics engine and commits decisions that a verifier checks.

Laboratory worlds. A replication world gives the agent a scientist’s brief: a claimed treatment effect and the smallest difference that matters; the lab’s own experiment as a data table, with its design and a quality-control log; a budget for new experiments, priced per experimental unit and per measurement; and the lab’s decision standard. The design states how animals, wells or donors were assigned to conditions, whether whole containers such as cages or plates were assigned, and how repeated measurements nest within units. The standard maps the 95% interval of the treatment effect, estimated at the experimental unit [16, 20] after exactly the exclusions that the log documents, to one of four decisions on the claim (retain, revise, reject or insufficient evidence), and it requires the final decision to rest on an independent replication; when the design does not determine the unit or an exclusion, the required decision is insufficient evidence, with a statement of what cannot be determined. Within seven tool calls the agent analyses the table, commits a decision on the claim, sizes and runs a replication within the budget and analyses its data, and then replies with a final decision. Its tools are those of a laboratory data-analysis assistant: table inspection, validated statistical analyses such as tt tests at a chosen unit and mixed models, design checks, power calculations, a calculator, the replication experiment and the decision commit; a malformed or refused call returns its reason. A deterministic verifier recomputes every analysis that a decision cites and accepts the episode only if each decision follows the lab’s rule from an analysis at the right unit with the documented exclusions, the replication fits the budget, and every number in the answer comes from a tool result or from the brief. Worlds span ten design strata, from one measurement per unit and repeated measurements through treatments assigned to containers and nested labels that persist in every container to quality-control judgements, replication sizing and designs that leave the unit undetermined. A method-comparison world, built from the same laboratory engine as the coupled campaigns of § 7.3, is a two-stage campaign on a hidden analyser: within a sample budget the agent chooses how many patient samples to measure with the new and a reference method and over what range, locks the analysis set with any exclusions the data justify, and commits whether the two methods are interchangeable at two medical decision points, with the bias at each, under the laboratory’s Deming-regression procedure [22]. It has eight tool calls, and its verifier checks the commit against the hidden analyser and accepts it only if the locked data cover and settle both decision points. Both families state their decision rules and the checks on an answer in the brief. They are training environments outside the benchmark suite, built in the same way, with a hidden truth, a stated rule and a deterministic verifier, but not part of its calibrated tiers.

Method. Each update draws 24 worlds that no earlier update used, plays each 8 times with the current policy and takes one optimiser step on all 192 episodes. The reward is binary and terminal, 1 when the verifier accepts the episode and 0 otherwise, without length shaping. An episode’s advantage is its reward minus the mean reward of its group of 8 attempts at the same world [34], without division by the group’s standard deviation [23]. The loss clips the importance ratio [33], bounds negative-advantage tokens with a dual clip [46] and adds a per-token penalty on the divergence from the starting model; it is divided by a fixed token count, so that an episode’s length does not change the weight of its tokens, and only the model’s own tokens are trained, not observations or tool results. All parameters are trained at a learning rate of 3×10−63 \times10^{-6} for 16 updates, with sampling at temperature 1.0 without truncation and at most 4,096 tokens per turn and 16,384 per episode. The starting model, Warm-9B, is Concise-9B after a short supervised warm-up on turns from its own episodes in training replication worlds. Two runs start from Warm-9B with the same seed. LabRL-9B trains on 24 replication worlds per update (384 in all). MixRL-9B trains on 16 of those replication worlds per update, in the same order and with the same environment seeds, plus 8 method-comparison worlds (256 and 128 in all), so that it replaces a third of LabRL-9B’s replication practice with method comparison. The runs took 9.4 and 8.9 hours on one B200 GPU; Appendix § 20 gives every setting.

Registration and evaluation. The design, its checkpoints and a rule for reading the results were registered before the runs began, and the registration treats this round of training as exploratory. Its primary comparison, MixRL-9B against LabRL-9B on a locked test set of complete laboratory workflows built from public replication data, has not been read and is not reported here. We report the registered secondary evaluations: a held-out panel of the two training families, a benchmark of laboratory data analysis, a development set of complete workflows and GPQA Diamond. The panel holds 30 replication worlds, three from each of the ten design strata, and 8 method-comparison worlds, four at each of two levels, drawn from the families’ development splits. It is disjoint from training: no panel world, scenario, parent study or task content occurs among the 1,726 training-split worlds that the runs and the warm-up drew from, which include every world either run trained on (Appendix § 20). Every system plays each panel world 8 times with the same environment and sampling seeds; success per world is the mean over its episodes, averaged over worlds, and differences are paired by world. The checkpoints after 16 updates are the registered ones; those after 8 and 12 updates were evaluated and not selected on. For reference, DeepSeek V4.1 Flash plays the panel through the same tools, prompts, verifier and per-turn caps, counted in the local model’s tokens, at low reasoning effort, the setting fixed before its run as its reference reading, and at its default effort (“high”), at which more of its turns reach the cap (66 capped turns on the panel, against 24 at low effort). These comparisons with Flash are not part of the registration.

Reinforcement learning raises held-out replication success above Flash under the 9B’s caps. On the 30 held-out replication worlds, LabRL-9B succeeds on 0.721, against 0.221 for Warm-9B, +0.500 [+0.400, +0.596], and its point estimate rises at each evaluated checkpoint: 0.579, 0.646 and 0.721 after 8, 12 and 16 updates (Table 11; Figure 9a). It is above DeepSeek V4.1 Flash at low effort (0.446), +0.275 [+0.150, +0.404], and at its default effort (0.371), +0.350 [+0.217, +0.483]. Both Flash readings hold it to the 9B’s caps of 4,096 tokens per turn and 16,384 per episode, counted in the 9B’s tokenizer; at its default effort Flash reaches the turn cap more often (66 against 24 capped turns) and scores lower than at low effort, so these readings understate Flash, which leads at a larger budget on the broader evaluations below. In training, the mean reward per update rose from 0.20 at the first update to 0.61 at the last.

SystemTrained onReplicationMethod comparison
STARTING MODEL
Warm-9B–0.2210.141
REINFORCEMENT LEARNING, ONE TRAINING SEED
LabRL-9B, after 8 updates192 lab0.5790.172
LabRL-9B, after 12 updates288 lab0.6460.141
LabRL-9B, after 16 updates384 lab0.7210.125
MixRL-9B, after 8 updates128 lab + 64 MC0.5080.219
MixRL-9B, after 12 updates192 lab + 96 MC0.6460.359
MixRL-9B, after 16 updates256 lab + 128 MC0.5830.328
API MODEL, SAME WORLDS, TOOLS AND CAPS
DeepSeek V4.1 Flash, low effort–0.4460.359
DeepSeek V4.1 Flash, default effort–0.3710.094
PAIRED DIFFERENCES, 95% BOOTSTRAP OVER WORLDS
LabRL-9B − Warm-9B+0.500 [+0.400, +0.596]−0.016 [−0.078, +0.062]
MixRL-9B − Warm-9B+0.362 [+0.283, +0.446]+0.188 [+0.078, +0.281]
LabRL-9B − DeepSeek V4.1 Flash, low+0.275 [+0.150, +0.404]−0.234 [−0.375, −0.062]
MixRL-9B − DeepSeek V4.1 Flash, low+0.138 [+0.017, +0.254]−0.031 [−0.281, +0.203]
MixRL-9B − LabRL-9B−0.138 [−0.221, −0.058]+0.203 [+0.078, +0.328]

Table 11. Table 11. Reinforcement learning on laboratory worlds: success on the held-out panel. Success per world, the mean over 8 paired-seed episodes, averaged over the 30 replication and the 8 method-comparison worlds (240 and 64 episodes per system), none of which occurs in training. Trained on: distinct training worlds up to that checkpoint (MC: method comparison). DeepSeek V4.1 Flash plays the same worlds through the same tools, prompts, verifier and per-turn caps. Differences are paired by world, with 95% intervals from 5,000 bootstrap resamples of worlds. The best system in each column is in bold. One training seed per run; the checkpoints after 16 updates are the registered ones.

Figure 9. Held-out panel success over training, from Table 11. (a) Replication and (b) method comparison: success per world for LabRL-9B and MixRL-9B after 8, 12 and 16 updates, both starting from Warm-9B at 0 updates, with 95% bootstrap intervals over worlds; markers are offset slightly so that intervals do not overlap. The horizontal lines mark DeepSeek V4.1 Flash at low and at default reasoning effort on the same worlds.

Each run improves the family it trains on. LabRL-9B does not move on method comparison (0.141 to 0.125, −0.016 [−0.078, +0.062]). MixRL-9B rises there to 0.359 after 12 updates and 0.328 after 16, +0.188 [+0.078, +0.281] over Warm-9B, and its difference from DeepSeek V4.1 Flash at low effort (0.359) is not resolved, −0.031 [−0.281, +0.203]; on replication it reaches 0.583, +0.362 [+0.283, +0.446] over Warm-9B and +0.138 [+0.017, +0.254] over Flash at low effort. After 12 updates, MixRL-9B matches LabRL-9B on replication (0.646 each; +0.000 [−0.067, +0.071]) with 192 replication worlds against 288, and leads on method comparison, +0.219 [+0.109, +0.344]. At the registered checkpoint after 16 updates, it is lower on replication, −0.138 [−0.221, −0.058], and higher on method comparison, +0.203 [+0.078, +0.328] (Figure 9); with one training seed per run, these checkpoints do not separate a cost of mixing the families from variation between checkpoints. The panel measures learning in the training families on held-out worlds; the broader evaluations below and the locked test address other laboratory work.

On broader laboratory analysis, the gain comes from more reliable finishing, and Flash stays ahead. On a first-party benchmark of laboratory data analysis answered through the same statistics engine (P+X; Table 12), LabRL-9B raises the score from 0.501 to 0.605, +0.104 [+0.062, +0.146], with gains in both suites (projects 0.403 to 0.478, design decisions 0.599 to 0.732); after 12 updates its gain is +0.127 [+0.086, +0.169], and MixRL-9B gains +0.108 [+0.066, +0.151]. The gain comes from finishing: the share of items answered in the required format within the token limit rises from 0.813 to 0.976, and on the 505 items that both models answered, accuracy is 0.642 against 0.645. The registered checkpoint of LabRL-9B after 16 updates, fails one of the benchmark’s registered guards against the starting model: it solves 4 of the 7 real-data items (R) against 6 for Warm-9B, and the guard allows at most one fewer; its other guards hold. Every registered guard holds for LabRL-9B after 12 updates and for MixRL-9B after 16. Without the power-calculation items, which the registration reports separately, the differences from Warm-9B are +0.114 [+0.070, +0.158] for LabRL-9B, +0.125 [+0.082, +0.171] after 12 updates and +0.104 [+0.058, +0.151] for MixRL-9B (Appendix § 20). With up to 24,000 tokens per call at its default effort, Flash stays well ahead on P+X, at 0.849 against 0.605 for LabRL-9B, +0.245 [+0.206, +0.284]. On the development set of a complete-workflow evaluation, 38 workflows built from the public data of three experiments in one study of the Reproducibility Project: Cancer Biology [11], each asking in turn for the experimental unit, in most workflows an exclusion decision, whether the claim holds and the sample size of the next experiment, LabRL-9B scores 0.454 against 0.447 for Warm-9B (+0.007 [−0.105, +0.118]), and the checkpoints after 12 updates score higher: 0.586 for LabRL-9B (+0.138 [+0.013, +0.263]) and 0.658 for MixRL-9B (+0.211 [+0.086, +0.336]), against 0.553 for MixRL-9B after 16 (+0.105 [−0.013, +0.230]); Flash scores 0.895. These intervals resample workflows, which come from three source experiments, so they understate the uncertainty of this small set. Both sets are development splits that informed earlier design choices, and none of the 9B models was trained on their items.

SystemP+XPXAnsweredRWorkflowGPQA
Warm-9B0.5010.4030.5990.81360.447159
LabRL-9B, after 8 updates0.6260.4810.7720.94860.474–
LabRL-9B, after 12 updates0.6280.4920.7650.96460.586–
LabRL-9B, after 16 updates0.6050.4780.7320.97640.454161
MixRL-9B, after 8 updates0.5950.4500.7390.91660.480–
MixRL-9B, after 12 updates0.5960.4190.7720.91860.658–
MixRL-9B, after 16 updates0.6090.4610.7570.94950.553–
DeepSeek V4.1 Flash, default effort0.8490.7940.9040.99860.895–

Table 12. Broader evaluations of the laboratory runs. P+X: a first-party benchmark of laboratory data analysis through the same statistics engine, on its 632 development items (P: 360 multi-step projects; X: 272 experiment-design decisions), one attempt each, scored as the macro mean of the two suites. Answered: share of P+X items answered in the required format within the token limit. R: the benchmark’s 7 real-data items solved. Workflow: the mean step score on 38 development workflows built from public replication data, one attempt each. GPQA: correct answers of 198 under the protocol of Appendix § 19. DeepSeek V4.1 Flash at its default reasoning effort; – not evaluated. P+X minus Warm-9B (paired by item, 95% bootstrap intervals): LabRL-9B +0.125 [+0.081, +0.169], +0.127 [+0.086, +0.169] and +0.104 [+0.062, +0.146] after 8, 12 and 16 updates; MixRL-9B +0.093 [+0.051, +0.137], +0.095 [+0.053, +0.138] and +0.108 [+0.066, +0.151]; MixRL-9B − LabRL-9B after 16 updates: +0.004 [−0.035, +0.045].

General reasoning. On GPQA Diamond under the protocol of Appendix § 19, scored as in Table 19, Warm-9B answers 159 of 198 questions and LabRL-9B 161, with a median generation of 10,560 tokens, 86% of Warm-9B’s 12,348. The two are equal within the run-to-run variation of this measurement: two runs of Concise-9B under the same protocol answered 155 (Table 19) and 162.

Limitations

The families idealise laboratory and reasoning problems, and calibration cannot exclude shortcuts that nobody has written, nor rule out that a route below the gate still shapes training (§ 9.2). The classic-puzzle families measure experimental reasoning in a pure form, not domain science; the suite’s simulator-grounded science is in PhysicsWorld and ChemistryWorld, whose simulators have the known limits recorded in their datasheets. The frontier tier was tuned against the one API model it reports: its target band was a range of Flash success, read from screens of Flash on final instances, and its action parsers, part of its campaign wording, one budget display and the re-ask after a cut turn were revised after Flash’s first screen of those instances, which were then played again (§ 3.2). Selecting rungs against Flash biases Flash’s absolute rates on that tier downwards and favours other models in comparisons with Flash; confirming Flash’s frontier rates on an untouched block of final instances is future work. Flash’s tier-wide results use the provider’s default reasoning effort and 16,384 tokens per call and understate it: on the 84 o=0o = 0 frontier instances, one per family and rung, its maximum effort with up to 393,216 tokens per call raises its successes from 32 to 57, below the pilot model’s 71 and the oracle’s 79, and its maximum-effort rates on the other frontier instances and on the core tier are not measured. The headroom we claim against the oracle, with the pilot model at xhigh effort as the strongest system tested; it lies mainly in the eight-stage campaigns, where neither that model nor Flash at maximum effort solves more than 8 of 14, and, in development-instance probes, in the longer campaigns. The candidate rungs above the frontier tier were probed on development instances with one model at xhigh effort and eight or sixteen instances per rung, and two of them also on 32 final instances; four rungs at 3–5 of 8 were not extended to sixteen as the probe protocol prescribes, and of the partly-holding rungs only one was rebuilt harder, so the probes show a direction for the ladder rather than results on the evaluated tiers. Flash’s PhysicsWorld run covers the first 16 of 24 final instances per level, paired with the other systems on those instances. The pilot plays one final instance per family and rung through the Codex CLI rather than the API protocol, without our per-call token cap, so its rates are pooled estimates with wide intervals that support no per-family claim. The tier’s main results rest on one API model at up to 24 instances per family and rung (19 on H3, 18 on L3), one locally run model at 8, whose thinking is near its 4,096-token cap on most decisions (a mean of 3,813–4,039 tokens per decision), and a one-instance pilot of a third model; a broader model comparison is a natural next step. Scores depend on the evaluation budgets: 4,096 thinking tokens per decision for the local models and 16,384 tokens per call for the API model, which reaches that limit on 27% of its frontier decisions; episodes that end on a re-asked turn succeed far less often, and its maximum effort with a larger budget raises Flash’s success on one instance per family and rung from 0.38 to 0.68 (§ 6), so much of the frontier tier’s difficulty for Flash is a budget effect. The training study is one model and one run. Its intervals cover the sampling of evaluation episodes, not variation across training runs, and its two phases differ in family mix, teacher share, protocol and number of rounds, so their separate contributions are not identified. GPQA Diamond with one response per question detects a large loss of general reasoning but not a small one, and transfer rests on three held-out discovery worlds, two of which improve in point estimate, and one held-out PhysicsWorld family.

The reinforcement-learning study (§ 9.3) trains one seed per run, and its intervals describe the panel’s worlds and episodes, not variation across training runs. Its panel holds 30 and 8 held-out worlds of the two training families, so it measures learning within those families; the registered test on locked real-data workflows has not been read, and the broader evaluations use development splits with one attempt per item, on which the gain carries over through more reliable finishing, with accuracy on the items both models answered unchanged, while Flash stays ahead. The laboratory families are training environments outside the benchmark suite, built in the same way but not part of its calibrated tiers, and we make no claim that the trained models learned to identify the experimental unit from the stated design; the locked workflow test addresses that. Flash’s reference setting on the panel is its low reasoning effort; at its default effort its success is lower on both families (0.371 and 0.094), and the comparison with it is not part of the registration. Both panel readings hold Flash to the 9B’s per-turn and per-episode token caps, which bind it more at its default effort, so the panel understates it; with up to 24,000 tokens per call it leads on the broader evaluations. The registered checkpoint of LabRL-9B fails one registered guard of the broader benchmark (R).

Conclusion

We presented Openhagen Worlds, a suite of calibrated, procedurally generated discovery environments (with finite truth spaces in some families) in which hidden truths are checked exactly, every scoring rule is published and a gate at every level holds the naive policy to at most 0.25 and every gated shortcut to at most 0.35 of development instances while the oracle succeeds on at least 0.75. Its core tier separates the local models from the oracle and approaches saturation for frontier models; on both tiers, Flash’s tier-wide results are measured under a capped per-call budget and understate it: at its maximum effort it solves 57 rather than 32 of 84 frontier instances, one per family and rung, still below the oracle’s 79. In a pilot, a stronger model at xhigh reasoning effort solves most of the frontier tier, and Flash at its maximum effort closes about two thirds of the gap to it (57 against 71 of 84; 7 against 8 of 14 on the eight-stage campaigns), leaving headroom mainly in those campaigns and in the gap to the oracle, so the ladder has to keep climbing; development-instance probes show that most longer campaigns admitted by the same gate, which make memory and planning bind, hold that model where added hidden structure does not. Under Flash’s per-call cap, failed episodes mostly end on wrong answers rather than malformed output, and campaign errors start early and compound. On-policy expert iteration on the suite gives a 9B model modest gains, resolved on the trained families and carried by two of three held-out families, without a detectable loss on GPQA Diamond, and its episode records let a training filter check how each answer was reached. In a separate study, online reinforcement learning on laboratory worlds outside the suite raises a 9B model’s success on held-out replication worlds from 0.221 to 0.721 in one training run, above Flash held to the 9B’s token caps but not above Flash at a larger budget on broader laboratory analysis (0.849 against 0.605), and its registered checkpoint fails one registered guard. Beyond single worlds, we have composed the families into multi-laboratory campaigns, in which samples and measurements pass between laboratories, runs fail or drift, and requests change partway through; training runs that included these campaigns were accompanied by gains on held-out laboratory tasks, and we will report a controlled comparison separately. We hope the suite serves as a calibrated measure of experimental reasoning; we plan to extend the frontier tier’s evaluation to more models and to the full set of withheld instances at maximum-effort budgets, to admit the longer campaigns as rungs of the evaluated suite, and to read the locked workflow test of the laboratory runs and repeat them over training seeds.

Availability

Openhagen Worlds, including the datasheet of every family and the frontier tier, is available to research and industry partners under licence; the multi-laboratory campaigns are available on the same terms. A hosted evaluation service (https://eval.openhagen.ai; its front page and https://eval.openhagen.ai/evaluation document the API, and access tokens are issued on request to [email protected]) plays exactly the core tier’s 2,208 final instances with the same family code and protocol, and reveals only the observation, reward, termination and success, so any model can be scored on the instances behind Table 10 and Table 23 without access to the generators. Played through the service, the scripted oracle and naive policy reproduce all 30 of their per-family final-instance rates on the discovery worlds. A frozen sample of 48 development-split instances from two families, trace debugging and causal intervention, each with its observation, hidden truth and a verified gold answer, and one scripted-oracle episode per family, is released with this report under CC BY 4.0, as ancillary files of this preprint and at https://eval.openhagen.ai/sample/ (Appendix § 14). The keys of the final instances and of the protected-structure filter are held by Openhagen and never distributed.

FamilyConcise-9B successConcise-9B invalidQwen3.8-27B successQwen3.8-27B invalidOracleNaiveHeadroom
heat inclusion0.0000.0690.0000.0000.9170.0140.917
rule induction0.0000.1390.0280.0000.9170.0000.889
impedance topology0.0000.0280.1250.0140.9580.1110.833
pathway fault0.0970.0140.1670.0140.9860.0140.819
automaton learning†0.0000.0000.1940.1251.0000.0000.806
knockout network0.0000.0970.2500.0001.0000.0970.750
control plant0.0830.0000.2500.0000.9860.0000.736
counterfeit coin†0.0970.2920.3330.0001.0000.0690.667
circuit fault0.0970.0280.3890.0000.9720.1670.583
reaction network0.0970.0000.3750.0280.9310.0140.556
causal intervention0.1110.0000.4030.0000.9580.0000.556
trace debugging0.1670.3470.5000.0001.0000.0280.500
mechanism discrimination†0.1530.0000.6530.0000.9860.0690.333
hidden law0.1810.0140.8060.0000.9440.0280.139
group testing0.3470.5141.0000.0001.0000.2500.000
twelve trained families0.0980.1040.3580.0050.9640.0600.606
three held-out families†0.0830.0970.3940.0420.9950.0460.602

Table 23. The core tier on all 1,080 final instances of the discovery worlds (24 per level, 72 per family); the scripted policies played the same instances. Invalid: share of episodes ending on replies without a legal action. Headroom: oracle minus Qwen3.8-27B. Rows sorted by headroom. † held out of all training.

Acknowledgements

The author is not a native English speaker and used AI writing assistance to help write this paper; AI coding assistants were also used to implement and run experiments under the author’s direction. The research, including its questions, design, decisions and conclusions, is the author’s own, and the author takes full responsibility for the content.

Sample episodes from the public sample

The public sample holds eight development-split instances per core level of two families, each with its observation, hidden truth and a verified gold commit, and one scripted-oracle episode per family. Its four files, README.md, MANIFEST.json (record counts and SHA-256 digests), instances.jsonl and oracle_episodes.jsonl, ship as ancillary files of this preprint and at https://eval.openhagen.ai/sample/, under CC BY 4.0. We show the first level-0 instance of each family, with its observation abridged only where marked, and the oracle’s recorded episode on it.

Causal intervention. The agent must find the direct causes of a target in a hidden linear structural causal model with an unobserved common cause, using at most four single-variable interventions.

Observation (abridged)

variables:     growth, humidity, pressure, turbidity             target: growth
candidates:    humidity, pressure, turbidity
intervention_range:  humidity [3.5, 13.3]  pressure [6.6, 18.2]  growth [10.6, 33.2]
                   turbidity [16.2, 27.8]
model:         hidden linear structural causal model over the observed variables: each variable =
               intercept + sum of coefficient x (its direct causes) + Gaussian noise; the causal graph
               is a random DAG (names carry no information); an unobserved common cause may drive
               several variables; every direct effect on growth has |coefficient| >= 0.6.
               INTERVENE X=v sets X to v in a fresh batch of 20 samples (X’s own causes are cut; the
               other variables respond through the graph)
objective:     find the set of direct causes of growth: the variables that appear with a nonzero
               coefficient in growth’s own structural equation (not causes acting only through other
               variables, not effects of growth, not merely correlated variables)
success_rule:  the COMMIT succeeds only if the committed set equals the set of direct causes exactly;
               an unknown variable, a value outside the intervention range or an intervention beyond
               the budget ends the episode at zero
grammar:       INTERVENE <variable>=<value>; at most 4 interventions.
               Then: COMMIT causes=<comma-separated variable names> (or COMMIT causes=none).
history:       observational record, n = 200: means growth 21.88, humidity 8.38, pressure 12.39,
               turbidity 21.94, and the 4 x 4 covariance matrix
Scripted-oracle episode

1  INTERVENE humidity=13.3   -> means: growth 28.57, pressure 17.90, turbidity 26.89 (n = 20)
2  INTERVENE pressure=18.2   -> means: growth 29.41, humidity 8.61, turbidity 27.01
3  INTERVENE turbidity=27.8  -> means: growth 21.29, humidity 8.23, pressure 12.05
4  COMMIT causes=pressure    -> correct; reward 1

Setting humidity moves growth, but only through pressure: the hidden truth is humidity → pressure → growth and pressure → turbidity, with the unobserved cause driving growth and humidity, so an observational regression would credit humidity as well.

Trace debugging. The agent must find the one buggy line of a short program and its fix, using at most five runs or line traces.

Suite details

Observation (abridged)

program:    def f(c, w):
            1|  bonus = 0
            2|  gap = 0
            3|  for j from 2 to c + 3:
            4|      if j % 3 == 0:
            5|          gap = gap - c - c
            6|      gap = gap + w * bonus
            7|  return gap + bonus * 2
input_domain: f(c, w) with every argument an integer from 0 to 6
examples:   f(0, 1) = 0   f(1, 2) = -2   f(1, 3) = -2   f(2, 5) = -4   f(4, 1) = -16   f(6, 0) = -24
bug:        exactly one line differs from the intended program, by one edit: an operator swapped
            within its kind, an integer constant off by one, or a variable read replaced by another
            variable defined at that line; some examples may already pass
tools:      RUN: the shown program’s output. TRACE: the variables defined at line n just before
            each run of line n (first 12 runs)
success_rule: the COMMIT succeeds only if the program with that line’s expression replaced by the
              fix returns the intended value for every input of the domain; an input outside the
              domain, a variable not defined at that line or an action beyond the budget ends the
              episode at zero
grammar:    RUN input=(<int>, <int>) or TRACE input=(<int>, <int>) line=<n>; at most 5 in total.
            Then: COMMIT line=<n> fix=<that line’s expression>
Scripted-oracle episode

1   TRACE input=(6, 0) line=4  ->  line 4 ran 8 times, j = 2, 3, ..., 9; gap 0 -> -12 -> -24
2   COMMIT line=3 fix=2 to c + 2  ->  correct; reward 1

The trace shows the loop running to j=9=c+3j = 9 = c + 3, one step past the intended bound; with the loop ending at c+2c + 2, every example holds.

Suite details

Calibration details

Figure 10 and Table 15 to Table 17 give the calibration of every core level, and Table 18 that of every frontier rung. Two diagnostics outside the gate refine the grouping of § 4.1: a fixed sweep of a control plant with the oracle’s fit scores 0.62–0.75, so analysis carries much of that family, and knockout network is solved by a plan fixed in advance from the interaction map, so its skill is deducing that plan rather than reacting to outcomes. Rigid-body identification’s exhaustive design reaches 0.94 at d0 and 0.56–0.66 above it.

Figure 10. Calibration of all 27 families per core level, 64 development instances per level (first of two blocks for the PhysicsWorld and ChemistryWorld families). Shaded bands mark the oracle and naive gates; the dashed line is the shortcut gate (0.35). † held out of all training.

FamilyOracle d0Oracle d1Oracle d2Oracle d3Naive d0Naive d1Naive d2Naive d3Shortcut d0Shortcut d1Shortcut d2Shortcut d3Exhaustive d0Exhaustive d1Exhaustive d2Exhaustive d3
rigid-body identification1.000.940.890.830.140.110.170.140.200.160.280.160.940.620.660.56
airfoil inference1.000.970.950.920.140.080.080.060.000.030.020.000.120.120.090.03
optical bench0.950.840.910.800.140.050.080.030.000.000.000.000.160.190.270.16
coupled oscillators1.000.980.970.910.000.000.000.000.000.000.000.000.000.060.160.00
fluid network†0.970.940.880.860.140.140.170.090.300.220.220.190.620.470.450.45
structure elucidation1.001.000.97–0.000.000.00–0.020.000.00–0.910.800.64–
titration equilibria0.940.860.84–0.120.160.11–0.050.020.00–0.450.340.22–
kinetics process design0.910.860.94–0.000.000.02–0.060.050.03–0.750.580.61–
chromatography method0.940.920.88–0.020.050.00–0.090.030.00–0.920.880.75–
electrochemical mechanism1.000.890.83–0.020.120.14–0.090.200.17–1.000.860.69–

Table 17. Calibration of the PhysicsWorld (d0–d3) and ChemistryWorld (d0–d2) families on the first of two blocks of 64 development instances per level; every cell also passes on the second block. † held out of all training.

FamilyH1 OH1 NH1 SH2 OH2 NH2 SH3 OH3 NH3 SL1 OL1 NL1 SL2 OL2 NL2 SL3 OL3 NL3 S
circuit fault0.910.000.170.840.000.080.860.000.200.830.000.270.810.000.120.890.000.12
impedance topology0.840.120.310.860.050.120.890.090.230.890.080.000.780.000.020.860.000.03
heat inclusion0.890.000.190.830.000.170.810.000.060.810.000.160.780.000.030.920.000.27
hidden law0.880.000.050.840.000.020.810.000.230.920.000.120.890.000.160.840.000.02
control plant0.830.000.080.880.000.000.920.000.000.940.000.270.860.000.270.860.000.22
reaction network0.800.020.170.840.000.000.840.030.250.860.000.200.860.000.080.810.000.00
mechanism discrimination†0.830.090.300.780.000.280.840.000.000.860.000.230.840.000.190.830.000.03
pathway fault–––––––––0.880.050.23–––0.880.000.02
knockout network1.000.020.121.000.000.051.000.000.031.000.000.051.000.000.021.000.000.00
group testing0.920.030.060.970.000.090.880.000.081.000.020.051.000.000.050.970.000.03
counterfeit coin†1.000.140.090.970.110.111.000.080.121.000.000.271.000.000.221.000.000.09
rule induction0.910.000.190.890.030.140.840.030.201.000.080.021.000.000.001.000.000.00
causal intervention0.940.000.000.920.000.080.920.000.080.910.000.000.950.000.120.860.000.09
trace debugging1.000.000.091.000.020.031.000.000.051.000.000.191.000.000.021.000.000.02
automaton learning†0.840.000.280.880.000.000.880.000.000.910.000.020.920.000.000.940.000.00

Table 18. Admission of the frontier rungs on two blocks of 64 development instances per rung. For each rung: the oracle’s lower block (O), the naive policy’s higher block (N) and the strongest gated shortcut’s higher block (S). Gated shortcuts include the next easier rung’s strategy on H rungs, the stage-by-stage policy on L rungs and the oracle without notes on L3. Reported policies are not gated and not shown: they apply the oracle’s exact analysis to a simpler design and are deductive, except one control-plant habit described in § 3.1. A dash marks a rung that is not admitted. † held out of all training.

FamilyPolicyd0d1d2d3d4d5
certificate auditoracle1.001.001.001.001.001.00
naive0.020.090.190.120.030.00
shortcut0.300.280.330.300.330.25
exhaustive0.300.450.480.300.310.34
exact executionoracle1.001.001.001.001.001.00
naive0.000.000.000.000.000.00
shortcut0.020.000.000.000.000.00

Table 16. Calibration of the two MicroWorld families on 64 development instances per level of their six-level ladders. Policies as in Table 15; exact execution defines no exhaustive design.

Evaluation settings

Robustness of the calibration. A second audit re-runs the calibration, determinism and shortcut checks on the distributed build of the suite rather than the research code. Replaying the original calibration instances reproduces 88 of the 90 oracle and naive entries of the fifteen discovery worlds exactly, and the other two differ by a single instance in a simulator-backed family, through floating-point differences between processor architectures. On a separate development split, sixteen families pass every gate at every audited level, 88 family levels in all with entry and harder levels included; 50 of those were run on both macOS arm64 and Linux x86_64 and pass on both, and every update replays each previously audited level byte for byte. On the ten PhysicsWorld and ChemistryWorld families the audit passes 34 of 35 levels; the exception is a rigid-body level whose naive policy reached 17 of 64 on that split, one episode above the gate, and which passes on both calibration blocks. The frontier rungs are admitted on the machine that plays their final instances.

Simulators and determinism. Most families compute their readings exactly: nodal analysis of resistor networks, complex impedance, closed-form kinetics checked against a numerical ODE solver, an exact discretisation of a control loop with dead time, Boolean reachability in signalling networks, charge-balance equilibria, modal analysis, and interpreters for programs and machines. Heat inclusion’s finite-volume solver agrees with a finite-element reference ([15]) to within 19 mK over all 2,916 candidate inclusions, under a quarter of the noise at every core level (at least 80 mK), and the rigid-body integrator agrees with MuJoCo [41] to within 0.04° on 128 fixed release comparisons. The hydraulic solver implements the global-gradient method of EPANET [32]; pathway fault simulates curated BioModels kinetic models [25] with libRoadRunner [43]; and structure elucidation draws from an isomer bank enumerated with RDKit [30]. External simulators run out of process in pinned environments, all randomness is seeded, and every admission run records a replay digest over the truth, every prompt, every oracle action and the return.

Contamination controls. Every distributed data file carries a canary string, every observation an instance hash, and a deduplication tool checks a training corpus against a split. The exact-execution generator rejects every program whose structure matches a protected program-execution benchmark; the protected list exists only as keyed digests under a key that is never distributed, so it cannot be enumerated from the code, and a copy of the generator without the key draws from a different instance space. Comparing 3,000 training truths with 64 development truths per level finds no repeated truth in 22 families. Four families draw their truths from a finite bank by construction and show one opening observation per level, so what can be learned there is the mapping from observations to answers; in pathway fault, the published kinetic models recur across splits, so its splits separate instances, not models.

Evaluation settings

Local models. Every evaluation of Concise-9B, Narrow-9B, Broad-9B and Qwen3.8-27B uses thinking on, a 4,096-token thinking cap per decision with a forced close, temperature 1.0, top-pp 0.95 and top-kk 20, under the protocol of § 3.3. After a forced close, the answer budget is just large enough for the answer, so that the model cannot keep reasoning past the thinking cap.

API model. DeepSeek V4.1 Flash is the snapshot V4.1-Flash-1017, served under the deepseek-flash API name since 17 October 2026; every run reported here used it with thinking on, and the default-effort suite runs used a limit of 16,384 tokens per call. The maximum-effort comparison of Table 6 used reasoning effort “max” with up to 393,216 tokens per call and no re-ask (Table 4). On the core tier, including PhysicsWorld and ChemistryWorld, a turn cut at the limit returns no action, and two such turns in a row end the episode, as for any two replies without a legal action. On the frontier tier a cut turn gets one re-ask of the same messages with thinking off and the episode’s answer budget, asking for the action line only, and the reply counts as the turn’s action; the re-ask carries the cut turn’s reasoning, up to its last 90,000 characters. Its runs on the laboratory worlds of § 9.3 use the same API name with the effort settings and caps given in Appendix § 20, not the 16,384-token limit above. Every response is logged with its exact messages, so every episode can be replayed without new calls.

Pilot model. The pilot (GPT-6.1 Sol via the Codex CLI’s “ultra” preset (server-reported reasoning effort xhigh), not the API protocol) plays the instance with ordinal o=0o = 0 of every family and frontier rung, and of every core level of the discovery worlds, PhysicsWorld and ChemistryWorld. Each decision is one stateless call that carries exactly the messages of the API protocol; the observation already contains the history (on L3, its window and notebook), every tool is disabled except the collaboration (sub-agent) tools that this CLI build offers regardless, which no stored call record shows in use, and the model’s final message is passed to the action parser unchanged. The output length is not ours to cap, so the frontier re-ask does not apply. The pilot runs the same evaluation code on a second machine of a different processor architecture. A replay feeds a recorded episode’s actions to the environment again; its outcome matches when it gives the same return (and, for Flash, the same ending). On that machine, 129 of 130 replayed Flash episodes and 80 of 80 replayed 27B episodes match (49 of Flash’s are byte-identical at every turn; the others differ only after a model-written Python run, whose printed output differs in its last digits across architectures). The one that does not match is a pathway-fault L1 instance that draws a different hidden model, as the hosted-service paragraph below describes for that family, and is excluded from every pilot comparison. Two rigid-body instances show small numerical differences in some joint readings and are kept. The oracle results in Table 7 were played on that machine.

Hosted service. The service plays the core tier’s final instances under the protocol of § 3.3, so a model run there with the settings above is directly comparable to Table 10 and Table 23. One family, pathway fault, relies on a numerical library whose floating-point results differ across processor architectures; the service runs on x86_64, like the evaluation here, and its scripted-policy rates match. Generator internals and the composition of the training mixture are withheld.

GPQA Diamond. All 198 questions with identical presented options, question order, per-question seeds and native thinking-mode prompt for every model; temperature 1.0, top-pp 0.95, top-kk 20, presence penalty 1.5; a 32,768-token reasoning allowance followed, when needed, by a forced close with at most 512 answer tokens; one response per question per model, paired by question and scored on the last boxed answer (Table 19).

Family-level bootstrap. Main-text intervals resample paired episodes, treating those of one family as independent. Table 20 repeats each headline comparison with a bootstrap that resamples whole families with all their paired episodes, computed exactly by enumerating every resample (no Monte Carlo error); with all mmm^m ordered draws of the mm families equally likely, its bounds are the smallest means with cumulative probability above 0.025 and at least 0.975. The family-level intervals are at least as wide, and all stay resolved except Flash minus the 27B on L3 alone and the held-out training gains of the 9B and the 27B, which rest on three families; automaton learning does not change, so a resample that draws only that family shows no gain. On the pilot’s 84 frontier instances, one per family and rung in 14 families, every pilot row stays resolved except the pilot model’s gap to the oracle over all rungs, −0.10 [−0.19, 0.00]; on the long-horizon rungs that gap stays resolved, −0.17 [−0.29, −0.07].

Training details

Optimisation. Both phases fine-tune all parameters for one epoch per round on every decision of the kept episodes (Table 21), at a learning rate of order 10−610^{-6} with batches of a few dozen sequences (exact values withheld). Rollouts use a 4,096-token thinking cap per decision; the narrow phase samples at temperature 1.0 without top-pp/top-kk truncation, and the broad phase samples like the evaluation. In the broad phase’s training rollouts, pooled over all trained families, invalid endings fell from 0.118 in the first round to 0.026, 0.020 and 0.017.

Table 21. The two training phases. Kept episodes are successful episodes per round, pooled over all trained families.

PhaseRoundsFamiliesOwn episodes keptTeacher episodes keptUpdatesTarget tokens
narrow31248, 47, 84413.1M
broad423333, 423, 468, 47581, 45, 24, 1732325.7M

Table 21.

Pre-specified criteria. The four criteria of § 9.1 were fixed before the broad phase’s evaluation: (i) fewer invalid endings than Narrow-9B over all final instances; (ii) higher success than Narrow-9B on the twelve trained discovery worlds, with a paired lower bound above zero; (iii) the same on the six families where Narrow-9B succeeded on fewer than 0.10; and (iv) GPQA Diamond within eight questions of Concise-9B, with median length at least 70% of Concise-9B’s. All four pass: (i) −0.043 [−0.054, −0.032]; (ii) +0.045 [+0.019, +0.072]; (iii) +0.026 [+0.007, +0.046], mostly control plant; (iv) 156 against 155. Because criterion (iii) selects families on the same instances it tests, part of that gain may be regression to the mean.

Entry levels. In the last round’s training rollouts, knockout network succeeded on about half of its entry-level episodes and heat inclusion on about a third, with at most two successes at any other level. Control plant had the only resolved gain among the weak families; it first rose on an upper level of its training ladder, and its final success rose from 0 for Narrow-9B to 0.097 for Broad-9B, with seven episodes gained and none lost.

Reinforcement learning on laboratory worlds. Table 22 gives the settings of both runs. Before each step the trainer recomputes the log-probability of every sampled token and rejects the batch if it differs from the sampler’s beyond fixed tolerances; a token whose two log-probabilities differ by more than 5 nats is dropped from the loss, which happened to 2 tokens in all, both in MixRL-9B. Every update of both runs committed, no episode was excluded, and gradient clipping did not bind in any update.

SettingValue
Starting modelWarm-9B: Concise-9B after a short supervised warm-up on its own turns in training replication worlds
Policy updateadvantage = reward − mean reward of the episode’s group; clipped importance ratio, ϵ=0.2\epsilon= 0.2; dual clip at 3 for negative advantages; per-token penalty on the divergence from Warm-9B, weight 0.03
Rewardbinary and terminal, from the family’s verifier; no length shaping
Group size8 episodes per world
Worlds per update24, none used twice; LabRL-9B: replication only; MixRL-9B: 16 of the same replication worlds and 8 method comparison
Episodes192 per update, 3,072 per run
Updates16, one optimiser step each, on all parameters
OptimiserAdam, (β1,β2)=(0.9,0.95)(\beta_1, \beta_2) = (0.9, 0.95); learning rate 3×10−63 \times10^{-6} after a linear warm-up over 2 updates; no weight decay; gradient norm clipped at 1.0
Samplingtemperature 1.0, no top-pp or top-kk truncation; at most 4,096 tokens per turn and 16,384 generated tokens per episode; context 40,960 tokens
Loss normalisationsum over the batch’s sampled tokens, divided by 192 × 16,384
Hardware and timeone B200 GPU; 9.4 h for LabRL-9B, 8.9 h for MixRL-9B

Table 22. Settings of the two reinforcement-learning runs of Section 9.3, which differ only in their training worlds.

The held-out panel. The panel’s worlds come from each family’s development split, one world per scenario, drawn at random within each stratum with a fixed seed. We checked that they are disjoint from training by world and scenario identifiers, by the study each world derives from, by a hash of its full public task content (brief and data tables) and, for method comparison, by its hidden analyser: none of the 38 panel worlds matches any of the 1,726 training-split worlds, which hold the full training pools of both families, every world drawn for either run (384 replication worlds for LabRL-9B; 256 replication and 128 method-comparison worlds for MixRL-9B) and the 56 worlds behind the warm-up’s rows. Every system plays each panel world with at most 4,096 tokens per turn (8,192 in method comparison, as the panel’s fixed configuration sets for every system), 16,384 generated tokens per episode, and sampling as in training.

Flash and the broader evaluations. On the panel, DeepSeek V4.1 Flash runs under the API name of Appendix § 19 with thinking on, at reasoning effort “low” (the reference setting) or “high” (the default). It receives the same system and user prompts, tool schema, statistics engine, budgets, parser and verifier as the local models, and its output in a turn is cut at the panel’s per-turn cap, counted in the local models’ tokenizer. On P+X and the workflow set it runs at its default effort with up to 24,000 output tokens per call; the 9B models run there with thinking on, temperature 0.6, top-pp 0.95 and at most 12,000 tokens per call. The registration reports P+X both with and without its power-calculation items, which need a pooled standard deviation that no evaluation tool reports: the 86 power steps of the projects and the 51 sample-size decisions. Without them, a project scores 1 when all of its other steps pass and the X suite keeps its other items (581 items in all); the macro score is 0.506 for Warm-9B, 0.631 and 0.620 for LabRL-9B after 12 and 16 updates and 0.592 and 0.610 for MixRL-9B, and the differences from Warm-9B are +0.125 [+0.082, +0.171], +0.114 [+0.070, +0.158], +0.085 [+0.038, +0.132] and +0.104 [+0.058, +0.151], against +0.127 [+0.086, +0.169], +0.104 [+0.062, +0.146], +0.095 [+0.053, +0.138] and +0.108 [+0.066, +0.151] with them. The workflow set has 38 workflows with 152 steps from three source experiments; a step passes when its decisions are right, its values match the engine’s analysis and its numbers come from a tool result or the materials, and a workflow scores its share of passed steps.

Per-family results

References

  1. [1]Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. doi: 10.48550/arXiv.1606.06565.arxiv.org/abs/1606.06565
  2. [2]Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/hash/d8e1344e27a5b08cddfd5d027d9b8d6de-Abstract.html.arxiv.org/abs/1705.08439
  3. [3]Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Statistical Science, 10 (3):273–304, 1995. doi: 10.1214/ss/1177009939.DOI
  4. [4]Howard Chen, Noam Razin, Karthik R Narasimhan, and Danqi Chen. Retaining by doing: The role of on-policy data in mitigating forgetting. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 16479–16494, 2026. URL https://proceedings.mlr.press/v306/chen26do.html.arxiv.org/abs/2510.18874
  5. [5]Yimeng Chen, Piotr Piekos, Mateusz Ostaszewski, Firas Laakom, and Jürgen Schmidhuber. PhysGym: Benchmarking LLMs in interactive physics discovery with controlled priors. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, volume 38, pages 52179–52217, 2025. doi: 10.52202/085713-1556.DOI
  6. [6]Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foundation model post-training. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 10818–10838, 2025. URL https://proceedings.mlr.press/v267/chu25c.html.arxiv.org/abs/2501.17161
  7. [7]Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 2048–2056, 2020. URL https://proceedings.mlr.press/v119/cobbe20a.html.
  8. [9]DeepSeek-AI. DeepSeek V4.1 Flash. Model served through the DeepSeek API, 2026. URL https://api-docs.deepseek.com/updates. Released 10 September 2026; model version DeepSeek-V4.1-Flash-1017, served under the deepseek-flash API name since 17 October 2026; accessed 18 November 2026.
  9. [10]Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: Reward RAnked FineTuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023. URL https://openreview.net/forum?id=m7p5O7zb1Y.arxiv.org/abs/2304.06767
  10. [11]Timothy M. Errington, Maya Mathur, Courtney K. Soderberg, Alexandria Denis, Nicole Perfito, Elizabeth Iorns, and Brian A. Nosek. Investigating the replicability of preclinical cancer biology. eLife, 10:e71601, 2021. doi: 10.7554/eLife.71601.DOI
  11. [12]Kanishk Gandhi, Michael Y. Li, Lyle Goodyear, Agam Bhatia, Louise Li, Aditi Bhaskar, Mohammed Zaman, and Noah D. Goodman. BoxingGym: Benchmarking progress in automated experimental design and model discovery. arXiv preprint arXiv:2501.01540, 2025. doi: 10.48550/arXiv.2501.01540.DOI
  12. [13]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):633–638, 2025. doi: 10.1038/s41586-025-09422-z.DOI
  13. [14]Suchin Gururangan, Swabha Sweyamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, 2018. doi: 10.18653/v1/N18-2017.DOI
  14. [15]Tom Gustafsson and G. D. McBain. scikit-fem: A Python package for finite element assembly. Journal of Open Source Software, 5(52):2369, 2020. doi: 10.21105/joss.02369.
  15. [16]Stuart H. Hurlbert. Pseudoreplication and the design of ecological field experiments. Ecological Monographs, 54(2):187–211, 1984. doi: 10.2307/1942661.DOI
  16. [17]Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. DiscoveryWorld: A virtual environment for developing and evaluating automated scientific discovery agents. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, volume 37, pages 10088–10116, 2024. doi: 10.52202/0790017-0324.arxiv.org/abs/2406.06769
  17. [18]Anita Kriz, Antonin Berthon, Nicolás Astorga, and Mihaela van der Schaar. Amortizing sequential experimental design in reasoning LLMs. arXiv preprint arXiv:2610.20832, 2026. doi: 10.48550/arXiv.2610.20832.arxiv.org/abs/2610.20832
  18. [19]Nathan Lambert, Jacob Morrison, Valentina Pyatkin, et al. Tülu 3: Pushing frontiers in open language model post-training. In Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=i1uGbfHHpH.arxiv.org/abs/2411.15124
  19. [20]Stanley E. Lazic. The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis? BMC Neuroscience, 11:5, 2010. doi: 10.1186/1471-2202-11-5.DOI
  20. [21]Ronan Le Bras, Swabha Sweyamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E. Peters, Ashish Sabharwal, and Yejin Choi. Adversarial filters of dataset biases. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1078–1088, 2020. URL https://proceedings.mlr.press/v119/bras20a.html.arxiv.org/abs/2002.04108
  21. [22]K. Linnet. Evaluation of regression procedures for methods comparison studies. Clinical Chemistry, 39(3):424–432, 1993. doi: 10.1093/clinchem/39.3.424.DOI
  22. [23]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective. In Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=5PAF7PAY2Y.arxiv.org/abs/2503.20783
  23. [24]Alexander Boesgaard Lorup. Concise-9B: Removing the slack from a reasoning model’s thinking. arXiv preprint arXiv:2611.02278, 2026. doi: 10.48550/arXiv.2611.02278.DOI
  24. [25]Rahuman S. Malik-Sheriff, Mihai Glont, Tung V. N. Nguyen, et al. BioModels—15 years of sharing computational models in life science. Nucleic Acids Research, 48(D1):D407–D415, 2020. doi: 10.1093/nar/gkz1055.DOI
  25. [26]Harsh Patel. A verifiable search is not a learnable chain-of-thought. arXiv preprint arXiv:2606.21884, 2026. doi: 10.48550/arXiv.2606.21884.DOI
  26. [27]Stanislas Polu, Jesse Michael Han, Kunhao Zheng, Mantas Baksys, Igor Babuschkin, and Ilya Sutskever. Formal mathematics statement curriculum learning. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=P7G-8dmSh4.arxiv.org/abs/2202.01344
  27. [28]Qwen Team. Qwen3.5: Towards native multimodal agents. Qwen release blog, 2026. URL https://qwen.ai/blog?id=qwen3.5.
  28. [29]Qwen Team. Qwen3.8-27B model card. Hugging Face, 2026. URL https://huggingface.co/Qwen/Qwen3.8-27B. Accessed 23 October 2026.
  29. [30]RDKit contributors. RDKit: Open-source cheminformatics. Software, 2026. URL https://www.rdkit.org. Accessed 23 October 2026.
  30. [31]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanze Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98.arxiv.org/abs/2311.12022
  31. [32]Lewis A. Rossman. EPANET 2 users manual. Technical Report EPA/600/R-00/057, U.S. Environmental Protection Agency, Cincinnati, OH, 2000. URL https://nepis.epa.gov/Exe/ZyPURL.cgi?Dockey=P1007WWU.txt.
  32. [33]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. doi: 10.48550/arXiv.1707.06347.arxiv.org/abs/1707.06347
  33. [34]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. doi: 10.48550/arXiv.2402.03300.
  34. [35]Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RL’s razor: Why online reinforcement learning forgets less. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=7HNRYT4V44.arxiv.org/abs/2509.04259
  35. [36]Surya Shetty and Ulisses Braga-Neto. LLM-IDEA: Identifiability-driven experimental agent for autonomous discovery of mechanistic world models. arXiv preprint arXiv:2610.11253, 2026. doi: 10.48550/arXiv.2610.11253.DOI
  36. [37]Avi Singh, John D. Co-Reyes, Rishabh Agarwal, et al. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=1NAyUngGFK.arxiv.org/abs/2312.06585
  37. [38]Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. In Advances in Neural Information Processing Systems, volume 35, pages 9460–9471, 2022. doi: 10.52202/068431-0687.DOI
  38. [39]Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. URL https://openreview.net/forum?id=uyTl5Bvosj.
  39. [40]Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, and Andreas Köpпf. Reasoning Gym: Reasoning environments for reinforcement learning with verifiable rewards. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, volume 38, pages 63612–63655, 2025. doi: 10.52202/085713-1913.DOI
  40. [41]Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012. doi: 10.1109/IROS.2012.6386109.DOI
  41. [42]Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. ScienceWorld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279–11298. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.emnlp-main.775.DOI
  42. [43]Ciaran Welsh, Jin Xu, Lucian Smith, Matthias König, Kiri Choi, and Herbert M. Sauro. libRoadRunner 2.0: A high performance SBML simulation and analysis library. Bioinformatics, 39(1):btac770, 2023. doi: 10.1093/bioinformatics/btac770.DOI
  43. [44]Matt L. Wiemann, Lindsay M. Smith, Peter Melchior, Siddharth Mishra-Sharma, Andrew Gordon Wilson, Pavel Izmailov, and Carolina Cuesta-Lázaro. DiscoverPhysics: Benchmarking LLMs for out-of-the-box scientific thinking. arXiv preprint arXiv:2605.26087, 2026. doi: 10.48550/arXiv.2605.26087.DOI
  44. [45]Yucheng Xu, Keyi Zhang, Yuyang Yu, Min Zhang, Shiyuan Meng, Pei Chu, and Zhongying Tu. Scaling scientific discovery environments for turn-level agentic RL. arXiv preprint arXiv:2607.28990, 2026. doi: 10.48550/arXiv.2607.28990.DOI
  45. [46]Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, Qiaobo Chen, Yinyuting Yin, Hao Zhang, Tengfei Shi, Liang Wang, Qiang Fu, Wei Yang, and Lanxiao Huang. Mastering complex control in MOBA games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6672–6679, 2020. doi: 10.1609/aaai.v34i04.6144.DOI
  46. [47]Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuangqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023. doi: 10.48550/arXiv.2308.01825.DOI
  47. [48]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, volume 35, pages 15476–15488, 2022. doi: 10.52202/068431-1126.DOI
  48. [49]Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104, 2018. doi: 10.18653/v1/D18-1009.DOI
  49. [50]Tianshi Zheng, Kelvin Kiu-Wai Tam, Newt Hue-Nam K. Nguyen, Baixuan Xu, Zhaowei Wang, Jiayang Cheng, Hong Ting Tsang, Weiqi Wang, Jiaxin Bai, Tianqing Fang, Yangqiu Song, Ginny Y. Wong, and Simon See. NewtonBench: Benchmarking generalizable scientific law discovery in LLM agents. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=Gk6umqW74m.arxiv.org/abs/2510.07172

Paper details

vlnc:W_jbdwvff8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

AI writing assistance, AI coding assistants

View statement
Looking for related papers…
Contents