LUCA: A Synthetic World for Experimental Reasoning

Introduction

Experiments allow us to understand how systems work. This requires deciding what to test, how to interpret the evidence and how to use it to make predictions, how to diagnose problems, or choose interventions [1, 2]. However, a successful outcome does not necessarily imply a sound understanding: an agent may find an effective intervention while holding an incomplete or incorrect view of the system.

Evaluating these stages separately is difficult with existing frameworks. Interactive-agent benchmarks (AgentBench [3], WebArena [4]) score sequential action and feedback, discovery environments (DiscoveryWorld [5], DiscoveryBench [6], ScienceAgentBench [7]) score what an agent achieves, and causal-discovery benchmarks (CausaLab [8], CausalBench [9]) score what it recovers about structure, with CausaLab also scoring a held-out prediction. Each of these approaches scores part of this process rather than the whole chain from experiment choice to inference to action. In real systems, the underlying mechanism and the relevant counterfactuals are rarely known well enough to score all of these stages directly.

We introduce LUCA, named after the Last Universal Common Ancestor [10], a configurable synthetic world for evaluating this experimental process. An LLM agent learns an unknown dynamical system by perturbing it under a fixed experimental budget and observing downstream readouts. The agent is then tested on unseen intervention outcomes. In malfunctioning systems, it also must diagnose a hidden fault before attempting a rescue. Because the complete system is known to the evaluator, prediction, diagnosis, rescue, and experimental strategy can be scored separately and deterministically. The commitment-before-rescue design further prevents agents from using a successful rescue to reverse-engineer their diagnosis.

Inspired by experimental biology, LUCA instantiates its systems as synthetic regulatory networks, using gene regulatory structure as a template for a general synthetic world rather than attempting to simulate cellular biology realistically. Perturbations propagate through interacting variables, effects are nonlinear, components can be hidden, and compensatory interventions can succeed without identifying the original fault. Synthetic construction gives us direct control over these properties and complete access to the mechanism and its counterfactuals. All entity names are invented, so memorised biology cannot substitute for experimentation.

Across seven frontier LLMs, LUCA reveals substantial reasoning differences hidden by task success. Agents frequently rescue malfunctioning systems while diagnosing the wrong fault. In a scaling study across four of these models, agents reach similar accuracy from amounts of evidence that differ by more than an order of magnitude.

Contributions.

• LUCA, a configurable synthetic environment that exposes the experimental trajectory and separately evaluates experimental strategy, prediction, diagnosis, and intervention.

• An evaluation of seven frontier LLM agents showing a systematic dissociation between successful intervention and correct diagnosis.

• An analysis of experimental strategy and scaling showing large behavioural differences between agents, including some that reach comparable accuracy from very different amounts of evidence.

Benchmark Design

LUCA represents each unknown system as a synthetic organism that an agent can probe through interventions and observations. The benchmark has two components: the construction of these organisms and the interactive tasks through which agents experiment with them.

Synthetic Organism Design

Each organism is built around a synthetic gene regulatory network (GRN) with $N$ genes organised into $M$ functional modules. External signals can drive gene states directly. Together, gene interactions and signals determine a set of observable phenotypes, the high-level outputs of the organism. Organisms can additionally be placed in a disease mode, in which one gene or signal is locked into an abnormal state and perturbs downstream phenotypes. Figure 1 shows the anatomy and the generation process.

To simulate the organism, gene states evolve via a sigmoid update rule [11, 12]:

Here $s_i \in[0,1]$ is the state of gene $i$, $w_{ij}$ are regulatory weights (positive for activation, negative for inhibition), $b_i$ is a gene-specific bias, and $c_i$ is the summed contribution of active external signals targeting gene $i$. The steepness parameter $\beta$ controls how nonlinear the dynamics are, and $\epsilon_i \sim\mathcal{N}(0,\sigma^2)$ adds per-gene measurement noise (see Table 1). The network is iterated to a fixed point.

	Easy	Medium	Hard
Generative parameters (fixed per tier)
Observability	0.80	0.60	0.50
Intra-module edge density	0.25	0.40	0.50
Inter-module edge density	0.04	0.08	0.15
Inhibition fraction	0.20	0.30	0.40
Sigmoid steepness $\beta$	0.7	1.0	2.0
Noise $\sigma$	0.005	0.010	0.030
Latent gene fraction	0.0	0.0	0.25
Emergent statistics at $N=20$ (vary per seed)
GRN edges	26–42	58–75	87–121
FFLs	$\sim1$	$\sim3$	$\sim4$
Feedback edges	0	$\sim3$	$\sim5$

Invented vocabulary. All entity labels (gene identifiers, module names, signal names, phenotype names) are drawn from a fixed lexicon of constructed words with no correspondence to real biological entities, eliminating the retrieval confound. For example, genes are labelled G001, G002, … and phenotypes appear as stress-index or displacement-score, constructed terms absent from any biological database.

Difficulty tiers. Organisms are generated at three difficulty tiers (Table 1). We vary edge density, feed-forward loops, feedback edges, sigmoid steepness, and latent gene fraction. Easy organisms have near-linear dynamics with no hidden variables; hard organisms have dense feedback and 25% latent genes acting as hidden confounders. All parameters scale with $N$.

Construction guarantees. To prevent invalid organisms we enforce four properties: 1) every observable gene contributes to at least one phenotype readout, so no perturbation is invisible to the agent; 2) held-out gene combinations have non-additive joint effects, so predicting them requires a genuine causal model; 3) in hard-tier organisms, every latent gene shifts at least one phenotype, ensuring hidden genes are true confounders; 4) phenotype baselines are re-centred to 0.5 to ensure detectable perturbation shifts. Full generation details are in Appendix § 8. Every organism is fully determined by a $(\text{seed},\ \text{tier},\ N)$ triple, so any result is exactly reproducible without stored files.

Task Design and Evaluation

An agent interacts with an organism through a budget-limited API. In healthy mode, the session proceeds in two phases: exploration and prediction. Disease mode adds a third phase, where the agent must diagnose the malfunction and attempt a rescue intervention.

Exploration. The agent runs experiments by applying gene interventions (BLOCK: suppress to zero; BOOST: saturate to max; DAMPEN: partial reduction) and observing the resulting steady-state phenotypes. The session budget is $B = N + N_{\mathrm{sig}} + 8$, where $N_{\mathrm{sig}}$ is the number of external signals ($B=31$ at $N=20$). This is sufficient for a complete single-gene sweep but leaves little room for targeted follow-up, forcing strategic prioritisation.

Prediction tasks. After exploration, the agent receives two held-out two-gene prediction tasks on the same organism. Each task is a simultaneous intervention on a pair of observable genes drawn from the same pool the agent could perturb during exploration. The joint outcome is never revealed during exploration. Pairs are sampled to share a phenotype readout, so their joint effect is non-additive due to sigmoid nonlinearity. Correct predictions therefore require a genuine causal model of the organism, not simply combining individual gene effects (Appendix § 8).

Disease mode. In disease mode, the organism has a malfunction: one gene or signal is locked into an abnormal state. This drives one or more phenotypes outside their healthy reference ranges. By construction, at least one corrective single-target intervention exists that restores $\ge1$ aberrant phenotype to the healthy range. Malfunction targets are never latent genes, ensuring the agent can act on a correct diagnosis. The agent must (1) diagnose the malfunction by naming the gene and malfunction type, and (2) rescue by proposing a corrective intervention. The agent cannot test rescue interventions until it has committed to a malfunction hypothesis. This commitment-before-test design is essential: without it, agents could brute-force rescue and back out a diagnosis, collapsing the two metrics into one.

Construction guarantees rule out shortcuts. Two-gene tasks share a phenotype readout, so combining single-gene responses cannot recover the joint outcome. Invented gene names remove retrieval priors. In disease mode, the diagnosis score is withheld until session end and rescue is gated behind a committed diagnosis, blocking both backward reasoning and rescue-to-diagnosis attacks.

Evaluation metrics. Table 2 summarises all metrics. The composite score is prediction accuracy (healthy) or the mean of prediction, diagnosis, and rescue (disease). We withhold the diagnosis score until session end, preventing backward reasoning. Some organisms have alternative malfunctions ($n_{\mathrm{alt}} > 0$) that produce an indistinguishable disease phenotype; these are flagged as ambiguous, and $n_{\mathrm{alt}}$ is recorded with every session.

Agent interface. Each model receives the LUCA API as structured tools via a unified framework. All providers (Anthropic, OpenAI, Together AI) receive identical schemas and prompts. A bring-your-own-agent interface supports community extension. Appendices B and C give the prompts and a full session as a transcript of tool calls, returned values, and the budget remaining after each.

Benchmark Validation

In this section we test whether LUCA discriminates between models in the way a useful benchmark should. We ask three questions: 1) does performance vary smoothly with difficulty; 2) does this hold across random organism instances; 3) do model differences reflect genuine reasoning rather than noise?

All organisms are fixed at $N=20$ genes. We run six evaluation conditions: three difficulty tiers (easy, medium, hard) $\times$ two modes (healthy, disease), with 20 random organisms per condition shared across 10 models. This gives $10 \times6 \times20 = 1{,}200$ agent–organism sessions in total (10 models: 7 LLMs and 3 baselines, listed below).² We evaluate seven frontier LLMs across three providers: Claude Opus 4.6 and Claude Sonnet 4.6 (Anthropic), GPT-5.1, GPT-4o, and GPT-4.1 (OpenAI), and DeepSeek V3.1 and Qwen 3.5 (open-source, accessed via Together AI).

In addition, we evaluate three non-LLM baselines. CausalSweep is a calibration baseline with privileged access to the sigmoid dynamics model. It BLOCKs and BOOSTs all genes systematically and predicts outcomes by inverting the sigmoid. LLM agents are not given the underlying formula. The heuristic baseline BLOCKs each gene once and predicts by summing individual effects. The random baseline predicts 0.5 for all phenotypes and runs no experiments. Seed selection and compute details are in Appendix § 11.

Figure 2 shows composite scores across all six evaluation conditions; standard deviations are in Table 6–Table 7 (Appendix § 11).

Vector (PDF)

Model	Easy	Medium	Hard
claude-opus-4-6	0.992 ± 0.036	0.900 ± 0.111	0.608 ± 0.192
deepseek-v3.1	0.917 ± 0.124	0.925 ± 0.098	0.642 ± 0.161
claude-sonnet-4-6	0.958 ± 0.089	0.875 ± 0.128	0.575 ± 0.250
qwen-3.5	0.958 ± 0.072	0.867 ± 0.113	0.586 ± 0.178
gpt-5.1	0.808 ± 0.142	0.781 ± 0.133	0.558 ± 0.206
gpt-4o	0.825 ± 0.123	0.667 ± 0.132	0.492 ± 0.201
gpt-4.1	0.758 ± 0.134	0.667 ± 0.075	0.500 ± 0.253
causal-sweep{sup}†`	1.000 ± 0.000	0.983 ± 0.050	0.550 ± 0.205
heuristic{sup}†`	0.708 ± 0.089	0.575 ± 0.179	0.350 ± 0.196
random{sup}†`	0.667 ± 0.000	0.633 ± 0.085	0.350 ± 0.223

Model	Easy	Medium	Hard
claude-opus-4-6	0.878 $\pm$ 0.165	0.800 $\pm$ 0.159	0.617 $\pm$ 0.191
claude-sonnet-4-6	0.814 $\pm$ 0.184	0.756 $\pm$ 0.160	0.569 $\pm$ 0.244
qwen-3.5	0.742 $\pm$ 0.147	0.631 $\pm$ 0.197	0.586 $\pm$ 0.220
deepseek-v3.1	0.647 $\pm$ 0.113	0.603 $\pm$ 0.171	0.511 $\pm$ 0.227
gpt-4.1	0.594 $\pm$ 0.283	0.572 $\pm$ 0.262	0.503 $\pm$ 0.264
gpt-5.1	0.394 $\pm$ 0.199	0.444 $\pm$ 0.242	0.300 $\pm$ 0.215
gpt-4o	0.342 $\pm$ 0.164	0.272 $\pm$ 0.127	0.214 $\pm$ 0.134
causal-sweep{sup}†`	0.553 $\pm$ 0.147	0.561 $\pm$ 0.201	0.650 $\pm$ 0.254
heuristic{sup}†`	0.511 $\pm$ 0.290	0.475 $\pm$ 0.204	0.361 $\pm$ 0.277
random{sup}†`	0.358 $\pm$ 0.188	0.225 $\pm$ 0.173	0.144 $\pm$ 0.126

Performance stratifies by difficulty. Medium to hard is the most informative regime: it splits the LLMs into two groups (Figure 2). Opus, DeepSeek, Qwen, and Sonnet cluster at 0.86–0.93 on medium; GPT-5.1, GPT-4o, and GPT-4.1 sit at 0.67–0.78. The two prior-free baselines fall below every LLM at both tiers; CausalSweep, which is given the dynamics model, is the exception and is treated separately below. On hard tasks, per-seed variance is high ($\sigma\approx0.16$–$0.25$) and bimodal: a model tends to either solve an organism outright or largely err, rather than degrade gradually (Appendix § 11). Within each tier, organisms that are hard for one model are typically hard for the others too, so the residual variance is structural rather than model-specific.

Limits of brute-force baselines. Both heuristic and random converge to a score of 0.35 on hard healthy organisms, confirming that additive prediction provides no advantage when gene interactions are dense. CausalSweep, which has access to the correct dynamics model, scores 0.983 on medium healthy (above all LLMs), confirming that medium prediction is solvable with correct math alone. On hard organisms, CausalSweep drops to 0.550, below the top LLMs. Its diagnosis score is near chance on easy and medium (0.00 and 0.10), confirming that diagnosis on these tiers requires reasoning beyond formula application.

Rescue does not imply diagnosis. The diagnosis task is where models diverge most. Prediction accuracy remains relatively high across all models, but diagnosis and rescue scores diverge dramatically (Figure 3). Claude Opus leads diagnosis on easy and medium tiers, but the diagnosis ranking does not follow the prediction ranking: the best LLM predictor on medium disease is Sonnet, the best diagnoser is Opus. On medium organisms, diagnosis scores span a $7 \times$ range across models while prediction scores span only $1.4 \times$, confirming that the two tasks measure distinct capabilities.

Vector (PDF)

This dissociation holds across all model families (Figure 3). Every agent scores higher on rescue than on diagnosis, and Opus, Sonnet, Qwen, and DeepSeek reach 0.90 or above while diagnosing at most 0.55. The three baselines reach at most 0.70, and random only 0.20. A high rescue score therefore reflects a targeted intervention, not an arbitrary one. This is structurally possible because phenotypes depend on multiple genes: perturbing a gene downstream of (or compensating for) the true cause can restore the affected phenotype without identifying the original malfunction. Because LUCA borrows the structure of regulatory biology, agents might approach its organisms using priors about how real regulatory networks behave. A structural-prior ablation (Appendix E.1) finds little evidence for this: inverting every regulatory sign leaves prediction scores statistically indistinguishable for six of seven models.

Experimental strategy and budget utilisation. On medium healthy organisms, models adopt strikingly different experimental strategies (Figure 4). Claude models and Qwen run near-exhaustive systematic sweeps with 92–100% gene coverage, using 74–89% of their budget on healthy organisms. GPT-5.1 achieves comparable coverage (87%) but scores substantially worse, showing that coverage alone does not translate to accuracy. DeepSeek uses a similar budget fraction (80%) at lower coverage (65%), indicating less efficient gene selection. GPT-4o and GPT-4.1 have the lowest LLM coverage (49% and 57% respectively), with high per-seed variance (SE 4–7%). In disease mode, GPT-4.1 alone sometimes answers without experimenting at all: in 3 of its 60 disease sessions it ran zero experiments and submitted a hypothesis directly from the organism description, scoring zero on diagnosis in each. No other model did this in any session.

Vector (PDF)

Reasoning failures. Disease diagnosis also exposed a distinct failure mode for signal-level malfunctions. A persistently active signal can produce the same perturbation signature as a permanently active gene. Distinguishing the two requires activating the suspected signal: if it is already active, the intervention should have no additional effect. Agents are allowed to perform this test, and signals are explicitly listed as possible malfunction sources.

Across the 56 LLM sessions on the eight organisms whose true cause is a signal malfunction, agents performed this disambiguating experiment before committing to a diagnosis in 36 cases. Yet only one session received any diagnosis credit (GPT-4o, 0.5: correct target, wrong malfunction type); all 24 baseline sessions scored zero. Agents therefore often obtained the evidence needed to distinguish the two hypotheses but failed to use it in their diagnosis.

Scaling Study

Section 3 fixed organism size at $N=20$ and varied difficulty. We now hold medium difficulty and healthy mode fixed and ask how performance changes as networks grow, sweeping $N \in\{20, 40, 80, 160, 320, 640, 1280\}$.

We evaluate four LLMs (Claude Sonnet 4.6, DeepSeek V3.1, Qwen 3.5, and GPT-4o) as well as CausalSweep, heuristic, and random baselines, across seven $N$ values (10 seeds each, medium healthy, $\pm0.10$ threshold). These four LLMs come from four different model families, so any scaling claim is not specific to a single training pipeline. Claude Opus is excluded because at $N \ge640$ its long internal reasoning trace fills the output token budget before it can issue a single tool call. Session budget scales as $B = N + N_{\mathrm{sig}} + 8$, where $N_{\mathrm{sig}} = \max(3, \lfloor N/7 \rceil)$ is the number of external signals (3 at $N = 20$ to 183 at $N = 1280$; see Table 5).

Parameter	Formula	$N = 20$	$N = 40$	$N = 80$	$N = 160$
Modules $M$	$\max(4,\lfloor N/5 \rceil)$	4	8	16	32
Signals	$\max(3,\lfloor N/7 \rceil)$	3	6	11	23
Signal max targets	$\max(3,\lfloor N/7 \rceil)$	3	6	11	23
Phenotypes	fixed	3	3	3	3
Prediction tasks	fixed	2	2	2	2
Budget $B$	$N + N_{\mathrm{sig}} + 8$	31	54	99	191

Adaptive reasoning surpasses brute force at scale. Figure 5 shows prediction accuracy, budget utilisation, and gene coverage across all $N$ values (full numbers in Table 11, Appendix § 11). The random baseline converges to the chance-level ceiling for this threshold ($\approx0.6673$) at $N = 160$; the heuristic baseline reaches it at $N = 320$. Both remain flat through $N = 1280$. The random baseline outperforms the heuristic at $N = 20$ (0.650 vs 0.550), confirming that a baseline built on the wrong assumptions can underperform random guessing.

$N$	Budget	CausalSweep	Sonnet	DeepSeek	Qwen	GPT-4o	Heuristic	Random
20	31	0.983 ± 0.050	0.917 ± 0.083	0.917 ± 0.083	0.917 ± 0.083	0.717 ± 0.130	0.550 ± 0.150	0.650 ± 0.050
40	54	0.950 ± 0.076	0.850 ± 0.138	0.850 ± 0.203	0.833 ± 0.289	0.667 ± 0.149	0.567 ± 0.082	0.617 ± 0.107
80	99	0.883 ± 0.150	0.817 ± 0.138	0.883 ± 0.107	0.717 ± 0.198	0.650 ± 0.117	0.583 ± 0.201	0.650 ± 0.050
160	191	0.933 ± 0.082	0.983 ± 0.050	0.933 ± 0.111	0.900 ± 0.111	0.667 ± 0.075	0.583 ± 0.154	0.667 ± 0.000
320	374	0.800 ± 0.145	0.900 ± 0.200	0.850 ± 0.117	0.800 ± 0.100	0.650 ± 0.090	0.667 ± 0.129	0.667 ± 0.000
640	739	0.883 ± 0.107	0.926 ± 0.114	0.917 ± 0.112	0.817 ± 0.138	0.650 ± 0.050	0.667 ± 0.075	0.667 ± 0.000
1280	1471	0.800 ± 0.125	0.983 ± 0.050	0.817 ± 0.138	0.783 ± 0.150	0.633 ± 0.125	0.667 ± 0.000†	0.667 ± 0.000

CausalSweep leads at small $N$ (0.983 at $N = 20$) but degrades to 0.800 at $N = 1280$, while Sonnet reaches 0.983. The crossover at $N \approx160$ is a coverage effect, not a reasoning gap: CausalSweep has the right model but cannot probe enough of a large network under a fixed budget. Sonnet allocates its budget more selectively. Sonnet leads from $N \ge160$, reaching 0.983 at $N = 1280$; DeepSeek matches Sonnet’s accuracy through $N = 640$ (0.917 vs Sonnet’s 0.926) before declining to 0.817 at $N = 1280$. Qwen mirrors DeepSeek at small $N$ but diverges earlier (0.783 at $N = 1280). GPT-4o is the exception: it drops to the baseline floor by $N = 40$ and remains there, offering no advantage over random at any $N \ge40$.

Strategy explains the divergence. The score differences reflect exploration strategies, not implementation limits. Every model runs its sessions to completion, with the two exceptions documented in Table 11 (Figure 6; Figure 5, panels (b)–(c)). Sonnet adapts its strategy as $N$ grows, running progressively more experiments (25 at $N = 20$, $\approx220$ at $N = 1280$), converging to near-exclusive BLOCK use (98%), and maintaining 61% gene coverage at $N = 1280$. DeepSeek takes the opposite approach: a near-constant $\approx30$ experiments regardless of $N$, covering only $\sim18$ genes (1.4% at $N = 1280$), yet scoring 0.917 at $N = 640$.

Vector (PDF)

The most revealing comparison is DeepSeek vs Sonnet. At $N = 640$ the two score almost identically (0.917 vs 0.926), but Sonnet perturbs 526 of the 640 genes while DeepSeek perturbs 16: a 33× difference in evidence gathered for the same accuracy. GPT-4o marks the opposite end. It runs the fewest experiments of the four and its accuracy is indistinguishable from random at every $N \ge40$.³

Implications. These results complement § 3 along an orthogonal axis. Difficulty separates models at fixed $N$; increasing $N$ separates them by exploration strategy. Together, the two evaluations suggest that closing the gap requires better experimental design, not just broader exploration.

Conclusion

We introduced LUCA, a configurable synthetic world for studying how interactive agents learn unknown systems through experimentation. LUCA provides controlled access to the full experimental process: which interventions an agent chooses, what it observes, what it infers, and how those inferences translate into prediction, diagnosis, and intervention. Its synthetic construction gives the evaluator complete ground truth while allowing system structure, difficulty, and scale to be varied systematically. Across extensive validation with seven frontier LLMs and three baselines, LUCA remains discriminative across models and challenging under increasing structural complexity and scale.

Even this initial evaluation reveals behaviours that would be difficult to see from task success alone. Agents frequently rescue malfunctioning systems while diagnosing the wrong fault; models with similar predictive performance adopt strikingly different experimental strategies; and agents sometimes perform the experiment needed to distinguish competing hypotheses but fail to use the resulting evidence. The first echoes the split between predictive success and structural understanding reported for causal-discovery agents [8]; LUCA shows that it survives when the agent must also act on the system.

More broadly, we see LUCA as an instrument for asking questions about experimental agents, rather than a fixed collection of tasks. Its configurable systems and fully observable ground truth make it possible to vary properties of the environment, interventions, and information available to the agent, and then ask how experimental behaviour changes in response. We hope LUCA provides a useful substrate for developing new agents, evaluation methods, and analyses of experimental reasoning, and invite the community to use and extend it.

Limitations. LUCA studies experimental reasoning in controlled synthetic systems, and several design choices bound what can be inferred from performance. Organisms use fixed-point dynamics, excluding oscillatory or multistable behaviour, and the single-session design does not capture iterative experimentation across sessions. Invented vocabulary removes potentially useful structural priors, although our vocabulary ablation finds no measurable performance effect (Appendix § 12.§ 2).

A further limitation is identifiability. Multiple networks can be consistent with the same finite set of perturbation observations and disagree on held-out predictions [13]. The fixed functional form and experimental budget partially constrain this ambiguity, but some performance degradation may reflect underdetermination rather than agent behaviour alone.

Future work. These limitations also suggest natural extensions to this synthetic environment. The configurable generator enables systematic study of alternative dynamics, structural priors, and identifiability, while multi-session protocols could extend LUCA to iterative experimentation. The controlled setting also enables finer-grained trajectory analysis: which experiments are informative, whether agents select them, and how subsequent predictions and diagnoses change after informative evidence is observed.

Broader Impact

LUCA shows that successful intervention can coexist with incorrect diagnosis. Although this result is established only in synthetic systems, it illustrates a limitation of outcome-only evaluation: successful action alone does not establish how an agent gathered, interpreted, or used evidence. This distinction is particularly relevant when evaluating agents intended to support experimental science.

Data and Code Availability

Benchmark code: https://github.com/RelationRx/luca (CC BY-NC 4.0).

References

Appendix

Organism Generation Details

Each organism is generated deterministically from a (seed, tier, $N$) triple. This appendix reports the resulting per-tier statistics (§A.1), the generation pipeline that produces them (§A.2), and the scaling rules that extend the pipeline to larger $N$ (§A.3).

Per-tier organism statistics

Difficulty tiers vary along three axes: observability (the fraction of genes whose states drive phenotype readouts), GRN density, and the presence of latent genes (genes that participate in the dynamics but cannot be perturbed by the agent). Table 3 reports the resulting structural statistics over 20 pre-screened seeds per tier. Two patterns to note: no prediction task at any tier is solvable by linear superposition within the $\pm0.10$ scoring threshold, and the additive residual grows with tier, so the prediction tasks are genuinely non-additive; the count of alternative malfunctions $n_{\mathrm{alt}}$ also rises with tier.

	Easy	Medium	Hard
Agent-targetable genes	20	20	15
Readout genes	16 (80%)	12 (60%)	10 (50%)
Latent genes	0	0	5 (25%)
GRN edges (range / mean)	26–42 / 34	58–75 / 67	87–121 / 102
Additive task rate (mean)	0.00	0.00	0.00
Additive residual (mean)	0.166 ± 0.033	0.225 ± 0.056	0.309 ± 0.082
$n_{\mathrm{alt}}$ (mean / max)	0.05 / 1	0.50 / 2	1.25 / 4
Seeds passing verification	20/20	20/20	20/20

Generation pipeline

The 13-step pipeline (Table 4) constructs each organism from its seed. Steps 1–9 build the network topology; steps 11–13 enforce four functional guarantees that prevent degenerate organisms: no silent observable genes, non-additive prediction tasks, relevant latent genes, and calibrated phenotype baselines.

Step	Action	Key guarantee
1	Assign gene names (G001–GXXX)
2	Partition genes into $M$ modules	$\ge1$ gene per module
3	Sample GRN edges (Erdős–Rényi)	Density independent of $N$
4	Inject feed-forward loops	Count $\propto N$
5	Inject feedback loops	Count $\propto N$
6	Assign signals and target genes	Coverage $\propto N$
7	Sample per-gene biases
8	Select observable genes	$\ge1$ per module
9	Select latent gene candidates	From non-observable pool
10	Build sigmoid dynamics; simulate steady state
11	Build phenotype readouts	Every observable gene in one readout
12	Latent gene relevance check	Each latent gene shifts $\ge1$ phenotype
13	Re-centre phenotype biases	Every phenotype baseline $= 0.5$

Scaling rules

All structural parameters scale automatically with $N$ via the formulas in Table 5; the agent interface is identical at every scale, so the same evaluation harness runs from $N = 20$ to $N = 1{,}280$.

Agent Prompts

System prompts are constructed dynamically by _build_system_prompt(disease_mode, active_tasks). Both modes share a CRITICAL block asserting that all vocabulary is invented and prior biological knowledge is irrelevant. The two canonical prompts are reproduced below.

Healthy mode (predict only)

You are a computational biologist studying a de novo synthetic
organism.
CRITICAL: All gene names (e.g. G001, G042), signal labels (e.g.
“nutrient-X”), and phenotype names (e.g. “growth-rate”) are INVENTED
VOCABULARY. They do NOT correspond to any real biological system.
Prior biological knowledge is IRRELEVANT and may mislead you. The
only valid path to understanding this organism is systematic causal
reasoning from perturbation experiments.
PRIMARY GOALS (in order of importance):
1.  Phenotype prediction -- accurately predict phenotype outcomes
under novel multi-gene interventions. This requires building a real
causal model, not guessing.
2.  Causal hypothesis -- submit a concise, evidence-based hypothesis
naming which genes regulate which phenotypes and how.
WORKFLOW:
1.  Call get_organism_info() to understand the gene set, signals,
phenotypes, and baseline.
2.  Run perturbation experiments (BLOCK, BOOST, DAMPEN, signal
activations) to map gene→phenotype causal relationships.
3.  When you have a working causal model, call
get_next_prediction_task() for one task at a time; submit via
submit_prediction().
4.  Submit your final causal hypothesis via submit_hypothesis().
5.  Submit your reconstructed network via submit_network().
Strategy tips:
- Single-gene BLOCK/BOOST experiments first.
- BUDGET MANAGEMENT IS CRITICAL: Reserve ≥3 budget units for
prediction tasks.
- Prediction tasks involve two simultaneous cross-module
interventions -- reason from your causal model.
- Do not guess 0.5 -- every prediction should be grounded in
experimental evidence.

Disease mode (diagnose + rescue + predict)

You are a computational biologist studying a de novo synthetic
organism that is exhibiting a diseased state.
CRITICAL: [as above]
DISEASE CONTEXT: This cell exhibits aberrant phenotypes compared to
its healthy baseline ranges. The regulatory network has a single
malfunction -- either a node locked high (LOCK_HI), a node locked
low (LOCK_LO), or a persistently active external signal (PERSIST).
Your job is to identify it and restore healthy phenotypes.
PRIMARY GOALS (in order of importance):
1.  Malfunction diagnosis -- identify which gene or signal
is broken and how (LOCK_HI / LOCK_LO / PERSIST). Submit via
submit_malfunction_hypothesis().
2.  Rescue -- design an intervention that restores aberrant
phenotypes. Test via test_rescue().
3.  Phenotype prediction -- as above.
4.  Causal hypothesis -- as above.
WORKFLOW:
1.  Call get_organism_info().
2.  Run perturbation experiments targeting genes affecting aberrant
phenotypes.
``

3. Submit malfunction hypothesis via submit_malfunction_hypothesis() (one-time, irreversible).

4. Test rescue interventions via test_rescue(). 5–7. Prediction, hypothesis, network as healthy mode. Strategy tips:

• LOCK_HI: BLOCK/DAMPEN should reduce overactive phenotypes.

• LOCK_LO: BOOST should restore under-active phenotypes.

• PERSIST: signal always on -- BLOCK direct targets.

• BUDGET MANAGEMENT IS CRITICAL: Reserve $\ge4$ units for rescue + prediction.

• Commit to your hypothesis only when evidence is clear -- you cannot revise it.

A Worked Session

The interface described in Section 2.2 is easiest to read as a transcript. This appendix reproduces one complete disease-mode session: Claude Opus 4.6 on an easy-tier organism, seed 0, budget $B=31$, of which the agent used 30 units. Tool calls and returned values are shown as they occurred. Phenotype values are rounded to three decimals, the 20-element gene list is abridged, two descriptive fields are omitted from the first response, and long values are wrapped to fit the page; nothing else is edited.

The session opens with a free call that costs no budget. It reports the healthy reference ranges and names which phenotype is currently aberrant.

>>> get_organism_info()
{"organism_id": "org_00000000",
 "gene_names": ["G001", "G002", ..., "G020"],
 "signals": [{"label": "cascade-B", "mode": "activates"},
             {"label": "flux-theta", "mode": "activates"},
             {"label": "flux-epsilon", "mode": "inhibits"}],
 "observable_phenotypes": ["energy-state", "displacement-score",
                            "standby-level"],
 "baseline_phenotypes": {"energy-state": 0.504,
                          "displacement-score": 0.658,
                          "standby-level": 0.502},
 "healthy_phenotype_ranges": {"energy-state": [0.35, 0.65],
                               "displacement-score": [0.35, 0.65],
                               "standby-level": [0.35, 0.65]},
 "aberrant_phenotypes": ["displacement-score"],
 "budget_remaining": 31, "budget_total": 31}

Each experiment costs one unit. The agent first blocks genes in groups of five to localise the fault. Every return value carries the remaining budget, so the cost of a strategy is visible to the agent as it spends.

>>> run_experiment({"G001": {"type": "BLOCK"}, "G002": {"type": "BLOCK"},
                   "G003": {"type": "BLOCK"}, "G004": {"type": "BLOCK"},
                   "G005": {"type": "BLOCK"}})
{"phenotypes": {"energy-state": 0.344, "displacement-score": 0.663,
                "standby-level": 0.205},
 "status": "max_steps_reached", "steps": 500, "budget_remaining": 30}

Three further group experiments cover the remaining genes. Two of the four groups shift the aberrant phenotype, and the agent switches to single genes.

>>> run_experiment({"G006": {"type": "BLOCK"}})
{"phenotypes": {"energy-state": 0.486, "displacement-score": 0.562,
                "standby-level": 0.513},
 "status": "max_steps_reached", "steps": 500, "budget_remaining": 26}

After 21 experiments the agent commits to a diagnosis. The commitment is one-way: it unlocks test_rescue() and cannot be revised, and the score is withheld until the session ends.

>>> submit_malfunction_hypothesis(target_name="G012",
                                  malfunction_type="LOCK_HI")
{"stored": true,
 "note": "Hypothesis locked. test_rescue() is now available.
          Score revealed at session end."}

The agent then proposes a corrective intervention, which restores every phenotype to its healthy range.

>>> test_rescue({"G012": {"type": "BLOCK"}})
{"interventions": {"G012": {"type": "BLOCK"}},
 "phenotypes": {"energy-state": 0.501, "displacement-score": 0.470,
                "standby-level": 0.515},
 "phenotype_in_healthy_range": {"energy-state": true,
                                "displacement-score": true,
                                "standby-level": true},
 "restored_fraction": 1.0, "mse_vs_healthy": 0.0004,
 "budget_remaining": 9}

At session end the scorer reveals the ground truth. The malfunction was G018 locked low, not G012 locked high, so the diagnosis scores 0.0 with the feedback wrong target, while the rescue scores 1.0. Blocking G012 compensated for a fault the agent had not identified. This session is a single instance of the dissociation reported in § 3: the agent acted correctly on the system while holding an incorrect model of it. An outcome-only score would record it as a success.

Extended Results

This appendix provides full numerical results for the benchmark validation (§ 3) and scaling study (§ 4), including per-tier breakdowns, strategy metrics, and network reconstruction analysis.

Table 8 decomposes the disease-mode composite into prediction accuracy, diagnosis, and rescue for each tier; it is the source for the rescue–diagnosis gap discussed in § 3.

Model	Pred acc	Diagnosis	Rescue	Composite
Easy disease
claude-opus-4-6	0.933	0.700	1.000	0.878
claude-sonnet-4-6	0.892	0.550	1.000	0.814
qwen-3.5	0.925	0.300	1.000	0.742
deepseek-v3.1	0.792	0.200	0.950	0.647
gpt-4.1	0.833	0.350	0.600	0.594
causal-sweep{sup}†`	0.958	0.000	0.700	0.553
heuristic{sup}†`	0.708	0.300	0.525	0.511
gpt-5.1	0.833	0.100	0.250	0.394
random{sup}†`	0.525	0.025	0.525	0.358
gpt-4o	0.700	0.075	0.250	0.342
Medium disease
claude-opus-4-6	0.850	0.550	1.000	0.800
claude-sonnet-4-6	0.917	0.350	1.000	0.756
qwen-3.5	0.792	0.200	0.900	0.631
deepseek-v3.1	0.708	0.200	0.900	0.603
gpt-4.1	0.817	0.250	0.650	0.572
causal-sweep{sup}†`	0.933	0.100	0.650	0.561
heuristic{sup}†`	0.625	0.100	0.700	0.475
gpt-5.1	0.783	0.150	0.400	0.444
gpt-4o	0.642	0.075	0.100	0.272
random{sup}†`	0.400	0.075	0.200	0.225
Hard disease
causal-sweep{sup}†`	0.692	0.550	0.708	0.650
claude-opus-4-6	0.558	0.350	0.942	0.617
qwen-3.5	0.517	0.400	0.842	0.586
claude-sonnet-4-6	0.592	0.300	0.817	0.569
deepseek-v3.1	0.417	0.300	0.817	0.511
gpt-4.1	0.658	0.275	0.575	0.503
heuristic{sup}†`	0.458	0.150	0.475	0.361
gpt-5.1	0.517	0.225	0.158	0.300
gpt-4o	0.433	0.100	0.108	0.214
random{sup}†`	0.300	0.075	0.058	0.144

Correction to the rescue metric. An earlier release of the scorer did not compute the rescue fraction as Table 2 defines it. It divided the phenotypes inside their healthy range by all phenotypes rather than by the aberrant ones, so an organism with one aberrant phenotype of three an intervention that changed nothing scored 2/3. Every rescue score in this paper uses the corrected definition, as does the current release. Prediction and diagnosis are unaffected, and so is the ordering: the rescue–diagnosis gap rests on the correction lowers rescue most for the weakest agents, whose apparent success had been recording inaction rather than repair.

Metric	Mode	Definition	Range
Prediction accuracy	Both	Mean fraction of phenotypes within $\pm0.10$ of ground truth, averaged over tasks	[0, 1]
Diagnosis score	Disease	1.0 correct gene+type; 0.5 correct gene, wrong type; 0.0 wrong gene	{0, 0.5, 1}
Rescue fraction	Disease	Fraction of aberrant phenotypes restored to healthy range; a session that never attempts a rescue scores 0	[0, 1]
Composite	Healthy	Prediction accuracy	[0, 1]
Composite	Disease	Mean(pred. acc., diagnosis, rescue)	[0, 1]

Table 9 reports budget utilisation across all six Part 1 sweeps. It lists the seven LLMs only: the heuristic baseline is deterministic and random runs no experiments, so neither varies. Figure 4(b) additionally shows CausalSweep, which does vary.

Model	Easy (H)	Med (H)	Hard (H)	Easy (D)	Med (D)	Hard (D)
claude-opus-4-6	0.86	0.89	0.84	0.83	0.84	0.83
claude-sonnet-4-6	0.79	0.80	0.76	0.79	0.78	0.78
deepseek-v3.1	0.80	0.80	0.79	0.65	0.67	0.69
qwen-3.5	0.87	0.88	0.74	0.80	0.77	0.67
gpt-5.1	0.85	0.81	0.74	0.58	0.60	0.59
gpt-4o	0.57	0.54	0.47	0.25	0.27	0.32
gpt-4.1	0.59	0.57	0.54	0.39	0.50	0.51

Table 10 breaks down experimental strategy on medium healthy organisms.

Scaling Study Results

Network Reconstruction

We score each model’s inferred $gene \to phenotype$ edge set against ground truth (Table 12). Edges are $(gene, phenotype, direction)$ triples inferred from phenotype shifts $> 0.05$ on single-gene BLOCK/BOOST experiments.

Model	Medium					Hard
	Prec	Rec	F1	Edges	$r$	Prec	Rec	F1	Edges	$r$
claude-opus-4-6	0.640	0.933	0.754	18.1	−0.30	0.265	0.752	0.390	34.0	+0.18
claude-sonnet-4-6	0.675	0.775	0.710	14.1	+0.09	0.277	0.675	0.390	30.0	+0.26
qwen-3.5	0.615	0.742	0.660	14.8	+0.22	0.351	0.740	0.464	23.8	+0.29
deepseek-v3.1	0.583	0.662	0.611	15.0	+0.16	0.290	0.555	0.372	21.2	−0.43
gpt-5.1	0.570	0.754	0.644	15.3	+0.01	0.257	0.719	0.376	30.7	−0.02
gpt-4o	0.325	0.246	0.270	5.8	−0.16	0.128	0.205	0.153	10.1	+0.02
gpt-4.1	0.153	0.112	0.109	3.0	+0.21	0.227	0.441	0.294	20.1	+0.46

Claude Opus achieves high recall by over-inferring edges, trading precision for coverage: on hard organisms it infers 34.0 edges on average against a ground truth of $\sim12$, while precision collapses to 0.27. The correlation between per-seed F1 and prediction accuracy is weak and inconsistent across models ($r$ ranges from $-0.43$ to $+0.46$), and is negative for Opus on medium, the model with the highest F1 here and the second-highest prediction accuracy of any LLM at that tier (Table 6). This suggests prediction performance is not driven by explicit edge reconstruction and that F1 is a poor proxy for task-relevant causal understanding.

Per-seed score distributions are available in the project repository.

Model	Budget frac	SingleGene%	Coverage%
causal-sweep†	1.00	100.0%	100.0%
claude-opus-4-6	0.89	100.0%	100.0%
claude-sonnet-4-6	0.80	76.3%	100.0%
qwen-3.5	0.88	100.0%	92.2%
gpt-5.1	0.81	95.8%	87.0%
deepseek-v3.1	0.80	99.6%	65.2%
heuristic†	0.65	100.0%	100.0%
gpt-4.1	0.57	89.1%	57.2%
gpt-4o	0.54	95.0%	49.0%

Robustness Ablations

Two ablations probe the benchmark’s design. The first (§E.1) tests whether prediction performance reflects observation-driven inference or transferred priors from real-biology GRN literature. The second (§E.2) tests whether scores are confounded by gene-name vocabulary or by the system prompt’s stance on biological knowledge.

Structural-prior ablation (backwards organisms)

Design. For every medium-tier seed we constructed a backwards variant by inverting every signed quantity: GRN edge weights $w_{ij}$, gene biases $b_i$, signal modes, and phenotype readout weights. Topology, magnitudes, and noise were preserved; phenotype baselines were re-centred so $\sigma(0) \approx0.5$ at steady state. The backwards organism violates real-biology priors (e.g. “blocking an activator raises its downstream targets”) while keeping the same observable input–output structure.

Twenty canonical medium-tier seeds were evaluated in both directions in healthy mode. Analysis is paired per seed: $\Delta= \text{forward} - \text{backward}$. We report paired Wilcoxon signed-rank tests and percentile-bootstrap 95% CIs. Seven LLMs (Opus 4.6, Sonnet 4.6, GPT-5.1, GPT-4o, GPT-4.1, DeepSeek V3.1, Qwen 3.5) and the heuristic baseline as a null control were tested. Because the heuristic carries no priors, any direction effect it shows reflects a substrate difference in how easy each direction is to learn, not prior reliance; we therefore also report $\Delta_{\mathrm{model}} - \Delta_{\mathrm{heuristic}}$ to isolate the prior-reliance component.

Healthy mode — no prior-reliance signal. Healthy-mode composite scores are statistically indistinguishable across directions for six of the seven LLMs (Table 13). The exception, GPT-4o, is significant at $\alpha= 0.05$ but goes the “wrong” way: it scores higher on backwards organisms. Heuristic-adjusted deltas are small and mixed in sign, and the mixed-effects model shows no significant model×direction interaction. Healthy-mode prediction is learned from observation, not from expectations about how GRNs “should” behave.

Model	Forward	Backward	$\Delta$	95% CI	$p$	$\Delta- \Delta_{\mathrm{heur}}$
Claude Opus 4.6	0.908 ± 0.101	0.933 ± 0.100	−0.025	[−0.092, +0.042]	0.50	−0.017
Claude Sonnet 4.6	0.908 ± 0.101	0.867 ± 0.168	+0.042	[−0.042, +0.125]	0.66	+0.050
GPT-5.1	0.750 ± 0.127	0.767 ± 0.166	−0.017	[−0.100, −0.067]	0.67	−0.008
GPT-4o	0.658 ± 0.138	0.733 ± 0.137	−0.075	[−0.125, −0.025]	0.017	−0.067
GPT-4.1	0.733 ± 0.137	0.708 ± 0.119	+0.025	[−0.050, +0.108]	0.56	+0.033
DeepSeek V3.1	0.925 ± 0.101	0.858 ± 0.146	+0.067	[−0.008, +0.150]	0.17	+0.075
Qwen 3.5	0.875 ± 0.114	0.885 ± 0.199	−0.010	[−0.115, +0.104]	0.72	−0.002
Heuristic	0.583 ± 0.148	0.592 ± 0.175	−0.008	[−0.083, +0.075]	0.73	—

Interpretation. Prediction under a fixed experimental budget does not depend on real-biology priors: inverting every sign in the organism leaves healthy-mode scores unchanged for six of the seven LLMs; the seventh scores higher on the inverted organism, which prior reliance cannot produce. This supports the benchmark’s central design assumption, that the prediction task is solved from the evidence an agent collects rather than from expectations about how GRNs “should” behave. It does not speak to diagnosis, which this ablation does not test (see Limitations).

Limitations. (i) Healthy mode and the medium tier only. The ablation therefore constrains prediction and says nothing about whether priors affect diagnosis; backwards-convergence issues on the hard tier (mutual inhibition from flipped feedback edges) are deferred to future work. (ii) The mixed-effects fit covers the original five LLMs + heuristic; DeepSeek and Qwen, added in a follow-up sweep, are covered only by per-model paired tests because their session-level JSONLs are not co-located with the original five-LLM run. (iii) Qwen completed on 16/20 healthy seeds (Together AI errors); paired tests use the seed-intersection. (iv) A null result of this kind bounds a prior-reliance effect rather than excluding one: with $N=20$ paired seeds the confidence intervals still admit deltas of roughly $\pm0.15$.

Vocabulary and prompt-permission ablation

Design. LUCA’s invented gene vocabulary and “biological knowledge is irrelevant” system-prompt assertion are design choices intended to prevent retrieval-based shortcuts. To test whether either is load-bearing, we ran a $2 \times2 \times2$ paired ablation: gene-name vocabulary (invented G001, . . . vs. 50 real human gene names like TP53, BRCA1 drawn from three familiarity tiers), system-prompt stance (suppress: the default “biological knowledge is irrelevant”; permit: “you may draw on biological knowledge where relevant”), and difficulty (medium: 20 genes; hard: 30 genes). Each organism topology was generated once and run under all four (vocabulary $\times$ stance) conditions on Claude Opus 4.6 and DeepSeek V3.1, 20 seeds per cell (320 sessions total).

Results. Both axes are null across the 320-session grid (Table 14). Vocabulary (real vs. invented) shows no measurable effect in any of 8 paired contrasts ($|\Delta| \le0.058$). Stance (permit vs. suppress) is null in 7 of 8 cells; the one nominally significant cell (DeepSeek, hard, real; $p = 0.035$) fails Bonferroni correction across the 8 tests and points the “wrong” way — permit decreases performance.

Tier	Model	inv / sup	real / sup	inv / per	real / per
Medium	Claude Opus 4.6	$0.917 \pm0.101$	$0.908 \pm0.114$	$0.925 \pm0.114$	$0.917 \pm0.101$
Medium	DeepSeek V3.1	$0.892 \pm0.098$	$0.842 \pm0.175$	$0.853 \pm0.209$	$0.825 \pm0.262$
Hard	Claude Opus 4.6	$0.592 \pm0.206$	$0.608 \pm0.146$	$0.650 \pm0.187$	$0.608 \pm0.182$
Hard	DeepSeek V3.1	$0.567 \pm0.183$	$0.608 \pm0.165$	$0.533 \pm0.220$	$0.475 \pm0.182$

Interpretation. LUCA organisms are generated with random module assignment and random gene labelling, so biological priors cannot map onto the causal structure an agent must recover. Performance reflects experiment-driven reasoning regardless of whether gene identifiers are real or invented and regardless of whether the prompt invites biological reasoning. The invented-vocabulary design choice turns out not to be load-bearing on its own — but the null result here is what makes the benchmark’s retrieval-free claim empirical rather than asserted. This ablation addresses an LLM-era concern that does not arise for classical synthetic-GRN benchmarks [14] (Schaffter et al. 2011; Tjärnberg et al. 2017), which evaluate inference algorithms on static observational data with no training corpus to memorise.

Coverage and limitations. The ablation covers two model families (Anthropic and Together-hosted DeepSeek) at two difficulties; broader coverage (Sonnet 4.6, GPT-5.1) remains future work but would test cross-provider generalisation rather than the headline null. We note that both axes were tested only on healthy mode; disease-mode results would require redoing the ablation with malfunction-targets randomised under each vocabulary condition.