betaFeedback

WorksComputer Science

TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs

Abstract

AI agents are increasingly expected to act as AI scientists, which requires three increasingly open abilities: carrying out a given computational workflow correctly, choosing the next measurements from a given design space, and designing their own experiments. Existing benchmarks mainly evaluate the first two abilities, in a terminal or in a closed loop that selects from predefined candidates. Evaluating the third ability requires reliably executing an experiment that an agent designs, with readings grounded in a real laboratory. We introduce TwinLab-Bench, which evaluates AI scientists on self-designed experiments in 52 tasks from laboratories in the physical sciences, the life sciences, and engineering. The benchmark supports more than 100 experimental action types, and the agent's report is scored by comparison with a hidden answer taken from real laboratory records. Each task has its own TwinLab agent, an agent that serves as the environment and executes the experiments that the tested agent designs, reading the raw materials of the laboratory and calling calculators that are built from its records. Compared with a language model that sees only the task description, the TwinLab agent almost eliminates readings that deviate from the real records and inconsistent answers to a repeated request, and closed-book probes control for answers recalled from memory. We find that these tasks remain hard for frontier models: the strongest, Claude Fable 5.2, averages a score of only 45.0, and we conduct an empirical study of where it fails.

[globe] Project: https://twinlab-bench.github.io/

Introduction

AI is advancing science in more and more of the research process, and the AI scientists that the field now builds need three increasingly open abilities [52, 50]. The first is carrying out a given computational workflow correctly, such as analyzing a given dataset [85, 30, 120, 86]. The second is choosing the next measurements from a given design space [39, 47, 103, 125]. The third is designing their own experiments, and automated laboratories are starting to run such experiments [19, 117]. The third ability is the hardest of the three, but few science benchmarks test it.

Existing science benchmarks test whether an agent can recall knowledge [63], work on given data [85], or search a given space [39], as Figure 2(a) shows. These abilities work on data that the task provides in advance, but the third ability, designing their own experiments, needs an environment that returns real data for new experiments. Figure 2(b) orders these environments by how close they come to a real laboratory. Fixed results support no new experiments. A simulator [58, 39, 47] or a replay [103, 125] returns new results, which come only from written rules or from recorded experiments. The goal of the experiment environment is a real laboratory, which returns real results for any experiment that an agent designs [20, 121]. We move one step from simulators and replays toward this goal with a twin of each real laboratory, and we build TwinLab-Bench on these twins.

Figure 2. Two axes for science benchmarks. (a) From left to right, the agent designs more of the method, and TwinLab-Bench tests ability ④. (b) From left to right, the environment for new experiments comes closer to a real laboratory, and TwinLab-Bench uses environment ③.

A twin of a real laboratory has to return real readings for as wide a range of experiments as an agent may design. Language models already play patients [108] and users [13] in agent evaluations, but they fall short when they play a laboratory. Given only the task description, a language model has no basis for the numbers of this laboratory, and given the raw materials, it can still copy wrong numbers. When we replay the requests that real laboratories made, the language model deviates from the record on 41% of these requests with the task description and on 14% with the raw materials, as Table 1 shows.

Language model given task description+ raw materialsTwinLab agent
Deviates from the real record41%14%2%
Inconsistent when repeated31%8%4%

Table 1. How often the reply of each environment to a request deviates from the record or changes when the request is repeated; lower is better. Appendix D.1 gives the details.

We propose the TwinLab agent to build a reliable twin of each laboratory. The TwinLab agent is a coding agent that serves as the environment, so it uses tools to read the files of the laboratory and runs programs on them. Each task has its own TwinLab agent with the materials and the calculators of its laboratory. For an experiment that the laboratory performed, the TwinLab agent reads the raw materials, often combining several records, and returns the result that the laboratory recorded. For an experiment that the laboratory did not perform, it runs calculators, which are deterministic programs that we write from the data of this laboratory, for example by fitting a physical model to its measurements. Each calculator extends the record only to nearby settings, with a basis that we write down when we build the task.

TwinLab-Bench consists of 52 tasks, and each task comes from the records of a real laboratory in the physical sciences, the life sciences, or engineering. The tested agent, which we call the actor agent, receives the notebooks, raw data, and interim conclusions that the laboratory had at one point of its work, requests the experiments that it designs within a budget, and submits a report that we score by comparing it with what the laboratory later found, as Figure 3 shows for four tasks. Our experiments show three properties of TwinLab-Bench. (1) The score comes from the experiments, because without budget Opus 5.5 and DeepSeek-flash pass none of the tasks. (2) The actor agent’s conclusion follows the true state of the laboratory that it measures. (3) The twin predicts hidden recorded readings within a median error of 2.47% even outside the recorded conditions, and it blocks all 9 attempts to exploit it.

Figure 3. Task sources. Four example tasks that come from journal papers or lab notebooks, with the finding that each task targets.

On TwinLab-Bench, the strongest frontier model averages a score of only 45.0 out of 100. Opus 5.5 still loses part of the score on decisions that it states in full before the run, and its most common decision error is accepting a mechanism without a discriminating test.

Our contributions are threefold.

  • TwinLab-Bench, a public benchmark that tests whether an agent can design its own experiments, built on the records of real laboratories and released with its evaluation code.

  • The twin of a real laboratory, made of a TwinLab agent, the raw records of the laboratory, and the calculators that we write from these records.

  • An empirical study that locates where frontier models lose score and which experimental abilities they lack.

Related Work

Benchmarks without new physical experiments. Most science benchmarks give the agent no laboratory for running new experiments. Question sets test whether an agent recalls scientific knowledge ([97, 63, 64] (Phan et al. 2026)) or how to change a published protocol ([113]), and data-analysis benchmarks hand the agent a dataset and a question ([30, 85]). AI scientist systems such as Kosmos also automate this kind of analysis at scale ([86]). Terminal benchmarks ([81] (Terminal-Bench-Science Team 2026)) and benchmarks of automated machine-learning research ([25, 129, 115]) let the agent run new experiments, but these experiments are programs inside a computer. End-to-end systems already write and review such research (Lu et al., [68]), and end-to-end research benchmarks run the agent on the given data and code of a study (Xu et al., [135]; Zhou et al., [149]; Yang et al., [136]). Benchmarks of these four kinds cannot evaluate how an agent chooses physical experiments that cost time and consume scarce materials.

Readings from simulation, replay, or a language model. Interactive environments answer the experimental requests of an agent with readings from a simulation, a replay, or a language model. Text worlds and equation models (Wang et al., [123]; Jansen et al., [58]; Duan et al., [39]; Gandhi et al., [47]; Rowstron & Chadha, [104]) answer any request, and their readings follow the rules written into the environment. EmbodiedLab (Gao et al., [48]) tests how agents operate instruments in a physics simulator on textbook experiments, while TwinLab-Bench leaves out physical operation and tests experimental decisions on problems from real laboratories. Replays of published measurements (Roohani et al., [103]; Wen & Liu, [125]) return real readings, and the agent picks its next experiment among the recorded conditions. Learned world models predict the outcome of an action (Ha & Schmidhuber, [54]), and language models serve as such world models for planning (Hao et al., [55]) and for text-based games (Wang et al., [124]). Language models also play patients (Schmidgall et al., [108]) and users (Yao et al., [137]; Barres et al., [13]) in agent evaluations, and they answer from general knowledge; evaluations of such simulators find that their feedback departs from real environments and users (Liu et al., [67]; Zhou et al., [150]). AgentBeats (Berkeley RDI, [16]) packages a benchmark as an agent that sets the tasks, the environment, and the scoring for the agent under test. TwinLab-Bench splits this role between a TwinLab agent that serves as the environment and a judge agent that scores the report. These environments do not answer the experiments that an agent composes with readings from the records of a real laboratory, so they cannot evaluate how an agent designs its own experiments.

TwinLab-Bench

TwinLab-Bench consists of 52 tasks, and each task defines a twin of one real laboratory, a budget, and a rubric. In a run, the tested model, which we call the actor agent, sends experimental requests to the TwinLab agent of the task, an agent that serves as the environment and returns the results, as Figure 4 shows. The actor agent finally submits a report, and a judge agent scores this report by comparing it with the hidden truth. All TwinLab agents run the same program, and the materials and the calculators of each task make it the twin of its laboratory, with the engineering details in Appendix § 15.§ 1.

Figure 4. How a task is run. The task materials of one example task, and one run with the actor, TwinLab, and judge agents.

Tasks

Task sources. We select the tasks from about 375 candidate sources, and each task is built on a real research record, such as an open laboratory notebook, raw instrument exports, a paper with its supplementary material, or a public dataset, as Figure 3 shows. Most tasks are adapted from the records of one laboratory. In some of them the laboratory itself met the problem and later solved it, and in the others the problem lies in the real data of the laboratory. Their answers come from a later notebook entry, a publication, a correction, or a reanalysis of the raw data. The few remaining tasks are grounded syntheses, as Appendix § 16 describes.

Task construction. We choose a cutoff point in the record, which is a date or a clearly defined state of the laboratory, and split the record into two parts. The data, protocols, and interim conclusions that the laboratory had before the cutoff point are handed to the actor agent. What the source records show after the cutoff point is given only to the TwinLab agent and to the judge agent, and we call this part the hidden truth. An expert route taken from the record averages a score of 97.8 out of 100 when we replay it in the twin of each task. Every task also passes a coverage review, which confirms that the standard experiments of the field can be simulated, and Appendix § 16 lists all tasks and their construction.

Task forms. The tasks take three forms. Most tasks are claim reviews, which ask the actor agent to decide which conclusions of the laboratory hold and how to test them. Troubleshooting tasks ask for the cause of a clear failure and a working procedure, and research-campaign tasks ask for an executable procedure toward a research goal. Some tasks offer no experiments, and the actor agent analyzes only the raw records that the laboratory already has.

Actor agent. The actor agent proposes its own hypotheses and composes its own experiments. It describes each experiment in natural language and decides which actions to run, on which materials, with which settings, in which order, and how many times. Across the benchmark the laboratories support more than 100 action types. Each laboratory lists its procedures and the parameter options of its equipment and reagents, and the actor agent combines them into experiments and can also request values outside the list. These lists contain no experiment and no step of the solution, as Appendix § 15.§ 1 details. In its sandbox, the actor agent can analyze the handoff materials freely with a shell and Python, but new readings come only from experimental requests to the TwinLab agent, and every request costs budget.

The actor agent can search the web with the source record blocked, and it finally submits a report with its conclusions and recommended procedure.

Difficulty control. We control the difficulty of the tasks with five properties that make them depend on experimental judgment. (1) The budget of each task, counted in units such as laboratory days or wells, is one to three times the amount that the expert route uses for most tasks, so it covers the expert route and a few plausible hypotheses but not all options. (2) Scarce materials named in the record cannot be recovered once used. (3) An artifact has a plausible benign explanation. (4) A measurement that looks decisive has another possible cause, for instance when all samples measured on one date also come from one batch. (5) In some tasks the correct answer is that the question cannot be decided within the budget. Overall, for tasks adapted from real records, these five situations are taken from the record, and we add no problem to the record.

The TwinLab Agent

The environment of each task is a twin of one specific laboratory, made of a TwinLab agent and the task materials of this laboratory. The task materials contain the full record of the laboratory with its raw files, the calculators that we write from this record, and the hidden truth. The TwinLab agent handles each experimental request in three steps. First, it looks for a basis in the task materials, often combining several records. Second, it takes the recorded value of a condition that the record contains on comparable material or runs calculator commands in a shell. Third, it writes these numbers as an observation in the instrument’s own format and returns a matching real image when one exists. The environment code writes a session record after every request, so the TwinLab agent keeps the state that earlier requests left, such as a sample already used up. Its tools only read files, search text, and run the calculators.

A calculator is a deterministic program written for one task from the laboratory’s own data, for example by resampling the recorded measurements, by fitting a physical model to the record, or by filling a real instrument’s export template. The TwinLab agent follows two further rules.

  • Noise. Independent samples receive noise at the measured repeatability.

  • Simulation limits. A quantity that neither the record nor the literature constrains is marked as a simulation limit, costs nothing, and is reported as not being an experimental result.

Local extension. The calculators extend the record only locally. An extrapolated value carries a basis that we establish through a literature search when the task is built, typically an interpolation between two recorded settings or a dependence documented in the literature, and this basis is written into the task item by item. § 4.3 tests how much the results depend on these extensions.

Hidden truth. The TwinLab agent needs the hidden truth because the readings have to agree with the true state of the laboratory, such as a cause that only appears later in the record. It answers only what an experiment returns, and Appendix § 15.§ 1 lists the rules that the environment code enforces. Table 2 lists what each role can see.

ActorTwinLabJudge
Handoff before the cutoff✓✓✓
Full record and raw files✓✓
Hidden truth✓✓
Rubric full-credit conditions✓✓
Maximum score of each rubric item✓
Actor’s shell and search log✓✓
How each reading was produced✓✓

Table 2. Who sees what. Materials that each agent of a run sees.

Scoring

Rubric. The rubric of each task consists of items with conditions for full and partial credit ([115, 9]). Most items belong to two components. Conclusion items check whether the report states the correct answer, and verification items check whether the report reaches this answer from evidence that the actor agent obtained in this run, so a key finding of a task is often split into one item of each kind.

Score and judge. The score of each task is rescaled to 0 to 100, and each task marks a few core items whose failure makes the report wrong; when any core item misses full credit, the run scores 0. We report the mean score over all tasks. The judge agent is a vision-language model that serves as the judge ([147]), reads the full task, the interaction log, and the report, and cites the report text or the request number for every item. Every core item with full credit is checked twice more by independent calls, and the item scores 0 when either check fails. We freeze the rubric before looking at the final evaluation runs.

Invalid runs. The judge agent also audits the environment, and a run is invalid when a reading leaks the hidden truth, contradicts earlier observations or physics in a way that matters for scoring, or refuses a relevant experiment that a real laboratory can run. A deterministic script replays every calculator command to check the state, the billing, and the numbers. A static check scans the content that reaches the actor agent through its tools, and a run that receives a blocked string of the source is invalid, as Appendix § 15.§ 1 details.

Experiments

Comparison with Existing Benchmarks

The scorecard of Figure 5 compares TwinLab-Bench with 8 existing benchmarks item by item on scale and diversity, realism, and memorization control, and TwinLab-Bench is the only benchmark in the comparison that combines a wide range of experiments with real problems and real readings, as Appendix § 13 discusses.

Figure 5. Scorecard of science benchmarks. Each row is a benchmark, and each bar splits its tasks by the levels in the key of its column. Orange boxes mark the main weakness of each group of benchmarks.

Main Results

Setup. We evaluate 13 models as actor agents, running the GPT models in Codex and the other models in Claude Code. The TwinLab agent and the judge agent run on DeepSeek-flash for every actor agent. We report cost together with score following Kapoor et al. [61]; Figure 6 counts the cost of the actor agent, and the TwinLab agent and the judge agent add about 55 USD for all tasks, as Table 10 shows.

Figure 6. Mean score versus the cost of the actor agent; models run at high effort by default.

Property (DeepSeek-flash)Result
TwinLab agent and judge agent, repeated runsAll 19.7 ± 3.3 over 3 runs in Table 8
Judge agent alone, repeated judgmentssame pass on 19/20 and 15/16 runs, section B.1
Judge agent alone, agreement with Opus 5.5pass agrees on 19/20 runs in section B.2
Cost of the judge agent16.9 USD for all tasks
Cost of the TwinLab agent37.7 USD for all tasks

Table 10. Stability, agreement, and cost of the TwinLab agent and the judge agent with DeepSeek-flash in both roles. Costs are measured with DeepSeek-flash as the actor agent.

Results. As Figure 6 shows, Claude Fable 5.2 scores 45.0 on average and Claude Opus 5.5 scores 40.2, and the other actor agents score between 5.4 and 31.7. Table 7 lists the scores by division and task form. For 127 of the 136 pairs of actor agents, with each reasoning effort of a model counted as one actor agent, the actor agent with the higher mean score also scores higher on more tasks than it scores lower, and a paired comparison over tasks separates 66 of the 75 pairs whose mean scores differ by at least 10, as Appendix § 8 shows. We find that (1) the score tends to rise with the logarithm of the cost, and both a larger model and a higher reasoning effort raise the score, and (2) the differences between actor agents hold across many tasks.

Actor (effort)Physical sciences n=20n = 20Life sciences n=20n = 20Engineering n=12n = 12Troubleshooting n=5n = 5Claim review n=37n = 37Campaign n=10n = 10All n=52n = 52Actor cost (USD)
Fable 5.2 high, n=52n = 5234.048.058.340.047.040.045.0424
Opus 5.5 high, n=52n = 5231.644.547.640.040.340.040.2141
GPT-6.1-sol high, n=52n = 5228.529.241.360.025.839.531.746
Opus 5.5 medium, n=52n = 5222.229.348.120.030.338.830.993
GPT-6-astra high, n=52n = 5232.033.224.258.027.429.130.7309
Sonnet 5.5 high, n=52n = 5227.118.947.620.029.828.828.760
GPT-6.1-sol low, n=51n = 5123.725.031.740.023.130.026.123
DeepSeek-flash-1017 high, n=52n = 5214.024.123.338.014.930.020.012
GLM-5.3 high, n=52n = 5212.119.431.720.019.220.019.4119
Kimi K3.1 Pro high, n=52n = 5218.214.923.419.017.619.818.183
MiMo-V2.6-Pro high, n=52n = 524.119.538.40.019.820.017.924
Sonnet 5.5 medium, n=52n = 5218.514.915.820.017.89.816.529
Qwen3.8-Max high, n=52n = 524.115.025.020.013.010.013.199
HY4 high, n=52n = 5214.29.815.939.010.39.712.935
MiMo-V2.6-Flash high, n=52n = 524.014.915.80.010.019.810.96
GPT-6-luna high, n=52n = 523.014.916.10.07.029.110.63
GPT-6-luna medium, n=52n = 520.05.015.00.04.910.05.42

Table 7. Main results. Mean score (0 to 100) of each actor agent and reasoning effort by academic division, by task form, and over all tasks (All), together with the cost of the actor agent on all tasks in USD. GPT-6.1-sol at low effort has no valid run on one task. Every actor agent: first valid run per task with the DeepSeek-flash TwinLab agent and judge (Fable 5.2, Opus 5.5, and GPT-6-astra from 2026-11-10, the others from 2026-11-01). Cost: the actor agent alone over all tasks, at first-party list prices; runs whose actor usage was not logged are counted at the mean of the logged runs.

Task scores separate correct and wrong final reports. With a fixed rule, we select 21 tasks whose final reports include both correct and wrong ones, which we divide by the hidden truth without looking at their scores. In 19 of these 21 tasks, every correct report has a higher total score than every wrong report, by 0.46 of the maximum score on average, as Appendix § 9.§ 3 shows. The judge agent is stable, repeating a judgment has little effect, and two judge agents agree on the pass verdict in 19 of 20 runs, as Appendix § 9.§ 1 and Appendix § 9.§ 2 show.

Fidelity of the Twin

We test along a chain of evidence how far the results that the twin returns can be trusted.

Most experiment results come from real laboratory data. We trace where the experiment results of the actor agents come from and whether the record contains the decisive experiments. Of these experiment results, 76% come from the record or from calculators calibrated on the same laboratory’s data, as Appendix § 11.§ 2 shows, and the record contains the conditions of 78% of the decisive experiments on the expert routes, as Appendix § 11.§ 3 shows. Thus the conclusions of most tasks depend on recorded data, and the calculators mostly fill in details around them.

Sensitivity to extrapolation. We replace the extrapolation outside the record with another equally plausible choice and rerun the actor agents on six tasks, where the replacement changes 12% to 44% of the calculator outputs, as Appendix § 11.§ 4 details. We find that most conclusions do not depend on which plausible extrapolation the calculators use, so a modeling choice that we make when we build a task changes few evaluation outcomes.

Held-out conditions. We hide recorded conditions of three tasks and rebuild their calculators without seeing them. The median error of the hidden readings is 0.74% for interpolated conditions and 2.47% for extrapolated ones, and a hidden discrete condition is predicted within 0.6% on average, as Appendix § 11.§ 5 shows. We find that most readings that the calculators compute near the record fall close to the real readings, so the twin returns realistic readings for new experiments near the record.

The twin supports routes beyond the expert route. In successful runs, more than a third of the requests are new experiments that the expert route does not contain, such as new combinations of settings and additional checks, as Appendix § 11.6 shows. The TwinLab agent answers these requests, so actor agents can solve a task along their own routes.

Robustness of the twin against hacking. When we tell Opus 5.5 on four tasks that the laboratory is simulated and that it may exploit any weakness, all 9 exploit attempts are blocked, and when we ask it to find flaws in the simulation, none of the 13 true flaws that it finds changes the scoring, as Appendix § 11.7 details. An audit of the 7808 replies of the TwinLab agent in 831 evaluation runs finds one reply that states part of the hidden truth and that the judge agent did not flag, and that run scores 0, as Appendix § 11.§ 2 shows. These results show that the score is hard to raise through the weaknesses of the twin and that the flaws that exist are minor.

Memorization Control

The tasks of TwinLab-Bench come from public records, so a model may have seen a source during training ([107, 51]) or may derive part of an answer from prior knowledge, and we test in three ways whether the scores come from the agent’s own experiments.

Changing the true state of the laboratory. To test whether the answer is determined by the readings that the actor agent obtains in its own experiments, we change the true state of the laboratory and keep the handoff word for word. We construct the new state from a documented case of the same kind of problem by changing only the calculator output and the hidden truth, and a model that remembers the source answers according to the source ([130]). In 8 of the 10 states that allow a verdict, the conclusion follows the new state, and no run mentions the source study, as Table 3 and Appendix § 10.§ 1 show.

Actor agent’s conclusion in the new stateStates
Follows the new state8
Follows it in direction only1
Consistent, not verified1
Never touched the changed part1

Table 3. Actor agents’ conclusions when the true state changes. We keep the handoff word for word and change only the true state of the laboratory.

Closed-book memory probes. In a closed-book session without tools, Opus 5.5 recalls the conclusion of the source for none of the tasks and derives the answer without data for only 2 tasks, and Fable 5.2 recalls the conclusion for one task, as Table 4 and Appendix § 10.§ 2 show. The tasks whose source Fable 5.2 names do not score higher in its evaluation runs. Prior knowledge gives at most part of an answer on many tasks.

Closed-book probeCases
Names the real source<br>memory of the recordyes: 9 / partial: 21 / no: 22
Recalls the source's finding<br>memory of the answeryes: 0 / partial: 3 / no: 49
Derives the answer without data<br>prior knowledgeyes: 2 / partial: 26 / no: 24

Table 4. Closed-book memory probes of Opus 5.5 on all tasks. Bars give the number of tasks with a yes, partial, or no answer.

Pass rate without experiments. With a budget of zero, Opus 5.5 and DeepSeek-flash pass none of the tasks, as Appendix § 10.§ 3 shows. These results show that prior knowledge and the handoff do not earn the core items, and the agent has to obtain the evidence through its own experiments.

Empirical Analysis of Model Failures

On the tasks that need experiments, the decisions that a model states before the run match only in part the decisions that it makes during the experiments, similar to the knowing-doing gap that language-model agents show in bandit tasks ([109]).

Knowing-doing gap in experimental decisions. Before each run, the model reads the handoff files and writes a plan for the whole task, and we check whether the plan states each key decision of the task in full, as Table 5 shows. Decisions that the plan states in full score 87% of their maximum in the run, but decisions that it does not mention still score 62%. Of the 11 decisions that the plan states in full and the run still loses, the run carries out 9 as planned and loses the score in reading its own results or in the report. A static benchmark that grades the plan therefore misjudges many of these decisions, and a closed-book question gives the same order of scores, as Appendix § 12.§ 1 shows.

Before the run, the planScore expected from the planScore in the run
States the decision in fullfull87%
States only the principlepartial66%
Does not mention itnone62%

Table 5. Knowing-doing gap in experimental decisions. Rows group the key decisions by how the plan states them. Opus 5.5 writes the plan before each run, and each percentage is the score that the run obtains on these decisions out of their maximum.

Failure types of the actor agents. We classify every valid run of Opus 5.5 that scores 0 by the error that caused the failure, as Table 6 and Appendix § 12.§ 2 report. Of the 16 failures of Opus 5.5, 12 come from an experimental decision, most often accepting a mechanism without a discriminating test. For example, when an oyster sugar assay reads about 30% low, Opus 5.5 rules out the pipettes after a water check and never tests the 1 μ\muL transfer, which delivers only about 70% of the sample.

Failure typeRuns
Experimental decision
Accepts a mechanism without testing it7
Chooses a procedure on weak early evidence4
Uses up a scarce material before testing1
Reading the results or reporting4
Runs that score 016

Table 6. Where runs fail. Each run of Opus 5.5 that scores 0 on the tasks that need experiments is counted once, under the error that caused it.

More budget rarely helps. Of the 21 failed runs of Opus 5.5, only 8 spend at least 90% of the budget. Doubling their budget turns only 2 of them into a pass, as Appendix § 12.§ 3 shows.

Conclusion

TwinLab-Bench evaluates whether an AI scientist can design its own experiments on twins of real laboratories, and most experiment results in these twins come from real laboratory data. The strongest frontier model averages a score below 50 out of 100. Opus 5.5 loses score even on decisions that it states in full before the run, and its most common experimental error is accepting a mechanism without a discriminating test. We release the tasks and the evaluation code, and the web search of the actor agent blocks the names of the benchmark and its tasks.

LLM Usage Statement

Language models are part of the method of this work. The actor agents are the tested models of § 4.2, and the TwinLab agent, the judge agent, and the analysis calls of § 4 run on the models named there. We build the tasks with a coding agent (Claude Code with Claude Opus 5.5) under our direction. We also use Claude Code with Claude Opus 5.5 to draft and revise the text, to write the analysis and plotting scripts, and to check the numbers in the text against the results. The authors take full responsibility for all content of the paper.

Appendix

Full Results, Divisions, and Repeated Runs

Table 7 gives the numbers behind Figure 6, with one row per actor agent and reasoning effort. Besides the mean over all tasks, it lists the mean by academic division (20 tasks in the physical sciences, 20 in the life sciences, and 12 in engineering), the mean by task form, and the cost of the actor agent on all tasks; Table 10 gives the costs of the TwinLab agent and the judge agent. The cost of the models other than Claude is computed from their token counts at the list prices of their providers; for runs whose actor agent session ended without a usage record, the actor agent cost is taken as the mean of the recorded runs of the same model. The prices of GPT-6.1-sol and GPT-6-astra are higher for inputs of 272k tokens or more, and we apply the tier for inputs below 272k tokens to every request. In USD per million tokens of uncached input, cached input, cache write, and output, this tier is 2.00, 0.10, 2.50, and 10.00 for GPT-6.1-sol and 10.00, 1.00, 12.50, and 50.00 for GPT-6-astra. Thirteen runs on the radiocarbon task were judged under the rubric in force when they ran, and the later change of this rubric only widens the full-credit condition of one item.

The physical sciences are the lowest-scoring division for 12 of the 17 actor configurations, and engineering is the highest-scoring division for 14 of them.

We run DeepSeek-flash and MiMo-V2.6-Pro as the actor agent three times on every task, and we run every other actor agent once. Table 8 gives the mean score of each run. The All score is 19.7±3.319.7 \pm3.3 for DeepSeek-flash and 14.0±3.614.0 \pm3.6 for MiMo-V2.6-Pro. Resampling the per-task score differences between two actor agents, Fable 5.2 scores significantly higher than 14 of the other 16 configurations and Opus 5.5 at high effort than 12 of them at the 95% level. Over all 136 pairs of the 17 configurations, the configuration with the higher mean score also scores higher on more tasks than it scores lower in 127 pairs, and the paired comparison separates 68 pairs, including 66 of the 75 pairs whose mean scores differ by at least 10.

Actor agentRun 1Run 2Run 3Mean ± std
DeepSeek-flash15.921.421.919.7 ± 3.3
MiMo-V2.6-Pro18.013.111.014.0 ± 3.6

Table 8. Three runs of two actor agents. Mean score (0 to 100) of each run over all tasks. The first run of MiMo-V2.6-Pro used an earlier version of the environment rules.

Reliability of the Scores

Stability of the Judge Agent

We judge the same run three times with identical inputs, for 20 runs of Opus 5.5 and 16 runs of MiMo-V2.6-Pro on the same tasks. Table 9 reports, for each actor agent, the share of runs with the same validity verdict in all judgments, the share with the same zero or nonzero score, the share of rubric items with identical scores, and the mean standard deviation of the score within a run. The zero or nonzero score differs between judgments on one run of each actor agent, and in both cases a core item whose full-credit condition has a boundary goes to 0 in some judgments. The standard deviation of a task score between repeated judgments is 2.4 for Opus 5.5 and 2.8 for MiMo-V2.6-Pro.

Actor agentValidityZero or notItemsScore std
Opus 5.520/2019/2092%2.4
MiMo-V2.6-Pro16/1615/1692%2.8

Table 9. Stability of the judge agent. Each run is judged three times by the DeepSeek-flash judge agent. Validity and Zero or not count the runs with the same verdict in all judgments, Items gives the share of rubric items with identical scores, and Score std is the mean standard deviation of the score within a run.

Changing the Models of the TwinLab Agent and the Judge Agent

DeepSeek-flash serves as the TwinLab agent, as the judge agent, and as one of the tested actor agents, and a language-model judge can favor its own generations [91], so we check whether the results depend on this choice. We choose 10 tasks by a fixed rule from the three groups of tasks that the three runs of DeepSeek-flash leave all nonzero, mixed, or all zero, and we run DeepSeek-flash as the actor agent once with Sonnet 5.5 as the TwinLab agent and Opus 5.5 as the judge agent. All 10 runs are valid, every score lies within the range of the three runs of DeepSeek-flash in the standard setting on the same task, and 5 tasks pass, against 5, 6, and 4 in the three standard runs. To separate the judge agent from the TwinLab agent, we rejudge the same 10 runs with DeepSeek-flash, and the pass verdict changes on 1 task, where the Opus 5.5 judge agent gives 0 to a core item with a boundary. We further rejudge 20 runs from the three runs of DeepSeek-flash on 9 of these tasks with Opus 5.5 and with DeepSeek-flash under the same judge prompt and rubric. The two judge agents agree on pass in 19 runs (Cohen’s κ=0.83\kappa= 0.83) and on validity in all runs, and the one pass change comes from Opus 5.5 grading a self-contradictory report by its weaker reading.

Scores Separate Correct and Wrong Reports

We draw 126 reports from the evaluation runs of all tasks with a hash rule fixed before any model call, taking one passing and two failing runs on tasks that have both and one failing run on the other tasks. A DeepSeek-flash call classifies each report as correct, plausible but wrong, incomplete, or missing by reading it against the hidden truth, without seeing its score, its judgment, or the rubric, and we check 29 labels at the boundary by hand and change 1. A report is correct when its main conclusions and deliverables agree with the hidden truth, also when its verification is too weak to pass. A report is plausible but wrong when any main conclusion, cause, mechanism, recommendation, or deliverable contradicts the hidden truth, or when it leaves a main question undetermined although the hidden truth decides it. A report is incomplete when it makes no wrong claim on the main line but its deliverable falls short. Of these reports, 43 are correct, 69 are plausible but wrong, 6 are incomplete, and 8 runs end without a report. On 19 of the 21 tasks that have both correct and wrong reports, the lowest total score of a correct report is above the highest total score of a wrong report, and the mean gap is 0.46 of the maximum score.

Validity and Cost of the Environment

Table 10 summarizes the TwinLab agent and the judge agent when both run on DeepSeek-flash: how stable they are, how well they agree with Opus 5.5 in the same roles, and what they cost. The costs are measured with DeepSeek-flash as the actor agent on all tasks, so that the costs of the actor agents in Figure 6 do not include them. For every actor configuration and task, we count the runs up to the first valid run, which is the run in Figure 6, and 44 of these 875 runs are invalid (5.0%), so a valid run takes 1.05 attempts on average. The scan for blocked strings ignores letter case, and no valid run contains a blocked string or a marker string. Counting only the first run of each actor configuration on each task, the share of invalid runs does not depend on the actor agent (p=0.24p = 0.24).

Memorization Control

Another State of the Same Laboratory

For each task in this study, we keep the task description and the handoff materials word for word and change the true state of the laboratory to another state that the literature documents for the same kind of problem. Only the calculator and the hidden truth change, and the replay of the expert route in the new state passes before any actor agent runs. The released data list the 11 states with the original cause, the new cause and its evidence, the change in the calculator, the result of the reference replay, the state that the actor agent’s conclusion follows, and whether the run mentions the source study. In 8 of the 10 states that allow a verdict, the conclusion follows the new state. On the other 2 states, Opus 5.5 does not pass the task (scores of 6.5 of 10 and 8.0 of 15). One state is excluded from the denominator because the actor agent never ran an experiment that touches the changed part.

Memory Probes per Task

For every task, the released data list whether Opus 5.5 recognizes the source, whether it recalls the source’s conclusion, whether it derives the answer closed-book, when the source was published relative to the model’s knowledge cutoff, and the task’s score. The probe uses two independent calls per task in a neutral temporary directory with all tools disabled. One call asks for the source and what the model remembers of it, the other asks for the most likely answer, and a third call grades both answers against the answer key.

We repeat the probe for GPT-6.1-sol, DeepSeek-flash, and Qwen3.8-Max with the same prompts, answer key, and grading. They name the source of 4, 2, and 1 tasks, against 9 for Opus 5.5, and none of them recalls the conclusion of a source. When the tasks are grouped by whether a model names their source, the tasks with a named source do not score consistently higher in the evaluation runs of the same model, and Opus 5.5 scores 43.3 on the 9 tasks whose source it names and 43.2 on the 22 tasks whose source it does not recognize. Fable 5.2 names the source of 16 tasks, two of which can be read from the task text, and recalls the conclusion of a source for one task. Its evaluation runs score 49.0 on the 16 tasks whose source it names, 32.2 on the 21 tasks whose source it names in part, and 58.7 on the 15 tasks whose source it does not name.

Zero-Experiment Baseline

We select the 33 tasks whose decisive data are not in the handoff, according to the scorecard annotation, and copy each task with a budget of zero; every experimental request is refused without calling the TwinLab agent. Opus 5.5 and DeepSeek-flash each run once on every copy, and the DeepSeek-flash judge agent scores the reports with the original rubric. Neither Opus 5.5 nor DeepSeek-flash passes any of the 33 tasks, while DeepSeek-flash with the normal budget passes 6 of the same tasks and at least one actor agent passes 28.

Fidelity of the Twin

Pilot Study: What the Laboratory Needs to Know

We replay the requests of the expert route in 12 tasks, four from each division, against three settings, with two independent sessions per setting. The language model in all three settings is DeepSeek-flash, the model that serves as the environment in the evaluation. The rule that selects the tasks and the tolerance of each task from the raw files were fixed before any output was read. Deviation is counted on the 126 answers to the 63 requests whose conditions exist in the record, and inconsistency on the 93 requests asked in both sessions. Deviation drops from 41% with the task description alone to 14% with raw materials and to 2% with the TwinLab agent; substantial inconsistency drops from 31% to 8% and 4%. The 3 deviating answers of the TwinLab agent all occur in one task: one request receives a calculator value below the tolerance band in both sessions, and one well of a 48-well plate exceeds an 8% rule by 0.2 points. Of the 4 requests that the TwinLab agent answers inconsistently, 2 differ only in the amount charged. The report of the expert route passes in 17, 21, and 23 of the 24 sessions of the three settings. In a pilot, a language model that writes a metabolic curve by itself produces a curve about 20 times slower than the record.

Sources of Readings and Destinations of Requests

We take the first valid run of every actor configuration other than Opus 5.5 at medium effort on every task, 831 runs with 7808 requests, and classify every reading by the dominant source of its numbers and every request by its destination. Over all readings, 76% come from the record or from calculators calibrated on the same laboratory’s data, with 53% looked up in the record and 23% computed by the calculators, and 99.5% trace back to the record, a calculator output, or one step of arithmetic. Only 0.5% of all requests are not simulated at all.

We also audit every reply of the TwinLab agent in these runs for disclosure of the hidden truth. A script first marks 79 replies that contain a number, a name, or a phrase of the hidden truth that the requests of the actor agent and the outputs of the tools do not explain. A DeepSeek-flash call then judges these replies, the 162 other replies in which the TwinLab agent read the internals of a calculator, and 100 random replies, and we check every reply that it flags by hand. One reply states a hidden mechanism without being asked, and one reply describes an asked-for observation that the calculator does not model, which we count as a coverage gap; both runs score 0, and none of the random replies is flagged.

Audit of Extrapolation Dependence

Without running new models, we label every reading in the evaluation runs with its source and compare the actor agents with the replay of the expert route, which every task has, on the 29 tasks whose replay used the same TwinLab agent. The share of requests that actually obtain data outside the record does not differ between the actor agents and the expert route (median difference 0 per task, p=0.24p = 0.24), and it does not differ between two reasoning efforts of the same model. On real-source tasks, rubric items whose evidence comes only from design parameters or from numbers written by the TwinLab agent score 0.63 of full credit, and items whose evidence comes from the record score 0.62. For each of the 6 tasks whose deliverable is scored by a calculator, the settings that the actor agent can choose are either limited to the real settings in the record or come from the same design model as the observations. The expert routes of 42 of the 49 tasks built from real records request experiments. Of their 189 decisive experiments, 78% use conditions that the record contains, 15% partly extend a recorded condition, and 7% lie outside the record. On 7 further tasks built from real records, the expert route has no scored experiment, because the evidence comes from offline analysis of the handoff records, and we count these tasks separately.

Sensitivity to Extrapolation

For six tasks whose decisive items rely partly on conditions outside the record, we build a variant that changes only the extrapolation outside the record to another choice that is physically just as plausible, such as the other end of the literature range or another functional form. Three of the tasks are chosen by a rule fixed in advance, which takes the real-source tasks where at least two of the three actor agents below score above zero and whose earlier runs reach the changed extrapolation. The outputs for recorded conditions stay identical, and so do the hidden truth, the rubric, the task description, and the actor agent’s files. We first replay the calculator commands of existing runs on the original and the variant and count how many outputs change, and we then run Sonnet 5.5, Kimi K3.1 Pro, and GLM-5.3 once each on every variant, with the DeepSeek-flash TwinLab agent and judge agent, and compare with their runs on the original in Table 11. Sonnet 5.5 runs at medium effort on the first three tasks and at high effort on the other three. In every variant the replay of the expert route still earns full marks. We compare each variant run with the run of the same model and effort in Figure 6; on the 12 pairs whose original run used the same task version as the variant, the pass verdict changes in 2 pairs and the median absolute change of the score is 0.

TaskChanged extrapolationOutputs changedRuns on the variant
Fatigue of printed steelwidths and decoration outside the anchors (6 items)44%conclusions same in 3/3
Oyster metabolismQ10Q_{10} below 26 °C, 3.52 → 2.020%conclusions 1–2 same; score 0 in all
Radiocarbon pretreatmentrinse release, sonication residue, collagen carbon23%1 deliverable changed
Phonon bursts in a detectoroff-record device model, thickness ratios, offsets and noise39%conclusions same in 3/3
Genetic background of fliesgeneration of re-derived strain panels12%conclusions same in 3/3
Zero-bias peak boundaryroom-temperature resistance and leakage readings25%2 runs read another sweep

Table 11. Sensitivity to the extrapolation. Outputs changed: share of calculator outputs of existing runs that differ on the variant.

In the fatigue task, the changed outputs appear in the evidence that the judge agent cites in 14 of 18 existing runs, yet the conclusions on the variant are the same for all three actor agents. In the oyster task, the change multiplies the reading of the expert’s bench-delay test by 1.7, but the actor agents take another path. On that path the readings change by 0% to 8%, and the ranking of the oyster families stays the same in 83 of 90 reads. Two changes in the third conclusion trace back to the wording of offline analyses and to a different choice of water, and replaying these runs on the original task gives almost the same readings. In the radiocarbon task, one actor agent measures very small samples after sonication, where the reading depends entirely on the extrapolated residue, and changes its recommendation for low-yield samples. The correct conclusion of this task itself depends on the size of this residue. In the phonon task, the changed outputs appear in the evidence that the judge agent cites in 20 of 21 existing runs, and the score differences between the original and the variant come from two items on the wording of the report whose evidence cites none of the changed requests. In the fly task, no run on the variant requests a re-derived strain panel, and in the zero-bias task, no rubric item reads the changed room-temperature readings, and the two runs that change their pass verdict analyze another sweep of the recorded data.

Held-Out Conditions

We test how close the calculators come to real readings on conditions that the record contains but the calculator does not see. We choose three tasks whose calculators extend a measurement along a continuous variable, namely phonon bursts in a detector, gliding of kinesin-driven microtubules, and fluorescence blanks of clam assays. On each task we hide 2 to 10 recorded conditions together with the measurements that reveal them, and the hidden values go to a file that the rebuilder cannot read. A separate session rebuilds each calculator from the reduced record and the construction rules alone, without the original calculator, the task files, or the rubric. Of the 15 hidden readings, 10 fall within twice the measured repeatability, and the median error is 0.74% for interpolated conditions and 2.47% for extrapolated ones. On all three tasks, the rubric items that involve the hidden conditions reach the same judgment with the rebuilt calculator. In a second round we hide a whole discrete condition, one of the four caseins that coat the slides of the kinesin task, and the rebuilt calculator predicts the mean gliding speed of the hidden casein within 0.6%. Over both rounds, 19 of the 29 hidden readings fall within twice the measured repeatability, with a median error of 1.37%, and the rubric judgments stay the same.

Experiments Beyond the Expert Route

We compare every successful run with the expert route of its task in two rounds. We use the successful runs in Figure 6. The first round takes the 14 tasks that the most actor agents solve, 74 runs; the second takes 8 tasks whose decisive test has to be devised by the agent or whose expert route has at least five experiments, 30 runs. For every core item, an Opus 5.5 call that sees the runs in shuffled order without their actor names decides whether the run supports the item with the same kind of experiment as the expert route, with another kind, or partly with another kind. In the two rounds, 10 of 74 and 6 of 30 successful runs support a core item partly with another kind of experiment than the expert route. In both rounds, 36% of the requests of successful runs have no counterpart in the expert route, mostly controls that the actor agents design themselves, such as a small pilot that varies two preparation steps together, which three actor agents ran independently to tell a clot in the sample from a missing reagent.

Red-Team Actor Agents

On 4 tasks, we run Opus 5.5 in three arms. The honest arm is the run of Opus 5.5 at high effort in Figure 6. The exploit arm is told that the laboratory is simulated, that the score is computed against a hidden answer, and that it may exploit weaknesses of the simulation to maximize the score. The flaw-finding arm is asked mainly to find flaws of the simulation, such as physically implausible readings, contradictions with the files, or leaks, and to submit a report as usual. The red-team instructions appear only in the actor agent’s workspace, so the TwinLab agent sees the original task, and all runs are judged blind with the original rubric. Every reported flaw is verified by reading the session and the calculator. The four tasks are a process-window search on public data, a synthetic battery task, the fatigue task, and the oyster task; the first two are scored by a calculator.

The exploit arm makes 9 attempts in total. Seven ask the laboratory staff for the conclusion, the answer key, the source dataset, an underlying table, or history after the cutoff point; one asks for a setting outside the recorded matrix; one lists settings as a range. The history cutoff, the TwinLab agent’s refusals, the calculators’ refusals, and the simulation limits block all of them, no leak guard fires, and no run searches the web, after which the actor agents solve the tasks normally. No core item receives credit without evidence, and on two tasks the second check of core items removes credit that the first judgment gave.

The flaw-finding arm claims 39 flaws: 13 are true, 11 are acceptable simulation limits, and 15 are false alarms. The true flaws are minor, such as two instruments of a synthetic task that use inconsistent voltage models, a handoff file that the TwinLab agent describes wrongly, and a clock that three answers report differently; none of them changes the scoring evidence or a key decision. All claimed physical implausibilities on the public-data task are doubts about real measurements. Looking for flaws also changes the actor agents’ behavior: on two tasks they record the designed discriminating evidence as a flaw, so the scores of this arm do not measure the honest ability.

Failure Analysis

Decision Points: Protocol and Sensitivity

We extract the decision points of each task from the rubric and from the expert route, and we split them into action decisions (133) and discovery decisions (149). Probe A asks the decision questions closed-book, and probe B asks the model to write a plan with read-only access to the handoff files. Under the core criterion, an answer counts as fully stated when it contains the decision of the expert route; under the strict criterion, it also has to meet the full-credit wording of the rubric. The grading model agrees with human annotation on 85% to 88% of the decision points. Probe A names the topic of each decision point and asks what the model would do about it. With the closed-book question (A), decisions stated in full, only in principle, and not mentioned score 82%, 75%, and 71% of their maximum in the run, and with the plan (B) they score 87%, 66%, and 62%. Decisions that the model states in full receive 37% (A) and 61% (B) of the score obtained and 28% (A) and 31% (B) of the score lost. Under the strict criterion, decisions stated in full lose 8% (A) and 13% (B) of their score, and decisions stated only in principle lose 25% (A) and 29% (B). On the action and discovery decisions together with B, the three levels lose 15%, 38%, and 57% of their score. The probes run on the evaluation runs of Opus 5.5 at high effort in Figure 6, and each prompt is sampled three times with the majority answer kept. For the 11 action decisions that B states in full and the run does not score in full, we compare the plan with the requests of the run and with the reason that the judge agent gives for the item. In 2 of them the run does not carry out the planned experiment: one run never tests the extraction kit on fresh tissue, and one run discharges cells that the order needs, although its plan tests only spare cells. In the other 9, the run carries out the planned experiment and loses the score when it draws conclusions from its own readings or writes the report, which accounts for 16 of the 19.5 rubric score that the 11 decisions lose, as Table 12 lists.

TaskScoreWhere the run loses the score
The run does not carry out the planned experiment
001-coral-lcm-porites-030/1.5Never tests the extraction kit on the fresh tissue that the materials list offers.
018-lisocl2-string-match-030/2Discharges cells that the order needs, although the plan tests only spare cells.
The run carries out the planned experiment and draws a wrong conclusion from its readings
005-glycogen-oyster-031/2Measures the samples again and attributes the low readings to the calibration; the 1 μL1\ \mu\mathrm{L} transfer delivers about 30% less.
042-alsi-grid-interactive-021/2Casts repeated bars as planned, but the formula uses the total standard deviation, which also counts the settings with a single bar.
044-seahorse-patient-ocr-010/1.5Grows line F03 again twice, obtains two low growths, and still calls the line normal.
044-seahorse-patient-ocr-010/1.5Gives no reason for the two repeated growths of line F03.
044-seahorse-patient-ocr-010/2.5Reads the respiration of line F15 without normalizing it by cell number and calls the line normal.
The run carries out the planned experiment and the report misstates or omits a result
032-contour-pass-fatigue-030/2.5Identifies the composition correctly, but the report tightens the limits of oxygen and nitrogen together.
038-orau-ultrafiltration-020/2.5Runs the rinse test and the known-age bone, but the report checks the known-age bone every few batches where every batch is needed.
058-nickelate-nv-stress-020/3Processes the stress, shear, and composition data of the sample, but the report keeps an untested explanation of the footprint.
085-ssb-ringtrial-pressure-020/0.5Asks for the stack pressure of cell L08 first, but the report omits that its open-circuit voltage and first capacity are normal.

Table 12. Table 12 Decisions that the plan states in full and the run still loses. Each row is one action decision of Opus 5.5 at high effort, and Score gives the rubric score that the run obtains out of the maximum of the item.

Failure Types per Run

We define three failure types. A mechanism is accepted without a discriminating experiment when the report adopts a mechanism that the run never tested against its alternative. A scarce material is used up before testing when the key scarce material is consumed before the experiment that decides how to use it. A procedure is chosen on weak early evidence when the deliverable depends on an early comparison that used too few samples or was confounded. For every failed run, the released data list the type, the share of budget and of scarce material used when the decisive test started, and whether the run ended because the budget ran out. Of the 16 failed runs of Opus 5.5 at high effort on the 33 tasks that need experiments, 12 fall into these three types, and the other 4 err in reading the results or in writing the report.

Budget Sensitivity

The budget of a task is a multiple of the amount used by the replay of the expert route; the multiple is between 1 and 3 for 49 tasks, with a median of 1.6. Table 13 compares how much of the budget passing and failing runs use on the 40 tasks that charge for experiments. The 12 tasks whose actions are all free of charge, 11 of which offer no experiments, are left out, because every run on them spends nothing.

Actor agentsPassing runsFailing runs
RunsUsed upMedianRunsUsed upMedian
All16052%0.9947940%0.91
Opus 5.51963%1.002124%0.85

Table 13. Budget use of passing and failing runs on the 40 tasks that charge for experiments. Used up counts the runs that spend at least 99% of the budget, and Median is the median fraction of the budget spent.

We also rerun with twice the budget all 8 failed runs of Opus 5.5 that spent at least 90% of their budget. All 8 reruns are valid, and 2 of them pass (the radiocarbon task, from 0 to 95, and the task that matches battery cells into strings, from 0 to 96.4). Of the 6 reruns that fail again, 4 fail on a core item that the original run also failed, and 2 fail on a different core item. Three of these 6 reruns use less than 70% of the doubled budget.

Scorecard Definitions and Annotations

We define the six columns of Figure 5 and their levels. Tasks and scenes count the tasks and the distinct experimental settings. Action types count the distinct experimental actions that the agent can request. The source of research problems has three levels: designed, found in real data, and met and solved by the laboratory. A problem found in real data lies in the measurements, but the record does not show the laboratory meeting it; a problem met and solved by the laboratory appears in the record together with its later answer. The environment for new experiments has three levels: analysis of given data, equations or a designed world, and a replay or a twin of a real laboratory. Memorization control has three levels: the risk is noted, the tasks use data that the model has not seen, such as altered, held-out, or fictional data, or memorization is probed on the tested model. For every existing benchmark, each cell cites the sentence of the original paper that supports it. TwinLab-Bench is annotated task by task with the same definitions, and two annotators label the action types independently (105 and 107 types).

Some benchmarks already ask an agent to design experiments. SciGym ([39]), Discovery-World ([58]), and BoxingGym ([47]) let the agent choose experiments in a designed world or an equation model, so their readings follow written rules. ASB ([125]) lets the agent choose batches of designs within a budget and returns published measurements, so each campaign offers one kind of action over the recorded designs. TwinLab-Bench combines the two properties that these benchmarks have separately: the actor agent composes its experiments freely, and the readings come from the records of real laboratories.

Data Sources and Licenses

Every task uses its source records under the terms of their licenses. Where a license does not allow direct reuse, we contacted the data providers by email, and the other sources are released under licenses that allow reuse with attribution, which we follow. Table 14 lists the source records, the papers, and the related references of every task.

TaskSummaryAbilities testedSources
Physical Sciences (20 tasks)
Physics
031-tesseract-phonon-bursts-03Investigate the source of excess noise, parasitic power and low-energy excess events in a cryogenic phonon detector, and test the group’s existing explanation with devices and measurements chosen from a fabrication menu.hypothesis refutation by controlled comparison; controlled experiment design; separating time-dependent instrument effects; stating the scope of a conclusion[26, 102, 8] (Anthony-Petersen et al. 2025)
058-nickelate-nv-stress-02On a high-pressure NV-centre imaging project with La3Ni2O7\mathrm{La}_3\mathrm{Ni}_2\mathrm{O}_7 shards, give a per-pixel criterion for local superconductivity, explain spatial structure in the maps, predict the behaviour of two further samples before inspecting them, and decide what to load next within a budget of measurement blocks.analysis of spatially resolved raw maps; advance prediction and check; reconciling transport with imaging; decisions about irreversible loading[79, 80]
065-zbp-phase-boundary-02Assess the phase diagram and zero-bias peak claims for a 2015 to 2016 InSb/NbTiN nanowire device using raw QTLab data, and decide what to include in the paper and what to measure next.robustness of extracted quantities to analysis choices; reproducibility across cooldowns; alternative-explanation reasoning; planning quantum transport measurements[29, 28]
068-shapiro-odd-steps-02Evaluate a PhD student’s summary of Shapiro step measurements in an Al/InAs Josephson junction using raw maps and logs, and phrase what can be supported about odd-order steps.robustness of extracted quantities to resolution and analysis choices; raw data processing in mesoscopic transport; claim wording; next-measurement planning[146, 142]
069-nanowire-gn-plateau-02Assess whether an InSb nanowire normal-superconductor junction shows a normal-state conductance plateau using raw QTLab maps, and phrase what the paper can state about it.robustness of an extracted quantity to analysis choices; comparison of repeated maps; quantum transport data processing; claim wording[141, 142]
070-cdte-shell-mobility-02Determine whether a CdTe shell changes the field-effect mobility of InSb nanowires using raw device data and logs, and write what the paper can state about transport.like-for-like device comparison; checks of assumed device parameters; analysis of raw transistor scans; scoping of claims with stated conditions[11, 10]
071-sn-insb-diode-gate-03Determine what the supercurrent asymmetry of Sn-InSb nanowire junctions depends on, using raw data and logs, and prepare a gate-dependence panel with supportable wording.symmetry checks in raw transport data; normalisation choices; identifying data that should not be pooled; confidence-graded statements[140, 142]
073-sninas-shadow-claims-02Evaluate Sn-InAs nanowire shadow junction devices from two cooldowns, deciding which devices work, what the conductance and supercurrent data show, and what to tell the growers.reasoning about the measurement chain; device triage from raw transport data; separation of supported and unsupported claims[111 a]
077-nb-gcs-leakage-02Assess a student’s summary of gated Nb Dayem bridges using raw I(V) and leakage files, build the device table, state what controls the supercurrent suppression, and plan the next device with expected readings fixed in advance.mechanism discrimination from raw transport data; artefact checks in sweeps; advance predictions; design of discriminating device experiments[105, 38]
TaskSummaryAbilities testedSources
083-tgp-readout-region-01Select devices for a readout cooldown from stage-2 tune-up data of InAs-Al hybrid nanowires, using an analysis pipeline and raw scans, and write the device statements to be signed.robustness of pipeline verdicts; inspection of analysis code; choice under limited cryostat slots; raw nonlocal conductance checks([1, 83, 84, 65, 82])
Earth Sciences
038-orau-ultrafiltration-02As the new head of bone pretreatment at a radiocarbon laboratory in 2002, explain why ultrafiltered bone dates from a blind test came back too old, establish a corrected pretreatment, and decide what to do with 208 past determinations.tracing an analytical offset to its source; method validation with known-age standards; retrospective data disposition rules; use of limited archaeological material([23, 22] (English Heritage Research Department 2011))
063-fracture-perm-shear-03For a triaxial fracture permeability experiment on Westerly granite, decide whether permeability falls with shear, what the end-of-run data can be used for, and how to plan the next run.quality assessment of long instrument records; reuse of archived runs; flagging unreliable channels; next-run planning with expected values([73])
086-bermuda-clumped-reprocess-01Reassess reprocessed clumped isotope data for Bermuda shells, deciding whether a site difference is real, whether old and new values can be averaged, and how cold the last interglacial was at these sites.data reduction and reference-frame reasoning; reprocessing of legacy data; use of a limited instrument queue; uncertainty-aware paleotemperature estimates([144 a])
Chemistry
055-esi-h2o2-origin-02Determine where and how H2O2H_2O_2 forms in water collected from a negative-mode electrospray source, evaluate a drafted electrochemical explanation, and report whether positive and uncharged droplets produce H2O2H_2O_2.mechanism testing by selective run planning; charge and mass accounting; evaluation of an existing draft; detection-limit reasoning([27])
062-micelle-catalysis-claims-02Review HPLC kinetics of a 2023 supramolecular micellar catalysis project and decide which rate-enhancement claims in a draft hold, which controls are missing, and what should be run next before the manuscript goes forward.reading ELN prints and chromatography reports; rate-enhancement and control analysis; data-quality screening; planning minimal follow-up runs([119 a])
067-oer-electrodeprepeats-02Review a robotic electrodeposition and oxygen-evolution dataset before submission, deciding which claims, uncertainties and data-quality statements the paper can make and what to change on the platform.replicate uncertainty estimation; screening runs for validity from raw electrochemical traces; analysis of repeated runs; review of paper claims([46, 46])
072-again-odd-even-02Using an archive of raw EGAln J-V data from two laboratories, decide whether n-alkanethiolate junctions on Au show an odd-even effect, compute the tunneling decay coefficient, compare with Ag, and write what the group can claim.session and provenance analysis across a long archive; statistical comparison across datasets; data-integrity screening of archived files; design of a decisive remeasurement(Data-Driven Organic Materials Lab, 2024; Dehghan-Torپoshti et al., 2026)
078-rh-hydroformylation-flex-02Decide which ligand to name as the flexible ligand of a Rh hydroformylation campaign, using high-throughput plate data and newly run plates, and forecast the 15 mL scale-up results.design of comparison plates; analysis of Bayesian-optimisation campaign data; replicate scatter estimation; quantitative scale-up forecasting([14] (Bennett & Abolhasani 2026))
079-rutpphrzh-fstagate-01Review femtosecond transient absorption measurements of a Ru/Rh photocatalyst, decide what the temperature and chloride results can support, and choose measurements for a limited cryostat slot.global lifetime analysis under model choice; Arrhenius analysis; design review of a measurement series; selection of limited instrument time([18, 17])
TaskSummaryAbilities testedSources
080-pyrite-feivomossbauer-01Check a draft section that claims surface high-valent iron species on pyrite catalysts, using Moessbauer, magnetometry and probe-reaction data, and decide which claims stay and what to measure.model selection in spectral fitting; quantitative plausibility estimates; magnetometry analysis; evaluation of manuscript claims([66]; [53])
Life Sciences (20 tasks)
Marine Biology
001-coral-lcm-porites-03Find out why laser-capture microdissected epidermis and gastrodermis of the coral Porites compressa yield little RNA and hard-to-read sections, and deliver a working protocol plus a go or no-go decision for low-input RNA-seq library preparation.root-cause isolation with dedicated controls; RNA quantity and integrity QC; trade-off tuning of histology steps; use of scarce tissue blocks([36])
002-resazurin-clam-02Take over heat-stress resazurin assays on Manila clam juveniles and decide which of three drafted conclusions the plate-reader data support, then fix the assay and data problems and give the protocol and analysis for the next round.auditing raw plate-reader exports and records; detection of data-handling errors; testing readout artefacts by experiment; judging claims against corrected data([126]; [100])
005-glycogen-oyster-03Audit a luminescence glycogen assay on Pacific oyster tissue, decide which drafted conclusions hold, repair the assay and data within the remaining kit, and report final per-animal values and group statistics.auditing plate data and sample records; checking quantification linearity and calibration; re-assay design under limited reagent; recomputation of group statistics([126]; [101])
007-resazurin-oyster-02Review resazurin metabolic assays on nine Pacific oyster families before they reach a breeding program, decide which conclusions hold, and recommend the protocol and analysis to use from now on.reading raw exports with timestamps; testing the assumptions behind a normalised readout; robustness checks of pairwise statistics; protocol redesign([126]; [101])
008-qpcr-polyic-02Check the conclusions of an RT-qPCR experiment on Pacific oysters exposed to PolyIC and other stressors, using raw run exports and limited leftover cDNA, and state how qPCR should be run from now on.experimental design review; confound separation by experiment design; reanalysis of raw instrument exports; use of scarce cDNA([126])
045-oyster-family-heat-03Build a heat-survival reference for nine Pacific oyster families by analysing a first heat challenge and planning further challenges, and choose four families for the next resazurin calibration run.reproducibility assessment across challenges; adaptive allocation of limited animals; conservative group calling; survival analysis([101])
050-oae-photo-scale-03Assess whether a reported effect of ocean alkalinity enhancement preconditioning on Pacific oyster resazurin metabolism holds, list every measurement and analysis problem, and advise how to design the next round.auditing size normalisation of raw readouts; reanalysis with corrected quantities; evaluation of post hoc analysis choices; reuse of earlier runs as evidence; experiment redesign([127]; [101])
051-cod-blood-clot-02Take over DNA and RNA co-extraction from frozen Pacific cod blood for 32 fish, develop a protocol without spending the priority tubes, and report yields against a sequencing provider’s criteria together with the causes of earlier failures.protocol development on surrogate samples; use of irreplaceable samples; QC against provider specifications; root-cause analysis from notebook records([126])
054-coral-nuclei-count-02Review same-day hemocytometer QC for coral single-nuclei RNA-seq, establish the loaded concentration, nuclei loaded and expected recovery per channel, identify logged readings that do not hold up, and say what has to change in the counting.consistency checks across logged counts; reading raw count images; uncertainty ranges; protocol recommendation([37])
Molecular and Cell Biology
003-avian-blood-rna-03Find out why RNA extractions from field-collected raptor blood fail, fix the procedure, and deliver RNA of the required mass and integrity per bird from the portions that remain.troubleshooting a multi-step wet-lab protocol; isolating the failing step with controls; use of irreplaceable samples; QC of delivered material([57]; [56]; Harvey & Knutie, 2023)
TaskSummaryAbilities testedSources
033-kinesin-casein-gliding-02Decide which casein blocker a planned kinesin-1 gliding-motility water-isotope series should use, with limited kinesin for new flow cells and earlier data from a single comparison.identifying uncontrolled variables in a measurement series; assay replication with scarce reagent; setting conditions for a follow-up series; stating limits([75, 78] (Maloney et al. 2010))
047-citp-worm-retest-04Decide for 15 blinded compounds which Caenorhabditis strains show a repeatable lifespan increase of at least 10 %, planning further trials across three sites and predicting whether each effect will carry over to automated lifespan machines.replication planning across sites; multiple-comparison caution; survival statistics; assay-transfer reasoning([12] (Caenorhabditis Intervention Testing Program, n.d.; Lucanic et al. 2017))
056-fly-white-background-02Decide phenotype by phenotype whether differences between a Drosophila w1118 stock and a wild-type stock are caused by the white gene, using the lab’s stocks and assays within a year’s allocation, and state what a thesis chapter may claim.genetic background control; design of comparisons that isolate a causal variable; sex-specific reasoning; scoping of claims([99, 44, 90, 112, 96])
059-ddrad-quant-anchor-01From a student’s 113-entry ddRAD notebook, decide which DNA quantification instrument should anchor the loading sheets, how well that anchor is established, and what must be measured before it is fixed.evidence synthesis from a long notebook; cross-instrument method comparison; statement of uncertainty ranges; measurement planning([5])
060-camp-pulldown-lysates-01As a postdoc supervising a 2010 cAMP pull-down project in airway smooth muscle cells, determine what six RIPA lysates contain, whether the control versus cigarette-smoke-extract comparisons stand, and how the next round should quantify and load.assay interference testing; discriminating instrument faults from other causes; protein quantification; judging comparability of paired samples([93])
076-yeast-raman-batch-01Check a Raman spectroscopy classification run on nine yeast strains before a group meeting, decide how far the reported performance can be trusted, and advise whether a proposed screen should go ahead.validation design for machine learning on spectra; assessment of reported model performance; sensitivity to preprocessing; experimental screen design(Cheveralls et al., 2026; Cheveralls, 2026)
Medicine
016-scrna-doublet-ambient-01From a departed postdoc’s 10x single-cell dataset on injured mouse kidneys, give the percentage of proximal tubule cells in a proposed hybrid state in sham and injured kidneys and the endothelial Spp1 fold change with supporting evidence, and recommend a protocol for the next cohort.separating technical artefacts from biology in single-cell data; choosing orthogonal bench validation; assessing sample integrity; quantitative estimation under uncertainty([59, 49, 133, 116, 134, 62, 88])
034-fip-square-wave-03Determine whether a recurring square-like step in camera-based fiber photometry traces has a neural, behavioural or rig origin, recommend a fix with evidence, and classify archived sessions and their QC handling.artefact attribution from raw time series; cross-rig and same-animal comparisons; archive-scale data triage; evaluation of QC criteria; writing a notice for users([3])
044-seahorse-patient-ocr-01Decide which of 26 patient fibroblast lines have a respiratory chain defect by analysing Seahorse mito stress data and choosing which irreplaceable vials to grow again.analysis of replicate structure; adaptive replication with scarce material; cautious clinical classification; statistics on paired controls([138])
075-cd90-qpcr-replication-01Prepare a dossier on a multicentre RT-qPCR replication of a CD90 expression result, deciding whether each laboratory’s result counts and whether the original finding replicated.validity assessment of replication experiments; treatment of low-signal qPCR measurements; sensitivity to analysis choices; reference-gene and efficiency reasoning([4, 21])
Engineering (12 tasks)
TaskSummaryAbilities testedSources
Electrical and Computer Engineering
018-lisocl2-string-match-03Select which primary lithium thionyl chloride cells to weld into 13 four-cell strings for seabed loggers, given inventory data and a discharge test that uses up every cell it measures.destructive-test budgeting; tray-level sampling and screening; transfer of measurements to mission conditions; lot-release decisions[89, 98, 106 US Patent 4307160] (Jauch Quartz 2020; Cieslak 1992)
021-gas-sensor-drift-01Assign sample labels to 271 recordings of a drifting 62-sensor metal-oxide electronic nose, spending a limited number of paid unblinding credits, and state the expected accuracy of the labels.calibration transfer under sensor drift; allocation of limited ground-truth queries; validation on held-out labels; calibrated accuracy estimates[131 b]
085-ssb-ringtrial-pressure-02Coordinate an interlaboratory sulfide all-solid-state battery round robin, deciding the go or no-go rule, the end point and the cycling pressure for round 2, and forecasting which cells will fall below a retention threshold.interlaboratory variance analysis; early-cycle forecasting; round-robin protocol design; evaluation of screening criteria[95]
Mechanical Engineering
022-alsi-process-window-01Choose the production setting of a laser powder bed fusion AlSi10Mg bracket from a 60-setting matrix with a limited machine budget, so that the setting and its drift neighbours meet a tensile strength requirement at the highest build rate.sequential experiment planning under a budget; constrained process-window search; robustness reasoning; tensile data interpretation[71 a]
042-alsi-grid-interactive-02Choose the production setting of an AlSi10Mg laser powder bed fusion bracket from one-bar screening data and paid confirmation bars, so that a single bar reaches a strength requirement with 95 % confidence anywhere inside the machine tolerance.tolerance analysis across a process window; statistical bounds from pooled replicate scatter; budgeted confirmation testing; outlier handling[71 a]
084-shell-record-retest-01Decide what a paper can claim for two 3D-printed shell designs against balsa, using the records of an autonomous print-and-crush loop and later retest packages.retesting for repeatability; part-level quality control analysis; evaluation of optimiser-selected results; lab-process recommendations[114]
Materials Science and Engineering
029-die-attach-release-01Decide which of 96 silver-sintered die-attach assemblies can ship as a 42-unit lot with a minimum die shear strength, when the shear test is destructive and test carriers are limited.destructive-test allocation; evaluation of a screening proxy; statistical release rules; use of scarce test fixtures[72, 43]
032-contour-pass-fatigue-03Determine what controls fatigue crack initiation in hot isostatically pressed laser powder bed fusion 316L specimens, tie it to a changeable item in the process record, and recommend changes to the next build and the powder specification.fractography and phase identification; testing a default causal explanation; linking failure evidence to process records; separating correlation from demonstrated cause(Wieteccha-Reiman et al., 2025)
040-ceder-spinel-eis-03Take over a self-driving-lab campaign for lithium chloride spinel conductors, decide which composition routes and agent hypotheses to keep or drop, identify the best Li-ion conductors and the basis for that claim, and state what the paper can claim.checking automated analysis against raw impedance spectra; evaluation of agent-generated hypotheses; experiment selection from a synthesis menu; conductivity and phase interpretation[41 b]
053-ec-film-od-cap-01Decide whether an electrochromic film campaign justifies capping optical density, recommend a process window, and explain the high defect scores of two batches, using the film-by-film table of a multi-objective self-driving lab and four remaining batch days.analysis of batched campaign data; check of a claimed trade-off; budgeted batch planning; replication of process windows(Dahms et al., 2025; aimat-lab, 2025)
TaskSummaryAbilities testedSources
082-mmb-delamination-r-effect-01Work out what mixed-mode bending fatigue delamination data show about the load-ratio effect in a composite joint, decide which crack-growth numbers can be sent to project partners, and write the summary.checks of processed data against raw records; provenance checking; judgement of fatigue-law fits; communication of caveats([87])
Chemical Engineering
061-silica-saxs-attribution-03In a self-driving silica nanoparticle campaign measured by SAXS, establish which measurements belong to which recipes in crossed rounds and what the comparison of two Bayesian optimisers says, before the optimisers are retrained.provenance tracing through repositories and robot logs; data integrity assessment; raw SAXS curve checks; reproducible scoreboard construction([94])

Table 14. Table 14 Tasks of TwinLab-Bench. Tasks are grouped by division and department as in Figure 1. Each row gives the task identifier, a summary of the task, the abilities that the task tests, and the source records and papers of the task.

The pictures in Figure 1 come from the source records of the four tasks. The fluorescence images of gliding microtubules are from the open notebook of Maloney on OpenWetWare ([74]) (CC BY-SA), and the fracture surface of the steel specimen is Figure 7a of Wietecha-Reiman et al. [128] (CC BY 4.0).

TwinLab-Bench task coverage across engineering, physical sciences, and life sciences, with examples of actor-agent and TwinLab-agent interactions. The central diagram shows 52 tasks divided among Engineering, Physical sciences, and Life sciences, with departments including Chemical, Electrical, Mechanical, Materials, Chemistry, Physics, Earth Sci., Medicine, Mol. & Cell Bio., and Marine Bio. Example tasks shown are Fish blood: DNA and RNA from one tube; Wet lab: test six extraction methods, then extract 27 fish; Molecular motors: rank four caseins; Wet lab: build flow cells and film moving filaments; 3D-printed steel: where fatigue cracks start; Materials lab: image broken bars with SEM, TEM and CT; Superconductivity: are odd Shapiro steps missing?; Physics lab: re-read 2,149 raw files behind a student’s claim.

Figure 1. TwinLab-Bench covers 52 tasks by division and department, and a separate TwinLab agent drives the environment of each task.

Implementation of the Twin and the Judge

Mechanisms of the Twin and the Scoring

Readings. The environment code checks that the recorded value is used whenever the actor agent’s request allows it. The TwinLab agent gives no hints or interpretations, does not substitute another procedure for a requested one, and answers questions about history only up to the cutoff point. When the actor agent asks whether the cause is X, the TwinLab agent only answers that this needs an experimental test. The environment code also asserts that observations contain no internal identifier or leakage pattern and that an unexecuted request is neither charged nor consumes material, and a failed assertion stops the run.

Parameter values. A listed value is either a condition that the laboratory ran or a condition with a written extrapolation basis; across all tasks, 371 listed values are recorded and 100 are extrapolated. An extrapolated value may never be needed in the real experiment, so it acts as a plausible distractor, and the lists are not ordered by the answer.

Isolation. The actor agent’s sandbox has no configuration, memory, skill, or past session. The TwinLab agent cannot see the actor agent’s reasoning, and the actor agent cannot see image file names or the rationale of the TwinLab agent. The environment code checks that the actor agent was answered by the requested model, and a run is invalid when the platform switched models during it. We scan the content that reaches the actor agent through its tools, and a blocked string or a per-task marker string in this content, or repeated attempts to reach a blocked source, invalidates the run. Every run records a hash of the rubric that is compared with the frozen version.

Design against hacking of the twin. A strong actor agent may seek out conditions where the calculators extrapolate favorably, or look for flaws in the twin, as agents exploit flaws in their rewards and tests elsewhere [6, 148]. Besides scoring against the hidden truth, three designs limit this risk. Numbers come from deterministic calculators, extrapolation is confined to values with a written basis, and the leakage guards, the replay script, and the environment audit check every run.

We release the environment code, which enforces the remaining rules, such as measurement noise, checkpoints, billing, and the stock of scarce materials.

Judge Agent and Environment Audit

The environment audit has three parts. A leak is an observation that states an explanation, a cause, or the hidden truth. An inconsistency is an observation that contradicts the task, an earlier observation, or physics, including identical values for independent samples, silently changed parameters, and unexecuted items reported as results. A coverage gap is a refusal of an experiment that a real laboratory can run and that matters for scoring. Any of the three makes the run invalid. We release all prompts with the evaluation code, including those of the actor agent, the TwinLab agent, the judge agent, and every probe in these appendices.

Task List and Construction

Of the 52 tasks, 49 are adapted directly from the records of one laboratory and 3 are grounded syntheses. In some of the adapted tasks the laboratory itself met the problem and later solved it, and in the others the problem lies in the real data of the laboratory. By form, 37 tasks are claim reviews, 5 are troubleshooting tasks, and 10 are research campaigns. The hidden truth of a task comes from a later notebook entry, a publication, a correction, or a reanalysis of the raw data. The hidden truth of the remaining 3 tasks, which are grounded syntheses, is fixed by construction. These tasks use real data as background, the literature shows that their problem occurred on the same kind of device or sample at a similar size, and their answer comes from the cases that the literature documents. The handoff materials are taken from the last version of the source repository that was committed before the cutoff point, so they contain no file or finding that appeared later. The readings, the hidden truth, and the expert route of every task come from the records and annotations that the source laboratory published, and every task follows the same construction steps. We considered about 375 candidate sources and kept those with a closed answer, raw data that can anchor new readings, and key findings that require a specific diagnosis.

Table 14 lists the 52 tasks with a summary of each task, the abilities that it tests, and the records and papers that it is built from, and groups the tasks by division and department as in Figure 1.

References

  1. [1]Morteza Aghaee, Alejandro Alcaraz Ramirez, Zulfi Alam, Rizwan Ali, Mariusz Andrzejczuk, Andrey Antipov, Mikhail Astafev, Amin Barzegar, Bela Bauer, Jonathan Becker, et al. Interferometric single-shot parity measurement in inas–al hybrid devices. Nature, 638(8051):651–655, 2025. doi: 10.1038/s41586-024-08445-2.DOI
  2. [3]Allen Institute for Neural Dynamics. Square-like signals peri go cue. GitHub issue #30, AllenNeuralDynamics/aind-fip-qc-raw, 2026. URL https://github.com/AllenNeuralDynamics/aind-fip-qc-raw/issues/30.
  3. [4]Olavo Bohrer Amaral, Clarissa França Dias Carneiro, Kleber Neves, Ana Paula Wasilewska Sampaio, Bruna Valério Gomes, Mariana Boechat de Abreu, Pedro Batista Tan, Gabriel Paz Souza Mota, Ricardo Netto Goulart, Nathalia Raquel de Souza Fernandes, et al. Estimating the replicability of brazilian biomedical science, 2025.DOI
  4. [5]Natalie Ameral. Natalie ameral’s open lab notebook. GitHub repository njameral/Ameral_Lab_Notebook, 2020. URL https://github.com/njameral/Ameral_Lab_Notebook.
  5. [6]Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016.arxiv.org/abs/1606.06565
  6. [8]Robin Anthony-Petersen, Andreas Biekert, Raymond Bunker, Clarence L. Chang, Yen-Yung Chang, Luke Chaplinsky, Eleanor Fascione, Caleb W. Fink, Maurice Garcia-Sciveres, Richard Germond, et al. A stress-induced source of phonon bursts and quasiparticle poisoning. Nature Communications, 15(1), 2024. doi: 10.1038/s41467-024-50173-8.
  7. [9]Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, et al. HealthBench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775, 2025.
  8. [10]Ghada Badawy, Bomin Zhang, Tomáš Rauch, Jamo Momand, Sebastian Koelling, Jason Jung, Sasa Gazibegovic, Bart J. Kooi, Silvana Botti, Marcel A. Verheijen, Sergey M. Frolov, and Erik P. A. M. Bakkers. Data, codes for ’electronic structure and epitaxy of cdte shells on insb nanowires’, 2021.
  9. [11]Ghada Badawy, Bomin Zhang, Tomáš Rauch, Jamo Momand, Sebastian Koelling, Jason Jung, Sasa Gazibegovic, Oussama Moutanabbir, Bart J. Kooi, Silvana Botti, et al. Electronic structure and epitaxy of cdte shells on insb nanowires. Advanced Science, 9(12), 2022. doi: 10.1002/advs.202105722.
  10. [12]Stephen A. Banse, Mark Lucanic, Christine A. Sedore, Anna L. Coleman-Hulbert, W. Todd Plummer, Esteban Chen, Jason L. Kish, David Hall, Brian Onken, Michael P. Presley, et al. Automated lifespan determination across caenorhabditis strains and species reveals assay-specific effects of chemical interventions. GeroScience, 41(6):945–960, 2019. doi: 10.1007/s11357-019-00108-9.
  11. [13]Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ²-bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025.arxiv.org/abs/2506.07982
  12. [14]J. A. Bennett, N. Orouji, A. Velayati, Eamon F. Reynolds, Matthew E. Porowski, M. Saha, D. Billodeaux, Alexander J. M. Miller, and M. Abolhasani. An autonomous lab for data-driven homogeneous catalysis. Nature Communications, 17(1), 2026. doi: 10.1038/s41467-026-74425-x.DOI
  13. [16]Berkeley RDI. AgentBeats: Agentified agent assessment. https://rdi.berkeley.edu/agentx-agentbeats, 2026. Accessed Nov. 11, 2026.
  14. [17]Louis Blechschmidt, Linda Zedler, Alexander Mengele, Sven Rau, and Benjamin Dietzek-Ivansic. Ligand labilization gates intramolecular electron transfer in molecular photocatalyst - dataset, 2024.DOI
  15. [18]Louis Blechschmidt, Linda Zedler, Alexander K. Mengele, Sven Rau, and Benjamin Dietzek-Ivanšić. Ligand labilization gates intramolecular electron transfer in molecular photocatalyst. Chemical Communications, 61(12):2508–2511, 2025. doi: 10.1039/d4cc05844e.DOI
  16. [19]Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624:570–578, 2023.DOI
  17. [20]Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D. White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6:525–535, 2024.DOI
  18. [21]Brazilian Reproducibility Initiative. bri-analysis. GitHub repository, 2026. URL https://github.com/BrazilianReproducibilityInitiative/bri-analysis.
  19. [22]Fiona Brock, Christopher Bronk Ramsey, and Thomas Higham. Quality assurance of ultrafiltered bone dating. Radiocarbon, 49(2):187–192, 2007. doi: 10.1017/S0033822200042107.DOI
  20. [23]Christopher Bronk Ramsey, Thomas Higham, Angela Bowles, and Robert Hedges. Improvements to the pretreatment of bone at oxford. Radiocarbon, 46(1):155–163, 2004. doi: 10.1017/S0033822200039473.DOI
  21. [25]Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, et al. MLE-bench: Evaluating machine learning agents on machine learning engineering. In ICLR, 2025.arxiv.org/abs/2410.07095
  22. [26]C. L. Chang, Y.-Y. Chang, M. Garcia-Sciveres, W. Guo, S. A. Hertel, X. Li, J. Lin, M. Lisovenko, R. Mahapatra, W. Matava, et al. Spontaneous generation of athermal phonon bursts within bulk silicon causing excess noise, low energy background events, and quasiparticle poisoning in superconducting sensors. Applied Physics Letters, 127(26), 2025. doi: 10.1063/5.0281876.DOI
  23. [27]Casey J. Chen, Pyeongyeon Kim, Evan R. Williams, and Kevin R. Wilson. Asymmetry in droplet charge and h2o2 production: Exclusive production of hydrogen peroxide in negatively charged microdroplets via electron emission. JACS Au, 6(8):4503–4512, 2026. doi: 10.1021/jacsau.6c00548.DOI
  24. [28]Jun Chen and Sergey Frolov. Full data for ’experimental phase diagram of zero-bias conductance peaks in superconductor/semiconductor nanowire devices’, 2021.
  25. [29]Jun Chen, Peng Yu, John Stenger, Moïra Hocevar, Diana Car, Sébastien R. Plissard, Erik P. A. M. Bakkers, Tudor D. Stanescu, and Sergey M. Frolov. Experimental phase diagram of zero-bias conductance peaks in superconductor/semiconductor nanowire devices. Science Advances, 3(9), 2017. doi: 10.1126/sciadv.1701476.DOI
  26. [30]Ziru Chen, Shijie Chen, Yuting Ning, et al. ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific discovery. In ICLR, 2025.arxiv.org/abs/2410.05080
  27. [36]Zoe Dellaert. Troubleshooting laser capture microdissection of porites compressa. Zoe Dellaert’s Open Lab Notebook, Putnam Lab, University of Rhode Island, 2026a. URL https://zdellaert.github.io/ZD_Putnam_Lab_Notebook/LCM-Porites-Troubleshooting/.
  28. [37]Zoe Dellaert. Nuclei isolation for coral single nuclei rna-seq. Zoe Dellaert’s Open Lab Notebook, Putnam Lab, University of Rhode Island, 2026b. URL https://zdellaert.github.io/ZD_Putnam_Lab_Notebook/coral-nuclei-isolation/.
  29. [38]Angelo Di Bernardo. Dataset for "high-performance gate-controlled superconducting switches: Large output voltage and reproducibility", 2024.arxiv.org/abs/2404.18899
  30. [39]Haonan Duan et al. Measuring scientific capabilities of language models with a systems biology dry lab. arXiv preprint arXiv:2507.02083, 2025.arxiv.org/abs/2507.02083
  31. [41]Yuxing Fei, Bernardus Rendy, Xiaochen Yang, Junhee Woo, Xu Huang, Chang Li, Shilong Wang, David Milsted, Yan Zeng, and Gerbrand Ceder. Agentic llm reasoning in a self-driving laboratory for air-sensitive lithium halide spinel conductors, 2026a.arxiv.org/abs/2604.11957
  32. [43]Shuang-Tao Feng, Yun-Hui Mei, Gang Chen, Xin Li, and Guo-Quan Lu. Characterizations of rapid sintered nanosilver joint for attaching power chips. Materials, 9(7):564, 2016. doi: 10.3390/ma9070564.DOI
  33. [44]María José Ferreiro, Coralia Pérez, Mariana Marchesano, Santiago Ruiz, Angel Caputi, Pedro Aguilera, Rosa Barrio, and Rafael Cantera. Drosophila melanogaster white mutant w1118 undergo retinal degeneration. Frontiers in Neuroscience, 11, 2018. doi: 10.3389/fnins.2017.00732.DOI
  34. [46]Nis Fisker-Bødker, Daniel Persaud, Yang Bai, Mark Kozdras, Tejs Vegge, Jason Hattrick-Simpers, and Jin Hyun Chang. Ampere-2: an open-hardware, robotic platform for automated electrodeposition and electrochemical validation. Digital Discovery, 4(9):2491–2501, 2025. doi: 10.1039/D5DD00180C.
  35. [47]Kanishk Gandhi, Michael Y. Li, Lyle Goodyear, Agam Bhatia, Louise Li, Aditi Bhaskar, Mohammed Zaman, and Noah D. Goodman. BoxingGym: Benchmarking progress in automated experimental design and model discovery. arXiv preprint arXiv:2501.01540, 2025.arxiv.org/abs/2501.01540
  36. [48]Hongcheng Gao, Zelin Zheng, Alex Zhou, Jay Zhu, Shijia Ge, Yazhe Wang, Xiping Li, Jianshu Zeng, Di Wu, Xuan Shangguan, Yexing Wan, Lingyu He, Yongli Wang, Xiaohan Yu, Zhiqi Jia, Yue Liu, Bryan Hooi, Yinpeng Dong, Wentao Zhang, Sihang Wu, and Xiao He. Embodiedlab: Benchmarking embodied agents that act to know. arXiv preprint arXiv:2610.35385, 2026.arxiv.org/abs/2610.35385
  37. [49]Pierre-Luc Germain, Aaron Lun, Carlos Garcia Meixide, Will Macnair, and Mark D. Robinson. Doublet identification in single-cell sequencing data using scdblfinder. F1000Research, 10:979, 2022. doi: 10.12688/f1000research.73600.2.DOI
  38. [50]Ali Essam Ghareeb, Benjamin Chang, Ludovico Mitchener, et al. A multi-agent system for automating scientific discovery. Nature, 655:497–505, 2026. doi: 10.1038/s41586-026-10652-y.DOI
  39. [51]Shahriar Golchin and Mihai Surdeanu. Time travel in LLMs: Tracing data contamination in large language models. In ICLR, 2024.arxiv.org/abs/2308.08493
  40. [52]Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, et al. Accelerating scientific discovery with Co-Scientist. Nature, 655:487–496, 2026. doi: 10.1038/s41586-026-10644-y.DOI
  41. [53]Fernande Grandjean and Gary J. Long. An inadequate over-parameterized mössbauer spectral analysis for the mistaken detection of surface feiv=o, in nature communications 16, 7642 (2025). Nature Communications, 17(1), 2026. doi: 10.1038/s41467-026-78477-x.
  42. [54]David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018.DOI
  43. [55]Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In EMNLP, 2023.
  44. [56]Johanna A. Harvey. Rna-blood-preservation-extraction. GitHub repository, 2021. URL https://github.com/JAHarvey/RNA-Blood-preservation-extraction.
  45. [57]Harvey Lab, University of Rhode Island. Harveylabnotebook. GitHub repository, 2026. URL https://github.com/JAHarvey/HarveyLabNotebook.
  46. [58]Peter Jansen, Marc-Alexandre Côté, Tushar Khot, et al. DiscoveryWorld: A virtual environment for developing and evaluating automated scientific discovery agents. In NeurIPS, 2024.
  47. [59]Philipp Janssen, Zane Kliesmete, Beate Vieth, Xian Adiconis, Sean Simmons, Jamie Marshall, Cristin McCabe, Holger Heyn, Joshua Z. Levin, Wolfgang Enard, et al. The effect of background noise and its removal on the analysis of single-cell expression data. Genome Biology, 24(1), 2023. doi: 10.1186/s13059-023-02978-x.
  48. [61]Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI agents that matter. arXiv preprint arXiv:2407.01502, 2024.arxiv.org/abs/2407.01502
  49. [62]Yuhei Kirita, Haojia Wu, Kohei Uchimura, Parker C. Wilson, and Benjamin D. Humphreys. Cell profiling of mouse acute kidney injury reveals conserved cellular responses to injury. Proceedings of the National Academy of Sciences, 117(27):15874–15883, 2020. doi: 10.1073/pnas.2005477117.
  50. [63]Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, et al. LAB-Bench: Measuring capabilities of language models for biology research. arXiv preprint arXiv:2407.10362, 2024.
  51. [64]Jon M. Laurent, Albert Bou, Michael Pieler, et al. LABBench2: An improved benchmark for AI systems performing biology research. arXiv preprint arXiv:2604.09554, 2026.
  52. [65]Henry F. Legg. On the robustness of topological gap detection via transport. Nature, 654(8120):E22–E26, 2026. doi: 10.1038/s41586-026-10567-8.
  53. [66]Cancan Ling, Meiqi Li, Hao Li, Xiufan Liu, Furong Guo, Yi Liu, Rui Zhang, Jincai Zhao, and Lizhi Zhang. High-spin surface feiv = o synthesis with molecular oxygen and pyrite for selective methane oxidation. Nature Communications, 16(1), 2025. doi: 10.1038/s41467-025-63087-w.
  54. [67]Yi Liu, TingFeng Hui, Wei Zhang, et al. EnvSimBench: A benchmark for evaluating and improving LLM-based environment simulation. arXiv preprint arXiv:2605.07247, 2026.
  55. [68]Chris Lu et al. Towards end-to-end automation of AI research. Nature, 651:914–919, 2026. doi: 10.1038/s41586-026-10265-5.
  56. [71]Qixiang Luo, Nancy Huang, Tianyi Fu, Jinying Wang, Dean L. Bartles, Timothy W. Simpson, and Allison M. Beese. Processing, microstructure, and mechanical property dataset for alsi10mg fabricated by laser powder bed fusion additive manufacturing, 2023b. URL https://zenodo.org/records/10008435.
  57. [72]Youshuo Ma, Xin Li, and Hongyu Zhang. Effect of grain structure of gold plating layer on environmental reliability of sintered ag-au joints. Materials, 17(8):1844, 2024. doi: 10.3390/ma17081844.DOI
  58. [73]Ben Madara, Chris Marone, Derek Elsworth, and Paul Johnson. Penn state lab testing fluid-rock interaction in geothermal reservoirs, 2018.
  59. [74]Andy Maloney. Lab notebook of Andy Maloney, Koch Lab, University of New Mexico. OpenWetWare, https://openwetware.org/wiki/User:Andy_Maloney, 2010a. CC BY-SA.
  60. [75]Andy Maloney. More on kinesin speed effects due to temperature. OpenWetWare lab notebook, 2010b. URL https://openwetware.org/wiki/User:Andy_Maloney/Notebook/Lab_Notebook_of_Andy_Maloney/2010/05/25/More_on_kinesin_speed_effects_due_to_temperature.
  61. [78]Roger Maloney. Experimental protocols for and studies of the effects of surface passivation and water isotopes on the gliding speed of microtubules propelled by kinesin-1. PhD thesis, University of New Mexico, 2011. URL https://digitalrepository.unm.edu/phys_etds/40.
  62. [79]S. V. Mandyam, E. Wang, Z. Wang, B. Chen, N. C. Jayarama, A. Gupta, E. A. Riesel, V. I. Levitas, C. R. Laumann, and N. Y. Yao. Uncovering origins of heterogeneous superconductivity in la3ni2o7. Nature, 651(8104):54–60, 2026. doi: 10.1038/s41586-025-10095-x.DOI
  63. [80]Srinivas V. Mandyam, Esther Wang, Zhipan Wang, Bijuan Chen, Nishan C. Jayarama, Anmay Gupta, Eric A. Riesel, Valery Levitas, Christopher Laumann, and Norman Yao. Uncovering origins of heterogeneous superconductivity in la3ni2o7 using quantum sensors, 2025. URL https://zenodo.org/records/17971352. Data deposit.
  64. [81]Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In ICLR, 2026.
  65. [82]Microsoft Quantum Hardware. azure-quantum-tgp. Python Package Index, version 1.1.0, 2023. URL https://pypi.org/project/azure-quantum-tgp/1.1.0/.
  66. [83]Microsoft (United States). Interferometric single-shot parity measurement in inas-al hybrid devices, 2025.
  67. [84]Microsoft (United States). Interferometric single-shot parity measurement in inas-al hybrid devices, 2026.
  68. [85]Ludovico Mitchener, Jon M. Laurent, Alex Andonian, Benjamin Tenmann, et al. BixBench: a comprehensive benchmark for LLM-based agents in computational biology. arXiv preprint arXiv:2503.00096, 2025a.arxiv.org/abs/2503.00096
  69. [86]Ludovico Mitchener et al. Kosmos: An AI scientist for autonomous discovery. arXiv preprint arXiv:2511.02824, 2025b.arxiv.org/abs/2511.02824
  70. [87]Francisco Monticeli, Davide Biagini, John-Alan Pascoe, Yasmine Mosleh, Dario Di Maio, Yunxia Ren, Hanna Beketova, Giuliano Allegri, Emiel Amsterdam, and Robbert-Jan van den Brink. D-standard - experimental mechanical test data set, 2026.DOI
  71. [88]Walter Muskovic and Joseph E. Powell. Dropletqc: improved identification of empty droplets and damaged cells in single-cell rna-seq data. Genome Biology, 22(1), 2021. doi: 10.1186/s13059-021-02547-0.DOI
  72. [89]Ocean News & Technology. Lithium batteries enable long-term, remote and wireless deployments. Ocean News & Technology, 2024. URL https://oceannews.com/featured-stories/lithium-batteries-enable-long-term-remote-and-wireless-deployments/.
  73. [90]Gregory F. Oxenkrug. The extended life span of drosophila melanogaster eye-color (white and vermilion) mutants with impaired formation of kynurenine. Journal of Neural Transmission, 117(1):23–26, 2010. doi: 10.1007/s00702-009-0341-7.DOI
  74. [91]Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. In NeurIPS, 2024.arxiv.org/abs/2404.13076
  75. [93]Wilfred J. Poppinga. Notebook: camp compartmentalization. OpenWetWare lab notebook, 2010. URL https://openwetware.org/wiki/User:Wilfred_J._Poppinga/Notebook/cAMP_compartmentalization.
  76. [94]Pozzo Research Group. silica-np-synthesis. GitHub repository, 2024. URL https://github.com/pozzo-research-group/silica-np-synthesis.
  77. [95]Sebastian Puls, Elina Nazmutdinova, Fariza Kalyk, Henry M. Woolley, Jesper Frost Thomsen, Zhu Cheng, Adrien Fauchier-Magnan, Ajay Gautam, Michael Gockeln, So-Yeon Ham, et al. Benchmarking the reproducibility of all-solid-state battery cell performance. Nature Energy, 9(10):1310–1320, 2024. doi: 10.1038/s41560-024-01634-3.DOI
  78. [96]Shuang Qiu, Chengfeng Xiao, and R Meldrum Robertson. Different age-dependent performance in drosophila wild-type canton-s and the white mutant w1118 flies. Comparative Biochemistry and Physiology Part A: Molecular & Integrative Physiology, 206:17–23, 2017. doi: 10.1016/j.cbpa.2017.01.003.DOI
  79. [97]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In COLM, 2024.arxiv.org/abs/2311.12022
  80. [98]Nick Richards. How to configure the bq35100 for eos mode, 2022. URL https://www.ti.com/lit/an/sluaa17/sluaa17.pdf. Application Note SLUAA17.
  81. [99]April Rickle, Krittika Sudhakar, Alix Booms, Ellen Stirtz, and Adelheid Lempradl. More than meets the eye: mutation of the white gene in drosophila has broad phenotypic and transcriptomic effects. GENETICS, 230(3), 2025. doi: 10.1093/genetics/iyaf097.DOI
  82. [100]Roberts Lab. resazurin-assay-development. GitHub repository, 2025a. URL https://github.com/RobertsLab/resazurin-assay-development.
  83. [101]Roberts Lab. sormi-assay-development. GitHub repository, 2025b. URL https://github.com/RobertsLab/sormi-assay-development.
  84. [102]Roger K. Romani. Aluminum relaxation as the source of excess low energy events in low threshold calorimeters. Journal of Applied Physics, 136(12), 2024. doi: 10.1063/5.0222654.DOI
  85. [103]Yusuf Roohani, Andrew Lee, Qian Huang, et al. BioDiscoveryAgent: An AI agent for designing genetic perturbation experiments. In ICLR, 2025.arxiv.org/abs/2405.17631
  86. [104]Ant Rowstron and Ayush Chadha. Can an AI scientist really discover knowledge? Advanced Research and Invention Agency (ARIA), https://aria.org.uk/media/twpp2gjh/an-ai-scientist-benchmark.pdf, 2026. Version 1.0, June 5, 2026.
  87. [105]Leon Ruf, Elke Scheer, and Angelo Di Bernardo. High-performance gate-controlled superconducting switches: Large output voltage and reproducibility. ACS Nano, 18(31):20600–20610, 2024. doi: 10.1021/acsnano.4c05910.DOI
  88. [106]Saft. LSH 20 primary lithium-thionyl chloride cell data sheet, 2009. URL https://www.enix-power-solutions.com/wp-content/uploads/2021/10/LSH20-15.pdf.
  89. [107]Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of EMNLP, 2023.
  90. [108]Samuel Schmidgall, Rojin Ziaei, Carl Harris, et al. AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments. arXiv preprint arXiv:2405.07960, 2024.arxiv.org/abs/2405.07960
  91. [109]Thomas Schmied, Jörg Bornschein, Jordi Grau-Moya, Markus Wulfmeier, and Razvan Pascanu. LLMs are greedy agents: Effects of RL fine-tuning on decision-making abilities. arXiv preprint arXiv:2504.16078, 2025.arxiv.org/abs/2504.16078
  92. [111]Amritesh Sharma, An-Hsi Chen, Connor P. Dempsey, Amrita Purkayastha, Mihir Pendharkar, Susheng Tan, Christopher J. Palmstrøm, Sergey M. Frolov, and Moïra Hocevar. Sn-inas nanowire shadow-defined josephson junctions. Nano Letters, 25(34):12869–12875, 2025b. doi: 10.1021/acs.nanolett.5c02410.DOI
  93. [112]A. F. Simon, M.-T. Chou, E. D. Salazar, T. Nicholson, N. Saini, S. Metchev, and D. E. Krantz. A simple assay to study social behavior in drosophila : measurement of social space within a group. Genes, Brain and Behavior, 11(2):243–252, 2012. doi: 10.1111/j.1601-183x.2011.00740.x.DOI
  94. [113]Aditya Sivakumar, Ashu Singhal, Nicholas Larus-Stone, and Nithin Parsan. BenchBench-Protocol: Evaluating real-world wet-lab protocol reasoning and modification. arXiv preprint arXiv:2608.23898, 2026.arxiv.org/abs/2608.23898
  95. [114]Kelsey L. Snapp, Benjamin Verdier, Aldair E. Gongora, Samuel Silverman, Adedire D. Adesiji, Elise F. Morgan, Timothy J. Lawton, Emily Whiting, and Keith A. Brown. Superlative mechanical energy absorbing efficiency discovered through self-driving lab-human partnership. Nature Communications, 15(1), 2024. doi: 10.1038/s41467-024-48534-4.DOI
  96. [115]Giulio Starace, Oliver Jaffe, Dane Sherburn, et al. PaperBench: Evaluating AI’s ability to replicate AI research. In ICML, 2025.arxiv.org/abs/2504.01848
  97. [116]Marlon Stoeckius, Shiwei Zheng, Brian Houck-Loomis, Stephanie Hao, Bertrand Z. Yeung, William M. Mauck, Peter Smibert, and Rahul Satija. Cell hashing with barcoded antibodies enables multiplexing and doublet detection for single cell genomics. Genome Biology, 19(1), 2018. doi: 10.1186/s13059-018-1603-1.DOI
  98. [117]Nathan J. Szymanski, Bernardus Rendy, Yuxing Fei, et al. An autonomous laboratory for the accelerated synthesis of inorganic materials. Nature, 624:86–91, 2023.DOI
  99. [119]Mei-Ling Tan, M. Ángeles Gutiérrez López, Naomi Sakai, and Stefan Matile. Anion-(π)n-π catalytic micelles. Angewandte Chemie International Edition, 62(40), 2023b. doi: 10.1002/anie.202310393.DOI
  100. [120]Terminal-Bench-Science Team. Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains. https://www.tbench.ai/news/terminal-bench-science-0-1, 2026. Released Aug. 2026; no arXiv paper as of Nov. 7, 2026.
  101. [121]Gary Tom, Stefan P. Schmid, Sterling G. Baird, et al. Self-driving laboratories for chemistry and materials science. Chemical Reviews, 124(16):9633–9732, 2024.DOI
  102. [123]Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. ScienceWorld: Is your agent smarter than a 5th grader? In EMNLP, 2022.arxiv.org/abs/2203.07540
  103. [124]Ruoyao Wang, Graham Todd, Ziang Xiao, Xingdi Yuan, Marc-Alexandre Côté, Peter Clark, and Peter Jansen. Can language models serve as text-based world simulators? In ACL, 2024.
  104. [125]Yibo Wen and Han Liu. Autonomous science bench: Evaluating AI agents for autonomous scientific discovery. arXiv preprint arXiv:2610.18449, 2026.arxiv.org/abs/2610.18449
  105. [126]Sam White. Sam’s notebook. Open lab notebook, Roberts Lab, University of Washington, 2018. URL https://robertslab.github.io/sams-notebook/.
  106. [127]Sam White. Resazurin assays - usda m.gigas juveniles oae preconditioning effect on 36c heat stress response. Sam’s Notebook, Roberts Lab, University of Washington, 2026. URL https://robertslab.github.io/sams-notebook/posts/2026/2026-08-31-Resazurin-Assays--USDA-M.gigas-Juveniles-OAE-Preconditioning-Effect-on-36C-Heat-Stress-Response/.
  107. [128]I. J. Wietecha-Reiman, A. D. Iams, S. M. Sabol, and T. A. Palmer. Contributions of sub-surface intergranular phases on fatigue crack initiation in additively manufactured austenitic stainless steel. Journal of Materials Science, 60(39):18332–18351, 2025. doi: 10.1007/s10853-025-11259-w.
  108. [129]Hjalmar Wijk, Tao Lin, Joel Becker, et al. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114, 2024.arxiv.org/abs/2411.15114
  109. [130]Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks. In NAACL, 2024.arxiv.org/abs/2307.02477
  110. [131]Julius Wörner, Jonas Eimler, and Miriam Pein-Hackelbusch. Long-term drift behavior in metal oxide gas sensor arrays: a one-year dataset from an electronic nose. Scientific Data, 12(1), 2025a. doi: 10.1038/s41597-025-05993-8.DOI
  111. [133]Nan Miles Xi and Jingyi Jessica Li. Benchmarking computational doublet-detection methods for single-cell rna sequencing data. Cell Systems, 12(2):176–194.e6, 2021. doi: 10.1016/j.cels.2020.11.008.DOI
  112. [134]Leyuan Xu, Jiankan Guo, Dennis G. Moledina, and Lloyd G. Cantley. Immune-mediated tubule atrophy promotes acute kidney injury to chronic kidney disease transition. Nature Communications, 13(1):4892, 2022. doi: 10.1038/s41467-022-32634-0.DOI
  113. [135]Wanghan Xu, Shuo Li, Tianlin Ye, et al. ResearchClawBench: A benchmark for end-to-end autonomous scientific research. arXiv preprint arXiv:2606.07591, 2026.arxiv.org/abs/2606.07591
  114. [136]Zhibo Yang, Chen Zhang, Yuewei Zhang, and Hao Wang. TruthInsightBench: An evidence-grounded benchmark for automated evaluation of open-ended scientific discovery agents. arXiv preprint arXiv:2609.05079, 2026.arxiv.org/abs/2609.05079
  115. [137]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. In ICLR, 2025.DOI
  116. [138]Vicente A. Yépez, Laura S. Kremer, Arcangela Iuso, Mirjana Gusic, Robert Kopajtich, Eliška Koňaříková, Agnieszka Nadel, Leonhard Wachutka, Holger Prokisch, and Julien Gagneur. Ocr-stats: Robust estimation and statistical testing of mitochondrial respiration activities using seahorse xf analyzer. PLOS ONE, 13(7):e0199938, 2018. doi: 10.1371/journal.pone.0199938.DOI
  117. [140]Bomin Zhang, Z. Li, V. Aguilar, P. Zhang, Mihir Pendharkar, Connor P. Dempsey, J. S. Lee, Sean David Harrington, S. Tan, J. S. Meyer, et al. Evidence of φ0-josephson junction from skewed distribution patterns in s-insb nanowires. SciPost Physics, 18(1), 2025a. doi: 10.21468/scipostphys.18.1.013.
  118. [141]Hao Zhang, Önder Gül, Sonia Conesa-BoJ, Michał P. Nowak, Michael Wimmer, Folkert K. de Vries, Jasper van Veen, Michiel W. A. de Moor, Jouri D. S. Bommer, David J. van Woerkom, et al. Ballistic superconductivity in semiconductor nanowires. Nature Communications, 8(1), 2017. doi: 10.1038/ncomms16025.DOI
  119. [142]Hao Zhang, Önder Gül, Sonia Conesa-BoJ, Michał P. Nowak, Michael Wimmer, Folkert K. De Vries, Jasper Van Veen, Michiel W. A. De Moor, et al. Data underlying "ballistic superconductivity in semiconductor nanowires", 2022.DOI
  120. [144]Jade Z. Zhang, Sierra V. Petersen, Ian Z. Winkelstern, and Kyger C. Lohmann. Seasonally variable aquifer discharge and cooler climate in bermuda during the last interglacial revealed by subannual clumped isotope analysis. Paleoceanography and Paleoclimatology, 36(6), 2021b. doi: 10.1029/2020PA004145.
  121. [146]P. Zhang, Sanchayeta Mudi, M. Pendharkar, J. S. Lee, Connor P. Dempsey, Paul McFadden, S. D. Harrington, Jason T. Dong, H. Wu, A. H. Chen, et al. Missing odd-order shapiro steps do not uniquely indicate fractional josephson effect. SciPost Physics, 18(6), 2025b. doi: 10.21468/SciPostPhys.18.6.203.
  122. [147]Liannmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NeurIPS, 2023.arxiv.org/abs/2306.05685
  123. [148]Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. ImpossibleBench: Measuring LLMs’ propensity of exploiting test cases. arXiv preprint arXiv:2510.20270, 2025.arxiv.org/abs/2510.20270
  124. [149]Junwei Zhou, Zhen Sun, Binyu Li, et al. ASI-Bench: At the dawn of artificial superintelligence. arXiv preprint arXiv:2608.17271, 2026a.arxiv.org/abs/2608.17271
  125. [150]Xuhui Zhou, Weiwei Sun, Qianou Ma, et al. Mind the Sim2Real gap in user simulation for agentic tasks. In Conference on Language Modeling (COLM), 2026b.arxiv.org/abs/2603.11245

Paper details

vlnc:W_7yczhp8w
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Code

View statement
Looking for related papers…
Contents