betaFeedback

WorksComputer Science

BenchBench-Cloning: Evaluating Molecular Cloning Design and Protocol Development

Abstract

Molecular cloning is one of the most routine wet-lab workflows, yet existing evaluations of language models in biology mostly test knowledge recall or isolated steps rather than end-to-end experimental planning. We introduce BenchBench-Cloning, a benchmark for large language models comprising 65 molecular cloning tasks grounded in sequence records, laboratory notebooks, and published experimental studies. Planning a molecular cloning experiment requires defining a set of constructs needed for a scientific objective and a construction route to produce them. BenchBench-Cloning contains 50 protocol-development tasks, in which agents produce an executable protocol for a specified construct, and 15 construct-design tasks, in which agents determine the constructs a scientific objective calls for. Protocol-development tasks are evaluated by execution in a cloning simulator; construct-design tasks are evaluated with task-specific deterministic checks and, where needed, rubric-based LLM-as-a-judge evaluation. Across eight systems with repeated attempts per task, GPT-6 Astra scores highest at 30.6%, with the others between 8.6% and 22.2%, and the benchmark remains unsaturated under best-of-ten sampling. As integrating individual cloning decisions into complete, valid plans remains challenging even for frontier models, we expect BenchBench-Cloning to serve as a useful benchmark for tracking progress in experimental planning.

Introduction

Designing and constructing plasmids is a common task in molecular biology. A scientist must determine the required sequence, identify the requisite starting materials, choose a construction strategy, and specify the reactions needed to produce the desired product. The choice of assembly method depends on the starting materials, and the products of each reaction determine which steps can follow.

Language models are commonly used to assist with individual steps of this process, including sequence analysis, primer design, and selection of cloning methods. Evaluating capabilities separately, however, leaves open whether models can successfully plan and execute a full experiment. For example, a proposed sequence may satisfy the requested design while an accompanying protocol may fail to produce it. Similarly, executable protocols may fail to address the scientist’s objective. This planning process is time-consuming even for experienced scientists, with human errors frequent and surfacing only after several days (or weeks) of work. A model that can reliably reason through entire cloning workflows stands to provide enormous scientific and commercial benefit for scientists.

To measure progress toward this goal, we introduce BenchBench-Cloning, a benchmark of 65 molecular cloning tasks designed to evaluate a model’s performance on correct and efficient cloning strategies. The protocol-development set contains 50 tasks where scientists specify the desired construct and the agent develops an executable protocol from the supplied inventory, and the construct-design set contains 15 tasks where the agent must design a construct that satisfies a scientific objective based on experimental motivations.

We contribute:

  • BenchBench-Cloning: a benchmark of 65 tasks testing AI capability in reasoning through complex cloning tasks.

  • Baseline results for current frontier models, indicating that current models struggle to develop complete, valid cloning plans.

We describe the design and construction of the benchmark, present results from evaluating current models against it, and discuss what these results suggest about the current state and future trajectory of AI-assisted experimental planning.

Related Work

Life-science benchmarks increasingly evaluate tasks that resemble parts of a scientist’s workflow. LAB-Bench and LABBench2 cover literature use, data interpretation, and sequence manipulation [1, 2]. LifeSciBench evaluates expert-authored research tasks using detailed rubrics across multiple scientific workflows [3]. These evaluations broaden the assessment of scientific assistance beyond factual recall to evidence interpretation and conclusion formulation. BenchBench-Protocol evaluates modifications to published laboratory protocols using tasks recovered from changes scientists made to experimental work [5].

Agentic benchmarks such as BixBench [4] and Terminal-Bench-Science [6] evaluate models on multistep computational analyses in biology and other scientific domains. LABBench2 CloningQA is the closest prior work, with 14 tasks requiring complete cloning protocols including methods like DNA and enzyme reagents, restriction-ligation cloning, Gibson assembly, and Golden Gate assembly. Submissions use a domain-specific language and are evaluated using component checks and end-to-end in silico validation [2]. BenchBench-Cloning expands this formulation to 50 protocol-development tasks in an environment supporting 42 operations. Our verification requirements include method-specific primer constraints, molecular-state and end-chemistry checks, and the ability to produce and isolate multiple products.

Benchmark Design and Construction

BenchBench-Cloning contains 65 tasks divided into two sets: 50 protocol-development tasks and 15 construct-design tasks. In protocol-development tasks, the desired construct is specified and the agent must determine how to produce it from the available materials. In construct-design, the scientific objective is specified and the agent must determine appropriate construct(s), specify their exact sequences, and provide a construction plan.

Both sets of tasks provided a fixed request and supporting files. Agents inspect these files, use analysis tools, and revise their work during an attempt. The two task sets differ in their inputs, outputs, and evaluation procedure Figure 1.

Figure 1. The two task formats. In protocol development (A) the agent writes a protocol that the simulator executes; in construct design (B) the agent returns constructs and a plan that deterministic checks and, on ten tasks, a model judge score.

Protocol-Development Tasks

Each protocol-development task provides a scientist’s request describing the desired construct and an inventory of starting materials. Products are circular plasmids between 2.8kb and 15.2kb long, with a median length of 5.4kb. All 50 protocol-development tasks are derived from laboratory notebooks created by contributors on Benchling, drawing on real construct requests and experimental context.

An inventory containing up to eight starting plasmid constructs is supplied as Genbank or FASTA files with accompanying sequence annotations. Notes of material properties (such as methylation sensitivity at restriction sites, GC-rich regions, end chemistry, acceptor vectors with ccdB counterselection cassettes, host strain) are also provided. The task set spans nine experimental contexts, each defined by the dominant constraint the task imposes (Table 1).

Dominant constraintTasksShareReasoning the task requires
Methylation state (unknown host / Dam-Dcm sites)1122%Starting plasmid grown in an unknown or mixed host, or a required site overlapping a Dam/Dcm motif; establish a known methylation state or avoid the blocked site before cutting
Homology-directed and seamless assembly714%Gibson, In-Fusion, in vivo assembly or recombineering with homology arms near the minimum length; in vitro transcription templates needing an oriented promoter
Repeats and tandem arrays714%Repeated elements (tandem tags, repeated promoters or terminators, homopolymer tracts) where a primer or homology arm binds in more than one place or where a homopolymer run makes length unstable through slippage
Backbone end chemistry and orientation612%Self-compatible vector ends that re-circularize or dimerize unless dephosphorylated; ends that must be filled or blunted; inserts whose blunt or palindromic ends give both orientations
Golden Gate / MoClo510%Assemblies with internal Type IIS sites to domesticate and overhang sets that must stay unique
GC-rich templates48%A window over the disclosed GC limit so PCR through it fails; the fragment must be released or built another way
Recombination, CRISPR and viral vectors48%Gateway and Cre/lox recombination, CRISPR guide and donor plasmids, lentiviral and AAV transfer plasmids
Counterselection48%An acceptor or destination vector carrying a cassette that the product must lose
Expression vectors and tag exchange24%Bacterial expression vectors in which an affinity or solubility tag is exchanged while the reading frame and protease site are preserved
Total50100%

Table 1. Protocol-development task families. Task counts and shares are shown for each dominant constraint with the reasoning it requires.

The environment includes documentation for a custom cloning protocol language supporting 42 operations. These cover amplification, partial and complete digestion, Gibson and Golden Gate assembly, oligonucleotide annealing and ligation, end modification, CRISPR-based operations, and other relevant techniques. The environment also includes tools such as Python, Biopython, Primer3-py, and plotting libraries for inspection and design.

During evaluation, the agent receives the request, inventory, operation documentation, and analysis tools. It submits a protocol that is then passed through the simulator and assigned a binary score. A protocol passes if it successfully executes and ends with one isolated product that matches the hidden target sequence. Invalid operations, incorrect sequences, or unresolved mixtures result in failure. Alternative construction routes are accepted when they pass through the simulator and produce the same final product as the reference protocol.

Construct-Design Tasks

The construct-design set contains 15 tasks derived from published scientific studies.1 Each task provides a scientist’s objective alongside sequence records and supporting materials. Six tasks additionally provide experimental records such as sequencing reads or gel images. Agents return an annotated construct sequence, orderable primers when required, a construct plan, and a written justification.

These tasks require the agent to determine which constructs are needed to address a scientific objective. Depending on the task, a successful response may require identifying a relevant sequence feature, applying a repair across related constructs, or including controls needed to interpret an experiment.

Each task has a deterministic grader that checks the returned sequences, their source, and the construction route. Additionally, 10 tasks use a rubric-based LLM-as-a-judge to assess the written reasoning and construction plan. For these tasks, the final score is the product of the deterministic grader and the rubric score. Task sources are listed in Appendix § 8.

Model Setup

We evaluate eight systems using Claude Opus 5.5, Claude Mythos 5.1, GPT-6 Astra, GPT-6.1 Sol, Gemini 3.8 Flash, Gemini 4 Argon, Grok 4.7, and Kimi K3. Models were tested in their native agent harnesses where available, and Kimi K3 was tested using the Pi harness. Each system completed ten individual attempts per protocol-development task and five per construct-design task. Appendix § 9 shows the specific settings of each run.

Runs were orchestrated using Harbor in Modal sandboxes with two vCPUs and 6 GiB of memory. Network access was restricted to calls to the model provider. Each agent received the task instructions and supporting files, used the available tools, and wrote its submission to a designated answer file. Attempts were limited to two hours. GPT-6 Astra served as the model judge for the ten construct-design tasks requiring rubric-based evaluation. We also evaluated the tasks with Claude Mythos 5.1 and report agreement metrics in Appendix § 10.

Evaluation Metrics

For the protocol-development tasks, we report pass rate, defined as the proportion of attempts whose submitted protocol executes successfully and produces the isolated target sequence. We average the ten attempt outcomes within each task and average across the fifty tasks. We estimate 95% confidence intervals using 20,000 bootstrap resamples of the tasks, retaining all attempts associated with each sampled task. For construct-design tasks, an attempt is classified as successful when its final score is at least 0.9. The pass rate is computed by averaging the five attempt outcomes within each task and then averaging across the 15 tasks. The aggregate score is a weighted average of the protocol-development and construct-design scores, with weights of 50/65 and 15/65, respectively, giving each task equal weight.

For protocol-development tasks, we additionally report pass@kk for up to ten attempts to measure the probability that at least one of kk attempts succeeds. This measures the availability of a valid protocol among repeated attempts.

Results

Performance on BenchBench-Cloning varies substantially across systems. GPT-6 Astra is the strongest overall system, with an aggregate score of 30.6%. Scores for the other evaluated systems range from 8.6% to 22.2% (Figure 2).

Figure 2. Aggregate score by system. Task-weighted means at top effort are shown with 95% confidence intervals.

Performance by Task Set

On the protocol-development set, GPT-6 Astra achieves the highest pass rate at 27.0% (95% CI 19.8–34.6%), followed by Claude Opus 5.5 at 19.0% and GPT-6.1 Sol at 16.4% (Figure 3). Grok 4.7, Kimi K3, and Gemini 3.8 Flash score 6.2%, 5.4%, and 4.0%, respectively (Table 2).

SystemEffortProtocol dev. %Construct design %Aggregate %
GPT-6 Astramax27.0 [19.8, 34.6]42.7 [22.7, 64.0]30.6 [23.2, 38.3]
GPT-6.1 Solmax16.4 [11.4, 22.2]41.3 [22.7, 61.3]22.2 [16.3, 28.5]
Claude Opus 5.5max19.0 [12.0, 27.0]25.3 [9.3, 44.0]20.5 [13.7, 27.8]
Gemini 4 Argonhigh15.2 [8.4, 23.2]30.7 [10.7, 52.0]18.8 [11.5, 26.8]
Claude Mythos 5.1max15.6 [9.6, 22.6]17.3 [5.3, 30.7]16.0 [10.5, 22.2]
Grok 4.7xhigh6.2 [3.4, 9.2]26.7 [9.3, 46.7]10.9 [6.3, 16.2]
Kimi K3high5.4 [2.8, 8.4]20.0 [6.7, 36.0]8.8 [4.9, 13.2]
Gemini 3.8 Flashhigh4.0 [1.0, 8.4]24.0 [8.0, 42.7]8.6 [4.0, 14.0]

Table 2. Performance by task set at top effort. Pass rates in percent are shown with 95% confidence intervals.

Figure 3. Pass rate by task set. Means at top effort are shown with 95% confidence intervals for protocol development (A) and construct design (B).

All evaluated systems produce successful protocols, but no system consistently completes the protocol-development tasks. Analysis of submitted designs shows that unsuccessful attempts can contain substantial correct work, with a small number of decisions preventing completion.

Performance Under Repeated Sampling

Because a scientist can compare several candidate answers before acting on one, expected best-of-kk performance is reported as an upper bound on single-system performance. System performance ranking is not stable in kk (Figure 4). Opus performs better than Sol on individual attempts, but Sol performs better at k=10k = 10. Opus’s successful attempts are concentrated on fewer tasks, while Sol succeeds less consistently across a broader set of tasks.

Figure 4. Protocol-development pass@kk. Probability that at least one of kk attempts passes is shown with 95% confidence bands for the three leading systems.

At ten attempts, Astra solves 36 of the 50 protocol-development tasks at least once, corresponding to a pass@10 of 72% (Table G1). Sol solves 33 tasks and Opus solves 29, corresponding to 66% and 58%, respectively. The strongest system still leaves 14 tasks unsolved, indicating that the benchmark is not saturated within ten attempts.

Systempass@11 % [95% CI]pass@55 % [95% CI]pass@1010 % [95% CI]Solved ≥1Solved allNote
GPT-6 Astra27.0 [19.8, 34.6]59.9 [48.3, 71.0]72.0 [60.0, 84.0]362
GPT-6.1 Sol16.4 [11.4, 22.2]48.3 [37.7, 58.8]66.0 [52.0, 78.0]330
Claude Opus 5.519.0 [12.0, 27.0]44.6 [33.4, 56.0]58.0 [44.0, 72.0]293
Gemini 4 Argon15.2 [8.4, 23.2]32.4 [21.4, 44.0]42.0 [28.0, 56.0]212
Claude Mythos 5.115.6 [9.6, 22.6]42.5 [32.1, 52.9]60.0 [46.0, 74.0]302
Grok 4.76.2 [3.4, 9.2]22.9 [13.8, 32.6]34.0 [22.0, 48.0]170k=9k = 9 cap (2 tasks)
Kimi K35.4 [2.8, 8.4]20.3 [12.1, 29.4]32.0 [20.0, 46.0]160
Gemini 3.8 Flash4.0 [1.0, 8.4]10.5 [4.0, 18.1]16.0 [6.0, 26.0]80k=9k = 9 cap (4 tasks)

Table G1. Table G1: Protocol-development pass@kk and task coverage at top effort. Pass rates in percent with 95% confidence intervals are shown at k=1k = 1, 5, and 10 with the number of tasks solved at least once and in every attempt.

Reasoning Effort and Cost

Average cost per attempt varies by more than an order of magnitude across systems at comparable performance (Figure 5). The pareto frontier identifies the system that offers the best observed trade-off between performance and cost. Token counts for each run are reported in Appendix § 11.

Figure 5. Aggregate score against cost per attempt. Each system is shown at its three reasoning settings with the top setting filled; the dashed line is the cost frontier.

Analysis of Model Behavior

Successful Work

Systems perform well on several tasks where the expected molecular outcome is spelled out clearly. Six of the fifteen construct-design tasks were passed in every attempt by at least two systems, and four systems passed the minimal-edit recoding task every time (Figure 6; Table G2). Many failed submissions also contained correct work. In the sodium-channel task, every scored attempt placed both introns at the junctions and kept the full channel protein intact, even though none of them passed the task.

Figure 6. Construct-design solves by task and system. Cells give the number of five attempts that passed at top effort.

TaskGPT-6 AstraGPT-6.1 SolClaude Opus 5.5Gemini 4 ArgonClaude Mythos 5.1Grok 4.7Kimi K3Gemini 3.8 Flash
Matched promoter panel0/50/55/55/54/55/53/53/5
GFP into EcoFlex vector5/55/53/53/51/53/51/50/5
Insulin intermediate construct5/54/50/51/50/54/52/55/5
Minimal-edit recoding5/55/51/50/50/55/55/50/5
Reporter choice from data3/52/55/50/53/50/53/55/5
Insulin two-chain constructs5/55/50/55/50/52/51/52/5
MPRA construct rebuild3/51/53/55/53/50/50/52/5
AG/AP1 chimera plasmid2/52/51/54/51/51/50/51/5
Detector choice from antibody data3/54/50/50/50/50/50/50/5
Alkaloid-pathway GoldenBraid set1/51/50/50/50/50/50/50/5
Tus/TerB HEK293 constructs0/51/50/50/51/50/50/50/5
Prime-assembly donor family0/51/50/50/50/50/50/50/5
Synthetic-chromosome locus edit0/50/51/50/50/50/50/50/5
Sodium-channel promoter repair0/50/50/50/50/50/50/50/5
Dual-luciferase frame panel0/50/50/50/50/50/50/50/5

Table G2. Table G2: Construct-design solves per task and system at top effort. Each cell gives the number of five attempts that passed.

The dual-luciferase task makes this especially clear. Across the eight systems, 38 of the 39 submitted panels had correctly built reference, frame 0, frame +1 and frame +2 constructs, and every construction step in those panels ran as written. None of them included the fifth construct, a control with the Firefly reading frame closed by a stop codon, which the experiment needs in order to separate readthrough from re-initiation (Figure 7). Several systems even mentioned re-initiation in their written answers, but they treated it as background shared by all constructs rather than something the panel had to measure.

Figure 7. The dual-luciferase frame-panel task. The request, the five reference constructs, and the number of the 39 submitted panels containing each construct are shown.

Taken together, these results suggest that agents can usually carry out the individual sequence edits a task requires. Where they fall short is in reasoning from the experimental question to the set of constructs required to exclude alternative explanations.

Protocol-Development Failure Modes

Analysis of protocol failures identifies recurring errors in primer design, backbone preparation, and methylation state. These errors often arise in systems that correctly identify the intended sequence changes but do not maintain compatibility between the materials and operations used to produce them.

In several cases, we found that systems would frequently select primers which target the correct sequence while remaining unsuitable for the proposed reaction conditions. Primer failures frequently involve pairs that cannot be used together under the simulated annealing conditions. For example, primer pairs were found to dimerize, or to have annealing points which were too far apart to be feasibly used in a reaction together. In these cases, while the system succeeded in designing single appropriate primers, it failed to evaluate whether the primer could be appropriately used as a pair. We also identified primer design failures involving secondary structure issues (e.g. hairpin loops), length limits, or homopolymer runs.

Backbone preparation and methylation produced additional failures. Systems ligated vectors with self-compatible ends without dephosphorylating them, leading to backbones recircularizing or dimerizing. In other cases, systems treated compatible or blunt ends as directional, carrying inserts forward in one orientation when the reaction yields both, or ignored methylation-sensitive sites in plasmids from a Dam/Dcm-methylating host, leaving the acceptor circular. These are mistakes a molecular biologist would ordinarily anticipate and prevent when designing the protocol. Each attempt is classified by the first error that blocks execution, reported by system in Appendix § 12.

Construct-Design Failure Modes

Most construct-design failures were incomplete designs. Systems usually made the change that was asked for, but they often missed a second change that the experiment needed. The sodium-channel task is a good example (Table G2). Two related expression constructs each carry a cryptic bacterial promoter that has to be removed. GPT-6 Astra and GPT-6.1 Sol fixed the first construct in all of their attempts, yet no system ever fixed the second one. The dual-luciferase task shows a similar pattern. Nearly all submitted panels had correct reference and frame-shifted reporters, but none of them included the closed-frame control.

Some designs went further than the task asked. The matched-panel task asks for the smallest set of constructs that can resolve a single discrepancy, and the grader checks for this. Astra and Sol built an eight-construct factorial panel in every attempt and so failed the check. Most of the other systems submitted the four-construct panel and passed.

When the grader also checked the construction route, many failures came from the construction record and not from the sequences themselves. We saw source references that could not be resolved, the same identifier used for different sequences, missing oligonucleotide tables, and fragments that no listed reaction actually produces. The designed sequences in these submissions were sometimes correct, but the grader had no way to confirm that the plan could build them from the materials provided.

Ten of the tasks also use a model judge, so passing the deterministic checks does not guarantee a pass. The judge scores the written plan, and an attempt needs at least 0.9 to pass. At the top effort setting, 34 of the 142 attempts that cleared the deterministic checks fell below this threshold (Table F3). Most often the judge’s main concern was that the plan did not lay out an orderable path from the supplied sources to the final product; in two cases it was a misreading of the experimental evidence. We list the reviewed submissions in Appendix § 13.

SystemAttempts on judged tasksPassed deterministic checksJudge score below 0.9
GPT-6 Astra50282
GPT-6.1 Sol50346
Claude Opus 5.55093
Gemini 4 Argon50218
Claude Mythos 5.15063
Grok 4.750183
Kimi K350156
Gemini 3.8 Flash50113
Total40014234

Table F3. Judge threshold on the ten rubric-judged tasks at top effort. Attempts that passed the deterministic checks and, of those, the number the judge scored below 0.9 are shown per system.

Discussion

Implications and Future Work

BenchBench-Cloning shows that frontier models can simulate many of the individual operations required by molecular cloning, but they do not yet reliably assemble those operations into a complete, executable experimental design. The strongest system, GPT-6 Astra, reaches an aggregate score of 30.6% and a protocol-development pass rate of 27.0%, and even at ten attempts it leaves 14 of 50 protocol-development tasks unsolved. However, it comes with significant costs; under the pareto frontier, it is observed that mid-range systems like GPT-6.1 Sol can achieve similar results at lower cost. Notably, while repeated sampling produces a greater likelihood of valid solutions, its practical value depends on identifying those solutions. Comparing individual attempts with multiple candidates and a selection procedure under the same inference budget would help determine how additional computation is best used.

Failures are rarely a matter of missing basic knowledge. Unsuccessful submissions are often the result of flawed critical thinking and understanding of the deeper underlying motivations, such as an omitted control. The failures are not at any single subproblem, but rather in the ability to maintain context across longer horizon tasks, such as understanding that materials have constraints, or that a primer pair cannot work together under the stated reaction conditions.

These findings bear on how such systems are used today. A failing submission typically reads as a complete and well-reasoned plan, with correct sequence edits, plausible primers, and a confident justification, and any mistakes are only visible to someone who already knows what the experiment demands. Current systems are therefore convincing enough to mislead the users most likely to rely on them. The near-miss character of these failures also points to the fix, with blocking errors being checks an experienced molecular biologist makes as a matter of course that a system should make on its own plan before presenting it. Until systems do so reliably, expert review of model-generated protocols remains necessary; the benchmark measures progress toward systems that reliably and correctly plan complete cloning workflows. Further analysis can assess whether explicit self-verification and candidate selection improve complete-task success.

For benchmark design, our results also show that grading choices matter: of 142 submissions that passed the deterministic gate, 34 fell below the pass threshold on the model judge, most often because the plan lacked an orderable construction route from the supplied sources. Correct sequences thus do not guarantee a usable plan, and some failures reflect incomplete documentation rather than flawed biology.

Limitations

While BenchBench-Cloning was designed as a single-turn benchmark evaluating whether a model could respond correctly to a fixed scientist request, this likely does not reflect how a scientist would collaborate with a model. Scientists often use LLMs in a multi-turn capacity, which would allow models to ask clarifying questions about available materials, revise plans after feedback, respond to experimental results obtained during construction, and emphasize practical limitations on suggested protocols. Future work can focus on a format that more closely reflects this reality.

Further, while BenchBench-Cloning measures planning for molecular cloning, it does not directly measure whether models improve experimental outcomes at the bench. The simulator cannot capture every source of variation in reagents, handling, growth, or reaction outcomes. As LLMs move into the physical world, future iterations of BenchBench-Cloning could involve actually conducting experiments designed by the models to test out the protocols designed.

Another limitation in analysis of traces is that our simulator inherently reports the first blocking error, so these diagnostics do not necessarily establish whether later steps would have succeeded. Failure frequencies also depend on the routes models choose. Models using PCR more frequently encounter primer-related failures, which may have skewed the occurrences of primer design failures amongst the reasoning traces.

Finally, a limitation of our construct-design tasks was the use of LLM-as-a-judge to assess written reasoning and construction plans. Because this kind of model grading is inherently stochastic, there were occasional inconsistencies in grading results. Notably, agreement between graders does not establish independent scientific validity.

Conclusion

BenchBench-Cloning measures whether models can determine the constructs addressing a scientific objective and plan a construction route from the available materials. GPT-6 Astra scores highest at 30.6%, with other systems’ aggregate scores ranging from 8.6% to 22.2%. Repeated sampling improves protocol-development performance, but the strongest system still leaves 14 out of 50 tasks unsolved after ten attempts.

A closer look at successes and failures identified trends. Systems demonstrate surprisingly skilled sequence work and successful construction planning on part of the benchmark. However, their failures frequently concern compatibility between operations, incomplete construct plans, or lack of consideration towards any changes needed for a broader scientific objective. These failures matter in practice because the scientists most likely to rely on a system for cloning judgment are those with the least bench experience, and the least able to catch its errors. Evaluating molecular cloning assistance therefore requires checking both whether a design addresses the experiment and whether the proposed operations can produce it. Closing the gap between individual cloning decisions and a complete experimental plan is the next step toward systems that scientists can rely on, and BenchBench-Cloning offers a way to measure progress toward it.

Construct-Design Tasks and Sources

Each construct-design task gives the agent a scientist’s objective, sequence records and supporting materials. Six tasks also supply experimental records from the source study. Every task has a deterministic grader that checks the returned sequences, their sources and the construction route; ten tasks also use a rubric-based model judge to assess the written reasoning and construction plan, and for these the final score is the product of the deterministic result and the rubric score. An attempt passes at a final score of at least 0.9. Table A1 lists the tasks; Figure 1 and Figure 7 show the dual-luciferase task in detail.

#TaskSourceExperimental records suppliedGrading
1MPRA construct rebuildDao et al. 2025, Nat. Commun.Sequencing reads from the plasmid pool and cellular RNADeterministic
2Matched promoter panelPicó et al. 2026, Nat. Commun.Per-well growth and GFP measurements, part mapDeterministic
3Sodium-channel promoter repairJ. Biol. Chem. 2021, cryptic prokaryotic promoters in neuronal sodium channelsGel and fragment-stability images, sequence changes, start-site and electrophysiology dataDeterministic
4Detector choice from antibody dataPLoS ONE 2021, CD55 C81A reporterAntibody and localisation images, oligonucleotide and genotype tablesDeterministic + rubric judge
5Reporter choice from dataGhodke et al. 2016, Biophys. J.Published microscopy and calibration figure panelsDeterministic
6Dual-luciferase frame panelAnnibaldis et al. 2020, Nucleic Acids Res.Published readthrough measurements (three panels)Deterministic
7AG/AP1 chimera plasmidLai et al. 2021, Nat. Commun.Source paper and supplementDeterministic + rubric judge
8Synthetic-chromosome locus editGoold et al. 2025, Nat. Commun.None (genome and design records)Deterministic + rubric judge
9Alkaloid-pathway GoldenBraid setGrzech et al. 2023, ACS Synth. Biol.Source paper and supplementDeterministic + rubric judge
10GFP into EcoFlex vectorEcoFlex kit (Addgene)NoneDeterministic + rubric judge
11Prime-assembly donor familyLiu et al. 2025, bioRxivSource preprint and supplementDeterministic + rubric judge
12Tus/TerB HEK293 constructsAddgene plasmids and genomic recordsNoneDeterministic + rubric judge
13Minimal-edit recodingAuthored sequenceNoneDeterministic + rubric judge
14Insulin two-chain constructsHuman INS and pMAL-c6T recordsNoneDeterministic + rubric judge
15Insulin intermediate constructHuman INS and pMAL-c6T recordsNoneDeterministic + rubric judge

Table A1. Construct-design tasks. Source, experimental records supplied, and grading route are listed for each task.

Model and Harness Settings

All attempts ran through Harbor in Modal sandboxes (2 vCPU, 6 GiB) with network access limited to the model provider. Each system completed 10 attempts per protocol-development task and 5 attempts per construct-design task at every reasoning setting.

SystemModel idHarness (version)Access routeReasoning settings
GPT-6 Astragpt-6-astraCodex CLI 0.156.1OpenAI APIlow / high / max
GPT-6.1 Solgpt-6.1-solCodex CLI 0.156.1OpenAI APIlow / high / max
Claude Opus 5.5claude-opus-5-5Claude Code 2.1.289Anthropic API (WIF)low / high / max
Gemini 4 Argongemini-4-argonGemini CLI 0.63.0, in-sandbox proxyVertex AI (global endpoint)low / medium / high
Claude Mythos 5.1claude-mythos-5-1Claude Code 2.1.289Anthropic API (WIF)low / high / max
Grok 4.7grok-4.7Grok Build 1.0.46xAI APIlow / high / xhigh
Kimi K3moonshotai/Kimi-K3Pi 1.0.3Basetenlow / medium / high
Gemini 3.8 Flashgemini-3.8-flashGemini CLI 0.63.0Vertex AI (API key)low / medium / high

Table B1. Table B1: Model and harness configurations. Model identifier, harness, access route, and reasoning settings are listed per system.

Judge Agreement

To check the rubric-based outcomes, every construct-design attempt from the first six systems evaluated was graded a second time by Claude Mythos 5.1 acting as an independent judge. The judge received the public task text, the private reference material and the submission files, and returned a score on the same 0 to 1 scale; agreement is measured on the binary pass outcome.

SliceTrials gradedAgreementCohen’s kappa
All trials38789.7%0.76
Tasks with experimental records (6)13799.3%0.98
Construction tasks (9)25084.4%0.65
GPT-6 Astra6285.5%0.72
GPT-6.1 Sol6384.1%0.69
Claude Opus 5.57485.1%0.67
Kimi K36296.8%0.90
Gemini 3.8 Flash6395.2%0.80
Grok 4.76392.1%0.79

Table C1. Table C1: Agreement between the Claude Mythos 5.1 judge and reported outcomes. Trials graded, agreement, and Cohen’s kappa are shown per slice.

Agreement between Claude Mythos 5.1, used as an independent judge, and the outcomes reported in the paper (pass = final score of at least 0.9 under both). The comparison covers the 450 construct-design attempts of the first six systems evaluated (15 tasks, 5 attempts each). The Mythos judge saw the public task text, the private reference material and the submission files, but not the reported outcome. It declined to grade 63 attempts under its own safety policy; those are excluded.

Token Usage and Cost

SystemEffortInput (k)Cached (k)Output (k)Cost/attempt ()∣Medianmin∣Caphits∣Designcost() | Median min | Cap hits | Design cost ()
GPT-6 Astralow14211830.571.101.20
GPT-6 Astrahigh346307111.353.802.34
GPT-6 Astramax428376212.087.903.72
GPT-6.1 Sollow19517140.121.400.21
GPT-6.1 Solhigh392354140.274.500.46
GPT-6.1 Solmax499444280.4614.0130.77
Claude Opus 5.5low24322580.291.400.66
Claude Opus 5.5high541499230.774.001.98
Claude Opus 5.5max3,5613,3211394.6924.2185.06
Gemini 4 Argonlow5,0764,620392.216.513.22
Gemini 4 Argonmedium4,1843,760742.5310.622.35
Gemini 4 Argonhigh4,1503,640802.7311.112.45
Claude Mythos 5.1low299269141.132.902.60
Claude Mythos 5.1high597529372.827.104.11
Claude Mythos 5.1max1,2241,050967.2318.837.23
Grok 4.7low2,6512,432541.9814.801.59
Grok 4.7high4,6584,2511654.0042.3153.47
Grok 4.7xhigh6,1845,5981995.3043.3304.20
Kimi K3low239185210.544.020.45
Kimi K3medium691587621.4210.242.09
Kimi K3high682561641.4912.821.39
Gemini 3.8 Flashlow10,95910,240801.6117.7371.71
Gemini 3.8 Flashmedium7,1486,6101091.3117.5131.52
Gemini 3.8 Flashhigh6,8626,3131071.2917.8231.68

Table D1. Token usage, cost, and runtime per protocol-development attempt. Mean tokens, cost, median minutes, and time-cap hits are shown per lane with the construct-design cost per attempt.

SystemTierEffortProtocol dev. % [95% CI]Construct design % [95% CI]Aggregate % [95% CI]Proto ()∣Const() | Const ()
GPT-6 Astralowlow16.8 [10.2, 24.0]28.0 [13.3, 45.3]19.4 [13.2, 26.2]0.571.20
GPT-6 Astramidhigh21.2 [14.8, 28.6]44.0 [22.7, 66.7]26.5 [19.4, 34.0]1.352.34
GPT-6 Astratopmax27.0 [19.8, 34.6]42.7 [22.7, 64.0]30.6 [23.2, 38.3]2.083.72
GPT-6.1 Sollowlow15.4 [9.0, 22.6]22.7 [9.3, 38.7]17.1 [11.2, 23.5]0.120.21
GPT-6.1 Solmidhigh17.0 [11.4, 23.4]37.3 [20.0, 56.0]21.7 [15.7, 28.0]0.270.46
GPT-6.1 Soltopmax16.4 [11.4, 22.2]41.3 [22.7, 61.3]22.2 [16.3, 28.5]0.460.77
Claude Opus 5.5lowlow7.2 [3.8, 11.2]29.3 [13.3, 46.7]12.3 [7.7, 17.2]0.290.66
Claude Opus 5.5midhigh15.6 [10.4, 21.6]34.7 [17.3, 53.3]20.0 [14.2, 26.3]0.771.98
Claude Opus 5.5topmax19.0 [12.0, 27.0]25.3 [9.3, 44.0]20.5 [13.7, 27.8]4.695.06
Gemini 4 Argonlowlow11.0 [7.4, 15.0]31.9 [10.7, 52.0]15.8 [10.4, 21.7]2.213.22
Gemini 4 Argonmidmedium14.2 [9.0, 20.0]33.3 [13.3, 54.7]18.6 [12.3, 25.4]2.532.35
Gemini 4 Argontophigh15.2 [8.4, 23.2]30.7 [10.7, 52.0]18.8 [11.5, 26.8]2.732.45
Claude Mythos 5.1lowlow9.0 [5.0, 13.8]16.0 [4.0, 29.3]10.6 [6.3, 15.4]1.132.60
Claude Mythos 5.1midhigh13.0 [7.6, 19.2]24.0 [8.0, 42.7]15.5 [9.8, 21.8]2.824.11
Claude Mythos 5.1topmax15.6 [9.6, 22.6]17.3 [5.3, 30.7]16.0 [10.5, 22.2]7.237.23
Grok 4.7lowlow1.4 [0.4, 2.8]10.7 [2.7, 21.3]3.5 [1.4, 6.2]1.981.59
Grok 4.7midhigh4.0 [2.0, 6.4]22.7 [6.7, 41.3]8.3 [4.3, 12.9]4.003.47
Grok 4.7topxhigh6.2 [3.4, 9.2]26.7 [9.3, 46.7]10.9 [6.3, 16.2]5.304.20
Kimi K3lowlow3.2 [1.2, 6.0]16.0 [2.7, 33.3]6.2 [2.5, 10.6]0.540.45
Kimi K3midmedium5.6 [3.2, 8.4]29.3 [12.0, 48.0]11.1 [6.6, 15.8]1.422.09
Kimi K3tophigh5.4 [2.8, 8.4]20.0 [6.7, 36.0]8.8 [4.9, 13.2]1.491.39
Gemini 3.8 Flashlowlow7.2 [3.2, 12.2]22.7 [5.3, 42.7]10.8 [5.7, 16.6]1.611.71
Gemini 3.8 Flashmidmedium5.2 [2.0, 9.0]22.7 [5.3, 42.7]9.2 [4.5, 14.6]1.311.52
Gemini 3.8 Flashtophigh4.0 [1.0, 8.4]24.0 [8.0, 42.7]8.6 [4.0, 14.0]1.291.68

Table D2. Complete performance metrics across evaluated lanes. Pass rates in percent with 95% confidence intervals and cost per attempt are shown for every system and reasoning setting.

Failure Categories and Attempt Outcomes

Figure E1. First blocking failure of protocol-development attempts at top effort. Shares of graded attempts are shown per system, with counts in Table E1.

SystemGradedPassedPrimer designFinal tube not single productBench step failedWrong sequenceParser rejectedNo protocol
GPT-6 Astra5001352469295130
GPT-6.1 Sol500822988223753
Claude Opus 5.5500952799420921
Gemini 4 Argon50076210907014103
Claude Mythos 5.150078323860346
Grok 4.749831347342303330
Kimi K35002738137324163
Gemini 3.8 Flash49620347791551020

Table E1. First blocking failures by system for protocol-development tasks at top effort.

SystemRecordedGradedNo protocolTime capRefusedParser rejectedUnscored
GPT-6 Astra500500000130
GPT-6.1 Sol500500313050
Claude Opus 5.55005002118090
Gemini 4 Argon5005001031104140
Claude Mythos 5.150050063040
Grok 4.7548498303003350
Kimi K3501500320161
Gemini 3.8 Flash50049620230104

Table E2. Breakdown of execution and grading statuses for protocol-development attempts.

Reviewed Construct-Design Submissions

Adjudication outcomes are summarized in Table F1, and every reviewed attempt at the top reasoning setting is listed in Table F2. Table F3 gives, for the ten judged tasks, how many top-effort attempts passed the deterministic checks and how many of those the judge scored below 0.9. Review decides only whether a submission passes the deterministic checks. On the ten judged tasks the final score also includes the rubric judge, so a reviewed attempt can pass the deterministic checks and still fall below the 0.9 threshold; five top-effort attempts did (four by Gemini 4 Argon and one by Gemini 3.8 Flash), and they are counted as failures in Table 2 and Table G2.

Adjudication outcomeTop-tier trials (8×75)All 24 lanes (24×75)
Shipped grader result stands5521626
A: passes after value-preserving re-expression of the record1743
A′: engine source checks verify product and route as written57
B: passes after manual verification of the engine evidence526
0: still fails after allowed re-expression2195
Unscored (harness failure)03
Total6001800

Table F1. Adjudication outcomes for construct-design submissions. Counts are shown for the eight top-effort lanes and for all 24 lanes.

SystemTaskDeterministicFinalBasis
Claude Mythos 5.1AG/AP1 chimera plasmidpasspasspasses after the record is re-expressed without changing any value
Claude Opus 5.5AG/AP1 chimera plasmidpasspasspasses after the record is re-expressed without changing any value
Gemini 4 ArgonAG/AP1 chimera plasmidpasspasspasses on manual check of the grader’s evidence
Gemini 4 ArgonAG/AP1 chimera plasmidpasspasspasses on manual check of the grader’s evidence
Gemini 4 ArgonAG/AP1 chimera plasmidfailfailstill fails after allowed re-expression
Gemini 4 ArgonAG/AP1 chimera plasmidpasspasspasses on manual check of the grader’s evidence
Gemini 4 ArgonAG/AP1 chimera plasmidpasspasspasses on manual check of the grader’s evidence
GPT-6.1 SolAlkaloid-pathway GoldenBraid setpasspassgrader’s own source checks verify the product and route as written
Gemini 4 ArgonAlkaloid-pathway GoldenBraid setpassfail (0.79)passes on manual check of the grader’s evidence
Claude Opus 5.5GFP into EcoFlex vectorpasspasspasses after the record is re-expressed without changing any value
Claude Opus 5.5GFP into EcoFlex vectorpasspasspasses after the record is re-expressed without changing any value
GPT-6 AstraGFP into EcoFlex vectorpasspasspasses after the record is re-expressed without changing any value
GPT-6 AstraGFP into EcoFlex vectorpasspasspasses after the record is re-expressed without changing any value
GPT-6.1 SolGFP into EcoFlex vectorpasspasspasses after the record is re-expressed without changing any value
Gemini 4 ArgonGFP into EcoFlex vectorpasspasspasses after the record is re-expressed without changing any value
Gemini 4 ArgonGFP into EcoFlex vectorpasspasspasses after the record is re-expressed without changing any value
Gemini 4 ArgonGFP into EcoFlex vectorpasspasspasses after the record is re-expressed without changing any value
Grok 4.7GFP into EcoFlex vectorpasspassgrader’s own source checks verify the product and route as written
Grok 4.7GFP into EcoFlex vectorpasspassgrader’s own source checks verify the product and route as written
Grok 4.7GFP into EcoFlex vectorpasspassgrader’s own source checks verify the product and route as written
Claude Mythos 5.1Prime-assembly donor familyfailfailstill fails after allowed re-expression
Claude Mythos 5.1Prime-assembly donor familyfailfailstill fails after allowed re-expression
Claude Opus 5.5Prime-assembly donor familyfailfailstill fails after allowed re-expression
Claude Opus 5.5Prime-assembly donor familyfailfailstill fails after allowed re-expression
GPT-6.1 SolPrime-assembly donor familypasspasspasses after the record is re-expressed without changing any value
Gemini 3.8 FlashPrime-assembly donor familyfailfailstill fails after allowed re-expression
Gemini 3.8 FlashPrime-assembly donor familypassfail (0.67)passes after the record is re-expressed without changing any value
Gemini 3.8 FlashPrime-assembly donor familyfailfailstill fails after allowed re-expression
Gemini 3.8 FlashPrime-assembly donor familyfailfailstill fails after allowed re-expression
Gemini 3.8 FlashPrime-assembly donor familyfailfailstill fails after allowed re-expression
Gemini 4 ArgonPrime-assembly donor familypassfail (0.80)passes after the record is re-expressed without changing any value
Gemini 4 ArgonPrime-assembly donor familypassfail (0.81)passes after the record is re-expressed without changing any value
Gemini 4 ArgonPrime-assembly donor familypassfail (0.88)passes after the record is re-expressed without changing any value
Grok 4.7Prime-assembly donor familyfailfailstill fails after allowed re-expression
Grok 4.7Prime-assembly donor familyfailfailstill fails after allowed re-expression
Grok 4.7Prime-assembly donor familyfailfailstill fails after allowed re-expression
Kimi K3Prime-assembly donor familyfailfailstill fails after allowed re-expression
Kimi K3Prime-assembly donor familyfailfailstill fails after allowed re-expression
Kimi K3Prime-assembly donor familyfailfailstill fails after allowed re-expression
Kimi K3Prime-assembly donor familyfailfailstill fails after allowed re-expression
Claude Opus 5.5Synthetic-chromosome locus editpasspassgrader’s own source checks verify the product and route as written
Claude Mythos 5.1Tus/TerB HEK293 constructsfailfailstill fails after allowed re-expression
Claude Mythos 5.1Tus/TerB HEK293 constructspasspasspasses after the record is re-expressed without changing any value
GPT-6.1 SolTus/TerB HEK293 constructspasspasspasses after the record is re-expressed without changing any value
Gemini 3.8 FlashTus/TerB HEK293 constructsfailfailstill fails after allowed re-expression
Gemini 3.8 FlashTus/TerB HEK293 constructsfailfailstill fails after allowed re-expression
Gemini 4 ArgonTus/TerB HEK293 constructsfailfailstill fails after allowed re-expression
Grok 4.7Tus/TerB HEK293 constructsfailfailstill fails after allowed re-expression

Table F2. Table F2: Reviewed construct-design attempts at top effort. Deterministic results after review and final results at the 0.9 threshold are listed with the basis for each decision.

Repeated Sampling and Per-Task Results

Confidence intervals in Table G1 use the same bootstrap draws as Table 2.

References

  1. [1]Jon M. Laurent, Joseph D. Janizek, Miguel Ruzo, Madeline M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodrigues. LAB-Bench: Measuring capabilities of language models for biology research, 2024.arxiv.org/abs/2407.10362
  2. [2]Jon M. Laurent, Albert Bou, Marlon Pieler, Conor Igoe, Alex Andonian, Siddharth Narayanan, Jonathan Braza, Alejandro Sanchez Vassopoulos, Jacob L. Steenwyk, Benjamin Lash, Andrew D. White, and Samuel G. Rodrigues. LABBench2: An improved benchmark for AI systems performing biology research, 2026.arxiv.org/abs/2604.09554
  3. [3]Anthony Liu, Andrew Ho, Alexander M. Droste, David Martin, Eric Wong, Eric Zhou, Ian Zhou, Jeffrey Park, Jialu Jiao, Kai-Rou Skelly, Kevin Kim, Karthik Rao, Masatoshi Uehara, Madison Marion, Nicholas Fitzgerald, Ricardo Dias, Suyash Shringarpure, Yiqing Yuan, and Yana Wang. LifeSciBench: Evaluating language models on realistic, expert-level tasks in the life sciences, 2026. Preprint.DOI
  4. [4]Lindsay Mitchener, Jon M. Laurent, Benjamin Tenmann, Siddharth Narayanan, Geemi P. Wellawatte, Andrew D. White, Lorenzo Sani, and Samuel G. Rodrigues. BixBench: A comprehensive benchmark for LLM-based agents in computational biology, 2025.arxiv.org/abs/2503.00096
  5. [5]Aditya Sivakumar, Ashu Singhal, Nicholas Larus-Stone, and Nithin Parsan. BenchBench-Protocol: Evaluating real-world wet-lab protocol reasoning and modification, 2026.arxiv.org/abs/2608.23898
  6. [6]Terminal-Bench-Science Team. Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains. https://github.com/harbor-framework/terminal-bench-science, August 2026. Version v0.1.0.DOI

Paper details

vlnc:W_eeubbewp
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents