WorksComputer Science
Evaluating and improving AI agents for sequencing protocol understanding
Abstract
Existing scientific-agent benchmarks rarely evaluate whether agents can reconstruct the exact molecular structures and generation workflows specified by sequencing protocols, a capability critical for analyzing customized sequencing assays. We introduce LibStructBench, a document-grounded benchmark for reconstructing sequencing-library structures and molecular workflows. Our experiments show that frontier agents often produce approximately correct reconstructions but remain unreliable at exact reconstruction, particularly in molecular-state representation and workflow connectivity. Switching the underlying language model produced larger average performance differences than switching agent harnesses, with a statistically significant difference only for molecular-state reconstruction. Additional knowledge improved oligo-family recovery in a small pilot without improving mean molecular-workflow scores. Autonomous accumulation of procedural rules and cross-protocol examples showed non-monotonic validation performance and no mean held-out molecular-workflow improvement in the tested trajectory. These findings highlight the need for more controlled and verifiable approaches to scientific reasoning about molecular processes in complex experimental protocols.
Introduction
Correct analysis of sequencing data depends on knowing the molecular structure of the library being sequenced. Adapter sequences, cell and sample barcodes, unique molecular identifiers (UMIs), their positions and strand orientation determine how raw sequencing reads should be extracted, trimmed, demultiplexed, and quantified. This is particularly important for customized or newly developed single-cell assays, where barcode and UMI layouts may differ substantially from standard commercial protocols. Machine-readable specifications such as SEQSPEC [6] demonstrate that explicit library descriptions can support standardized preprocessing across genomics assays. However, such approaches assume that the underlying library structure has already been established.
For new or customized assays, establishing this structure can itself require substantial manual interpretation of the experimental design, followed by custom code for cell and molecule recovery and assessment of assay sensitivity and specificity. Moreover, diagnosing a failed sequencing run is even harder: it requires not only the final read structure but also an understanding of all molecular intermediates and experimental operation steps. AI agents based on frontier large language models (LLMs) could potentially automate this process by reconstructing the library, generating assay-specific preprocessing logic, inspecting raw sequencing data, and tracing quality-control failures back to candidate experimental steps. Such capability could substantially accelerate the development and debugging of new sequencing technologies.
Benchmark construction and validation
LibStructBench was built from scg_lib_structs, an open resource of manually reconstructed sequencing-library primer and adapter elements (hereafter, oligos) and step-by-step library-generation procedures for single-cell genomics methods (https://teichlab.github.io/scg_lib_structs/). The resource spans diverse assay classes, including gene expression, chromatin accessibility, DNA methylation, genomic DNA, and multi-omic protocols. We selected more than 70 curated commercial and academic sequencing protocols with diverse library architectures as the initial legacy records for LibStructBench.
For each protocol, we assembled an agent-visible evidence bundle containing available experimental protocol or technical documentation, primary journal article, supplementary files, oligo tables, and molecular figures. These bundles were frozen and content-addressed before evaluation and defined the evidence boundary for reconstruction. Agents could inspect and process the supplied files but were not permitted to search the web or retrieve external protocol-specific information. Final ground-truth records were produced only after the agent-audit and human-adjudication procedure described below.
The benchmark datasets and implementation are publicly available:
Protocol ground truth: https://huggingface.co/datasets/sequencing
Benchmark implementation: https://github.com/seqmachines/libstruct-bench
Workflow visualization and agent traces: https://www.acolytics.com/proofread
The ground-truth release manifests retain recorded source URLs, document versions where available, and content hashes. These manifests do not grant redistribution rights for the underlying documents, which retain their original licensing restrictions.
Benchmark tasks and output representations
Task 1: final sequencing-library reconstruction. The final library structure is a key piece of information for raw sequencing-data processing and sequencing run quality assessment. Given a protocol bundle, the agent must reconstruct the canonical structure of each defined final library. Fixed nucleotide segments and typed variable regions are normalized before scoring. Reconstruction accuracy is evaluated using normalized sequence similarity based on Levenshtein distance (edit distance),
Task 2: workflow-linked oligo recovery. Oligos are the physical reagents that include adapters, priming sites, barcodes, and other technical elements into a sequencing library. Extracting them therefore tests whether the reconstructed molecular workflow is grounded in the correct building blocks, and the ground truth of this task is mainly the "Adapter and primer sequences" section in scg_lib_structs. The agent returns one structured record per oligo family, including its sequence, name and alias, role, molecular kind, orientation, ordered components, and modifications. Ordered components are used for assembled, double-stranded, or hairpin oligos that cannot be represented faithfully as a single flat sequence.
Required ground-truth oligo families are those linked to Task 3 with source-recoverable sequences. Agent-extracted oligos with compatible sequence scaffolds and metadata are collapsed into predicted families and matched one-to-one to required families, with eligible optional or non-recoverable matches neutralized. For each extraction-ground-truth pair, we compute a weighted similarity
The weights summarize sequence, modification, molecular kind, orientation, and role similarity, and are renormalized over supported components. If denotes the resulting one-to-one matching, the primary Task 2 score is
Here, denotes required, source-recoverable ground-truth families, all collapsed predicted families, and predictions neutralized by eligible optional or non-recoverable matches. Neutralization uses exact matches or a correspondence threshold of 0.50 for used non-recoverable families, while preserving required matches. Thus, and , counting both matched and unmatched non-neutralized predictions. When , F1 is 1 if , and 0 otherwise. These protocols are excluded from primary Task 2 macro-F1.
Task 3: molecular library-generation workflow. Task 3 reconstructs the “Step-by-step library generation” section in scg_lib_structs as a directed state-transition graph. Shared upstream steps are represented once, with modality-specific or alternative routes branching toward their terminal states. Each state records the physical molecular strands and their 5′ → 3′ architecture, while each transition records the input states, molecular operation, oligos, products, and whether those products are carried forward or discarded. For example, reverse transcription (operation) converts an RNA template (substrate) using an RT primer (oligo) into an RNA–cDNA intermediate (product). And in the fragmentation example shown in Figure 1, fragmentation generates multiple product classes from double-stranded cDNA (substrate), with the barcode-bearing cDNA fragment carried forward and internal or TSO-end fragments discarded.
Figure 1. LibStructBench overview. Given a protocol bundle containing primary source evidence, an AI agent reconstructs three linked representations of sequencing-library construction, corresponding to Tasks 1–3. The schematic illustrates these tasks using a simplified workflow for 10x Chromium 3′ Gene Expression v4 [1].
A transition is represented as , where and are the input and output molecular states (e.g. RNA and RNA-cDNA intermediate), is the molecular operation, and is the set of linked Task 2 oligo families. Agent-reconstructed and ground-truth workflows are first matched using their terminal modalities, states, and transitions. Within each matched workflow, states and transitions are aligned through global partial one-to-one matching, with candidate pairs below a context-adjusted assignment score of 0.25 left unmatched. For each matched transition pair, scientific similarity is
which compares the operation, aligned input and product states, carried-versus-discarded disposition, and transition-linked oligos.
For entity type , representing transitions or states, let denote the matched pairs, the number of scored agent-output entities, and the number of scorable ground-truth entities. Transition and state F1 are computed using the same soft-F1 form:
Matched entities receive fractional credit, unmatched scored outputs reduce precision, and missing ground-truth entities reduce recall. For states, weights reference-strand sequence 0.50, molecular architecture 0.15, ordered strand and segment structure 0.20, and pairing or discontinuities 0.15.
To evaluate graph topology separately, we compare typed substrate, carried-product, and discarded-product edges between aligned states. Let denote the number of matching typed edges. Using the same effective output and scorable ground-truth counts, edge F1 is
Ground-truth construction, audit, and adjudication
Human-curated records in scg_lib_structs may contain reconstruction errors, while the underlying protocol documents may themselves be ambiguous or internally inconsistent. We therefore audited every candidate benchmark record before evaluation. For each protocol, the legacy curation was first converted into the canonical Task 1–3 ground-truth schema. An AI agent then compared this candidate record against the frozen source bundle and generated proposed corrections for human review.
Codex implemented the deterministic audit framework, including source manifests, schemas, Task 1–3 validation, review checkpoints, and application of approved changes. Claude Code performed the protocol-level audit, checking each candidate against the protocol sources and flagging unsupported claims, ambiguities, protocol-version conflicts, and proposed corrections. A human expert adjudicated every substantive finding by accepting, revising, or rejecting the proposed change and directly reviewed the oligos, molecular states, and transitions against the primary sources. Only human-approved changes were applied to the benchmark record (Figure S1). In the summarized audit, 54 of 69 confirmed findings (78.3%) were categorized as legacy-curation errors (Figure S1). This statistic describes error causes rather than the origin of accepted corrections, which could not be reliably determined from the available adjudication records. Future ground-truth audit updates and corrections will be documented at https://acolytics.com/proofread.
Figure S1. Human-agent ground-truth audit and adjudication. (a) Existing human curation is converted to the canonical representation and verified against primary evidence. Claude Code proposes corrections, a human curator adjudicates substantive findings, and only approved decisions are applied by deterministic code. (b) Distribution of confirmed audit findings of errors from 73 usable protocols collected in scg_lib_structs.
Experiments and results
Interface and PDF-conversion controls
Direct LLM API and coding-agent comparison. We first assessed final-library reconstruction accuracy and whether native coding-agent interfaces improved performance over direct LLM APIs. Three models were evaluated across 20 sequencing protocols: Claude Opus 5, GPT-5.6 Sol, and Gemini 3.7 Flash, using their direct APIs and corresponding native agent interfaces (Claude Code, Codex, and Antigravity CLI) with high reasoning effort via Harbor [16]. Direct APIs received protocol PDFs without tools, whereas agents could use native file, shell, and document-processing tools. Excel sheets were excluded in inputs here because direct APIs do not accept this format.
Mean paired similarity differences (agent minus API) were +0.033, +0.005, and -0.030 for Claude, GPT, and Gemini, respectively. None was statistically significant in two-sided paired Wilcoxon signed-rank tests with Bonferroni correction across the three model comparisons ( for all; Figure 2). Exact final-library reconstruction remained uncommon, with only 3-5 of 20 protocols (15-25%) reconstructed exactly across configurations; API/agent win counts were 4/3 (13 ties), 1/1 (18 ties), and 3/1 (16 ties), respectively (Figure 2). Therefore, native coding agents showed no statistically detectable improvement in Task 1 accuracy.

Figure 2. Task 1 performance for direct APIs versus native coding agents. Each point represents one sequencing protocol completed by both interfaces. denotes the mean paired similarity difference (agent minus API), and denotes the adjusted -value for two-sided Wilcoxon signed-rank test. The inset table reports mean normalized sequence similarity and exact reconstruction counts for each interface.
As final-library similarity alone cannot characterize errors in molecular components and workflow reconstruction, we further evaluated Tasks 2 and 3 using coding agents, whose tool-calling and structured-execution capabilities support more complex evaluations and improvement.
Text-layer nucleotide sequences are reliably recovered from PDFs. Since sequence reconstruction errors could arise from technical PDF text conversion rather than downstream protocol reasoning, we tested sequence recovery across five representative protocols containing 80 human-verified, source-visible nucleotide strings. Curated human text, PyMuPDF [3], pypdf [11], Docling without OCR [20], and MinerU OCR [36] were compared, and a fixed Codex independently recovered sequence strings. Each ground-truth sequence was scored by its best match among the recovered candidates, , using the normalized Levenshtein sequence similarity defined in Eq. (1).
Text from human, PyMuPDF, pypdf, and Docling recovered all 80 sequences exactly (Figure S2), but MinerU OCR recovered only 71 of 80 exact sequences. The OCR errors included corruption of repetitive character strings such as "GGGGGG". These results suggest that text-layer nucleotide extraction alone is unlikely to explain the observed reconstruction errors. However, this does not assess sequences embedded in figures or the interpretation of graphical molecular structures.

Figure S2. PDF-parser-control sequence recovery. Sequence recovery was evaluated across 80 verified nucleotide strings from 5 protocols. For each reference sequence, we report the highest normalized Levenshtein similarity to any sequence extracted. The table summarizes exact recovery and mean similarity.
Baseline molecular-workflow reconstruction
Baseline performance across models and harnesses. To assess how reliably coding agents could reconstruct the underlying molecular workflow and components, we benchmarked four native coding-agent systems on the same 20-protocol panel: GPT-5.6 Sol Max with Codex, Claude Opus 5 Max with Claude Code, Gemini 3.7 Flash High with Antigravity CLI, and Kimi K3 Max with Kimi Code. Each system received the same frozen protocol bundles, task instructions, output schema, and deterministic verifier similarly through Harbor [16]. For Task 2, ddSEQ was excluded from the macro-average because it contained no required, source-recoverable oligo families (). Mean Task 2 oligo-family F1 ranged from 0.724 to 0.774 with required recoverable oligo families. Task 3 molecular-transition, molecular-state, and typed-edge F1 ranged from 0.767–0.814, 0.495–0.723, and 0.721–0.754, respectively, across all 20 protocols (Figure 3a). No single system dominated all metrics. Kimi obtained the highest Task 2 F1, Gemini the highest transition F1, Claude the highest state F1, and GPT the highest typed-edge F1. Molecular-state F1 showed the largest between-system variation.
Figure 3. Baseline agent comparison. (a) Mean Task 2 oligo-family and Task 3 transition, state, and typed-edge F1 for native CLI systems. (b) Task 3 molecular-transition F1 across 10 common protocols and nine model-harness systems, clustered by protocol-level performance.
We then asked whether the agent harness altered performance for the same underlying model. GPT, Claude, and Gemini were additionally paired with Pi [39] and mini-SWE-agent [38], yielding nine model-harness systems when combined with their native interfaces. These systems were compared on 10 protocols common to all configurations, drawn from the 20-protocol baseline panel. Since molecular-transition F1 jointly evaluates the operation, substrates, products, linked oligos, and product disposition, we used it as the primary Task 3 metric for this comparison. Mean molecular-transition F1 ranged from 0.714 to 0.805 across the nine model-harness systems (Figure 3b). Hierarchical clustering did not clearly separate systems by model or harness, although molecular-state performance showed more pronounced model-associated patterns. (Supplementary Figure S4).
Estimated costs varied across systems (Figure S5) and were not monotonically related to reconstruction performance (Figure S5). To quantify model and harness effects, we performed a protocol-paired analysis across three models and two shared harnesses (Pi and mini-SWE-agent). Model-switch differences exceeded harness-switch differences on average across all four metrics, although only molecular-state F1 remained statistically significant after Bonferroni correction (, point-wise 95% CI [0.038, 0.124], ; Supplementary Figure S6).
Figure S5. Estimated API-equivalent cost and reconstruction performance. (a) Per-protocol costs and system means across ten protocols. (b) Cost versus mean performance across four F1 metrics.
Variability across repeated executions. As our baseline comparisons used a single execution per model-harness system and protocol, we assessed how aggregate performance and individual-protocol scores varied across repeated executions of the same agent configuration. We evaluated GPT-Codex across three executions per protocol on the 20-protocol baseline panel. Averaging the three executions changed the original oligo-family, molecular-transition, molecular-state, and typed-edge macro-F1 scores by −0.005, −0.014, −0.012, and −0.024, respectively. Despite these relatively small aggregate differences, within-protocol F1 ranges exceeded 0.10 for 5/19 informative oligo protocols and 9/20, 8/20, and 12/20 protocols for the three Task 3 metrics, respectively (Appendix § 9). Although individual protocols showed execution-to-execution variability, the aggregate performance estimates remained broadly consistent across the three executions, with substantial room for improvement in molecular-workflow reconstruction. Given the computational cost of repeated evaluations across multiple models and harnesses, we retained single-execution comparisons for the broader benchmark.
Sensitivity to scoring assumptions. Similarity weights and matching thresholds may influence reported scores and system rankings, we therefore recomputed matching and scoring across 640 configurations (Appendix § 16). The leading native systems for oligo-family, molecular-state, and typed-edge F1 remained unchanged. Typed-edge F1 was also more sensitive to matching thresholds than transition or state F1. Overall, the broad baseline findings were robust to the tested scoring assumptions, although absolute scores and close system rankings warrant cautious interpretation.
Reconstruction without protocol source bundles. Many sequencing protocols and scg_lib_structs are publicly available and may have appeared in LLM training data, we further evaluated protocol-name-only controls on the 20-protocol panel (Appendix § 10). In a separate Task 1 API experiment, name-only prompting increased mean sequence similarity for all three models but yielded fewer exact reconstructions. Only Claude’s similarity improvement remained significant after Bonferroni correction (). For native Tasks 2-3, name-only prompting yielded lower mean F1 across all three systems and four metrics, although none of the 12 paired comparisons remained significant after Bonferroni correction. These results indicate substantial reconstruction ability without supplied evidence but do not establish training-data memorization or that source bundles are unnecessary.
Failure modes. To characterize where agent reconstructions disagreed with the ground truth, we applied deterministic error analysis to 100 outputs from ten systems across ten common protocols. For each output, the verifier recorded whether at least one discrepancy occurred in each of six predefined categories (Table S3). Molecular-state assembly and workflow-topology/disposition discrepancies were identified in all 100 outputs (Figure S7). Missing recoverable information occurred in 88/100 outputs, unsupported additions in 77/100, operation or target mismatches in 24/100, and strand or orientation errors in only 8/100. Exploratory clustering of system-level discrepancy profiles tended to group configurations by underlying LLM (Figure S7), while protocol-level profiles showed variation in missing information, unsupported additions, and operation or target mismatches (Figure S7). These categories are nonexclusive, as molecular-state alignment errors can propagate into topology discrepancies; their frequencies therefore describe disagreement with the canonical reconstruction rather than independent failure mechanisms.
Figure S7. Baseline output-discrepancy profiles. (a) The fraction of the ten protocols with at least one discrepancy. (b) The fraction of the ten systems with at least one discrepancy. Values range from 0 to 1, where 1 indicates this error appeared in 100% protocols or LLM-harness runs.
| Error category | Description | Main effect |
| Missing recoverable information | Required source-supported content is missing or incomplete. | Lower recall. |
| Unsupported addition | The output adds content unsupported by the source and not marked optional or neutral. | Lower precision. |
| Operation / target mismatch | An operation is incorrect or applied to the wrong molecular target. | Lower transition F1. |
| Strand / orientation error | A sequence or strand has an incorrect orientation or strand relationship. | Lower oligo/state score. |
| Molecular-state assembly | A state has incorrect strands, segment order, pairing, or architecture. | Lower state F1. |
| Workflow topology / disposition | A workflow connection or carried/discarded relationship is incorrect. | Lower edge F1. |
Table S3. Deterministic output-discrepancy categories used in the baseline error analysis.
Improving molecular-workflow reconstruction
Accurate protocol understanding is important for downstream sequencing quality control, where errors in library structure and oligo definitions can directly affect read parsing and quantitative analyses. Baseline results showed that none of the native model-harness systems achieved consistently accurate reconstruction, we next explored whether supplying additional domain knowledge or accumulating reviewed experience could improve agent performance.
External knowledge had task-dependent effects in this pilot. GPT-5.6 Sol Max with Codex was used to examine whether external domain knowledge or cross-protocol examples could improve Tasks 2 and 3 performance on a fixed five-protocol validation panel (sci-ATAC-seq, scRRBS, SMART-seq, SHARE-seq, ddSEQ). Each protocol was tested under four paired conditions: 1) the protocol bundle alone (baseline); 2) a general molecular-method primer derived from textbooks and review articles (Table S4); 3) ground-truth records from five separate donor protocols, reformatted as example agent outputs and supplied as cross-protocol exemplar memory (Table S5); and 4) both knowledge sources. The baseline achieved mean oligo-family, molecular-transition, molecular-state, and typed-edge F1 scores of 0.826, 0.800, 0.636, and 0.673, respectively (Figure 4a). Combining general knowledge with cross-protocol memory increased oligo-family F1 to 0.877, a paired improvement of 0.050 (pointwise 95% protocol-bootstrap CI [0.022, 0.079]). General knowledge alone and memory alone achieved oligo-family F1 scores of 0.777 and 0.814, respectively. All three augmented conditions had lower Task 3 mean scores than baseline, although their paired confidence intervals included zero. These results show a task-dependent response in the saved outputs: combined knowledge improved informative-protocol oligo recovery, while no augmentation exceeded baseline Task 3 point estimates.
Figure 4. Improving molecular-workflow reconstruction. (a) Mean Task 2 and Task 3 scores under the baseline, general-knowledge, cross-protocol memory, and combined conditions. (b) Autonomous cumulative-improvement framework. (c) Validation performance across autonomous-training checkpoints. (d) Held-out test performance of baseline and the validation-selected checkpoint with and without knowledge base (KB) supplement. For panels (a) and (c), Task 2 includes four informative validation protocols and Task 3 includes five; the two experiments use different validation panels. Panel (d) includes nine informative held-out protocols for Task 2 and ten protocols for Task 3.
| Source type | Source | Primer content |
| Textbook | Molecular Cell Biology, Lodish et al. 2003 [21] | PCR, primer tails, restriction–ligation, reverse transcription, and DNA-end chemistry. |
| Textbook | PCR Protocols, Bartlett & Stirling 2003 [4] | End repair, ligation, terminal modifications, and phosphate-dependent sealing. |
| Textbook | Protocols for Gene Analysis, Harwood 1994 [17] | Primer tails, A-tailing, T-vector cloning, and blunt-end preparation. |
| Review | Single-cell genome sequencing: current state of the science, Gawad et al. 2016 [12] | Whole-genome amplification, amplification artifacts, and library construction. |
| Review | Scaling by shrinking: empowering single-cell ‘omics’ with microfluidic devices, Prakadan et al. 2017 [27] | Microfluidic compartmentalization, reaction splitting, and pooled processing. |
| Review | Single-cell epigenomics: techniques and emerging applications, Schwartzman & Tanay 2015 [31] | Bisulfite conversion, reduced-representation libraries, and size selection. |
| Review | RNA sequencing: the teenage years, Stark et al. 2019, [32] | RNA-seq library preparation, reverse transcription, and amplification. |
| Review | Methods and applications for single-cell and spatial multi-omics, Vandereyken et al. 2023 [34] | Multimodal branching, combinatorial indexing, tagmentation, and template switching. |
| Review | Chromatin accessibility profiling methods, Minnoye et al. 2021 [24] | Fragmentation, Tn5 tagmentation, adaptor addition, indexing, and amplification. |
| Research article | Suppression of artifacts and barcode bias in high-throughput transcriptome analyses utilizing template switching, Tang et al. 2013 [33] | Template switching, primer tails, barcode placement, and associated artifacts. |
| Review | Time-resolved single-cell RNA-seq using metabolic RNA labelling, Erhard et al. 2022 [10] | Affinity capture, reversible modification, elution, and molecular-fraction handling. |
Table S4. Sources for the general molecular-method knowledge primer. The primer summarizes general molecular operations instead of protocol-specific answers.
| Validation protocol | Donor protocol | Shared feature |
| sci-ATAC-seq | s3-ATAC | Tn5-based chromatin accessibility and combinatorial indexing [9, 25]. |
| scRRBS | 10x Chromium 3′ Gene Expression v4 | Shared Illumina adapter and library-completion oligos [15, 1]. |
| SMART-seq | Drop-seq | Template-switching reverse transcription and cDNA amplification [29, 23]. |
| SHARE-seq | SPLiT-seq | Combinatorial indexing with repeated barcode-incorporation steps [22, 30]. |
| ddSEQ Single-Cell 3′ RNA-Seq Kit | sci-RNA-seq | Barcoded reverse transcription, cDNA amplification, and shared library components [5, 7]. |
Table S5. Protocols used in the cross-protocol memory pilot. Donor protocols were selected as conceptual matches to the validation protocols based on shared molecular features.
Autonomous cumulative learning. We tested whether an agent could improve molecular-workflow reconstruction by accumulating experience across previously verified protocols. The 73 usable protocols were split into 58 training, 5 validation, and 10 held-out test protocols. Validation panel includes CEL-seq, chATAC-seq, ddSEQ, HyDrop-RNA, scRRBS. Training proceeded in batches of five protocols. We denote the initial agent as checkpoint (baseline), and each subsequent checkpoint as the agent state after learning from training protocols. After each batch, the agent autonomously updated two components: 1) a protocol-neutral procedural rule set derived from observed agent errors and 2) a knowledge base of oligos and molecular graphs derived from completed training ground truth. These components were frozen as the next checkpoint, and each checkpoint was evaluated on the same five validation protocols without using validation outputs for learning (Figure 4b).
Validation performance was non-monotonic across checkpoints. Using the aggregate validation criterion, was selected, whereas performance declined at later checkpoints, with substantial changes appearing by –. The final checkpoint scored below on all four validation metrics (Figure 4c). We then evaluated the selected procedural rules on the 10 held-out protocols, both alone and supplemented with the accumulated training knowledge base. Relative to , the rules alone did not improve any Task 3 metric. Adding the knowledge base increased the held-out oligo-family F1 point estimate from 0.560 to 0.621 (paired difference 0.061; pointwise 95% protocol-bootstrap CI [-0.024, 0.159]). Transition, state, and typed-edge F1 means remained below baseline (0.684 vs. 0.721, 0.523 vs. 0.592, and 0.588 vs. 0.674, respectively; Figure 4d). Thus, this single learning trajectory showed non-monotonic validation performance and no held-out Task 3 point-estimate gain. These findings are conditional on the saved outputs and canonical graph representation; they do not establish that cumulative learning generally fails to improve molecular understanding.
Discussion
Our results highlight a gap between recovering sequencing-library components and faithfully reconstructing molecular workflows. Agents often identified experimental operations but struggled with intermediate molecular states and workflow connectivity, explaining relatively high transition F1 despite lower state F1. Task 2 and Task 3 scores measure agreement with canonical references rather than independently validated biological correctness or downstream usability. The broad findings were robust to tested scoring assumptions, while repeated GPT-Codex executions revealed protocol-level variability despite similar aggregate performance.
Model and harness effects. No native model-harness system consistently outperformed the others across all reconstruction metrics. In our paired comparison across three models and two shared harnesses, switching the underlying language model produced larger average performance differences than switching harnesses, with a statistically significant difference only for molecular-state reconstruction. This suggests that representing intermediate molecular architectures is particularly sensitive to the underlying model in the tested configurations.
Additional knowledge and autonomous learning had task-dependent effects. Combining general biological knowledge with cross-protocol examples improved oligo-family recovery in our small pilot without improving mean molecular-workflow scores. Similarly, accumulated knowledge increased the held-out oligo-family F1 point estimate, with uncertainty spanning zero, but did not improve Task 3 mean performance. These findings suggest that recovering additional molecular components does not necessarily translate into correctly integrating them into a molecular workflow.
One possible explanation is that models already possess substantial general sequencing knowledge, as suggested by the name-only controls. Additional context may therefore offer limited benefit while complicating the selection and integration of protocol-specific evidence. Moreover, cumulative learning showed non-monotonic validation performance, with some learned rules associated with omitted molecular intermediates or altered workflow connectivity. Some discrepancies may also reflect differences in representation granularity, as chemically valid intermediate steps can be penalized by canonical graph matching. Furthermore, the verifier may penalize chemically valid reconstructions when their molecular-state or workflow representations differ from the canonical ground truth.
Future improvement should emphasize controlled and verifiable learning. Our document-grounded setting reflects the intended application to new or unpublished sequencing protocols, for which protocol-specific information may not be publicly available. We therefore use publicly documented protocols from scg_lib_structs as an auditable benchmark, while restricting agents to frozen source bundles rather than external retrieval.
The name-only controls suggest that agents can leverage prior knowledge to generate candidate reconstructions, which could then be verified against protocol-specific evidence. Future agents could first propose a provisional molecular workflow, then extract evidence from supplied documents to verify, revise, or reject individual components and transitions. Model Context Protocol (MCP)-accessible tools could support document parsing, oligo validation, molecular-state construction, and graph-consistency checking, while cross-protocol memory could enable iterative improvement. Learned rules should be reversible and validated against previously solved protocols before retention. Future work should compare this prior-guided approach with direct document-grounded reconstruction and accommodate biologically equivalent workflow representations.
Acknowledgments
We thank Ruslan Soldatov (Memorial Sloan Kettering Cancer Center) and Xinming Tu (Phylo Bio) for valuable discussions and guidance. We also acknowledge the Google Cloud Research Credits Program for providing Google Cloud and Gemini API credits, and the MIT–Anthropic AI for Science program and Claude hackathon for providing Claude usage and API credits.
Related work
Sequencing-library specifications. Explicit descriptions of sequencing-library structure are important for reproducible upstream preprocessing and interpretation of single-cell sequencing data. Community resources such as scg_lib_structs (https://teichlab.github.io/scg_lib_structs/) manually curate primers, adapters, barcodes, final-library structures, and library-generation steps across single-cell protocols. SEQSPEC [6] formalizes this idea with a machine-readable specification for genomics assays, representing ordered library regions, technical sequences, and their relationships to sequencing reads. This enables more standardized preprocessing, assay comparison, and reanalysis across various sequencing technologies. LibStructBench addresses a complementary problem: rather than assuming that a structured assay description is already available, it evaluates whether an AI agent can autonomously reconstruct such information accurately from heterogeneous sequencing-protocol documents.
Biomedical agents and benchmarks. Recent agentic systems increasingly combine language-model reasoning with scientific tools, retrieval, and structured execution environments. Co-Scientist [14] uses a multi-agent architecture to generate, critique, and refine scientific hypotheses, and Robin [13] combines literature-search and data-analysis agents in an iterative lab-in-the-loop discovery workflow. BIOMNI [18] generalizes this idea to a broad biomedical environment spanning database retrieval, code execution, and other domain tools.
On the evaluation side, LAB-Bench [19], ScienceAgentBench [8], BiomniBench [28], and DrugDiscoveryBench [2] evaluate increasingly realistic biological and scientific tasks, from literature and sequence reasoning to executable multi-step workflows. LibStructBench differs by focusing on a narrow but exact reconstruction problem: recovering sequencing-library components and the molecular workflow from primary protocol files.
Deterministic verification and iterative improvement. Recent agent systems increasingly combine generative reasoning with deterministic feedback and persistent adaptation. VirBench [26] shows that coupling scientific agents to a deterministic programmatic retrieval layer substantially improves the reliability of viral-sequence retrieval. SAGE [37] develops a self-improving agent with a persistent skill library, in which skills learned from earlier tasks accumulate. DeepVerifier [35] instead uses rubric-based verification to generate feedback for iterative refinement at inference time. Together, these studies motivate closed-loop agent systems where deterministic tools and prior experience are converted into reusable capabilities. Our work follows this direction by testing whether self-improvement loops that learn from discrepancies on reviewed training protocols can improve molecular-workflow reconstruction of unseen sequencing protocols.
Human-agent ground-truth audit and adjudication
Representative errors.
Human curation error (dsciATAC-seq). The legacy record labeled the 6-nt sequence between the s5 adapter and mosaic end as a UMI. The Methods and oligo table identify it as one of 96 Tn5 well barcodes, so the placeholder was corrected from [UMI:6] to [TN5_INDEX:6] in Tasks 1 and 3.
Protocol ambiguity (scDamID). The supplement prints a 57-nt P5 primer beginning ATG, whereas the final-library structure in the same protocol begins AATG and requires the 58-nt primer. Direct PDF re-extraction confirmed that the source itself omits the leading A; we retained the 58-nt form and recorded the conflict.
PDF-parser-control detail
Variability across repeated agent executions
We assessed execution variability for GPT-5.6 Sol Max with Codex by comparing versus executions per protocol across the 20-protocol baseline panel (60 executions total). The analysis included the original execution. All outputs were evaluated using the same scoring rules and approved ground-truth references as the main experiments. For protocol , metric , and execution , let denote the corresponding F1 score. Within-protocol variability was measured as across the three executions. We summarized these ranges across protocols using the median and interquartile range (linearly interpolated empirical quartiles), and counted protocols with . Execution-specific protocol-macro F1 was calculated as , and the three-execution mean as , where denotes the number of informative protocols. Task 2 excludes ddSEQ because (), whereas Task 3 includes all 20 protocols ().
The three-execution macro-F1 estimates differed from the original scores by only 0.005-0.024 across metrics (Table S1). However, within-protocol F1 ranges exceeded 0.10 for 5/19 oligo-family, 9/20 transition, 8/20 state, and 12/20 typed-edge comparisons (Figure S3). The largest observed range was 0.471 for typed-edge F1 in s3-ATAC. Thus, aggregate performance remained broadly consistent despite variability in individual protocols.
Figure S3. GPT-Codex variability across three executions per protocol. (a) Within-protocol F1 ranges. Points represent protocols; diamonds and intervals indicate medians and interquartile ranges. The dashed line marks a range of 0.10, with annotations indicating counts above this threshold. (b) Protocol-macro F1 for each execution and their three-execution mean.
| Metric | Original | 3-run mean | Max. range | ||
| Oligo family | 0.760 | 0.755 | 0.177 | 5/19 | |
| Transition | 0.809 | 0.795 | 0.343 | 9/20 | |
| State | 0.672 | 0.660 | 0.394 | 8/20 | |
| Typed edge | 0.754 | 0.730 | 0.471 | 12/20 |
Table S1. Observed variability across three GPT-Codex executions per protocol. Macro-F1 averages 19 informative protocols for oligo recovery and 20 for each Task 3 metric.
This analysis was limited to GPT-Codex; Figure 3a and Figure 3b retain one selected execution per system and protocol. Protocol-bootstrap intervals in the main results reflect across-protocol uncertainty conditional on these outputs, rather than execution variability. The results therefore do not quantify variability for other systems or for the knowledge and learning interventions, including matched - comparisons.
Reconstruction without protocol source bundles
Publicly available sequencing protocols and resources such as scg_lib_structs may have appeared in LLM training data, we evaluated whether models could reconstruct sequencing libraries using only protocol names and versions, without access to source documents. Generic instructions, output schemas, and local tools remained available, but source documents and retrieval access were withheld. External network access was restricted except for model-provider connections, and agent worked within an isolated docker image. This test assesses performance without supplied evidence, rather than distinguishing memorization from general learned knowledge or reasoning.
Task 1 API control. We compared the PDF-input API predictions from Figure 2 with one independent name-only API execution per model and protocol. Interestingly, name prompting generally yielded increased sequence similarity for all models, although only Claude remained significant after Bonferroni correction across three comparisons (). Exact reconstructions decreased from 4 to 2 for GPT, 5 to 3 for Claude, and 4 to 3 for Gemini, out of 20 protocols per model.
Native-agent comparison (Tasks 2-3). For Codex, Claude Code, and Antigravity, we compared one Name execution with the original Bundle execution used in Figure 3a ( per condition). All outputs were scored against the same approved references. All 20 protocols had valid pairs, with ddSEQ excluded from Task 2 (). For each system and metric, we calculated the mean paired difference . We used two-sided paired sign-flip tests with Bonferroni correction across 12 comparisons. In contrast to the Task 1 similarity results, Name prompting yielded lower mean F1 across all systems and metrics, although none of the differences remained significant after correction (all ).
| Protocol | Model | Task 1 Similarity PDF | Task 1 Similarity Name | Task 2 Oligo F1 Bundle | Task 2 Oligo F1 Name | Task 3 Transition F1 Bundle | Task 3 Transition F1 Name | Task 3 State F1 Bundle | Task 3 State F1 Name | Task 3 Edge F1 Bundle | Task 3 Edge F1 Name |
| 10x ATAC v2 | GPT | 1.000 | 0.947 | 0.791 | 0.673 | 0.782 | 0.848 | 0.875 | 0.863 | 0.714 | 1.000 |
| Claude | 1.000 | 1.000 | 0.713 | 0.687 | 0.777 | 0.761 | 0.776 | 0.812 | 0.714 | 0.714 | |
| Gemini | 1.000 | 1.000 | 0.701 | 0.673 | 0.956 | 0.707 | 0.720 | 0.396 | 1.000 | 0.667 | |
| 10x 3’ GEX v4 | GPT | 1.000 | 1.000 | 0.724 | 0.867 | 0.881 | 0.950 | 0.868 | 0.874 | 0.846 | 1.000 |
| Claude | 1.000 | 0.970 | 0.658 | 0.849 | 0.881 | 0.999 | 0.847 | 0.875 | 0.846 | 1.000 | |
| Gemini | 1.000 | 1.000 | 0.811 | 0.845 | 0.858 | 0.869 | 0.578 | 0.462 | 0.667 | 0.818 | |
| 10x 3’ Feature Barcoding | GPT | 1.000 | 0.853 | 0.865 | 0.703 | 0.695 | 0.751 | 0.720 | 0.550 | 0.650 | 0.600 |
| Claude | 1.000 | 0.881 | 0.852 | 0.704 | 0.770 | 0.752 | 0.717 | 0.666 | 0.650 | 0.650 | |
| Gemini | 0.995 | 0.903 | 0.817 | 0.712 | 0.759 | 0.650 | 0.513 | 0.469 | 0.684 | 0.684 | |
| Drop-seq | GPT | 0.897 | 0.989 | 0.675 | 0.733 | 0.766 | 0.918 | 0.534 | 0.739 | 0.750 | 0.960 |
| Claude | 0.886 | 0.989 | 0.528 | 0.746 | 0.853 | 0.942 | 0.770 | 0.800 | 0.889 | 1.000 | |
| Gemini | 0.886 | 0.989 | 0.511 | 0.735 | 0.820 | 0.617 | 0.558 | 0.329 | 0.750 | 0.364 | |
| Seq-Well S3 | GPT | 0.681 | 1.000 | 0.834 | 0.733 | 0.829 | 0.676 | 0.801 | 0.643 | 0.778 | 0.643 |
| Claude | 0.681 | 1.000 | 0.767 | 0.723 | 0.850 | 0.909 | 0.805 | 0.750 | 0.824 | 0.848 | |
| Gemini | 1.000 | 1.000 | 0.855 | 0.698 | 0.564 | 0.874 | 0.321 | 0.511 | 0.133 | 0.848 | |
| inDrop v1 | GPT | 1.000 | 0.586 | 0.672 | 0.448 | 0.895 | 0.766 | 0.706 | 0.505 | 0.846 | 0.720 |
| Claude | 1.000 | 0.871 | 0.675 | 0.477 | 0.641 | 0.656 | 0.491 | 0.512 | 0.595 | 0.500 | |
| Gemini | 1.000 | 0.860 | 0.655 | 0.490 | 0.721 | 0.778 | 0.231 | 0.479 | 0.462 | 0.692 | |
| s3-ATAC | GPT | 0.104 | 0.768 | 0.927 | 0.632 | 0.938 | 0.489 | 0.909 | 0.529 | 0.941 | 0.400 |
| Claude | 0.098 | 0.793 | 0.897 | 0.618 | 0.988 | 0.486 | 0.840 | 0.553 | 0.941 | 0.400 | |
| Gemini | 0.000 | 0.817 | 0.884 | 0.611 | 0.986 | 0.488 | 0.474 | 0.239 | 0.941 | 0.400 | |
| CEL-seq | GPT | 0.391 | 0.938 | 0.600 | 0.658 | 0.878 | 0.896 | 0.544 | 0.673 | 0.815 | 0.846 |
| Claude | 0.391 | 0.993 | 0.600 | 0.518 | 0.900 | 0.876 | 0.718 | 0.751 | 0.815 | 0.846 | |
| Gemini | 0.384 | 0.968 | 0.353 | 0.683 | 0.835 | 0.832 | 0.315 | 0.333 | 0.692 | 0.692 | |
| SPLiT-seq | GPT | 0.026 | 0.627 | 0.897 | 0.589 | 0.900 | 0.692 | 0.636 | 0.462 | 0.867 | 0.643 |
| Claude | 0.132 | 0.756 | 0.901 | 0.731 | 0.771 | 0.752 | 0.816 | 0.791 | 0.714 | 0.857 | |
| Gemini | 0.170 | 0.667 | 0.878 | 0.546 | 0.800 | 0.785 | 0.406 | 0.417 | 0.714 | 0.846 | |
| Protocol | Model | Similarity PDF | Similarity Name | Oligo F1 Bundle | Oligo F1 Name | Transition F1 Bundle | Transition F1 Name | State F1 Bundle | State F1 Name | Edge F1 Bundle | Edge F1 Name |
| sci-RNA-seq | GPT | 0.900 | 0.874 | 0.771 | 0.754 | 0.664 | 0.824 | 0.825 | 0.829 | 0.636 | 0.778 |
| Claude | 1.000 | 1.000 | 0.797 | 0.730 | 0.698 | 0.825 | 0.818 | 0.727 | 0.636 | 0.778 | |
| Gemini | 0.900 | 0.542 | 0.743 | 0.760 | 0.811 | 0.824 | 0.466 | 0.461 | 0.778 | 0.778 | |
| sci-ATAC-seq | GPT | 0.049 | 0.663 | 0.661 | 0.675 | 0.564 | 0.677 | 0.373 | 0.329 | 0.235 | 0.471 |
| Claude | 0.098 | 0.741 | 0.804 | 0.647 | 0.694 | 0.492 | 0.463 | 0.402 | 0.316 | 0.211 | |
| Gemini | 0.000 | 0.741 | 0.729 | 0.662 | 0.694 | 0.590 | 0.179 | 0.188 | 0.471 | 0.235 | |
| Microwell-seq | GPT | 0.578 | 0.826 | 0.527 | 0.739 | 0.900 | 0.944 | 0.648 | 0.714 | 0.909 | 1.000 |
| Claude | 0.578 | 0.853 | 0.516 | 0.475 | 0.808 | 0.544 | 0.732 | 0.240 | 0.783 | 0.182 | |
| Gemini | 0.509 | 0.812 | 0.545 | 0.661 | 0.940 | 0.666 | 0.734 | 0.362 | 1.000 | 0.600 | |
| scRRBS | GPT | 0.953 | 0.953 | 0.838 | 0.710 | 0.949 | 0.926 | 0.741 | 0.645 | 0.909 | 0.909 |
| Claude | 0.953 | 0.953 | 0.790 | 0.418 | 0.813 | 0.925 | 0.845 | 0.754 | 0.750 | 0.909 | |
| Gemini | 0.945 | 0.953 | 0.816 | 0.528 | 0.809 | 0.915 | 0.718 | 0.371 | 0.833 | 0.909 | |
| PIP-seq v4 | GPT | 0.829 | 0.842 | 0.575 | 0.713 | 0.765 | 0.888 | 0.316 | 0.637 | 0.750 | 0.960 |
| Claude | 0.829 | 0.838 | 0.697 | 0.731 | 0.771 | 0.747 | 0.589 | 0.658 | 0.686 | 0.720 | |
| Gemini | 0.829 | 0.820 | 0.767 | 0.635 | 0.775 | 0.682 | 0.363 | 0.382 | 0.783 | 0.720 | |
| scDamID | GPT | 0.994 | 0.595 | 0.643 | 0.496 | 0.783 | 0.547 | 0.644 | 0.546 | 0.692 | 0.480 |
| Claude | 0.994 | 0.637 | 0.629 | 0.728 | 0.881 | 0.789 | 0.672 | 0.627 | 0.846 | 0.750 | |
| Gemini | 0.994 | 0.619 | 0.661 | 0.607 | 0.992 | 0.490 | 0.452 | 0.255 | 1.000 | 0.364 | |
| ddSEQ | GPT | 0.063 | 0.401 | – | – | 0.936 | 0.701 | 0.898 | 0.669 | 0.938 | 0.519 |
| Claude | 0.181 | 0.382 | – | – | 0.971 | 0.700 | 0.914 | 0.649 | 1.000 | 0.583 | |
| Gemini | 0.063 | 0.384 | – | – | 0.664 | 0.700 | 0.648 | 0.490 | 0.519 | 0.583 | |
| SMART-seq | GPT | 0.000 | 0.342 | 0.859 | 0.811 | 0.791 | 0.604 | 0.573 | 0.499 | 0.722 | 0.643 |
| Claude | 0.240 | 0.500 | 0.662 | 0.833 | 0.879 | 0.800 | 0.630 | 0.631 | 0.833 | 0.800 | |
| Gemini | 0.000 | 0.500 | 0.757 | 0.866 | 0.803 | 0.557 | 0.501 | 0.276 | 0.765 | 0.462 | |
| SHARE-seq | GPT | 0.083 | 0.457 | 0.948 | 0.542 | 0.760 | 0.674 | 0.596 | 0.399 | 0.562 | 0.536 |
| Claude | 0.083 | 0.770 | 0.955 | 0.653 | 0.821 | 0.789 | 0.754 | 0.598 | 0.812 | 0.714 | |
| Gemini | 0.167 | 0.609 | 0.950 | 0.534 | 0.784 | 0.349 | 0.591 | 0.243 | 0.583 | 0.231 | |
| DR-seq | GPT | 0.109 | 0.630 | 0.850 | 0.425 | 0.565 | 0.323 | 0.435 | 0.465 | 0.511 | 0.300 |
| Claude | 0.109 | 0.771 | 0.497 | 0.338 | 0.482 | 0.422 | 0.472 | 0.484 | 0.226 | 0.213 | |
| Gemini | 0.109 | 0.882 | 0.600 | 0.444 | 0.722 | 0.483 | 0.329 | 0.336 | 0.643 | 0.426 | |
| PETRI-seq | GPT | 0.568 | 0.668 | 0.782 | 0.424 | 0.939 | 0.765 | 0.793 | 0.638 | 1.000 | 0.818 |
| Claude | 0.563 | 0.569 | 0.814 | 0.525 | 0.869 | 0.687 | 0.796 | 0.589 | 0.846 | 0.667 | |
| Gemini | 0.985 | 0.596 | 0.897 | 0.092 | 0.983 | 0.844 | 0.811 | 0.523 | 1.000 | 1.000 | |
| Macro mean | GPT | 0.561 | 0.748 | 0.760 | 0.649 | 0.809 | 0.743 | 0.672 | 0.610 | 0.754 | 0.711 |
| Claude | 0.591 | 0.813 | 0.724 | 0.638 | 0.806 | 0.743 | 0.723 | 0.644 | 0.736 | 0.667 | |
| Gemini | 0.597 | 0.783 | 0.733 | 0.620 | 0.814 | 0.685 | 0.495 | 0.376 | 0.721 | 0.616 | |
| (Name−source) | GPT | +0.187 | −0.111 | −0.066 | −0.061 | −0.042 | |||||
| Claude | +0.223 | −0.085 | −0.063 | −0.080 | −0.069 | ||||||
| Gemini | +0.186 | −0.113 | −0.129 | −0.119 | −0.105 | ||||||
| GPT | 0.0566 | 0.2535 | 1.0000 | 1.0000 | 1.0000 | ||||||
| Claude | 0.0215 | 0.5449 | 0.9889 | 0.1331 | 1.0000 | ||||||
| Gemini | 0.1329 | 0.7261 | 0.1420 | 0.1123 | 1.0000 | ||||||
| Task 1 exact | GPT | 4/20 | 2/20 | ||||||||
| Claude | 5/20 | 3/20 | |||||||||
| Gemini | 4/20 | 3/20 |
Table 1. Table S2: Protocol-name-only reconstruction across Tasks 1–3. Task 1 reports API sequence similarity (PDF vs. Name); Tasks 2–3 report native-agent F1 (Bundle vs. Name), using the Figure 3a bundle executions. Bold indicates the higher score within each pair; ties are unbolded. denotes Name minus source-input score. denotes Bonferroni-adjusted -values, correcting separately across three Task 1 and 12 Tasks 2–3 comparisons.
Complementary model-harness performance metrics
Figure S4. Complementary performance metrics across models and harnesses. Heatmaps show performance across the ten protocols and nine model–harness systems used in Figure 3b. System and protocol orders are fixed to the molecular-transition-F1 clustering in the main figure to permit direct comparison across metrics. All color scales range from 0 to 1.
Cost analysis and model-harness effects
Model-harness effect analysis. To test the differences of model and harness selection on reconstruction performance, we compared GPT, Claude, and Gemini in a fully crossed design with Pi and mini-SWE-agent, using one execution per configuration on 10 common protocols (excluding ddSEQ for Task 2). For protocol , model , and harness , let denote F1. Model- and harness-switch magnitudes were , . The factors 1/6 and 1/3 average over the three unique model pairs within each of two harnesses, and the single harness pair within each of three models, respectively. We estimated the mean of and its pointwise 95% CI using 10,000 paired protocol-bootstrap resamples. Mean model-switch magnitudes exceeded harness-switch magnitudes across all four metrics, but only molecular-state F1 showed a significant difference (, 95% CI [0.038, 0.124], ; Figure S6).
Figure S6. Model-harness effects. Mean absolute switch magnitudes (left) and paired model-minus-harness contrasts (right). Intervals are pointwise 95% protocol-bootstrap CIs, conditional on the selected executions. The state annotation gives for the primary four-metric comparison.
Secondary analyses and results. For each metric, we fitted a linear mixed model with model, harness, and their interaction as fixed effects and a protocol random intercept. Protocol-blocked repeated-measures ANOVA with Greenhouse-Geisser correction served as a robustness check. Bonferroni correction covered 12 omnibus tests (three effects × four metrics) within each analysis. Only molecular-state F1 showed a detectable model main effect (mixed model: , ; blocked analysis: ). State F1 averaged over the two harnesses was 0.655 for Claude, 0.612 for GPT, and 0.459 for Gemini. Neither analysis detected a harness main effect or model-harness interaction for any metric. These secondary tests concern average scores; nonsignificance does not establish equivalence or negligible harness effects.
Baseline output failure modes
External knowledge for agent improvement
Trajectory-based behavioral changes across autonomous-learning checkpoints
To interpret the non-monotonic learning curve, we compared the agent-visible context, execution trajectories, and predictions at the baseline, , , and checkpoints on the same frozen five-protocol validation panel. For each checkpoint, we analyzed one selected execution per validation protocol. The observed changes support behavioral interpretation but do not alone establish the causal effect of an individual rule or retrieved example.
| Checkpoint | Learned context | Observed reconstruction behavior | Validation performance |
| Baseline | No learned rules or cross-protocol memory. | Direct source-to-output reconstruction followed by the benchmark validator. | Task 2 F1: 0.758. Task 3 transition, state, and typed-edge F1: 0.846, 0.625, and 0.822. |
| 18 learned rules and 30 available exemplar records. | The agent constructed an evidence inventory, retrieved selected analogies, and compiled Task 2 and Task 3 from a linked work record. The predicted graphs averaged 10.0 states, 7.2 transitions, and 16.2 typed edges. | Highest Task 2 F1 (0.821), with transition, state, and typed-edge F1 of 0.843, 0.603, and 0.790. Selected checkpoint for held-out evaluation. | |
| 23 learned rules and 40 exemplar records. | Two newly learned rules may have contributed to the observed decline. A rule excluding oligo-reagent fabrication from the workflow was associated with omitted bead-construction steps in HyDrop-RNA, while a rule separating co-processed but physically unconnected molecules was associated with the separation of the ddSEQ workflow into disconnected branches. These behaviors suggest that learned rules may generalize poorly to the benchmark’s canonical representation. | Task 2 F1 decreased to 0.736; transition, state, and typed-edge F1 decreased to 0.743, 0.497, and 0.662. | |
| 28 learned rules and 58 exemplar records. | The agent represented oligo chemistry, non-templated tail addition, modified bases, and uncertain fragment fates more explicitly. Graphs again became more granular. | State F1 partially recovered to 0.539, while Task 2, transition, and typed-edge F1 were 0.745, 0.720, and 0.638, respectively. |
Table S6. Behavioral comparison of selected autonomous-learning checkpoints. Learned rules are protocol-neutral instructions accumulated from previous training protocols; exemplar records are ground-truth-derived cross-protocol examples available for retrieval. Task 2 scores are macro-averaged over four informative protocols; ddSEQ has no required recoverable oligo families and is reported separately. Task 3 scores include all five validation protocols.
The Task 3 deterioration visible at was already apparent at , with pronounced losses for ddSEQ and HyDrop-RNA. The decline in informative Task 2 F1 also involved CEL-seq, whose score decreased from 0.606 at to 0.450 at .
For ddSEQ, a rule learned from a previous multi-omic protocol instructed the agent to keep co-processed but physically distinct molecules in separate workflows. In the observed trajectory, the agent separated the cDNA and deconvolution-oligo branches into two workflows, whereas the benchmark connects them through a shared fraction-separation transition. One predicted workflow was therefore left unmatched. From to , ddSEQ transition F1 decreased from 1.000 to 0.493, state F1 from 0.830 to 0.449, and typed-edge F1 from 1.000 to 0.364. Because ddSEQ has no required recoverable oligo families, it is excluded from the Task 2 macro-average. Both checkpoints produced zero effective spurious oligo families; produced one.
For HyDrop-RNA, a learned rule instructed the agent to begin the molecular workflow after oligo-reagent fabrication. In the observed reconstruction, the agent represented a completed bead construct instead of the separate Acrydite and split-pool barcode oligos, omitting early bead-construction states and transitions. The benchmark, however, explicitly scores these oligos and construction steps. From to , HyDrop-RNA Task 2 F1 decreased from 0.915 to 0.738, state F1 from 0.401 to 0.159, transition F1 from 0.666 to 0.587, and typed-edge F1 from 0.609 to 0.526. These cases are consistent with negative transfer or a mismatch between learned representation conventions and the canonical benchmark. The trajectories do not isolate the causal effect of an individual rule from execution variability.
At , the agent recovered some molecular-state detail but frequently used a finer event granularity than the canonical benchmark. For example, it represented MspI-end fill-in, dA addition, bisulfite denaturation, and bisulfite conversion as separate scRRBS events. This representation was associated with higher scRRBS state F1, increasing from 0.521 at to 0.712 at , while transition F1 decreased from 0.883 to 0.741 and typed-edge F1 from 0.696 to 0.615, reflecting additional unmatched transitions and edges. Similarly, the chATAC-seq output represented adapter reagents as independent molecular states, coinciding with lower state precision and typed-edge agreement.
Sensitivity analysis
Here we assessed the sensitivity of benchmark scores and system comparisons to similarity weights and matching thresholds used in Tasks 2 and 3. In total, we evaluated 640 distinct scoring configurations: one default configuration, three uniform-weight alternatives, 28 individual weight perturbations, 600 Dirichlet-sampled weight vectors, and eight additional threshold combinations. Weight and threshold variations were evaluated separately, rather than through a full factorial combination.
For each configuration, we recomputed the complete matching and scoring pipeline using the same saved agent predictions and ground-truth records, without additional agent executions. This captured changes in both partial-credit similarity scores and molecular-state or transition alignments. The resulting ranges characterize sensitivity to scoring assumptions rather than statistical confidence intervals.
Sensitivity to similarity weights. We examined three sets of scientific similarity weights for oligo-family, molecular-transition, and molecular-state similarity. Table S7 summarizes their default values and scientific rationale. The weights were chosen heuristically to prioritize molecular identity and relevant structural information, rather than estimated from experimental data.
| Metric | Default component weights | Scientific rationale |
| Oligo F1 | Sequence: 0.65; modifications: 0.15; molecular kind: 0.10; orientation: 0.05; role: 0.05. | Nucleotide sequence is central to oligo identity, while modifications and molecular annotations provide additional information. |
| Transition F1 | Operation: 0.30; substrates: 0.15; products: 0.20; product disposition: 0.15; linked oligos: 0.20. | Emphasizes the molecular operation and its products and linked oligos, while retaining information about substrates and product disposition. |
| State F1 | Reference sequence: 0.50; architecture: 0.15; ordered segments: 0.20; pairing/discontinuities: 0.15. | Prioritizes molecular sequence identity while accounting for strand architecture, segment organization, and pairing relationships. |
Table S7. Default component weights for pairwise scientific similarity. These similarities contribute to entity matching and subsequent F1 calculation. For oligo-family and molecular-state scoring, weights are renormalized over components enabled by the ground-truth support rules.
For each default weight vector (Table S7), we evaluated three types of alternatives: i) uniform weights ( for each of components); ii) one-at-a-time relative perturbations of individual components, followed by renormalization; and iii) 200 joint perturbations of all components, sampled from , with . Here, controls the dispersion of sampled weights around the default vector . The concentration parameter was chosen heuristically to explore moderate departures from the default weights. Each of the oligo, transition, and state weight vectors was varied separately, with all other weights and matching thresholds held fixed.
Figure S8 shows that absolute F1 scores varied with component weighting. For example, uniform molecular-state weights increased Gemini’s state F1 from 0.495 to 0.581 and Kimi’s from 0.500 to 0.594, while Claude remained the highest-scoring system for molecular-state reconstruction. The leading systems for oligo-family and molecular-state F1 remained unchanged across the tested weight configurations. Molecular-transition rankings were less stable: Gemini ranked first in 179 of 200 sampled transition-weight configurations, while GPT ranked first in the remaining 21. Uniform transition weights also favored GPT. These results indicate that the principal leading-system comparisons were largely robust to alternative similarity weights, although absolute F1 scores and the close Gemini-GPT transition comparison were sensitive to component weighting. The analysis supports the stability of our comparative findings without establishing that the default weights are biologically optimal.
Figure S8. Sensitivity of native-system performance to similarity weights. Macro-averaged oligo-family, molecular-transition, and molecular-state F1 scores for four native coding-agent systems. The displayed ranges characterize sensitivity to scoring assumptions, not statistical confidence intervals.
Sensitivity to matching thresholds. Next we assessed sensitivity to molecular-state and transition assignment thresholds, both set to 0.25 by default. Each threshold was independently varied over , yielding nine combinations. Thresholds were applied to context-adjusted assignment scores rather than raw scientific similarity scores. All similarity weights were kept at their default values, and matching and scoring were recomputed for each combination.
Table S8 summarizes the maximum absolute changes in macro-averaged F1 across the tested thresholds and systems or conditions within each experiment. For the native-system comparison (Figure 3a), the maximum changes in transition, state, and typed-edge F1 were 0.035, 0.031, and 0.069, respectively.
| Experiment | Transition F1 | State F1 | Typed-edge F1 |
| Fig. 3a: Native systems | 0.035 | 0.031 | 0.069 |
| Fig. 3b: Model-harness comparison | 0.023 | 0.020 | 0.045 |
| Fig. 4a: External knowledge | 0.029 | 0.013 | 0.054 |
| Fig. 4c: Learning validation | 0.055 | 0.025 | 0.132 |
| Fig. 4d: Held-out evaluation | 0.018 | 0.009 | 0.031 |
Table S8. Sensitivity to state and transition assignment thresholds. Maximum absolute change in protocol-macro F1 relative to the default thresholds (0.25, 0.25), across nine threshold combinations and all systems or conditions within each experiment.
Figure S9 illustrates threshold sensitivity in the autonomous-learning validation experiment (Figure 4c), which exhibited the largest changes in transition and typed-edge F1 (0.055 and 0.132, respectively). Notably, increasing only the state-matching threshold from 0.25 to 0.40 produced a maximum typed-edge F1 change of 0.117, compared with 0.025 for state F1.
Figure S9. Sensitivity of molecular-workflow scores to matching thresholds. Heatmaps show the maximum absolute changes in molecular-transition, molecular-state, and typed-edge F1 across autonomous-learning validation checkpoints relative to their respective scores under the default thresholds. State and transition assignment thresholds were independently varied over . Each cell represents the maximum across checkpoints, rather than the average change. The default combination is outlined.
Overall, transition and state F1 were less sensitive to the tested matching thresholds than typed-edge F1. This likely reflects the dependence of graph-connectivity scoring on molecular-state alignment, where changes in matched states can propagate to the evaluation of associated edges. These findings suggest that graph-topology scores should be interpreted alongside molecular-state alignment and the canonical graph representation.
Robustness of knowledge augmentation and cumulative learning. We further investigated whether the conclusions from the external-knowledge and autonomous-learning experiments were preserved under alternative scoring assumptions (Figure S10). In the external-knowledge pilot (Figure 4a), combining general knowledge with cross-protocol examples consistently increased oligo-family F1 relative to baseline (range: 0.022–0.129) across the tested scoring configurations. However, baseline retained the highest mean scores across all three Task 3 metrics. In the cumulative-learning validation experiment (Figure 4c), performance remained non-monotonic across all tested configurations. The final-checkpoint oligo-family F1 difference () ranged from −0.021 to 0.015, indicating sensitivity to component weighting. In contrast, all three Task 3 endpoint differences remained negative. On the held-out protocols (Figure 4d), supplementing the selected checkpoint with the knowledge base produced consistently positive oligo-family F1 differences relative to baseline (range: 0.049–0.075). However, the pointwise 95% protocol-bootstrap confidence interval under default scoring included zero (). Both evaluated conditions remained below baseline in mean transition, state, and typed-edge F1 across the tested scoring configurations.
Figure S10. Robustness of knowledge augmentation and autonomous-learning results. Differences in F1 relative to baseline for combined external knowledge versus baseline; final checkpoint versus on the learning-validation panel; and with knowledge-base supplementation versus on held-out protocols.
Together, these results indicate that the absence of improvement in canonical molecular-workflow reconstruction was robust to the tested scoring assumptions. However, the final-checkpoint oligo-family F1 comparison was sensitive to component weighting.
Limitations. These sensitivity analyses used fixed agent predictions and therefore assess robustness to scoring assumptions, rather than execution-to-execution variability (Appendix § 9). Robustness across the tested configurations does not establish that the default weights are biologically optimal or that canonical graph matching accommodates all chemically valid representations. Furthermore, the model-versus-harness statistical tests were not repeated across all scoring configurations.
References
- [1]10x Genomics. Chromium GEM-X Single Cell 3’ v4 Gene Expression User Guide. 10x Genomics, 2024. URL https://www.10xgenomics.com/support/universal-three-prime-gene-expression/documentation/steps/library-prep/chromium-gem-x-single-cell-3-v4-gene-expression-user-guide.
- [2]Afra Feyza Akyürek, Xinming Tu, Alec Gutmanstein, Jason Qin, Divyansh Agarwal, Sofia Monasdოტter, Sergey Chekhov, Brenda Hernandez Villegas, Kirill Chugunov, Judah Engel, Veronica Chatrath, Oscar Kavanagh, Geobio Boo, Ernesto Hernandez, Ying Liu, Yuan Xue, Aakash Sabharwal, Daniel Yue Zhang, Zainab Doctor, Yuanhao Qu, Yunzhong He, and Sami Hassaan. DrugDiscoveryBench: Can coding agents assist early-stage drug discovery? Scale Labs and Phylo research report, 2026. URL https://labs.scale.com/leaderboard/drugiscoverybench.
- [3]Artifex Software, Inc. PyMuPDF documentation, 2026. URL https://pymupdf.readthedocs.io/en/latest/.
- [4]John M. S. Bartlett and David Stirling, editors. PCR Protocols, volume 226 of Methods in Molecular Biology. Humana Press, Totowa, NJ, 2 edition, 2003. ISBN 978-0-89603-642-0.
- [5]Bio-Rad Laboratories. ddSEQ Single-Cell 3’ RNA-Seq Kit User Guide. Bio-Rad Laboratories, 2024. URL https://www.bio-rad.com/sites/default/files/2024-06/10000167449.pdf. Version 1.0, Ver. B.
- [6]Ali Sina Booeshaghi, Xi Chen, and Lior Pachter. A machine-readable specification for genomics assays. Bioinformatics, 40(4):btae168, 2024. doi: 10.1093/bioinformatics/btae168.
- [7]Junyue Cao, Jonathan S. Packer, Vijay Ramani, Darren A. Cusanovich, Chau Huynh, Riza Daza, Xiaojie Qiu, Choli Lee, Scott N. Furlan, Frank J. Steemers, Andrew Adey, Robert H. Waterston, Cole Trapnell, and Jay Shendure. Comprehensive single-cell transcriptional profiling of a multicellular organism. Science, 357(6352):661–667, 2017. doi: 10.1126/science.aam8940.DOI
- [8]Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific discovery. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2410.05080.
- [9]Darren A. Cusanovich, Riza Daza, Andrew Adey, Hannah A. Pliner, Lena Christiansen, Kevin L. Gunderson, Frank J. Steemers, Cole Trapnell, and Jay Shendure. Multiplex single cell profiling of chromatin accessibility by combinatorial cellular indexing. Science, 348(6237):910–914, 2015. doi: 10.1126/science.aab1601.DOI
- [10]Florian Erhard, Antoine-Emmanuel Saliba, Alexandra Lusser, Christophe Toussaint, Thomas Hennig, Bhupesh K. Prusty, Daniel Kirschbaum, Kathleen Abadie, Eric A. Miska, Caroline C. Friedel, Ido Amit, Ronald Micura, and Lars Dölken. Time-resolved single-cell RNA-seq using metabolic RNA labelling. Nature Reviews Methods Primers, 2:77, 2022. doi: 10.1038/s43586-022-00157-z. URL https://doi.org/10.1038/s43586-022-00157-z.
- [11]Mathieu Fenniak, Matthew Stamy, pypub zz, Martin Thoma, Matthew Peveler, exile dkingcc, and pypdf Contributors. The pypdf library, 2024. URL https://pypi.org/project/pypdf/. See the pypdf documentation for the complete contributor list.
- [12]Charles Gawad, Winston Koh, and Stephen R. Quake. Single-cell genome sequencing: current state of the science. Nature Reviews Genetics, 17:175–188, 2016. doi: 10.1038/nrg.2015.16. URL https://doi.org/10.1038/nrg.2015.16.
- [13]Ali E. Ghareeb, Benjamin Chang, Ludovico Mitchener, et al. A multi-agent system for automating scientific discovery. Nature, 655:497–505, 2026. doi: 10.1038/s41586-026-10652-y.DOI
- [14]Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, et al. Accelerating scientific discovery with co-scientist. Nature, 655:487–496, 2026. doi: 10.1038/s41586-026-10644-y.DOI
- [15]Hongshan Guo, Ping Zhu, Fan Guo, Xianlong Li, Xinglong Wu, Xiaoying Fan, Lu Wen, and Fuchou Tang. Profiling dna methylome landscapes of mammalian cells with single-cell reduced-representation bisulfite sequencing. Nature Protocols, 10:645–659, 2015. doi: 10.1038/nprot.2015.039.
- [16]Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments. Software framework, 2026. URL https://github.com/harbor-framework/harbor.DOI
- [17]Adrian J. Harwood, editor. Protocols for Gene Analysis, volume 31 of Methods in Molecular Biology. Humana Press, Totowa, NJ, 1994. ISBN 978-0-89603-258-3. doi: 10.1385/0896032582. URL https://doi.org/10.1385/0896032582.
- [18]Kexin Huang, Serena Zhang, Hanchen Wang, Yuanhao Qu, Yingzhou Lu, Ryan Li, Yusuf Roohani, Lin Qiu, Shiyi Cao, Gavin Li, Junze Zhang, Di Yin, Rick Wierenga, Deniz Kavi, Sherry Liu, Tianwei He, Shruti Marwaha, Jennefer N. Carter, Xin Zhou, Matthew T. Wheeler, Jonathan A. Bernstein, Mengdi Wang, Peng He, Jingtian Zhou, Michael P. Snyder, Le Cong, Aviv Regev, and Jure Leskovec. Autonomous biomedical research with an artificial intelligence agent. Science, 393(6813):eadz4351, 2026. doi: 10.1126/science.adz4351.
- [19]Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques. LAB-Bench: Measuring capabilities of language models for biology research. arXiv preprint arXiv:2407.10362, 2024.arxiv.org/abs/2407.10362
- [20]Nikolaos Livathinos, Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Panos Vagenas, Cesar Berrospi Ramis, Matteo Omenetti, Kasper Dinkla, Yusik Kim, Shubham Gupta, Rafael Teixeira de Lima, Valery Weber, Lucas Morin, Ingmar Meijer, Viktor Kuropiatnyk, and Peter W. J. Staar. Docling: An efficient open-source toolkit for ai-driven document conversion. arXiv preprint arXiv:2501.17887, 2025.arxiv.org/abs/2501.17887
- [21]Harvey Lodish, Arnold Berk, Paul Matsudaira, Chris A. Kaiser, Monty Krieger, Matthew P. Scott, S. Lawrence Zipursky, and James Darnell. Molecular Cell Biology. W. H. Freeman, New York, 5 edition, 2003. ISBN 978-0-7167-4366-8.
- [22]Sai Ma, Bing Zhang, Lindsay M. LaFave, Andrew S. Earl, Zachary Chiang, Yan Hu, Jiarui Ding, Alison Brack, Vinay K. Kartha, Tristan Tay, Travis Law, Caleb Lareau, Ya-Chieh Hsu, Aviv Regev, and Jason D. Buenrostro. Chromatin potential identified by shared single-cell profiling of rna and chromatin. Cell, 183(4):1103–1116.e20, 2020. doi: 10.1016/j.cell.2020.09.056.DOI
- [23]Evan Z. Macosko, Anindita Basu, Rahul Satija, James Nemesh, Karthik Shekhar, Melissa Goldman, Itay Tirosh, Allison R. Bialas, Nolan Kamitaki, Emily M. Martersteck, John J. Trombetta, David A. Weitz, Joshua R. Sanes, Alex K. Shalek, Aviv Regev, and Steven A. McCarroll. Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets. Cell, 161(5):1202–1214, 2015. doi: 10.1016/j.cell.2015.05.002.DOI
- [24]Liesbeth Minnoye, Georgi K. Marinov, Thomas Krausgruber, Lixia Pan, Alexandre P. Marand, Stefano Secchia, William J. Greenleaf, Eileen E. M. Furlong, Keji Zhao, Robert J. Schmitz, Christoph Bock, and Stein Aerts. Chromatin accessibility profiling methods. Nature Reviews Methods Primers, 1:10, 2021. doi: 10.1038/s43586-020-00008-9. URL https://doi.org/10.1038/s43586-020-00008-9.
- [25]Ryan M. Mulqueen, Dmitry Pokholok, Brendan L. O’Connell, Casey A. Thornton, Fan Zhang, Brian J. O’Roak, Jason Link, Galip G. Yardimci, Rosalie C. Sears, Frank J. Steemers, and Andrew C. Adey. High-content single-cell combinatorial indexing. Nature Biotechnology, 39:1574–1580, 2021. doi: 10.1038/s41587-021-00962-z.DOI
- [26]Ferdous Nasri, Sarah Gurev, Patrick Varilly, Krithik Ramesh, Nuala A. O’Leary, Jonah Cool, Bernhard Y. Renard, Pardis C. Sabeti, and Laura Luebbert. Deterministic access to global viral sequence data enables robust agentic scientific discovery, 2026.arxiv.org/abs/2606.06749
- [27]Sanjay M. Prakadan, Alex K. Shalek, and David A. Weitz. Scaling by shrinking: empowering single-cell ‘omics’ with microfluidic devices. Nature Reviews Genetics, 18:345–361, 2017. doi: 10.1038/nrg.2017.15. URL https://doi.org/10.1038/nrg.2017.15.
- [28]Yuanhao Qu, Yingzhou Lu, Xinming Tu, Serena Zhang, Tianwei She, Alexander Glenn Shaw, et al. BiomniBench: Process-level evaluation of llm agents for real-world biomedical research. bioRxiv, 2026. doi: 10.64898/2026.05.12.724604.DOI
- [29]Daniel Ramsköld, Shujun Luo, Yu-Chieh Wang, Robin Li, Qiaolin Deng, Omid R. Faridani, Gregory A. Daniels, Irina Khrebtukova, Jeanne F. Loring, Louise C. Laurent, Gary P. Schroth, and Rickard Sandberg. Full-length mrna-seq from single-cell levels of rna and individual circulating tumor cells. Nature Biotechnology, 30:777–782, 2012. doi: 10.1038/nbt.2282.DOI
- [30]Alexander B. Rosenberg, Charles M. Roco, Richard A. Muscat, Anna Kuchina, Paul Sample, Zizhen Yao, Lucas T. Graybuck, David J. Peeler, Sumit Mukherjee, Wei Chen, Suzie H. Pun, David L. Sellers, Bosiljka Tasic, and Georg Seelig. SPLiT-seq reveals cell types and lineages in the developing brain and spinal cord. Science, 360(6385):176–182, 2018. doi: 10.1126/science.aam8999.
- [31]Omer Schwartzman and Amos Tanay. Single-cell epigenomics: techniques and emerging applications. Nature Reviews Genetics, 16:716–726, 2015. doi: 10.1038/nrg3980. URL https://doi.org/10.1038/nrg3980.
- [32]Rory Stark, Marta Grzlak, and James Hadfield. RNA sequencing: the teenage years. Nature Reviews Genetics, 20:631–656, 2019. doi: 10.1038/s41576-019-0150-2. URL https://doi.org/10.1038/s41576-019-0150-2.
- [33]Dave T. P. Tang, Charles Plessy, Md Salimullah, Ana Maria Suzuki, Raffaella Calligaris, Stefano Gunstincich, and Piero Carninci. Suppression of artifacts and barcode bias in high-throughput transcriptome analyses utilizing template switching. Nucleic Acids Research, 41(3):e44, 2013. doi: 10.1093/nar/gks1128. URL https://doi.org/10.1093/nar/gks1128.
- [34]Katy Vandereyken, Alejandro Sifrim, Bernard Thienpont, and Thierry Voet. Methods and applications for single-cell and spatial multi-omics. Nature Reviews Genetics, 24(8):494–515, 2023. doi: 10.1038/s41576-023-00580-2. URL https://doi.org/10.1038/s41576-023-00580-2.
- [35]Yuxuan Wan, Tianqing Fang, Zaitang Li, Yintong Huo, Wenxuan Wang, Haitao Mi, Dong Yu, and Michael R. Lyu. Inference-time scaling of verification: Self-evolving deep research agents via test-time rubric-guided verification. In Findings of the Association for Computational Linguistics: ACL 2026, pages 24822–24835, 2026. doi: 10.18653/v1/2026.findings-acl.1243.DOI
- [36]Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, Bo Zhang, Liqun Wei, Zihao Sui, Wei Li, Botian Shi, Yu Qiao, Dahua Lin, and Conghui He. MinerU: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839, 2024.arxiv.org/abs/2409.18839
- [37]Jiongxiao Wang, Qiaojing Yan, Yawei Wang, Yijun Tian, Soumya Smruti Mishra, Zhichao Xu, Megha Gandhi, Panpan Xu, and Lin Lee Cheong. Reinforcement learning for self-improving agent with skill library. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, pages 1529–1550, 2026. doi: 10.18653/v1/2026.acl-long.69.DOI
- [38]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R. Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, volume 37, 2024. doi: 10.52202/tns.2022.1701.arxiv.org/abs/2405.15793
- [39]Mario Zechner. Pi: A minimal coding-agent harness and agent runtime. https://github.com/badlogic/pi-mono, 2026. Software.











