betaFeedback

WorksComputer Science

AI scientists produce results without reasoning scientifically

Abstract

Large language model (LLM)-based systems are increasingly deployed to conduct scientific research autonomously, yet whether their reasoning adheres to the epistemic norms that make scientific inquiry self-correcting is poorly understood. Here, we evaluate LLM-based scientific agents across eight domains, spanning workflow execution to hypothesis-driven inquiry, through more than 25,000 agent runs and two complementary lenses: (i) a systematic performance analysis that decomposes the contributions of the base model and the agent scaffold, and (ii) a behavioral analysis of the epistemological structure of agent reasoning. We observe that the base model is the primary determinant of both performance and behavior, accounting for 41.4% of explained variance versus 1.5% for the scaffold. Across all configurations, evidence is ignored in 68% of traces, refutation-driven belief revision occurs in 26%, and convergent multi-test evidence is rare. The same reasoning pattern appears whether the agent executes a computational workflow or conducts hypothesis-driven inquiry. They persist even when agents receive near-complete successful reasoning trajectories as context, and the resulting unreliability compounds across repeated trials in epistemically demanding domains. Thus, current LLM-based agents execute scientific workflows but do not exhibit the epistemic patterns that characterize scientific reasoning. Outcome-based evaluation cannot detect these failures, and scaffold engineering alone cannot repair them. Until reasoning itself becomes a training target, the scientific knowledge produced by such agents cannot be justified by the process that generated it.

Introduction

DENDRAL,1 one of the first artificial intelligence (AI) systems to produce a scientific result, showed its reasoning at every step. Developed by Feigenbaum, Lederberg, and Djerassi in 1965, it inferred molecular structures from mass spectra by generating candidate structures, matching them against spectral evidence, and eliminating those that failed. DENDRAL established a template for AI in science: epistemic machinery encoded in explicit rules, every inference step inspectable. Expert systems built on this template remain in use in laboratories today. Large language models (LLMs) have renewed this ambition. A growing number of LLM-based agents are now designed to conduct scientific research autonomously, from hypothesis generation through experimentation to manuscript writing (see Section A.6 for the rising interest in those systems).[2, 3, 4, 5, 6, 7, 8, 9]

Yet the nature of these systems differs fundamentally from their predecessors. Where DENDRAL showed its reasoning at every step, LLM-based agents do not. Their behavior derives from statistical regularities learned during self-supervised training. The epistemic process by which they arrive at scientific conclusions is largely inaccessible to scrutiny.

This opacity matters for science itself. The tools scientists use shape the science they produce.[10] Evans and colleagues recently confirmed this for AI: the adoption of machine learning (ML) is narrowing the hypotheses scientists pursue.[11] LLM-based agents go further, directing the inquiry itself. How these systems reason, and whether their reasoning adheres to the norms that make scientific inquiry self-correcting, spans computer science, philosophy of science, and metascience.[12, 13] It bears on whether AI systems can produce scientific understanding beyond correct predictions.[14]

Current evaluations do not address this. Benchmarks for scientific agents measure task completion: whether a system answers domain-specific questions,[15, 16, 17] executes a data-analysis or discovery workflow,[9, 18, 19, 20] reimplements a published result, or produces a manuscript that passes peer review.[21, 22, 23, 24, 25] These metrics are task-specific. They reveal whether an agent succeeded but not how or why, and they do not predict behavior on new problems.[26] The epistemic process underlying a result determines whether the knowledge produced is justified in the philosophical sense of justified true belief,[27] whether failures are predictable, and whether the system can be trusted beyond the conditions under which it was tested.

Here, we study LLM-based scientific agents, also referred to as AI scientists, through two complementary lenses. The first is a systematic performance evaluation across controlled environments with standardized tools, graded task scopes, and fine-grained subtask diagnostics, designed to separate the contributions of the base model from the agent scaffold.[28] The second is a behavioral analysis of the epistemological structure of agent reasoning: what cognitive operations an agent performs, how they relate to one another, and whether they constitute disciplined inquiry. We evaluate three frontier models paired with two scaffold architectures across eight scientific domains, from molecular simulation and surface construction to spectroscopic structure elucidation and qualitative chemical analysis (see Figure 1). We find that the base model is the primary determinant of both performance and behavior, with the scaffold contributing little. Agents routinely ignore evidence they have gathered, commit to hypotheses without testing them, and fail to revise beliefs when confronted with contradictory data. These patterns do not adapt to the demands of the problem. The same undisciplined reasoning appears whether the task requires executing a computational workflow or conducting a hypothesis-driven inquiry under uncertainty. These findings suggest that evaluating LLM-based agents requires direct assessment of their reasoning process, and that progress will require changes at the base-model level rather than further scaffold engineering.

Image not extracted

Figure 1. Benchmarking scientific reasoning across epistemic demand and problem scope. (A) Agents as language decision processes. A base LLM policy is augmented with an agent scaffold (e.g., reasoning, reflection, tool use) to form an agent that interacts with scientific environments via iterative act-observe loops. (B) Large-scale evaluation grounds our findings. The benchmark comprises 25,000+ agent runs across 15+ environment scopes and 90+ tools, enabling statistically grounded evaluation of model and scaffold contributions. (C) Mechanistic and epistemological analyses reveal failure modes. We analyze agent behavior through mechanistic analysis of trajectories and epistemological analysis of reasoning processes. (D) Environments are organized along an axis of increasing epistemic demand (left to right). Workflow execution tasks (e.g., Molecular Simulation, Adsorption Surface Construction, ML-based Property Prediction, AFM Experiment Execution) emphasize procedural correctness; strategic reasoning tasks (e.g., Retrosynthetic Planning) require multi-step planning under constraints; hypothesis-driven inquiry tasks (e.g., Spectroscopic Structure Elucidation, Inorganic Qualitative Analysis, Circuit Inference) require generating, testing, and revising competing hypotheses. Within each domain, task scope increases through graded configurations (sn→sn+1\mathrm{sn} \to\mathrm{sn+1}) that expand the search space. Axis labels WE and E mark workflow-execution and evidence anchors; H, G, and P mark hypothesis, goal, and protocol anchors; T, J, and U denote test, judge, and update operations.

Results

We model LLM-based agents as language decision processes,[29, 30] in which a composite policy factors into a base-model component governing token-level choices and a scaffold component governing prompting strategy, tool routing, and orchestration logic (Figure 1A, Methods Section 4.1). In practice, a scaffold wraps the base model in a loop: it formats prompts, routes tool calls, appends observations to the conversation history, and repeats until a termination criterion is reached. Common implementations include ReAct[31] and structured tool-calling interfaces. This decomposition allows us to attribute observed behavior to the base model or to the scaffold. We evaluate agents through two lenses: a systematic performance analysis supported by manually curated diagnostic assessments, and a behavioral analysis of the epistemological structure of reasoning traces.

We constructed environments in eight scientific domains that vary along two dimensions: epistemic demand and task scope (Figure 1D, § 4.4). Epistemic demand captures the kind of reasoning a problem requires, ranging from workflow to hypothesis-driven tasks. Workflow-construction tasks include molecular simulation, surface construction, machine learning, and atomic force microscopy, which are procedurally complex but epistemically straightforward—the solution path is well-defined and the agent must execute it correctly. Hypothesis-driven tasks in spectroscopic structure elucidation, qualitative chemical analysis, and circuit inference are different. The agent must first understand the problem space itself, then form hypotheses about hidden structure, design discriminating experiments, and revise beliefs as evidence accumulates. Retrosynthetic planning occupies an intermediate position, requiring strategic search through large combinatorial spaces under cost and availability constraints. Problem scope varies within each domain through graded configurations that expand problem scope by modulating the search space: from a few to many interacting ions, from simple to complex resistor networks, from spectral confirmation to full elucidation, from measuring friction to (re)discovering power-law behavior. Each environment further decomposes into subtasks isolating specific cognitive operations—retrieval, execution, reasoning, and validation (more details in § 4.2).

Within each group, specific environments were designed to isolate aspects of scientific reasoning that are difficult to probe with existing benchmarks. The qualitative analysis environment simulates chemical equilibria from thermodynamic data, requiring iterative experimental design under limited sample volume. The circuit inference environment isolates logical reasoning from domain knowledge. The spectroscopic structure elucidation environment poses the same class of problem that motivated DENDRAL—inferring molecular identity from spectral evidence—with access to multiple analytical techniques and a cost-information tradeoff. All environments share a common implementation framework with standardized tool interfaces and can be explored interactively at https://lamalab-org.github.io/corral/#environments.

The base model determines performance

We evaluated three frontier (two commercial closed-source and one open-source) models—GPT-4o (gpt-4o-2024-08-06), Claude Sonnet 4.5 (claude-sonnet-4-5-20250929), and GPT-OSS (gpt-oss-120b)—each paired with two scaffold architectures (ReAct and structured tool-calling), across all eight domains and their task scopes (Figure 2; per-configuration scores in Tables G.8 and G.9).

Image not extracted

Figure 2. Performance is primarily driven by model choice and degrades with epistemic demand. (A) Heatmap of average scores for all model-scaffold configurations across environment-scope combinations, grouped by epistemic demand category. (B) Average Pass@5 across subtask categories ordered by increasing cognitive complexity. Both scaffolds exhibit the same decline from retrieval to validation. (C) Model spread (performance range across models) versus scaffold spread (performance range across scaffolds) for each environment. Nearly all environments fall above the diagonal, indicating that model choice, not scaffold design, is the primary source of performance variation.

Agents solve workflow-construction tasks at guided scopes reliably: in molecular simulation and surface construction, leading configurations approach the performance ceiling (see Figure 2). Performance declines as the problem scope broadens within a domain and declines more steeply as epistemic demand increases. In Inorganic Qualitative Analysis and Spectroscopic Structure Elucidation at the broadest scope, even the strongest configuration falls below 60%. Across nearly all environments, the performance spread attributable to model choice exceeds the spread attributable to scaffold choice (Figure 2C). Both scaffolds yield the same decline in Pass@5 from retrieval through execution and reasoning to validation (Figure 2B). Manual annotation of reasoning traces by two expert annotators, using a custom tool and a taxonomy developed for this study (Section F), independently confirms this pattern (Extended Data Figure 2).

To move beyond task-level performance and identify the underlying capabilities that drive it, we constructed a diagnostic battery of question-answer pairs for each domain, separately targeting the knowledge and reasoning operations associated with each environment (§ 4.6). We applied item response theory (IRT) to these diagnostic items.[32, 33] IRT jointly estimates item difficulty and respondent ability from patterns of correct and incorrect answers, yielding latent ability parameters for each model-environment combination (Figure 3A, Section G.4). The resulting capability profiles differ across models (Figure 3B). The separation is most pronounced for reasoning ability in hypothesis-driven domains, where the gap between the strongest and weakest model exceeds 5.3 standard units in Inorganic Qualitative Analysis (see Figure G.6 for all environments). In Retrosynthetic Planning and Spectroscopic Structure Elucidation, two of the three models effectively lack the domain knowledge required. The ML-based Property Prediction environment shows the smallest

Image not extracted

Figure 3. Reasoning ability is the dominant predictor of task success. (A) A two-stage latent-factor framework separates knowledge and reasoning contributions to task success. Stage 1: item response theory estimates latent knowledge and reasoning abilities from domain-specific diagnostic assessments. Stage 2: a prediction model combines these ability estimates with environment, scope, scaffold, and verbosity to predict task success probability. (B) Models diverge most in hypothesis-driven domains. Reasoning capability profiles show the largest separation in Inorganic Qualitative Analysis and Spectroscopic Structure Elucidation. (C) Reasoning ability dominates; scaffold choice is negligible. Variance decomposition of the prediction model: reasoning ability (41.4%) and environment–scope interaction (30.1%) together account for over 70% of explained variance. Scaffold (1.5%) and verbosity (0.1%) contribute minimally.

knowledge gap (0.62 standard units, see Figure G.6 for knowledge capabilities), consistent with the broad coverage of machine-learning topics in LLM training data. We combined these IRT-derived abilities with environment, scope, scaffold, and verbosity-level (see Methods for details) as parameters in a latent factor model predicting task success (Figure 3A). We evaluated eight candidate model specifications of increasing structural complexity (M1–M8; Section G.5.1) and selected the best-fitting model (M7) by PSIS-LOO cross-validation (Section G.5, Figure G.9). The variance decomposition is clear (Figure 3C). Reasoning ability accounts for 41.4% of the explained variance, environment scope for 30.1%, scaffold for 1.5% and verbosity for 0.1%. All three independent analyses (raw performance comparisons, manual trace annotation, and latent factor modeling) converge on the same conclusion: the base model determines the performance of the agent, and the scaffold modulates the interaction only marginally.

Epistemological structure of agent reasoning

To examine how agents reason, we analyzed the full agent traces. We classified each step in an agent’s trajectory according to the epistemic operation it performs: forming a hypothesis (H\mathrm{H}), gathering evidence (E\mathrm{E}), testing a prediction (T\mathrm{T}), justifying a conclusion (J\mathrm{J}), updating a belief (U\mathrm{U}), or committing to a hypothesis (C\mathrm{C}). We then represented the dependencies among these operations as a directed graph in which each node is an epistemic operation and each edge a dependency or reference between operations (Figure 4A). We observe that different reasoning modes produce different structural signatures. A Popperian cycle of conjecture and refutation generates loops between H\mathrm{H}, T\mathrm{T}, and U\mathrm{U} nodes. Abductive inference produces converging structures where multiple E\mathrm{E} nodes feed into H\mathrm{H}. A linear chain of tool calls with no back-references produces a path graph with minimal branching (see Methods Section 4.8 for more details and Section H.1 for the full annotation protocol).

Image not extracted

Figure 4. Reasoning breakdowns dominate across all domain groups. (A) Epistemological annotation process. Agent traces are annotated in two stages using the model Claude Sonnet 4.5: first, each step is labeled with an epistemic operation (hypothesis, test, evidence, judgment, update, commitment); second, directed edges capture dependencies between operations. The annotation process is described in detail in Section H.1. (B) Frequency of each pattern across domain groups. Bar plots show that reasoning-breakdown rates exceed productive-motif rates in every domain group; detailed scores are in Table H.16.

Reasoning breakdowns predominate. We compiled a catalog of productive reasoning patterns and anti-patterns, defined as structural templates over the annotated graphs (see Table H.14 and Table H.15 for definitions and illustration of graphs), and measured their prevalence across traces (Figure 4B). Untested claims, hypotheses stated without experiments designed to test them, appear in 53% of traces overall and 63% in hypothesis-driven domains. Evidence non-uptake, gathering a result and proceeding without incorporating it, occurs in 68% of traces. In 71% of traces, beliefs are never updated. Only 26% of traces exhibit refutation-driven belief revision. Convergent multi-test evidence, multiple independent lines of evidence brought to bear on a single hypothesis, occurs rarely (7%). Evidence-led hypothesis generation occurs in 65% of traces but coexists with the high rates of evidence non-uptake and untested claims. While abductive reasoning occurs, downstream testing and revision, in most traces, do not. Representative trace excerpts for each breakdown category are shown in Extended Data Figure 4; the annotated traces are browsable at https://lamalab-org.github.io/corral/#explainers (see Section H.7).

Image not extracted

Extended Data Figure 4. Autonomous agents repeatedly fail to correct flawed reasoning: they ignore accessible evidence, leave hypotheses untested, cling to false beliefs, and accept contradictions. Each panel lists model, environment, scope, and trial index. Evidence non-uptake (top left): Agent retrieves 20 isomers (including the correct one) but never consults the list, guessing instead. Untested claim (top right): From proton NMR, the agent infers methylene groups, acetyl, and benzene ring, but designs no test, skipping to mass spectrometry. Fixed belief trace (bottom left): LAMMPS simulation: agent mistakes /potentials/SW (a directory) for a file, blames “lost atoms” on heating rate, and never revises any of six hypotheses across three errors. Contradiction without repair (bottom right): Agent’s proposed isopropyl ester predicts 6H6\mathrm{H} doublet at 1.46 ppm (NMR) vs. experimental 3H3\mathrm{H} at 1.43 ppm; agent notes the discrepancy but keeps structure as final answer. Full annotated traces are browsable at https://lamalab-org.github.io/corral/#explainers (see also Section H.7).

Reasoning does not adapt to epistemic demand. Agents do not adapt their reasoning to the task, as a human practitioner would. Ion identification rewards hypothesis-test cycles, retrosynthesis rewards combinatorial tree search, and molecular simulation rewards a simulate-validate-refine loop; yet the topology of agent reasoning is the same across all three. Evidence non-uptake rates are 82% in workflow domains, 66% in strategic reasoning, and 60% in hypothesis-driven domains (Figure 4C). Convergent multi-test evidence remains at or below 10% in every domain group. Increasing problem scope within a domain produces no corresponding shift in the agent’s epistemic operations (Figure H.18). Agents apply a single reasoning mode across the full epistemic range. This mode is adequate for workflow execution but not for the disciplined treatment of evidence that hypothesis-driven inquiry requires (Figure 2).

This invariance extends to the model level. Models differ in the raw frequency of individual operations, with stronger models producing more H and E nodes per trace, but the topology of their reasoning graphs is similar. Higher-performing models retrieve more relevant knowledge and execute procedures more accurately. The epistemic structure of their reasoning does not change correspondingly.

Trace interventions and reliability

The reasoning breakdowns documented above raise the question of whether they can be mitigated by providing agents with better context. To test this, we designed a trace intervention experiment that provides an extreme form of scaffolding: injecting partial reasoning trajectories from prior runs into the agent’s conversation history before it begins a new trial (methodology in § 4.9). These interventions inject actual sequences of tool calls, observations, and reasoning steps: the content a successful agent would have produced on its own. We varied both the source (successful or failed prior runs) and the extent (early steps only, or near-complete trajectories) of the injected traces (Figure 5A).

Image not extracted

Figure 5. Scaffold interventions rescue workflow execution but not hypothesis-driven reasoning; failures in the latter recur across repeated trials. (A) Success rates under trace interventions. Agents receive a partial trajectory from a prior successful run (success trace; blue) or prior failed run (failed trace; red) before attempting a new trial. In workflow-construction environments (mean log-probability −0.20-0.20), one or two injected successful steps suffice to exceed baseline. In hypothesis-driven and strategic-reasoning environments (Spectroscopic Structure Elucidation, Inorganic Qualitative Analysis, Retrosynthetic Planning; mean log-probability ≈−0.27\approx-0.27), gains emerge only at near-complete trajectories (steps n−2n-2 or n−1n-1). Shaded regions show ±1\pm1 standard deviation. (B) Pass∧kPass^{\wedge k} (all kk trials succeed) and Pass@kPass@k curves by environment group. In hypothesis-driven domains, Pass∧kPass^{\wedge k} falls below 0.10.1 by k=4k=4 under baseline conditions, and below 0.050.05 by k=4k=4–66 even with success-trace interventions at early steps. (C) Mean token-level log-probability per domain group, ordered by epistemic demand. Domains where partial interventions fail show the most negative log-probabilities, linking intervention unrecoverability to low token-level confidence.

The effectiveness of these interventions tracks epistemic demand. We compare each intervention to a no-injection baseline (§ 4.9) and relate the results to token-level GPT-OSS-120B log-probabilities (§ 4.10). In workflow construction domains such as ML-based Property Prediction, where models exhibit high token-level confidence (mean log-probability −0.2-0.2 across the group; per-environment values in Figure G.14, Figure 5C), injecting one or two successful steps suffices to exceed baseline performance. Failed-trace interventions have little effect: success rates remain near baseline. In Spectroscopic Structure Elucidation, Inorganic Qualitative Analysis, and Retrosynthetic Planning, where token-level confidence is lower (mean log-probability −0.27-0.27), partial interventions do not improve performance. Gains emerge only when agents receive near-complete successful trajectories (steps n−2n-2 or n−1n-1).

This has a direct consequence for reliability. Pass∧kPass^{\wedge k}, the probability that all kk independent trials succeed, decays rapidly in hypothesis-driven domains (Figure 5B). Under baseline conditions, Pass∧kPass^{\wedge k} drops below 0.05 by k=4k = 4–66 for Spectroscopic Structure Elucidation and Inorganic Qualitative Analysis. Even under success-trace interventions at early steps, the decay is similar. Repeated attempts do not yield consistent success.

This pattern holds across both scaffold architectures. ReAct and tool-calling agents exhibit similar dependence on late-stage successful steps and similar degradation under failed-trace interventions (Extended Data Figure 3). Across all four analyses (performance decomposition, latent factor modeling, epistemological trace analysis, and trace interventions), the same conclusion holds: the base model determines what LLM-based agents can do in scientific settings, and current base models do not exhibit the epistemic patterns characteristic of scientific reasoning.

Discussion

The question of whether machines can conduct scientific inquiry has accompanied AI since DENDRAL built explicit rules for generating and eliminating hypotheses from spectral evidence in 1965.[1] Our data provide a specific answer for current LLM-based agents. In workflow-construction domains, agents approach ceiling performance. In hypothesis-driven domains, they do not adapt their reasoning to the demands of the problem: evidence is ignored in 68% of traces, refutation-driven belief revision occurs in only 26%, and convergent multi-test evidence is rare. These patterns persist across scaffold architectures, across domains, and under interventions that provide near-complete successful trajectories. They are properties of the base model.

The epistemic process by which a result is reached determines whether the resulting knowledge is justified.[13, 14, 27] An agent that reaches a correct answer while ignoring contradictory evidence and leaving hypotheses untested produces a result whose reliability on new problems is unknown. Much current engineering effort focuses on scaffolding (prompting strategies, orchestration logic, tool interfaces), but the reasoning patterns we document persist across all scaffold conditions we tested, including extreme interventions. Addressing them will likely require changes to how base models are trained: a slower process than scaffold improvement, and one currently not guided by epistemic criteria. The framework we introduce can supply such guidance: each environment provides a reproducible task, tools, and a scoring function over agent trajectories, sufficient to define training signals on the reasoning process itself.

Human scientists operate within institutional structures that enforce epistemic norms: peer review, replication, reputational consequences. LLM-based agents are subject to no such constraints. Evidence that reliance on AI assistance can erode independent problem-solving[34] compounds this gap. Characterizing the epistemic behavior of these systems requires explicit measurement. The evaluation framework we introduce does this across tasks and configurations. New environments, models, scaffolds, and metrics can be contributed, allowing the community to track how the epistemic behavior of these systems changes as models evolve. The methods and findings extend beyond scientific agents to any setting where the reasoning process determines the value of the result.

The tools scientists use shape the science they produce.[10, 11] LLM-based agents are now among those tools. Our analysis shows that their epistemic behavior does not exhibit the patterns that characterize scientific reasoning. As long as they are evaluated only by the answers they produce, this difference will remain invisible, and it will shape the knowledge they help produce.

Methods

Formalism

We adopt the language decision process (LDP) framework,[29, 30] a formulation of a partially observable Markov decision process in which observations and actions are represented in text. An LLM-based agent implements a composite policy πoverall\pi_{\mathrm{overall}} that factors into two components:

πoverall(at∣st)=f(πLLM(at∣st), πscaffold(at∣st)),(1)\pi_{\mathrm{overall}}(a_t \mid s_t) = f\left(\pi_{\mathrm{LLM}}(a_t \mid s_t),\ \pi_{\mathrm{scaffold}}(a_t \mid s_t)\right), \tag*{(1)}

where πLLM\pi_{\mathrm{LLM}} is the base-model policy governing token-level choices and πscaffold\pi_{\mathrm{scaffold}} is the scaffold policy governing prompting strategy, tool routing, memory management, and orchestration logic. The state st=(P,ht)s_t = (P, h_t) comprises the task prompt PP and the conversation history hth_t at step tt, which accumulates prior actions, tool outputs, and observations. At each step, the agent selects an action ata_t (a tool call, a reasoning step, or a final answer) according to πoverall\pi_{\mathrm{overall}}. The environment returns an observation, the history grows, and the cycle repeats.

Tools are part of the environment, not the agent. Each environment defines a set of callable functions with standardized docstrings; the agent receives no information beyond what the tool descriptions and returned observations provide. This separation is mirrored directly in our code implementation (Section C): the environment serves tools, task descriptions, and scoring functions through a common interface, while the agent interacts solely through text-based requests. Because all agent configurations face identical tool interfaces and environment dynamics, observed differences in performance and behavior can be attributed to the base model or the scaffold.

Task and tool formalism

Tasks and subtasks. A benchmark item is either a single task or a task group G={τ1,…,τK}G = \{\tau_1, \ldots, \tau_K\} whose dependencies form a directed acyclic graph: each subtask consumes the outputs of its predecessors. The framework resolves this graph and presents subtasks to the agent sequentially, injecting predecessor outputs into the prompt at each step. An ML-based Property Prediction task, for example, decomposes into data retrieval, feature engineering, model training, and evaluation, with each subtask consuming the artifacts produced by the previous one.

Task prompt. The message sequence presented to the base model comprises a system prompt, an environment-specific task prompt (goal description, submission format, and any input data from predecessor subtasks), a scaffold-specific user template that adds response-format instructions, and tool descriptions filtered by the verbosity axis defined below (see § 4.2). Only the task prompt varies across environments; all other components are held constant. Full construction details are given in Section C.1.

Tool verbosity. Tool docstrings are authored with tagged sections (such as [BRIEF], [DETAILED], [EXAMPLES]; full tag definitions in Section C.2). Three levels expose cumulative subsets to the agent: brief includes only the [BRIEF] tag; workflow extends through [WORKFLOW_INTEGRATION]; comprehensive adds examples, error conditions, and limitations. Tool descriptions and task prompts are constructed independently, so verbosity varies without altering the task, enabling a controlled ablation over the information available to πLLM\pi_{\mathrm{LLM}}.

Agent implementations

All experiments use Corral, the evaluation framework we developed for this study (Section C). Only the model backend and agent scaffold vary across runs. We evaluated three models: GPT-4o (gpt-4o-2024-08-06) via the OpenAI API, Claude Sonnet 4.5 (claude-sonnet-4-5-20250929) via the Anthropic API, and GPT-OSS-120B via the Blablador API. Temperature was set to 0.0 for all runs, and each environment enforces a fixed iteration limit. Tool verbosity was set to brief for main-text results; workflow and comprehensive settings appear in the appendix.

We used two agent scaffolds. The first is ReAct:[31] the agent produces reasoning traces and tool calls in a single text stream. The second is structured tool calling: the API parses tool names and arguments from the model output, the agent executes the call, appends the result to the conversation history, and repeats until a termination criterion is met. The two scaffolds differ in how actions are expressed, but the surrounding benchmark logic is identical. Scope and measurement caveats, including compute budget and malformed-response rates, are detailed in § 19.

Domains and environments

Each domain is realized as one or more environments providing tools, a task description, and a scoring function. Observations combine structured data (numerical values, tables, spectra) with natural-language summaries at the selected verbosity level. Scoring functions evaluate both the final answer and, where applicable, intermediate subtask completions. Each environment is backed by a domain-specific engine, instrument, or dataset, ranging from LAMMPS simulations and a live atomic force microscope to a custom thermodynamic equilibrium engine and manually curated spectra (Table D.3). The inputs each environment receives and the artifacts each produces are shown in Extended Data Figure 1. Below, we summarize each domain. Full tool lists, scoring rubrics, and task specifications for the Inorganic Qualitative Analysis environment appear in Section D.1; details for every environment are at https://lamalab-org.github.io/corral/#environments.

Image not extracted

Extended Data Figure 1. Overview of the inputs and outputs of the environments available in Corral. In AFM Experiment Execution, the agent operates an atomic force microscope to capture material images and compute properties such as surface roughness. In Adsorption Surface Construction, the agent builds catalytic surfaces for simulation, producing surface structure files. In Molecular Simulation, the agent runs simulations to obtain material properties. In ML-based Property Prediction, the agent trains an XGBoost model to predict a property. In Circuit Inference, the agent infers the complete system topology from resistor measurements. In Retrosynthetic Planning, the agent constructs a synthesis path for a target molecule under economic constraints. In Spectroscopic Structure Elucidation, the agent elucidates the molecule from spectral data. In Inorganic Qualitative Analysis, the agent identifies the ions present in a provided sample.

Spectroscopic structure elucidation. The agent determines the molecular structure of an unknown compound by requesting and interpreting spectroscopic data.[35, 36, 37] Available tools include mass spectrometry, 1H{}^{1}\mathrm{H} and 13C{}^{13}\mathrm{C} NMR, HSQC, and infrared spectroscopy, alongside reference databases for chemical shift ranges, isotope distributions, and degree-of-unsaturation calculations. The agent can validate proposed structures and simulate spectra for comparison. Each task comprises 10 sequential subtasks that progress from molecular formula determination through functional group identification to complete SMILES assignment. Two scopes vary the molecular complexity: the first isolates individual analytical steps; the second requires integrating ambiguous, multi-modal evidence. HSQC acquisition is marked as expensive, introducing a cost-information tradeoff.

Inorganic qualitative analysis. In a custom-built simulation environment, the agent identifies unknown cations in solution using the systematic wet-laboratory procedure taught in introductory chemistry. Tools include adding reagents, performing flame tests, measuring pH, centrifuging, decanting, heating, and observing color and precipitate changes. The environment simulates real chemical equilibria; observations are computed from thermodynamic data (rather than scripted). Three scopes increase the number of candidate ions from 3–5 common species with distinct chemistry to 15 or more, including ions with overlapping reactivity (e.g. Ag+\mathrm{Ag}^{+}, Hg2+\mathrm{Hg}^{2+}), progressively requiring the agent to design discriminating experimental sequences rather than follow a decision tree. Sample volume is limited, penalizing redundant experiments.

Circuit inference. The agent recovers the topology and component values of a hidden resistor network from pairwise resistance measurements between nodes. Tools provide series and parallel resistance calculations, delta-wye and wye-delta transforms, resistance measurement between specified node pairs, and circuit validation. Six tasks progress from simple parallel-series networks (6–7 resistors) through linear chains (10 resistors) to complex topologies (13+ resistors with multiple interaction points). This domain isolates logical and mathematical reasoning from domain-specific scientific knowledge.

Retrosynthetic planning. The agent designs multi-step synthetic routes to a target molecule under cost, step-count, and commercial-availability constraints. Tools include a template catalog searchable by functional group or bond type, template application and step verification, commercial-availability lookup with CAS number conversion, and functional- and protecting-group detection. Three scopes increase target complexity from simple molecules with one-step solutions to complex natural products requiring up to five steps within a $9,999 budget, where all leaf nodes must be commercially available.

AFM experiment execution. The agent operates an atomic force microscope, executes the experimental procedure, and analyzes the resulting nanoscale surface characterization data. The four scopes vary in the degree of guidance: the first specifies all scan parameters for identical measurements, while the last provides only the objective and expects the agent to choose scan areas and derive a mathematical relationship. The Image Optimizer tool is removed at higher scopes, introducing a tradeoff between available assistance and independent reasoning about scan modes and parameters.

Molecular simulation. The agent designs and executes molecular dynamics simulations using the LAMMPS package[38] to predict materials properties. Tools cover the full workflow: retrieving crystal structures from the Materials Project[39, 40], converting to LAMMPS data format, querying force-field metadata, running simulations, and analyzing output logs. Two scopes vary the degree of procedural guidance: at the first, the simulation protocol is prescribed; at the second, the agent must select simulation parameters, design the heating or cooling schedule, and choose appropriate analysis methods. Tasks span quenching, surface energy computation, and melting simulations, with quantitative targets and specified tolerances (e.g. self-diffusivity of molten SiO2_2 within 20%).

Adsorption surface construction. The agent builds an adsorbate-slab configuration from bulk crystal structures and molecules for heterogeneous catalysis studies. Tools integrate with the Materials Project to retrieve structures and polymorph data, generate surface slabs for specified Miller indices, vacuum, and enumerate adsorption sites. The agent must select an appropriate bulk polymorph, generate a surface termination, identify relevant adsorption configurations, and place the adsorbate molecule on the adsorption site.

ML-based property prediction. The agent assembles a complete machine-learning pipeline to predict formation energies of material polymorphs. Tools cover data retrieval from the Materials Project (batch and single-composition queries), dataset filtering and consolidation, tabular feature engineering, XGBoost[41] training with configurable hyperparameters, evaluation, and cross-validation. Tasks span three material classes (oxides, nitrides, sulfides). Models are scored against specified R2R^2 thresholds and cross-validation targets.

Manual trace annotation

We built a custom web application to support the manual annotation described in Figure F.4 (see the annotation app at https://lamalab-org.github.io/corral/#annotate). The tool displays reconstructed agent traces as interactive graphs. Each node represents a single message-like event, such as user instructions, reasoning steps, or tool calls. Annotators process traces in order. They label only behavior-relevant nodes; by design, system messages, tool outputs, and the initial task description are not annotatable. For each eligible node, annotators select one or more markers from a taxonomy we developed for this study. The taxonomy distinguishes positive behaviors (e.g., planning statements, validation attempts), failure modes (e.g., hallucinations, unnecessary tool use), and neutral states (detailed in Section F). Annotators can also add free-text observations to each node and trace. The interface requires left-to-right progression and checks that all annotatable nodes receive at least one marker before submission. For downstream analysis, two expert annotators completed the full annotation process on 773 selected traces. Each selected trace received annotations from one domain expert. The two annotators first annotated and discussed several calibration traces and edge cases to align their understanding. This process produced one final annotation record per trace, which informed all later marker-level analyses.

Diagnostic question-answer pairs

For each domain, we designed two types of diagnostic questions. Knowledge questions target the specific facts needed to complete tasks; reasoning questions target the specific inference operations the environment requires. The designer of each environment compiled the initial set; submissions were made via pull requests and accepted only after two independent expert reviews. Knowledge question counts range from approximately twenty to over one hundred per environment; reasoning question counts are more uniform, with twenty to twenty-five per environment. All three models were evaluated on the full item bank at a temperature of 0.0. Full results are detailed in Table E.6; the item format, scoring rubric, and review protocol are described in Section E.

IRT and phenomenological model

We adopt a two-stage modeling strategy to (i) estimate the latent cognitive capabilities that each model brings to each scientific domain, and (ii) quantify how those capabilities, together with agent-design choices, predict benchmark success as illustrated in Figure 3 A.

Extracting model capabilities. A two-parameter logistic (2PL) IRT model estimates latent abilities from binary QA responses. The diagnostic questions are described in § 4.6. For model jj answering item ii:

P(Yij=1)=σ(ai(θj−bi)),(2)P(Y_{ij}=1)=\sigma\left(a_i\left(\theta_j-b_i\right)\right), \tag*{(2)}

where θj\theta_j is the latent ability of model jj, ai>0a_i>0 is item discrimination, and bib_i is item difficulty. We fit two separate 2PL models, one to knowledge items and one to reasoning items, yielding θj(K)\theta_j^{(K)} and θj(R)\theta_j^{(R)} per model-environment pair. Both are standardized to zero mean and unit variance before entering Stage 2.

Phenomenological latent factor models. The IRT-derived abilities serve as covariates in a second modeling stage that predicts agent success on the full benchmark. Raw trial outcomes are first balanced (equal task counts per environment) and aggregated into Binomial counts (ksuccess,ntrials)(k_{\mathrm{success}}, n_{\mathrm{trials}}) per unique combination of model, environment, scaffold, scope, tool-description verbosity, aggregation grain (task-level vs. subtask-level outcome), and task identifier (the specific task-scope combination). The standardized IRT abilities θ~K\tilde{\theta}_{K} and θ~R\tilde{\theta}_{R} are then merged as continuous predictors.

We specify a family of Bayesian generalised linear models[42] (M1–M8; Section G.5.1) of increasing structural complexity, all sharing the Binomial likelihood

k∼Binomial⁡(n,p),logit⁡(p)=η,(3)k \sim\operatorname{Binomial}(n,p), \qquad\operatorname{logit}(p)=\eta, \tag*{(3)}

and a common set of sum-to-zero categorical effects for scaffold (γ\gamma), verbosity (ξ\xi), and scope (κ\kappa). Models differ in how they partition the remaining variance. Candidate specifications are compared using Pareto-smoothed importance sampling leave-one-out cross-validation (PSIS-LOO; Figure G.9). The best-fitting model is M7, which allows knowledge and reasoning slopes to vary by environment ×\times scope. All main-text variance decompositions and predicted success probabilities are derived from M7 (see Section G.5.1).

Epistemological graphs

We sampled traces balanced across environments, models, and problem scope levels, prioritizing task diversity. Only ReAct traces were used, as they contain explicit reasoning steps. Tool description verbosity was set to brief; differences between verbosity levels are minimal (see Figure G.15).

Each trace was annotated using Claude 4.5 Sonnet in two stages. In the first stage, the model labels nodes from short, overlapping windows of messages. Node labels correspond to epistemic operations: hypothesis, test, evidence, judgment, update, and commitment. In the second stage, the model adds directed edges between related nodes. Edge types include testing, observing, using, contradicting, competing, and updating. In both stages, the temperature was set to 0.7. Every node and edge includes a supporting quote and the index of the source message, grounding annotations in the trace. After processing the full trace, repeated items are normalized and duplicates removed. Section H.1 details the annotation process.

Annotations were validated in two ways. Automated checks verify that each supporting quote appears in the cited message and that each edge connects valid nodes. Expert review guided the annotation design: an expert iteratively revised prompts, labels, and markers until annotations matched their own interpretation of the traces. Inter-annotator agreement analysis on a representative sample of 25 traces showed substantial human–human agreement (overall 92.6%, mean PABAK 0.853) and even higher human–LLM agreement (95.7%), confirming that the automated pipeline performs within the range of human expert judgment. Section H.5 provides a deeper analysis of the annotation results.

Productive patterns and anti-patterns are defined as explicit structural templates over the annotated graphs (Table H.14, Table H.15). Productive patterns include Popperian falsification,[43] exploratory-to-confirmatory transitions,[44] and active learning.[45, 46] Anti-patterns are drawn from established epistemic failures in scientific reasoning and from behaviors observed in the traces, including untested hypotheses, ignored evidence, and unresolved contradictions. Both are grouped into higher-level families for aggregated analysis (Table H.16).

Trace intervention experiment

The trace intervention experiment probes whether agent performance is driven by latent model capability or by accumulated conversational context. We first ran 15 trials per task across all environments at temperature 0.7 to populate a registry of successful and failed execution traces. A moderate temperature follows standard practice for balancing sample diversity against coherence.[47] These runs are distinct from the main evaluation, which uses temperature 0.0 for deterministic scores. For each new trial, we inject a partial trajectory from this registry into the agent’s conversation history before the trial begins. The injected traces are replayed, tool calls and environment interactions included, so that the agent resumes from a consistent intermediate state. We vary two factors: the source of the injected trace (successful or failed prior run) and its extent (early steps 1–2 versus near-complete trajectories at steps n−2n-2 or n−1n-1). The intervention supplies the actual reasoning and actions a successful agent would have produced rather than instructions or prompting strategies. Intervention trials also use temperature 0.7. Protocol details and pseudocode are in Section G.6.

Token-level log-probability analysis

To quantify the model’s token-level confidence across benchmark environments, we analyze the log-probabilities returned by the language model for each generated token. For every assistant message produced during a trial, we select and save the log-probability of the top-1 token at each decoding step. Formally, for a response consisting of TT tokens {w1,w2,…,wT}\{w_1, w_2, \ldots, w_T\},

ℓt=log⁡pθ(wt∣w<t,x),t=1,…,T,(4)\ell_t = \log p_\theta(w_t \mid w_{<t}, x), \qquad t = 1, \ldots, T, \tag*{(4)}

where pθp_\theta denotes the language model, w<tw_{<t} is the preceding token sequence, and xx is the input context (which includes system prompt, task prompt, tool definitions, and conversation history).

For each environment ee, we pool all non-special token log-probabilities across every assistant message from every trial in that environment. Tokens whose log-probability is exactly zero and that correspond to special control tokens (e.g., <|endoftext|>, <|im_end|>) are excluded. The per-environment mean log-probability is then,

ℓˉe=1Ne∑t=1Neℓt(e),(5)\bar{\ell}_e = \frac{1}{N_e} \sum_{t=1}^{N_e} \ell_t^{(e)}, \tag*{(5)}

where NeN_e is the total number of retained (non-zero, finite) tokens across the message or trial it belongs to, giving a vocabulary-level confidence estimate. More negative values of ℓˉe\bar{\ell}_e signal that the model distributes probability more diffusely, reflecting greater uncertainty. Per-environment values and interpretation are reported in Section G.7.

Acknowledgements

The research of M.R.G., N.A., and K.M.J. was supported by the Carl-Zeiss Foundation, Open Philanthropy, as well as Intel and Merck via the AWASES research center. K.M.J. is part of the NFDI consortium FAIRmat funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project 460197019. K.M.J.’s work was also supported by a Google Research Scholar Award. N.M.A.K, S.M, C.G, and I.M. thank the High Performance Computing Facility at IIT Delhi for computational and storage resources. S.M. acknowledges the partial funding support from the Prime Minister’s Research Fellowship. N.M.A.K. acknowledges the funding support received by Google Research Scholar Award, Anusandhan National Research Foundation (ANRF/ARG/2025/007405/ENS), and Alexander von Humboldt fellowship. We thank Nitya Nand Gosvami for providing the support with the Atomic Force Microscopy and valuable feedback associated with it. We thank Santiago Miret, Rob Jordan, and Vijay Narasimhan for valuable discussion. We thank Mara Schilling-Wilhelmi for feedback on a draft of this article and K. Sidharth Nambudiripad for support in human annotation.

Data availability

The data supporting this study, including environment tasks,[48] evaluation reports,[49] agent traces,[50] question-answer pairs,[51, 52, 53] token-level log probabilities,[54] and other data related to the study,[55, 56, 57, 58, 59, 60] are publicly available in a Hugging Face collection at https://huggingface.co/collections/jablonkagroup/corral under permissive license.

Code availability

The package built in this study is publicly available at GitHub (https://github.com/lamalab-org/corral). A version of the code corresponding to this manuscript is archived on Zenodo https://doi.org/10.5281/zenodo.19659851.

Competing interests

The authors declare no competing interests.

Declaration of AI-assisted technologies in the research and writing process

The authors used Claude Opus (Anthropic) for assistance with language and readability of the manuscript. During code development, the authors made use of several large language model assistants (Claude Code, GitHub Copilot). After using these services, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

Author contributions

Martiño Ríos-GarcíaNawaf AlampraChandan GuptaIndrajeet MandalSajid MannanAli Asghar AghajaniN. M. Anoop KrishnanKevin Maik Jablonka
Conceptualization■■■■■■■■
Data curation■■■■■■
Formal analysis■■■■■■■■
Funding acquisition■■
Investigation■■■■■■
Methodology■■■■■■■■
Project administration■■
Resources■■
Software■■■■■■
Supervision■■
Validation■■■■■■■■
Visualization■■■
Writing – original draft■■■■
Writing – review & editing■■■■■■■■

Table 1.

M.R.G., N.A., and K.M.J. jointly designed the Corral framework and the analysis pipeline; M.R.G. and N.A. led the implementation. Environments were developed as follows: spectroscopic structure elucidation and retrosynthetic planning by M.R.G.; catalyst surface construction, machine learning, and circuit inference by N.A.; atomic force microscopy by I.M. and S.M.; molecular dynamics by S.M. and C.G.; and inorganic qualitative analysis by A.A. The reasoning-trace analysis was led by M.R.G.; the intervention analysis by N.A.; and the performance-based analysis, enabled by the Corral core, was led jointly by C.G., N.A., and M.R.G. N.M.A.K. and K.M.J. acquired funding, administered the project, provided resources, and supervised the work. K.M.J. drafted the manuscript with substantial contributions from M.R.G. and N.A.; C.G. drafted the related-work section. All authors reviewed, edited, and approved the final version of the manuscript.

Image not extracted

Figure Extended Data Fig. 2. Trace quality markers are more strongly influenced by the model than by the scaffold. (A) Normalized effect sizes for each annotated marker. Most points lie above the diagonal, indicating that model differences exceed scaffold differences for the majority of markers. (B) Proportion of positive, negative, and neutral markers across scaffold-model combinations. The annotation process and the marker taxonomy are described in Section F. 773 traces were manually annotated across all environments; detailed results per marker are in Section G.3.

Image not extracted

Figure 3 (PDF p. 28). Pass∧k{}^{\wedge}k decay under trace interventions is consistent across scaffold architectures. Pass∧kPass^{\wedge}k as a function of kk under trace interventions, shown separately for ReAct and tool-calling agents. Rows correspond to interventions with traces from successful baseline runs (top two) and traces from failed baseline runs (bottom two), with columns representing each environment. Unlike Pass@kPass@k, Pass∧kPass^{\wedge}k requires all kk trials to succeed.

Related work

Corral sits at the intersection of five lines of research: benchmarks for scientific agents, end-to-end AI scientist systems, the attribution of agent capability to the base model versus the scaffold, psychometric measurement of model ability, and process-level epistemic evaluation of AI-generated science.

Benchmarks for scientific agents

Existing benchmarks for scientific agents primarily score outcomes. Domain-specific evaluations such as ChemBench, MaCBench, and Lab-Bench probe knowledge and reasoning in chemistry, materials science, and biology through curated question sets.[1, 2, 3] Workflow-oriented benchmarks including ScienceAgentBench, SciAgentGym, and Curie evaluate end-to-end execution of data analyses and discovery pipelines drawn from published work.[4, 5, 6] Replication-oriented evaluations such as Lessons-Learned, MLR-Bench, ResearchBench, and DeepResearch test whether agents can reproduce or extend specific results.[7, 8, 9, 10] AstaBench and BixBench evaluate agents across the full research pipeline, including manuscript-style outputs.[11, 12] Generalist agent benchmarks such as AgentBench, GAIA, Gaia2, MultiAgentBench, and ToolSandbox[13, 14, 15, 16, 17] measure capabilities that scientific agents draw on but were not designed for science.

These evaluations score what an agent produced, not how it produced it. A correct answer reached by lookup is scored identically to the same answer reached by hypothesis testing; an incorrect answer reached by careful inquiry is scored identically to one reached by confabulation. Concerns that benchmarks measure narrow skill rather than the underlying capability they purport to assess are long-standing.[18] Corral preserves the outcome metrics these benchmarks provide and adds standardized environments with graded task scopes and fine-grained subtask diagnostics.

AI scientists

A growing body of work develops LLM-based systems intended to conduct end-to-end research: hypothesis generation, experimental design, execution, and write-up. Sakana’s AI Scientist[19] produces submission-ready manuscripts; Kosmos[20] reports cross-domain discoveries; ChemCrow,[21] Coscientist,[22] the EI Agent series,[23] Organa,[24] and AlphaEvolve[25] target specific scientific or engineering domains. Multi-agent architectures such as SciAgents, ScienceClaw, and virtual laboratories structure the workflow with role specialization or knowledge-graph guidance,[26, 27, 28] building on earlier instantiations for protein design[29] and finite-element mechanics;[30] recent work also adapts the objective each sub-agent optimizes to mitigate reward hacking.[31] Complementary efforts target reasoning at the base-model level, training models to produce reflection and recursive refinement steps explicitly.[32] Alongside this systems-building effort, risk-focused perspectives argue that AI scientists introduce novel vulnerabilities requiring explicit safeguards across user intent, agent alignment, and environmental feedback.[33] These systems are deployed and reported on without a shared instrument for assessing the quality of the reasoning that produced their outputs. Corral is designed to fill this role: its environments, tools, scoring functions, and runner can be reused as building blocks for new agents and scaffolds, and the diagnostic and epistemic instruments introduced here can be applied to any system built on top. The environments can also serve as training substrates: each supplies reproducible tasks and scoring over agent trajectories, sufficient to optimize agents against criteria defined on the reasoning process.[34]

Scaffolds and the base-model/scaffold attribution

The capability of an LLM-based agent reflects two contributions: the base model’s token-level competence and the scaffold’s orchestration logic. Prompt-based scaffolds such as Chain-of-Thought, ReAct, and self-consistency,[35, 36, 37] and agentic scaffolds with tool access such as Toolformer, Voyager, and Reflexion,[38, 39, 40] are now standard, and gains over base models are routinely reported. The question of how much of any gain is the scaffold and how much is the base model is rarely answered directly. Meireles et al.[41] are among the few to address it, showing that effective scaffolding preserves coordination scaling laws that deteriorate without it. Corral varies model and scaffold orthogonally on a fixed task set, allowing each contribution to be read directly from the results.

Item response theory for model evaluation

Item response theory (IRT), developed for educational assessment,[42, 43, 44] models benchmark performance as the interaction between latent ability and item characteristics such as difficulty and discrimination. Early NLP applications used IRT to construct evaluation datasets and demonstrated that item difficulty predicts model behavior more reliably than aggregate accuracy.[45, 46] Bayesian IRT-based leaderboards introduced uncertainty-aware ability estimates,[47] and subsequent work showed that hidden implementation choices distort raw rankings in ways that IRT can correct.[48, 49] For LLM agents, IRT-inspired task selection has been used to preserve ranking fidelity across scaffolds and environments.[50] We apply IRT to our diagnostic subtask items to estimate latent capability profiles per model–environment pair, separating ability from item difficulty in a way that aggregate Pass@kPass@k does not.

Epistemic evaluation of AI in science

Process-level epistemic evaluation of AI-generated science remains underdeveloped. Field-level critiques are well established: Ioannidis [51] on irreproducibility in the empirical sciences, and Hao [52] on machine learning, narrowing the hypothesis space scientists pursue. More directly on large language models, Mitchell & Krakauer [53] survey the debate over whether such systems exhibit genuine understanding, noting that surface-level competence on benchmarks does not settle the underlying epistemic question. Thais [54] extend these concerns to LLM-based scientific systems specifically, arguing that current AI-for-science work conflates predictive accuracy with epistemic warrant and lacks operational instruments for assessing whether claims are justified in the philosophical sense. [55] Corral provides such an instrument. The behavioral analysis introduced here annotates reasoning traces as graphs of operations and detects structural patterns of productive inquiry, such as Popperian falsification, exploratory-to-confirmatory transitions, and active learning, [56, 57, 58, 59] as well as patterns of epistemic failure, such as untested hypotheses, ignored evidence, and unresolved contradictions. This shifts the evaluation from whether an agent reached the right answer to the process by which it arrived there.

Rise of AI Scientists

Following Hao [52], we quantified the emergence of AI-scientist research within chemistry and materials science and contextualized its growth against the broader AI/ML-for-chemistry-and-materials literature. We constructed a narrow corpus targeting AI scientists, LLM agents, autonomous discovery systems, and closely related agentic workflows. Using OpenAlex, [60] we first retrieved the broader set of AI- and ML-related papers in chemistry and materials science, then applied a stricter keyword-based filter to isolate publications matching our AI-scientist definition. Figure A.1 reports yearly publication counts for both corpora on a log scale, along with the share of AI-scientist papers within the broader AI-for-chemistry literature.

Image not extracted

Figure A.1. Sharp rise of AI-scientist publications within chemistry and materials AI literature. Annual publication counts (log scale) for the broader AI-for-chemistry-and-materials corpus versus a strict AI-scientist subset. The right axis shows the share of AI-scientist papers within the broader corpus, revealing a steep recent increase.

Two patterns are immediately apparent. First, the broader AI-for-chemistry literature has reached roughly 10410^{4} papers per year, while the strict AI-scientist subset has grown from about 10110^{1} papers in 2018 to the upper 10210^{2} range in the most recent partial year—already exceeding earlier full-year totals. Thus, although AI-scientist work remains much smaller than AI for chemistry as a whole, it is no longer a marginal subfield but a recognizable and rapidly growing research stream.

Second, the relative share of AI-scientist papers within the AI-for-chemistry literature is rising even faster than absolute counts. This strict subset accounted for only about 1% of AI-for-chemistry output through 2024, increased to roughly 2% in 2025, and approaches 5% in the current partial year. Consequently, while AI-scientist research still represents a very small fraction of all chemistry and materials science publishing—approximately 0.01% through 2024 and around 0.06% in 2025—its footprint within the AI segment of the field is expanding rapidly.

Glossary

Domain. One of the eight scientific areas covered by the benchmark (e.g., retrosynthetic planning, atomic force microscopy); see § 4.4.

Scope. A difficulty or ablation variant within a domain, varying task complexity, tool availability, or degree of agent guidance. Scopes are introduced per domain in § 4.4.

Environment. A concrete instance of a domain at a specific scope, formalized as a language decision process in § 4.1. Each environment exposes tools, a task description, and a scoring function through the Corral HTTP interface.

Task. A single problem an agent is asked to solve within an environment; may decompose into dependency-ordered subtasks (§ 4.1).

Subtask. A component of a task whose output is forwarded as input to downstream subtasks in the dependency graph (§ 4.1).

Scaffold. The orchestration layer wraps a base LLM: prompt formatting, tool routing, history management, and termination logic. The two scaffolds evaluated here are ReAct and structured tool calling (§ 4.3).

Tool-documentation verbosity. The level of detail retained in tool docstrings presented to the agent varied as an experimental axis (§ 4.2).

Configuration. A particular combination of base model, scaffold, tool-documentation verbosity, and environment.

Trial. A single end-to-end execution of an agent configuration on a task.

Trace. The conversation history produced during a trial is used downstream for epistemological annotation (§ 4.8).

Pass@kk. Probability that at least one of kk independent trials on a task succeeds. Estimated from n≥kn \ge k trials by the unbiased estimator 1−(n−ck)(nk)1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}, where cc is the number of successful trials.

Passk^{k}. Probability that all kk independent trials on a task succeed. A reliability measure, unlike Pass@kk, requires every trial to succeed rather than at least one.

Language decision process (LDP). The formal model of agent–environment interaction adopted here: a partially observable Markov decision process whose observations and actions are represented in text (§ 4.1).

ELPD-LOO. Expected log predictive density estimated by leave-one-out cross-validation; used to compare the candidate hierarchical models in Section G.5.

Inter-annotator agreement (κ\kappa). Cohen’s κ\kappa between two independent annotators on the same set of traces, reported in Section H, to quantify the reliability of the manual review.

Epistemic operation. A node type in the annotated reasoning graph, drawn from the fixed vocabulary {hypothesis, test, evidence, judgment, update, commitment}. Each operation corresponds to an identifiable epistemic act in the trace (§ 4.8).

Item response theory (IRT). A psychometric modeling framework that jointly estimates respondent ability and item difficulty from a matrix of correct/incorrect outcomes. We fit the two-parameter logistic (2PL) form, in which each item has a difficulty bb and a discrimination aa, and each respondent a latent ability θ\theta (Section G.4).

Latent ability (θ\theta). The IRT parameter representing a model’s capability on a set of items, separated from the items’ own difficulty. Higher θ\theta corresponds to a higher probability of success on items of a given difficulty.

Posterior predictive check (PPC). A fitted Bayesian model’s predictions for held-out observations are compared against the observed outcomes to assess calibration and generalization. Reported for the selected hierarchical model in Section G.5.

Capability item. A question–answer item used to estimate latent model capability through IRT (§ 4.6, Section E).

Productive motif. A graph template over the annotated trace that matches a recognized practice of disciplined inquiry: Popperian falsification, exploratory-to-confirmatory transition, active-learning-style test redesign (Section H.3).

Reasoning breakdown. A graph template over the annotated trace that matches an established epistemic failure mode: untested claim, evidence non-uptake, fixed belief trace, contradiction without repair (Section H.3).

Corral: Framework

Corral is a composable evaluation framework that separates environments from agents through a standard HTTP interface. Environments are hosted by a FastAPI server that exposes task descriptions, tools, and scoring through a small set of RESTful endpoints (Table C.2, Figure C.2). Agents interact with an environment solely by issuing HTTP requests. The Corral runner drives evaluation by iterating trials over registered agent–environment pairs, checkpointing intermediate state, and aggregating results into per-configuration scores and diagnostics. The snippets below illustrate the three main extension points; the canonical API reference is maintained at https://lamalab-org.github.io/corral.

Tools. Tools are formally subclasses of the Tool base class. The @tool decorator is a convenience shortcut that wraps a plain Python function into such a subclass. In both cases, the docstring of the tool is part of the contract between environment and agent: Corral exposes it to the agent as the tool’s natural-language description, and its tagged sections are filtered by the tool-documentation verbosity axis (§ 4.2).

Listing 1: Defining a tool via the decorator shortcut.

from corral.backend.tool import tool

@tool
def my_custom_tool(input_param: str) -> str:
    """[BRIEF] One-sentence summary.

    Args:
        input_param: description of the parameter.
    Returns:
        description of the return value.
    """
    return f"Processed: {input_param}"

Environments. An environment subclasses Environment, attaches its tools in the constructor, and overrides get_task_prompt and score. A dictionary of environment instances is then served over HTTP by create_benchmark_server.

Listing 2: Defining and serving an environment.

from corral.backend import Environment
from corral.backend.server import create_benchmark_server
import os, uvicorn

BASE_WORK_DIR = os.getenv("CORRAL_WORK_DIR")

class MyEnvironment(Environment):
    def __init__(self, task_id, problem, answer,
                 base_workdir=BASE_WORK_DIR):
        self.problem = problem
        self.correct_answer = answer
        super().__init__(task_id, base_workdir)
        self.add_tool(my_custom_tool)

    def get_task_prompt(self):
        return f"Solve this problem: {self.problem}"

    def score(self):
        if self.state.submitted_answer is None:
            return 0.0
        return 1.0 if self.state.submitted_answer == self.correct_answer else 0.0
environments = {
    "task_1": MyEnvironment("task_1", "Problem 1", "Answer 1"),
    "task_2": MyEnvironment("task_2", "Problem 2", "Answer 2"),
}

if __name__ == "__main__":
    app = create_benchmark_server(environments)
    uvicorn.run(app, host="0.0.0.0", port=8000)

Agents. An agent subclasses BaseAgent and implements run. The agent drives inquiry through the router interface: get_task_guide returns the task prompt, execute_tool invokes a tool, and get_llm_response routes a message list through the configured base model.

Listing 3: Defining a custom scaffold.

from corral.agents import BaseAgent
from corral.agents.prompt_utils import create_prompt
from corral.agents.utils import LiteLLMMessage

class MyAgent(BaseAgent):
    def __init__(self, model: str, **kwargs):
        super().__init__(model, **kwargs)

    def run(self, interface, task_id: str) -> str:
        task_guide = interface.get_task_guide(task_id)
        messages = create_prompt(task_guide)
        # loop: call base LLM, parse action, execute tool,
        # append observation, repeat until termination
        response = self.get_llm_response(messages)
        return response

Runner. The Corral runner iterates over tasks and trials, dispatches agent calls through the router, persists intermediate state for fault tolerance, and returns a unified result summary. Experiment execution and environment definition are kept separate.

Listing 4: Runner invocation with custom diagnostic metric.

from corral import CorralRouter, CorralRunner
from corral.agents import MyAgent


# Connect to the environment server
router = CorralRouter()

# Configure the agent (model + scaffold)
9  agent = MyAgent(model="gpt-4o", temperature=0.7, max_iterations
   =100)

11 # Create runner and register custom metrics
12 runner = CorralRunner(router, agent)

14 # Run evaluation
15 result = runner.bench(
16     ["task_1", "task_2"],
17     trials_per_task=3,
18     verbose=True
19 )

21 # Results include default + custom metrics
22 print(f"Average score: {result.metrics['average_score']}")

Prompt construction

The message sequence delivered to the base model is assembled from four independently authored components.

System prompt. A static instruction, identical across all configurations, that establishes the agent’s role and a context management directive.

Task prompt. Each environment implements a get_task_prompt() method that assembles a structured prompt from the task definition (see the endpoints in Table C.2). All environments follow a common template: (i) a task name, (ii) a natural-language description of the scientific goal, (iii) the required submission format (e.g., a SMILES string, a list of ions), and (iv) any available input data. For task groups, the input data include the artifacts forwarded from predecessor subtasks in the dependency graph, plus initial parameters such as target molecule or material composition. Some environments prepend a brief role preamble or append workspace instructions when filesystem tools are available.

Scaffold template. The task prompt is inserted into a scaffold-specific user template. For ReAct, the template embeds the task prompt together with a textual tool-usage guide and instructs the model to respond in XML-tagged thought—action—observation turns; the final answer is submitted via a <final_answer> tag. For structured tool calling, the template contains only the task prompt; tool schemas are passed separately through the native function-calling API, and the final answer is submitted with a “Final Answer:” keyword prefix.

Tool descriptions. Filtered by the verbosity axis (§ 4.2) and injected either inline in the user prompt (ReAct) or via the function-calling API (tool calling).

Only the task prompt varies across environments; the system prompt, scaffold template, and tool-injection mechanism are held constant.

Example of different verbosity

The example boxes below render the docstring of the tool search_by_smiles at each of the three verbosity levels. § 4.2 defines the basis on which information is controlled for different verbosity levels. Table C.1 lists the tagged sections and their content.

Table C.1: Tagged sections used in tool docstrings. Each tool’s documentation is authored with the sections below. At each verbosity level, a cumulative subset of these sections is retained (§ 4.2).

TagContent
[BRIEF]One-sentence functional summary
[DETAILED]Extended description of the function
[PROCEDURAL]Step-by-step usage instructions
[CONTEXTUAL]When and why to use this tool
[WORKFLOW_INTEGRATION]How this tool fits into multi-step workflows
[SYNTACTICAL]Argument formatting conventions
[EXAMPLES]Concrete input/output examples
[RAISES], [LIMITATIONS]Error conditions and known constraints

Table C.1. Tagged sections used in tool docstrings.

Verbosity: Brief

Search the NMRShift database for entries matching or chemically similar to the given SMILES.

ARGUMENTS

smiles (str)

The SMILES representation of the compound to search for in the NMRShift database

top_k (int)

The maximum number of results to return. Defaults to 10

RETURNS

A list of dictionaries containing the most relevant entries from the NMRShift database

Verbosity: Workflow

Search the NMRShift database for entries matching or chemically similar to the given SMILES.

This function searches the NMRShift database for entries that match or are chemically similar to the provided SMILES string. It uses a vector database search to find the top top_k results based on chemical similarity. The function returns a list of matching entries, each containing relevant information about the compound.

When to use this tool:

  • Use it to validate your proposed molecule against the NMRShift database.

  • Use it when you have a SMILES string and want to find related compounds in the NMRShift database.

  • When you need to retrieve chemical shifts or other NMR-related information for a specific compound.

How this tool works:

  • It uses a vector database search in the "nmrshiftdb2" collection.

  • The search uses chemical embeddings generated by the "ibm-research/MoLFormer-XL-both-10pct" model.

  • The function retrieves the top top_k results based on chemical similarity.

  • Each result contains relevant information about the compound, such as its SMILES, chemical shifts, and other properties.

Typical workflow integration:

  1. Have a SMILES representation of the molecule you want to search for, or the SMILES of a structure that you think can be valid.

  2. Apply this tool with the SMILES string to search for matching or chemically similar entries in the NMRShift database.

  3. Use the retrieved entries to validate your proposed molecule, compare chemical shifts, or gather additional information. You can also use get_formula_from_smiles to validate the molecular formula.

ARGUMENTS

smiles (str)

The SMILES representation of the compound to search for in the NMRShift database

It should be a valid SMILES notation that can be processed by the vector database search.

top_k (int)

The maximum number of results to return. Defaults to 10

It should be a positive integer.

RETURNS

A list of dictionaries containing the most relevant entries from the NMRShift database. Each dictionary contains relevant information about the compound. Results are sorted by similarity score in descending order.

Verbosity: Comprehensive

Search the NMRShift database for entries matching or chemically similar to the given SMILES.

This function searches the NMRShift database for entries that match or are chemically similar to the provided SMILES string. It uses a vector database search to find the top top_k results based on chemical similarity. The function returns a list of matching entries, each containing relevant information about the compound.

When to use this tool:

  • Use it to validate your proposed molecule against the NMRShift database.

  • Use it when you have a SMILES string and want to find related compounds in the NMRShift database.

  • When you need to retrieve chemical shifts or other NMR-related information for a specific compound.

How this tool works:

  • It uses a vector database search in the "nmrshiftdb2" collection.

  • The search uses chemical embeddings generated by the "ibm-research/MoLFormer-XL-both-10pct" model.

  • The function retrieves the top top_k results based on chemical similarity.

  • Each result contains relevant information about the compound, such as its SMILES, chemical shifts, and other properties.

  • The function returns a list of matching entries, each represented as a dictionary containing the relevant information.

Typical workflow integration:

  1. Have a SMILES representation of the molecule you want to search for, or the SMILES of a structure that you think can be valid.

  2. Apply this tool with the SMILES string to search for matching or chemically similar entries in the NMRShift database.

  3. Use the retrieved entries to validate your proposed molecule, compare chemical shifts, or gather additional information. You can also use get_formula_from_smiles to validate the molecular formula.

Usage examples:

search_by_smiles("CCO")
search_by_smiles("C1=CC=CC=C1")
search_by_smiles("C(C(=O)O)N")
search_by_smiles("C1=CC=C(C=C1)C(=O)O")

Exceptions: If an error occurs during the search process, such as errors in the embedding of the query. This can also occur if the top_k parameter is not a positive integer. Try another tool.

Known Limitations:

  • The function requires a valid SMILES string that can be processed by the vector database search.

  • The search results are limited to the top_k parameter, which defaults to 10.

  • The search is based on chemical embeddings, which may not capture all chemical similarities perfectly.

Arguments

smiles (str)

The SMILES representation of the compound to search for in the NMRShift database

The SMILES string representing the chemical structure of the molecule to search for in the NMRShift database. It should be a valid SMILES notation that can be processed by the vector database search.

Valid SMILES string

Examples: "CCO", "C1=CC=CC=C1", "C(C(=O)O)N", "C1=CC=C(C=C1)C(=O)O"

top_k (int)

The maximum number of results to return. Defaults to 10

The maximum number of search results to return from the NMRShift database. It should be a positive integer.

Any positive integer (e.g., 10, 20, 50)

Examples: 10, 20, 50

Returns

A list of dictionaries containing the most relevant entries from the NMRShift database. Each dictionary contains relevant information about the compound, such as its SMILES, chemical shifts, and other properties. The results are sorted by similarity score in descending order.

Example: [{"entry_id": "nmrshiftdb2:234", "compound_name": "Benzene", "smiles": "c1ccccc1", ...}]

Image not extracted

Figure C.2. Corral framework overview. Environments are hosted in a FastAPI server; each exposes a task description, tools, and a scoring function. Agents interact with an environment by issuing HTTP requests to query information, check state, or submit answers. The Corral runner drives evaluation, handling checkpointing, aggregation, and metric computation. New agents, environments, and metrics extend the framework through the three subclass interfaces described in this section.

MethodPathDescription
Task and Environment Discovery
GET/tasksRetrieves a list of all available task IDs in the benchmark.
GET/dependency_chainIndicates whether any tasks in the benchmark are part of a dependency chain.
GET/tasks/{task_id}/promptFetches the specific prompt for a given task. The {task_id} is a path parameter representing the unique identifier for the task.
GET/tasks/{task_id}/guideReturns the complete environment guide for a task, including the prompt and a detailed description of the available tools. Accepts an optional verbosity query parameter (e.g., MINIMAL or WORKFLOW) to control the level of detail in the tool descriptions.
Tool Interaction and Introspection
GET/tasks/{task_id}/toolsReturns a structured JSON object detailing the available tools for a task, compatible with function-calling APIs. The level of detail in the argument descriptions is controlled by the optional verbosity query parameter.
GET/tasks/{task_id}/tools/guideProvides a guide to the tools available for a specific task. Also accepts the optional verbosity query parameter.
POST/tasks/{task_id}/tools/executeExecutes a specified tool with the provided arguments within the task’s environment. The tool name and arguments are sent in the request body.
State Management and Task Submission
GET/tasks/{task_id}/stateRetrieves the current state of the specified task environment.

Table C.2. Benchmark Server API Endpoints. The table specifies all the endpoints that the server hosting the environments admits.

MethodPathDescription
POST/tasks/{task_id}/submitSubmits a final answer for the task. The submitted answer is in the request body. The response includes the score and the final state of the trial.
GET/tasks/{task_id}/statusGets the completion status of a task, including whether it is completed, the final score, the submitted answer, and tool usage statistics.
GET/tasks/{task_id}/trialsRetrieves the states of all trials that have been run for a specific task.
GET/tasks/{task_id}/trials/{trial_id}Fetches the state of a specific trial for a given task, identified by its unique {trial_id}.

Table C.2 (continued). from previous page

Domains and environments

Each of the eight domains described in Methods exposes multiple task scopes; each scope is realized as a separate environment. The eight domains are summarised in Extended Data Figure 1, which shows the input the agent receives and the artifact it produces in each case. An overview of the characteristics of each domain is provided in Table D.3. Full per-scope tool lists, scoring rubrics, and task specifications are maintained at https://lamalab-org.github.io/corral/#environments. One environment, the inorganic qualitative analysis (wetlab), is used as a worked example below (Section D.1).

Table D.3: Overview of the Corral environments. For each of the eight scientific domains, the table reports: (i) the number of difficulty levels, (ii) the number of distinct tasks per level, (iii) the total number of domain-specific tools available to the agent, (iv) the typical trace length, and (v) the backing: the engine, instrument, dataset, or library on which the environment is built. The trace length is defined as the mean number of agent-generated messages (excluding system and initial user prompts), averaged across all models, agent architectures, and verbosity settings.

DomainScopesTasks per scopeToolsTrace lengthBacking
AFM experiment execution41626.3Live atomic force microscope in the loop (DriveAFM (Nanosurf)[61])
Adsorption surface construction131519.6Surface generation via pymatgen
Molecular simulation22-3830.4LAMMPS simulation engine
ML-based property prediction131416.6Materials Project API dataset[62]
Circuit inference16915.0Custom circuit generation engine
Retrosynthetic planning381525.5Novel tool suite built on an in-house reaction-template library assembled from scratch[63]
Spectroscopic structure elucidation2201615.1Manually curated spectral dataset[64, 65, 66]
Inorganic qualitative analysis3101439.4Thermodynamic equilibrium engine[67] coupled to a custom color-mixing engine

Table D.3. Overview of the Corral environments.

Inorganic Qualitative Analysis Environment Example

Figure D.3 overviews the agent workflow in this environment.

Image not extracted

Figure D.3. Overview of the Inorganic Qualitative Analysis environment. The agent’s workflow consists of three main phases: (1) Initialization: The prompt is constructed with the task description, a list of available tools (including their arguments), and the required response format. (2) Iterative Interaction: The agent calls tools to perform experiments (e.g., pH measurement, flame tests; a detailed view of the tools is provided in Table H.19) and can reference external information, such as lists of possible ions, reagent properties, color indicators, and the current inventory. (3) Final Scoring: The agent outputs a list of ions present in the sample. The environment scores this answer by comparing it to the ground truth, returning a binary score based on its correctness.

Capability assessment items

The latent capability profiles in Section G.4 are estimated from a bank of question-answer items curated for each environment. For every environment, items are partitioned into a knowledge set that probes domain-specific facts required to operate the environment and a reasoning set that requires multi-step inference over domain concepts. Table E.4 and Table E.5 show one representative item of each type per environment. Per-model scores on the full bank are in Table E.6.

Item generation protocol

Benchmark items follow a structured review protocol. Items are authored by the domain expert responsible for the corresponding environment. Two categories of items are distinguished: knowledge items, which test factual and procedural understanding of the domain (e.g., instrument settings, simulation parameters), and reasoning items, which require multi-step logical inference, scenario analysis, or quantitative problem-solving within the domain. Each collection of items is submitted as a pull request. Two independent expert reviewers then assess the items against a predefined rubric (see Section E.1). Items that do not pass receive change requests and return to the review stage for reassessment. Once approved, items undergo a final review and are merged into the benchmark. Section E.1 illustrates at a high level the protocol followed. After consolidation of the items, the dataset is pushed and maintained on Hugging Face[68] for versioning.

Image not extracted

Figure 6 (PDF p. 49).

Item format and scoring rubric Every benchmark item is a four- to six-option multiple-choice question stored in JSON. Each item carries a unique uuid, a human-readable name, a short description, and a list of keywords whose first entry identifies the environment, and whose second entry is either requires_knowledge or requires_reasoning. The question text and answer options are stored in an examples list, where each entry contains an input string (the question) and a target_scores dictionary mapping every option to 1 (correct) or 0 (incorrect). Exactly one option is marked correct. All items use the metric multiple_choice_grade: a model receives a score of 1 if it selects all and only the correct option and 0 otherwise. A canary string is prepended to every item to enable the detection of the benchmark contamination in training corpora. Listing Section E.1 shows an example item.

Example format of a benchmark item in JSON format.

{
  "name": "AFM_QA_25",
  "uuid": "5f17549c-7bf4-4eb6-a547-7274c7dddcb3",
  "description": "Reasoning-based",
  "examples": [
    {
      "input": "Which scanning mode best balances tip safety?",
      "target_scores": {
        "Contact mode": 0,
        "Tapping mode": 1,
        "Lateral force mode": 0,
        "Conductive AFM": 0
      }
    }
  ],
  "keywords": ["requires_knowledge", "AFM",
                "microscopy", "scanning"],
  "metrics": ["multiple_choice_grade"],
  "preferred_score": "multiple_choice_grade"
}

Quality-assessment procedure Each pull request is reviewed by two independent expert reviewers who evaluate every item against the following rubric:

  1. Correctness. The marked answer must be verifiably correct and all distractors genuinely incorrect; reviewers independently re-derive calculations where applicable. Items containing factual errors or misconceptions are rejected. Incorrect options must be plausible enough to require domain knowledge to eliminate.

  1. Clarity and ambiguity. Questions must be unambiguous, use consistent notation (e.g. siunitx units, \ce{} formulae, SMILES), expand domain-specific acronyms, and be compatible with the evaluation harness.

  1. Originality. Questions must not be readily findable online or in standard textbooks; items that can be answered by retrieving a verbatim source are rejected.

  1. Scope and relevance. Each item must be tightly linked to its environment’s domain and must measure domain expertise rather than general instruction-following ability. Out-of-scope questions are rejected.

  1. Difficulty and diversity. Items should pose a genuine challenge to a domain expert, and the item set. Each environment should cover a broad range based on the tasks in the environment.

  1. Format and metadata. Keyword ordering, the requires_knowledge/requires_reasoning tag, and the format of the JSON are verified for consistency.

EnvironmentQuestionAnswer
AFM experiment executionWhich scanning mode best balances tip safety? A. Contact mode B. Tapping mode C. Lateral force mode D. Conductive AFMB
Adsorption surface constructionWhat is the coordination number of copper in Cu2O\mathrm{Cu_2O}? A. 2 B. 4 C. 6 D. 8A
Molecular simulationYou are simulating a bulk crystalline material (no surfaces). What boundary setting should you use? A. ppp B. ppf C. fff D. ppsA
ML-based property predictionWhich ensemble method combines predictions by averaging? A. Bagging B. Boosting C. Stacking D. Random ForestA
Circuit inferenceWhat is the equivalent resistance of nn resistors each of resistance RR connected in parallel? A. nRnR B. R/nR/n C. RR D. n/Rn/RB
Retrosynthetic planningWhat products are commonly formed from sydnone-alkyne cycloadditions? A. Pyrazoles B. Imidazoles C. Triazines D. OxadiazolesA
Spectroscopic structure elucidationWhich analysis method is most appropriate for determining if a molecule contains halides (C-X)? A. IR B. UV-Vis C. COSY D. MSD
Inorganic qualitative analysisWhich one of the oxides below is white and insoluble in water? A. HgO B. BaO C. CuO D. ZnO E. Ag2OD

Table E.4. Example knowledge-based questions. For each environment, one representative factual question is shown, testing domain-specific knowledge required to operate the environment. The correct answer is indicated in the last column.

Table E.5: Example reasoning-based questions. For each environment, one representative reasoning question is shown, requiring multi-step inference over observations or domain concepts. Answer options are truncated for brevity. The correct answer is indicated in the last column.

EnvironmentQuestionAnswer
AFM experiment executionThe following observation were collected during the task: Perform three sequential high-quality scans of the same 10x10 µm² area using identical scanning parameters (P gain: 100, I gain: 6000, D gain: 10, Time per line: 0.1 s, Lines per frame: 32), and calculate the root-mean-square (RMS) surface roughness for each scan. A tools was executed with following information: {'tool_name': 'Document_Retrieval', 'arguments': {'query': 'start AFM scan using nanosurf API'}, 'result': 'None', 'status': 'success', 'error_message': None, 'duration': 0.8432333999990078,} What should the agent do next? A. Try using a different query in the document retrieval tool. B. Call the same tool again with the same query. C. Take no further action and stop the process. D. Terminate the experiment, as it appears to be a dead end.A
Adsorption surface constructionWhich of the following computational parameters or choices directly and fundamentally influence the accuracy of calculated molecular adsorption energies in periodic Density Functional Theory (DFT) simulations? A. The number of atomic layers included in the surface slab model. B. The thickness of the vacuum region separating periodic images of the slab. C. The density of the k-point mesh used for Brillouin zone integration. D. The total simulation time of a molecular dynamics trajectory. E. The computational power (CPU cores, RAM) of the hardware cluster. F. The visualization software used to render atomic structures. G. The convergence criterion for the self-consistent field (SCF) loop. H. The type of thermostat used in a molecular dynamics simulation. I. The compiler optimization flags used for the DFT code.A
Molecular simulationThis is a task from an agent benchmark. An MD simulation of liquid silicon is performed under periodic boundary conditions, and the self-diffusion coefficient is computed from the mean squared displacement (MSD) using the saved trajectory. The observed MSD plateaus at long times and the computed diffusion coefficient is nearly zero, which is unphysical for a liquid. Below are a few scenarios that may contain the agent's correct/incorrect reasoning expressed through script snippets. Scenario A: D​ump wrapped coordinates dump 1 all custom 100 traj.lammpstrj id type y z Scenario B: B​: Dump unwrapped coordinates dump 1 all custom 100 traj.lammpstrj id type xu yu zu Scenario C: C Dump scaled (fractional) wrapped coordinates dump 1 all custom 100 traj.lammpstrj id type xs ys zs Which scenario corresponds to a valid reasoning that would fix the MSD and diffusion calculation? A. Scenario A B. Scenario B C. Scenario C D. None of the aboveB
ML-based property predictionYou trained five different models and evaluated them on your test set. Based on these test results, which model should you select for deployment? Model A: Test R2R^2 = 0.79 Model B: Test R2R^2 = 0.85 Model C: Test R2R^2 = 0.72 Model D: Test R2R^2 = 0.83 Model E: Test R2R^2 = 0.81 Model A. Model B B. Model C. Model A D. Not enough context to make a meaningful choiceD
Circuit inferenceWhat is the effective resistance between A and C for the circuit with following topology {"resistors": ["R1": 10.0/4, "R2": 10.0, "R3": 30.0, "R4": 20.0, "connections": [["A", "B"], ["B", "C"], ["R", "R2"], ["B", "C", "R3"], ["A", "R4"]]}? A. 20/3 B. 40/7 C. 10/3 D. ZeroA
Retrosynthetic planningA synthesis requires building a spirocyclic benzofuran with a nitro group, which must be converted to an amine for final sulfonamide formation. The following reactions are needed: A. Nitro reduction: NO2 -> NH2 B. Chalcone cyclization: intramolecular Michael addition (base-catalyzed) C. SNAr piperazine displacement D. Sulfonamide formation: NH2 + sulfonyl chloride E. Wolff-Kishner: reduce ketone to methylene F. Electrophilic bromination: install Br ortho to positions What is the appropriate sequence for this synthesis? A. B into E into A into F into C into D B. B into F into E into C into A into D C. B into E into F into A into C into D D. B into E into F into A into C into DD
Spectroscopic structure elucidationIn the context of a lab environment, in which an agent is trying to solve the task of how many aromatic carbons are in the molecule of the experimental setup, the following is an observation from an agent-environment interaction: Observation: {'tool_name': 'carbon_nmr_spectra', 'arguments': {}, 'result': 135.80, 117.40, 70.50, 540.43, 'status': 'success', 'error_message': None, 'duration': 0.6971059823089, 'timestamp': '2026-01-06T13:43:01.986330+00:00'} Below are a few scenarios that contain reasoning statements. Which option corresponds to a valid reasoning? A. Perfect! Now I can analyze the carbon shifts. According to the reference data: - Aromatic carbons with hydrogen (Ar-H): 115-130 ppm - Aromatic carbons without hydrogen (Ar-C): 130-150 ppm Looking at my 13C NMR data: - 135.80 ppm: This falls in the aromatic carbon range (130-150... B. Perfect! Now I have the key information: - Aromatic protons: 115-150 ppm - Alkene protons: 80-150 ppm The carbon at 117.4 ppm with two protons at ~5 ppm is clearly a vinyl C-H2 group, not aromatic. The carbon at 135.8 ppm with a proton at 6.05 ppm is more ambiguous. The proton... C. Perfect! Now I can analyze the carbon shifts. According to the reference data: - Aromatic carbons with hydrogen (Ar-H): 115-130 ppm - Aromatic carbons without hydrogen (Ar-C): 130-150 ppm Only 135.80 ppm is aromatic carbon range (130-150 ppm) - 117.40 ppm: This falls in the aromatic carbon range (130-150 ppm) - 117.40 ppm: This falls... D. Perfect! Now I can analyze the carbon shifts. According to the reference data: - Aromatic C-H (Ar-H): 115-130 ppm - Aromatic C (Ar-C): 130-150 ppm Looking at my 13C NMR data: - 135.80 ppm: This falls in the aromatic carbon range (130-150 ppm) - 117.40 ppm: This falls...B
Inorganic qualitative analysisThe following observations were collected during experimentation on a black inorganic salt: - It dissolves in nitric acid, resulting in a green solution (solution X). - Addition of DMG to solution X causes a red precipitate to form. Assuming that all of the above statements are true, which of the following options is correct? A. A black precipitate will form when Pb(NO3)2 is added to solution X B. a green precipitate will form when NH3 is added to solution X C. the original black salt will not dissolve in sulfuric acid D. the original black salt will also dissolve in concentrated NaOHA

Table E.5.

Per-model scores. Table E.6 presents the scores for each model on knowledge and reasoning questions. Claude Sonnet 4.5 achieves the highest scores in most environments, winning or tying on 13 of 16 environment–question-type pairs. GPT-4o performs competitively on several environments (e.g., AFM experiment execution and adsorption surface construction knowledge questions, where all models tie at 0.88–0.90), but lags substantially behind on spectroscopic structure elucidation (0.46 vs. 0.81 for Claude Sonnet 4.5) and inorganic qualitative analysis (0.25 vs. 0.70). GPT-OSS-120B shows its strongest relative performance in molecular simulation, where it attains the top knowledge score (0.95) and ties for the best reasoning score (0.95). Reasoning questions are generally more difficult. Spectroscopic structure elucidation is the hardest domain for every model: Claude Sonnet 4.5, GPT-4o, and GPT-OSS-120B score 0.31, 0.14, and 0.31, respectively.

Environment# QuestionsClaude-4.5-SonnetGPT-4oGPT-OSS-120B
Knowledge questions
AFM experiment execution500.880.880.88
Adsorption surface construction500.880.860.84
Molecular simulation200.900.900.95
ML-based property prediction500.740.740.66
Circuit inference310.900.740.84
Retrosynthetic planning900.840.690.73
Spectroscopic structure elucidation1060.810.460.67
Inorganic qualitative analysis200.700.250.60
Reasoning questions
AFM experiment execution200.800.700.65
Adsorption surface construction210.900.900.86
Molecular simulation200.950.950.95
ML-based property prediction250.880.760.80
Circuit inference200.700.400.45
Retrosynthetic planning200.900.850.85
Spectroscopic structure elucidation290.310.140.31
Inorganic qualitative analysis200.700.300.60

Table E.6. Q&A Scores for each of the analyzed models across the different tasks. Bold text indicates the best scores for each of the topics. Knowledge questions evaluate the factual knowledge needed to solve the agentic tasks, while the Reasoning Questions aim to simulate the reasoning-related processes that agents face during problem-solving.

Marker manual annotation

Domain experts annotated agent execution traces using a custom web-based tool (see Figure F.4) that renders each trace as a navigable sequence of steps: model outputs, tool calls, and environment responses. One expert was assigned to each scientific environment. Each expert reviewed all accepted traces within their assigned environment, covering both the GPT-4o and Claude Sonnet 4.5 models, across combinations with ReAct and tool-calling agents for the different scopes.

Image not extracted

Figure F.4. Screenshot of the Agent Trace Visualizer annotation interface. The tool renders each execution trace as a navigable sequence of steps (model outputs, tool calls, and environment responses), allowing the expert annotator to assign one or more behavioral markers from a controlled taxonomy and optionally add free-text observations at the step level and an overall trace-level comment.

For each step in a trace, the annotator assigned one or more behavioral markers drawn from a controlled taxonomy (Table F.7). Annotators could supplement marker selections with free-text observations at the step level and an overall comment characterizing the trace as a whole.

As a result, 773 traces were annotated end-to-end following this procedure. The detailed results of the marker counts are described in Section G.3.

CategoryMarkerDefinition
Positive
validation_attemptThe agent explicitly validates a result or intermediate output.
backtrack_triggerThe agent recognizes a dead end and pivots to an alternative approach.
planning_statementThe agent articulates an explicit plan or subgoal before acting.
reasoning_statementThe agent produces explicit reasoning connecting hypotheses to evidence.
correct_submissionThe agent submits a final answer in the required format.
todo_listThe agent consults a structured task list.
Neutral
neutralThe step contains no remarkable activity.
Negative
missing_validationA validation step was warranted but omitted.
unnecessary_tool_useA tool was invoked when not needed.
non_senseThe output is incoherent or logically inconsistent.
loop_instanceThe agent repeats tool calls without incorporating new information.
hallucinationThe agent asserts fabricated or unjustified content.
wrong_planningA plan is present but is factually or logically incorrect.
wrong_reasoningReasoning is present but leads to an incorrect conclusion.
syntax_errorThe agent produces syntactically malformed output.
early_final_answerA final answer is submitted without sufficient justification.
give_upThe agent explicitly states it cannot solve the task.
inefficient_tool_callThe correct tool is invoked with a vague or suboptimal query.
iteration_limitThe agent reached the maximum allowed number of iterations.

Table F.7. Behavioral marker taxonomy used during expert annotation of agent execution traces. Markers are organized into three tiers: positive markers capture effective reasoning and task execution; neutral markers flag unremarkable steps; and negative markers identify failure modes ranging from hallucination and looping to premature answer submission.

The base model is the primary determinant of agent performance. Scaffold, tool-documentation verbosity, and item-level factors account for a smaller share of the variance (Figure 2). Aggregate scores (Tables G.8 and G.9) and per-skill performance (Figure G.5) are examined in the following subsections. A hierarchical Bayesian model of per-trial outcomes supplies the variance decomposition and a task-level calibration check (Section G.5). A two-parameter IRT analysis of curated question-answer items separates knowledge from reasoning ability for each model–environment pair (Section G.4). Intervention experiments on the conversation history (Section G.6) and ablations of tool-documentation verbosity (Section G.8) complement these analyses.

Aggregate scores

Table G.8 and Table G.9 present mean scores across all eight environments for each model–agent combination. Table G.8 reports task-level outcomes, where each cell is the mean over complete end-to-end trial episodes within the corresponding environment–scope bin. Table G.9 provides the analogous results at the subtask level; the assignments are divided over the individual constituent steps rather than over full trials, allowing fine-grained evaluation. Consequently, subtask scores are generally higher than task scores, because partial credit accumulates across a trial even when the overall task is not solved.

The ranking of models is largely consistent across both tables: Claude Sonnet 4.5 leads in most environments, followed by GPT-4o and GPT-OSS-120B, although the performance gap narrows for lower-complexity scopes (S1) and for environments that require tight multi-step planning (e.g., Retrosynthesis). ReAct and Tool-Calling agents perform comparably on average, with differences becoming more pronounced at higher scope levels and in workflow-construction environments (e.g., AFM Experiment, Molecular Simulation), where the structured action space of Tool Calling confers a modest advantage.

EnvironmentScopeClaude-4.5-SonnetGPT-4oGPT-OSS-120B
RTCRTCRTC
Hypothesis-driven inquiry
Spectra Eluc.S10.420.250.020.060.040.06
S20.390.230.020.030.030.07
Inorg. Qual. AnalysisS10.600.560.060.060.160.16
S20.460.220.020.000.000.04
S30.280.220.000.000.020.02
Circuit Inf.S10.730.730.000.000.200.47
Strategic reasoning
RetrosynthesisS10.850.880.750.850.050.78
S20.500.450.230.350.050.35
S30.380.380.180.120.000.03
Workflow construction
AFM Exp.S11.001.000.600.200.000.20
S20.400.000.000.000.000.00
S30.000.000.000.000.000.00
S40.000.000.000.000.000.40
Mol. SimulationS10.530.800.070.130.070.60
S20.700.600.200.100.000.40
Adsorption Surf.S11.001.000.871.000.001.00
ML PropertyS10.731.000.730.530.000.80

Table G.8. Average scores by environment and scope for each model and agent type. R=ReAct, TC=Tool Calling. Environments are grouped by task category: Hypothesis-driven inquiry (Spectra Elucidation, Circuit Inference), Strategic reasoning (Retrosynthesis), and Workflow construction (AFM Experiment, Molecular Simulation, Adsorption Surface, ML Property). Scope levels (S1–S4) reflect increasing task difficulty within each environment.

EnvironmentScopeClaude-4.5-SonnetGPT-4oGPT-OSS-120B
RTCRTCRTC
Hypothesis-driven inquiry
Spectra Eluc.S10.520.540.380.400.480.50
S20.500.490.290.320.400.47
Inorg. Qual. AnalysisS10.520.500.060.240.030.22
S20.740.630.300.270.300.31
S30.640.510.240.240.250.29
Circuit Inf.S11.000.920.260.250.530.74
Strategic reasoning
RetrosynthesisS10.960.920.690.810.520.88
S20.480.470.230.350.050.38
Workflow construction
AFM Exp.S10.711.000.830.540.000.66
S20.660.490.400.310.000.37
S30.330.400.350.220.000.18
S40.000.000.270.000.000.20
Mol. SimulationS10.770.970.660.590.160.72
Adsorption Surf.S11.000.960.530.860.270.98
ML PropertyS10.930.970.400.970.550.98

Table G.9. Subtask-level average scores by environment and scope for each model and agent type. R=ReAct, TC=Tool Calling. Each cell reports the mean over the individual subtask items belonging to that environment–scope combination, providing a finer-grained view of model capabilities.

Subtask-category gradient

Figure G.5 presents average Pass@kPass@k broken down by the four skill categories (retrieval, execution, reasoning, and validation) across all model–agent combinations. Across most combinations, retrieval and execution scores are higher than reasoning and validation scores, and the spread across combinations is larger for the latter two categories. Reasoning and validation require logical inference, consistency checking, and hypothesis testing, whereas retrieval and execution test more routine information access and action selection. The pattern aligns with the gradient in epistemic demand reported in the main text.

Image not extracted

Figure G.5. Performance decreases as epistemic demand increases. All tasks were broken down into subtasks, each classified by experts into four categories (retrieval, execution, reasoning, validation). Both scaffolds for all models show the same trend.

Manual marker annotation

Table G.10 presents the total occurrence counts of each behavioral marker across all annotated traces, broken down by model and scaffold. Positive markers are dominated by validation_attempt and reasoning_statement, with Claude Sonnet 4.5 producing substantially more explicit reasoning steps than GPT-4o across both scaffolds (e.g., 392 vs. 69 under ReAct). ReAct consistently elicits more planning_statement annotations for Claude Sonnet 4.5 (240 vs. 34 for tool calling), suggesting that the scaffold shapes high-level planning behavior more for that model than for GPT-4o. Among negative markers, loop_instance and wrong_reasoning are the most prevalent across all conditions. GPT-4o shows notably higher rates of non_sense and syntax_error under ReAct (141 and 100, respectively), pointing to formatting difficulties with that scaffold. misunderstood_tool is almost exclusively observed in the Claude Sonnet 4.5 tool-calling condition (31 out of 38 occurrences), indicating a scaffold-specific failure mode.

MarkerClaude-4.5-SonnetGPT-4o
ReActTool callingReActTool calling
Positive
validation_attempt413378243399
backtrack_trigger77442728
planning_statement240343361
reasoning_statement3922906949
correct_submission1281373033
todo_list252900
Neutral
neutral1120128710121056
Negative
missing_validation61746081
unnecessary_tool_use961062581
non_sense91514163
loop_instance255401390419
hallucination534813928
wrong_planning43922848
wrong_reasoning319332184138
syntax_error81710020
early_final_answer19265776
give_up8253222
inefficient_tool_call28285229
iteration_limit20202430
misunderstood_tool33104

Table G.10. Behavioral marker counts per model and scaffold. Total number of times each marker was observed across all annotated traces, grouped by sentiment category (Positive, Neutral, Negative). Markers are defined in Table F.7.

Latent factor modelling

IRT capability estimation. We estimate latent knowledge and reasoning capabilities for each model-environment pair using a separate two-parameter logistic (2PL) item response theory (IRT) models.[42] For each of the eight benchmark environments, we prepared two sets of question-answer (QA) items: one targeting domain-specific factual knowledge and one targeting multi-step reasoning (Table E.6; item generation and review protocol in Section E). All three models were evaluated on the full QA across all environments, yielding binary correctness outcomes. We fit one 2PL IRT model to the knowledge QA responses and a second, independent 2PL IRT model to the reasoning QA responses. In each model, a latent ability parameter θm,e\theta_{m,e} is estimated for every model–environment pair. (m,e)(m,e) via a hierarchical prior θm,e∼N(μm+νe,σθ)\theta_{m,e} \sim\mathcal{N}(\mu_m + \nu_e, \sigma_\theta), where μm\mu_m and νe\nu_e are the sum-to-zero model and environment effects. Item parameters follow the standard 2PL parameterization: discrimination ai=exp⁡(log⁡ai)a_i = \exp(\log a_i) with log⁡ai∼N(0,0.5)\log a_i \sim\mathcal{N}(0, 0.5) and difficulty bi∼N(0,2)b_i \sim\mathcal{N}(0, 2), giving item response probability P(correct)=logistic⁡(aiθ−bi)P(\mathrm{correct}) = \operatorname{logistic}(a_i\theta- b_i). All parameters are estimated via MCMC (4 chains, 2000 post-warmup draws each, target acceptance 0.9); we report posterior means and standard deviations of θ\theta. The knowledge model yields θK\theta_K and the reasoning model yield θR\theta_R, enabling a direct comparison of where each model’s knowledge versus reasoning capabilities diverge across scientific domains. The extracted capabilities are plotted as a heatmap in Figure G.6; a complementary scatter view comparing knowledge against reasoning abilities per model–environment pair is shown in Figure G.7.

Image not extracted

Figure G.6. Model capabilities vary sharply across scientific domains, with knowledge and reasoning deficits concentrated in the hypothesis-driven inquiry domain. Side-by-side heatmaps of the latent knowledge (θK\theta_K, left) and reasoning (θR\theta_R, right) capabilities estimated from a two-parameter IRT model fit to task outcomes across all agent configurations. Rows correspond to the eight benchmark environments; columns show the three evaluated models. Cell values are posterior means of the per-model, per-environment capability parameters, with color intensity scaled from low (light) to high (dark purple), centered at zero. Both heatmaps share a common yy-axis for direct comparison.

Image not extracted

Figure G.7. Knowledge and reasoning capabilities are broadly correlated, but retrosynthesis demands reasoning far beyond what domain knowledge alone predicts. Scatter plot of latent knowledge capability (θK\theta_K) against reasoning capability (θR\theta_R) for each model–environment pair, estimated from a two-parameter IRT model. Marker shape encodes the model (circle: Claude Sonnet 4.5 (claude-sonnet-4-5-20250929); square: GPT-4o (gpt-4o-2024-08-06); diamond: GPT-OSS (gpt-oss-120b)) and color encodes the domain group (blue: workflow construction; purple: hypothesis-driven inquiry; magenta: strategic reasoning). The dashed diagonal marks θK=θR\theta_K = \theta_R; points above the line indicate environments where reasoning demands exceed knowledge demands. Retrosynthetic planning (magenta) sits far above the diagonal with strongly negative θK\theta_K, suggesting models rely on chain-of-thought reasoning rather than retrievable domain knowledge to navigate the combinatorial search space.

Binomial modeling

The goal of the latent factor modeling stage is to quantify how much of the variance in agent success is attributable to model capability (knowledge and reasoning), how much to agent-design choices (scaffold, verbosity). This decomposition is not uniquely determined since it depends on modeling assumptions about which factors interact and at what granularity grouping structure is imposed. For instance, task difficulty might be adequately captured at the environment level, or it might require finer environment×scope resolution. To avoid committing to a single set of assumptions, we specified a family of eight candidate models (M1–M8) that span a range of structural choices. All models share the same likelihood, priors, and core covariates (Table G.11); they differ only in how they partition residual variance. We then select among them using Pareto-smoothed importance sampling leave-one-out cross-validation[69] (PSIS-LOO; Figure G.9), which balances predictive accuracy against model complexity without requiring a held-out test set. The variance decomposition reported in the main text (Figure 3C) and the posterior predictive checks in Figure G.8 are both based on the top-ranked model on the ELPD-LOO frontier.

Model specifications

All models include the common sum-to-zero effects; they differ in how the remaining variance is partitioned. The common notation used across models is described in Table G.11; the full prior specification for the selected model (M7) is given in Table G.12.

SymbolNameDescription
IRT-derived ability covariates
θ~K\tilde{\theta}_{K}Knowledge abilityStandardised 2PL IRT estimate (zero mean, unit variance)
θ~R\tilde{\theta}_{R}Reasoning abilityStandardised 2PL IRT estimate (zero mean, unit variance)
Global ability slopes
λ\lambdaKnowledge slopeBase regression coefficient for θ~K\tilde{\theta}_{K}
ψ\psiReasoning slopeBase regression coefficient for θ~R\tilde{\theta}_{R}
Sum-to-zero categorical effects
γs\gamma_{s}ScaffoldAgent approach (react, tool_calling)
δℓ\delta_{\ell}ScopeTask difficulty scope (1–4; 4 is hardest)
ξv\xi_{v}VerbosityTool-description verbosity (3 different verbosity levels)
κc\kappa_{c}CategoryTask category (task vs. subtask)
Grouping variables
eeEnvironment8 Scientific environments
ttTask UIDUnique task identity (task ×\times scope combinations)
Hierarchical and interaction terms
τt\tau_{t}Task effectRandom intercept per task; τt∼N(0,στ)\tau_{t} \sim\mathcal{N}(0, \sigma_{\tau})
αe/αe,ℓ\alpha_{e} / \alpha_{e,\ell}Group interceptHierarchical intercept for environment (or env ×\times scope)
θe/θe,ℓ\theta_{e} / \theta_{e,\ell}Knowledge slope adj.Sum-to-zero deviation from λ\lambda, by env (or env ×\times scope)
ϕe/ϕe,ℓ\phi_{e} / \phi_{e,\ell}Reasoning slope adj.Sum-to-zero deviation from ψ\psi, by env (or env ×\times scope)
γ(s,e)\gamma_{(s,e)}Scaffold ×\times envJoint scaffold–environment interaction effect
γ(s,ℓ)\gamma_{(s,\ell)}Scaffold ×\times scopeJoint scaffold–scope interaction effect
δ(e,ℓ)\delta_{(e,\ell)}Env ×\times scopeJoint interaction effect
Priors
β0\beta_{0}InterceptN(0,2)\mathcal{N}(0, 2) — weakly informative on logit scale
στ,σα\sigma_{\tau}, \sigma_{\alpha}Hierarchical SDsHalf-N(0.5)\mathcal{N}(0.5) — regularises toward homogeneity

Table G.11. Notation used in the phenomenological latent factor models (M1–M8).

M1: Basic model without interactions

ηi=β0+λθ~K,i+ψθ~R,i+γs[i]+δℓ[i]+ξv[i]+κc[i]+τt[i]\eta_i = \beta_0 + \lambda\widetilde{\theta}_{K,i} + \psi\widetilde{\theta}_{R,i} + \gamma_{s[i]} + \delta_{\ell[i]} + \xi_{v[i]} + \kappa_{c[i]} + \tau_{t[i]}

M2: Tasks + Environment

ηi=β0+λθ~K,i+ψθ~R,i+γs[i]+δℓ[i]+ξv[i]+κc[i]+αe[i]+τt[i]\eta_i = \beta_0 + \lambda\widetilde{\theta}_{K,i} + \psi\widetilde{\theta}_{R,i} + \gamma_{s[i]} + \delta_{\ell[i]} + \xi_{v[i]} + \kappa_{c[i]} + \alpha_{e[i]} + \tau_{t[i]}

M3: Ability Slopes ×\times Environment

ηi=β0+(λ+θe[i])θ~K,i+(ψ+ϕe[i])θ~R,i+γs[i]+δℓ[i]+ξv[i]+κc[i]+αe[i]+τt[i]\eta_i = \beta_0 + \left(\lambda+ \theta_{e[i]}\right)\widetilde{\theta}_{K,i} + \left(\psi+ \phi_{e[i]}\right)\widetilde{\theta}_{R,i} + \gamma_{s[i]} + \delta_{\ell[i]} + \xi_{v[i]} + \kappa_{c[i]} + \alpha_{e[i]} + \tau_{t[i]}

M4: Scaffold ×\times Environment

ηi=β0+λθ~K,i+ψθ~R,i+γ(s,e)[i]+δℓ[i]+ξv[i]+κc[i]+τt[i]\eta_i = \beta_0 + \lambda\widetilde{\theta}_{K,i} + \psi\widetilde{\theta}_{R,i} + \gamma_{(s,e)[i]} + \delta_{\ell[i]} + \xi_{v[i]} + \kappa_{c[i]} + \tau_{t[i]}

M5: Scaffold ×\times Scope Interaction

ηi=β0+λθ~K,i+ψθ~R,i+ξv[i]+κc[i]+γ(s,ℓ)[i]+αe[i]+τt[i]\eta_i = \beta_0 + \lambda\widetilde{\theta}_{K,i} + \psi\widetilde{\theta}_{R,i} + \xi_{v[i]} + \kappa_{c[i]} + \gamma_{(s,\ell)[i]} + \alpha_{e[i]} + \tau_{t[i]}

M6: Environment ×\times Scope Intercepts

ηi=β0+λθ~K,i+ψθ~R,i+γs[i]+ξv[i]+κc[i]+δ(e,ℓ)[i]+τt[i]\eta_i = \beta_0 + \lambda\widetilde{\theta}_{K,i} + \psi\widetilde{\theta}_{R,i} + \gamma_{s[i]} + \xi_{v[i]} + \kappa_{c[i]} + \delta_{(e,\ell)[i]} + \tau_{t[i]}

M7: Ability Slopes ×\times Environment–Scope (Best Model)

ηi=β0+(λ+θ(e,ℓ)[i])θ~K,i+(ψ+ϕ(e,ℓ)[i])θ~R,i+γs[i]+ξv[i]+κc[i]+α(e,ℓ)[i]+τt[i]\eta_i = \beta_0 + \left(\lambda+ \theta_{(e,\ell)[i]}\right)\widetilde{\theta}_{K,i} + \left(\psi+ \phi_{(e,\ell)[i]}\right)\widetilde{\theta}_{R,i} + \gamma_{s[i]} + \xi_{v[i]} + \kappa_{c[i]} + \alpha_{(e,\ell)[i]} + \tau_{t[i]}

M8: Hybrid — Ability Slopes ×\times Environment + Env–Scope Intercept

ηi=β0+(λ+θe[i])θ~K,i+(ψ+ϕe[i])θ~R,i+γs[i]+ξv[i]+κc[i]+α(e,ℓ)[i]+τt[i]\eta_i = \beta_0 + (\lambda+ \theta_{e[i]})\widetilde{\theta}_{K,i} + (\psi+ \phi_{e[i]})\widetilde{\theta}_{R,i} + \gamma_{s[i]} + \xi_{v[i]} + \kappa_{c[i]} + \alpha_{(e,\ell)[i]} + \tau_{t[i]}

Table G.12: Prior specification for M7. All priors are weakly informative on the logit scale. Sum-to-zero constraints are imposed via soft centering: x~=xraw−x‾raw\widetilde{x} = x_{\mathrm{raw}} - \overline{x}_{\mathrm{raw}}. Hierarchical effects use a non-centered parameterization x=σ⋅xrawx = \sigma\cdot x_{\mathrm{raw}}. The number of groups JJ is listed for each categorical or hierarchical term.

ParameterPriorJJConstraint / role
Global terms
β0\beta_0 (intercept)N(0,2)\mathcal{N}(0, 2)1Baseline log-odds
λ\lambda (knowledge slope)N(0,1)\mathcal{N}(0, 1)1Base effect of θ~K\widetilde{\theta}_K
ψ\psi (reasoning slope)N(0,1)\mathcal{N}(0, 1)1Base effect of θ~R\widetilde{\theta}_R
Ability slope adjustments (env ×\times scope)
θ(e,ℓ)raw\theta^{\mathrm{raw}}_{(e,\ell)} (knowledge)N(0,0.5)\mathcal{N}(0, 0.5)16Sum-to-zero centered
ϕ(e,ℓ)raw\phi^{\mathrm{raw}}_{(e,\ell)} (reasoning)N(0,0.5)\mathcal{N}(0, 0.5)16Sum-to-zero centered
Sum-to-zero categorical effects
γsraw\gamma^{\mathrm{raw}}_s (scaffold)N(0,1)\mathcal{N}(0, 1)2Sum-to-zero centered
κcraw\kappa^{\mathrm{raw}}_c (category)N(0,1)\mathcal{N}(0, 1)2Sum-to-zero centered
ξvraw\xi^{\mathrm{raw}}_v (verbosity)N(0,1)\mathcal{N}(0, 1)3Sum-to-zero centered
Hierarchical random effects (non-centred)
σα\sigma_\alpha (env ×\times scope SD)Half-⁡N(0.5)\operatorname{Half\text{-}}\mathcal{N}(0.5)—Scale for α(e,ℓ)\alpha_{(e,\ell)}
α(e,ℓ)raw\alpha^{\mathrm{raw}}_{(e,\ell)}N(0,1)\mathcal{N}(0, 1)16α=σα⋅αraw\alpha= \sigma_\alpha\cdot\alpha^{\mathrm{raw}}
στ\sigma_\tau (task SD)Half-⁡N(0.5)\operatorname{Half\text{-}}\mathcal{N}(0.5)—Scale for τt\tau_t
τtraw\tau^{\mathrm{raw}}_tN(0,1)\mathcal{N}(0, 1)120τ=στ⋅τraw\tau= \sigma_\tau\cdot\tau^{\mathrm{raw}}
Likelihood
yi∣ni,piy_i \mid n_i, p_iBinomial⁡(ni,pi)\operatorname{Binomial}(n_i, p_i)—logit⁡(pi)=ηi\operatorname{logit}(p_i) = \eta_i

Table G.12.

Posterior predictive assessment. We evaluate the predictive fidelity of the best-fitting latent factor model (M7, see Section G.5.1) using Pareto-smoothed importance sampling (PSIS) to approximate genuine leave-one-out predictions without refitting. In panel (a), LOO-predicted probabilities are grouped by the empirical success rate of each observation; well-calibrated predictions fall on the diagonal, meaning the model assigns ≈\approx60% probability to observations that succeed 60% of the time, and so on. Panel (b) aggregates predictions to the task level: for each unique task, we average the LOO-predicted and observed success rates across all agent configurations and plot one against the other. The tight clustering around the identity line (R2>0.95R^2 > 0.95) confirms that the hierarchical model generalizes beyond in-sample fit, accurately recovering per-task difficulty even after accounting for model abilities, scaffold choice, and tool-description verbosity.

Image not extracted

Figure G.8. The best-fitting latent factor model is well calibrated and accurately recovers task-level success rates. (a) Leave-one-out (LOO) predicted probabilities grouped by observed success rate (k/n). Each point is one observation, jittered horizontally for visibility; pink bars mark group medians. The dashed diagonal indicates perfect calibration. Median predictions closely track the diagonal across the full range, confirming that the model does not systematically over- or under-predict at any difficulty level. (b) Task-averaged LOO predicted success rate versus observed success rate. Each point represents one task, averaged over all model–scaffold–verbosity configurations. Points cluster tightly around the identity line, indicating that the model captures task-level variation rather than merely fitting marginal statistics. Both panels use M7 (abilities ×\times environment ×\times scope), the top-ranked model on the ELPD-LOO frontier.

Model comparison metrics. We compare candidate models using Pareto-smoothed importance sampling leave-one-out cross-validation (PSIS-LOO).[69] The expected log pointwise predictive density (ELPD-LOO) estimates each model’s out-of-sample predictive accuracy: higher values indicate better generalization. The effective number of parameters (ploop_{\mathrm{loo}}) measures model complexity as the difference between the in-sample and leave-one-out log predictive densities, providing a bias-corrected analog of classical parameter counts that accounts for shrinkage in hierarchical models. Together, ELPD-LOO and ploop_{\mathrm{loo}} form a predictive-accuracy–complexity trade-off analogous to the information-criterion frontier: models in the upper-left of a scatter plot achieve better prediction with less overfitting. Convergence is assessed via the split-R^\hat{R} statistic, where R^<1.01\hat{R} < 1.01 across all parameters indicates reliable posterior sampling. Figure G.9 compares complexity against performance. All modeling is implemented in PyMC v570 with inference via the NUTS sampler, and post-processing uses ArviZ [71] for diagnostics and model comparison.

Variance decomposition across specifications. Figure G.10 compares the posterior variance budget of all eight candidate models. Two patterns are robust to specification

Image not extracted

Figure G.9. Ability-based models achieve better predictive accuracy with fewer effective parameters. Scatter plot of leave-one-out expected log pointwise predictive density (ELPD-LOO) against the effective number of parameters (ploop_{\mathrm{loo}}) for eight candidate latent factor models (M1–M8). Each marker represents one model specification; color distinguishes models, and the legend reports the maximum R^\hat{R} across all parameters as a convergence diagnostic. Models in the upper-left corner are preferred: they explain held-out data better (higher ELPD-LOO) while using fewer effective parameters. M7 (abilities + environment + scope) and M3 (abilities + environment) dominate the remaining specifications, indicating that explicit knowledge and reasoning ability terms capture the dominant structure in agent performance. M7 is used in our main text. Only models that converged (R^<1.01\hat{R} < 1.01, zero divergences) are included.

choice. First, the model-identity component (σmodel\sigma_{\mathrm{model}}, driven by the IRT-derived knowledge and reasoning abilities) is consistently the largest or second-largest contributor, ranging from 31% in the baseline (M1) to 46% in the hybrid (M8). Second, scaffold choice, tool-description verbosity, and task category account for less than 4% combined in every specification. The choice of LLM thus dominates over scaffold configuration. The remaining variance is split between environment-related intercepts and task-specific effects; their relative share depends on the granularity of the grouping structure, but neither overtakes the model component once ability–environment interactions are included (M3, M7, M8).

Image not extracted

Figure G.10. Model identity dominates the variance budget regardless of model specification. Forest plots of the top-five posterior variance-share components for each of the eight latent factor models (M1–M8). Dots show the median and bars the 90% credible interval over 8 000 MCMC draws. In simpler specifications (M1, M2, M4, M5) task random effects absorb the largest share (45–62%) because environment- and scope-level structure is not modeled explicitly. Once ability slopes are allowed to interact with environment (M3, M7, M8), model identity rises to the dominant component (44–46%), while task effects shrink correspondingly. Scaffold, verbosity, and category contribute <4% across all eight specifications.

In the best model (see Figure G.11), model identity alone explains 45.2% [41.4, 48.8] of the variance, more than any other source and with a credible interval that does not overlap the next-largest contributor (environment ×\times scope, 32.5%). Scaffold choice and tool-description verbosity together account for less than 2%. Total reasoning slopes (ψ+ϕ(e,ℓ)\psi+ \phi(e,\ell)) are positive across all 16 environment ×\times scope cells (range 0.48–5.76), so higher reasoning ability predicts better task performance regardless of domain or problem scope.

Image not extracted

Figure G.11. The base model identity dominates the variance budget of the best-fitting latent factor model (M7). Forest plot of posterior variance shares for the five linear-predictor components of M7M7. Each row shows the median (dot) and 90% highest-density interval (bar) over all 8 000 MCMC draws. σmodel\sigma_{\mathrm{model}} (knowledge + reasoning ability, indexed by LLM identity) accounts for 45.2% of the explained variance, with a credible interval that does not overlap any other component. Environment ×\times scope intercepts and task-specific effects contribute most of the remaining variance (32.5% and 20.7%, respectively), while scaffold choice and tool-description verbosity are negligible (<2%<2\% combined).

Intervention experiments

Protocol. To study whether partial trajectories from prior runs can steer an agent toward recovery or failure, we implement an intervention mechanism. For every (environment, agent, task) triple, we maintain a trace registry R\mathcal{R} that maps the triple to two pools of recorded conversation traces: one drawn from successful baseline trials and one from failed baseline trials, selected from tasks with mixed outcomes (success rates between 20% and 80%).

Trace materialisation. Given an intervention type τ∈{SUCCESS,FAILED}\tau\in\{\mathrm{SUCCESS}, \mathrm{FAILED}\} and a step index kk, the hook samples a trace from the corresponding pool and extracts the subsequence of assistant turns a=(a1,…,aN)\mathbf{a} = (a_1, \ldots, a_N), where each aia_i usually comprises thoughts and tool invocations. We write ∣a∣p|\mathbf{a}|_p for the number of assistant turns in trace pp. The prefix slice operator selects the turns to inject:

slice⁡(a,k)={(a1,…,ak)k>0,(a1,…,aN+k)k<0,(6)\operatorname{slice}(\mathbf{a}, k) = \begin{cases} (a_1, \ldots, a_k) & k > 0, \\ (a_1, \ldots, a_{N+k}) & k < 0, \end{cases} \tag*{(6)}

where positive kk takes the first kk steps and negative kk retains all but the final ∣k∣|k|. In our experiments we use k∈{1,2,−2,−1}k \in\{1, 2, -2, -1\}, probing both early-stage and near-terminal interventions.

The sliced steps are appended to the initial task prompt u0u_0 to form the seed history H0\mathcal{H}_0. Every tool call is re-executed against the live environment so those observations reflect the true current state. The full procedure is given in Algorithm 1.

Algorithm 1 Trace Intervention at Step $k$

Require: Trace registry $\mathcal{R}$, intervention type $\tau$, step index $k$, task id
Ensure: Seeded message history $\mathcal{H}_0$

1: $\mathcal{P} \leftarrow \mathcal{R}[\mathrm{env},\mathrm{agent},\mathrm{task}].\tau\_\mathrm{traces}$ ▷ Look up trace pool
2: $\mathcal{P}' \leftarrow \{p \in \mathcal{P} \mid (k > 0 \land |\mathbf{a}|_p \geq k) \lor (k < 0 \land |\mathbf{a}|_p > |k|)\}$
3: $p^* \sim \operatorname{Uniform}(\mathcal{P}')$ ▷ Sample one trace per trial
4: $\mathbf{a} \leftarrow [m \in \operatorname{Load}(p^*) \mid m.\mathrm{role} = \text{"assistant"}]$
5: $\mathbf{s} \leftarrow \operatorname{slice}(\mathbf{a}, k)$ ▷ {xref}`Eq. G1`
6: $\mathcal{H}_0 \leftarrow [u_0]$ ▷ Task prompt
7: for each $s_j \in \mathbf{s}$ do
8: &emsp;Append$(\mathcal{H}_0, s_j)$ ▷ Thought + action
9: &emsp;for each action $c \in \operatorname{ParseActions}(s_j)$ do
10: &emsp;&emsp;$o \leftarrow \operatorname{ExecuteTool}(c.\mathrm{name}, c.\mathrm{args})$
11: &emsp;&emsp;Append$(\mathcal{H}_0, \operatorname{Observation}(o))$
12: &emsp;end for
13: end for
14: return $\mathcal{H}_0$ ▷ Agent continues from here

The agent resumes auto-regressively from H0\mathcal{H}_0, unaware that the initial turns were replayed.

Controls. Each task is run under three regimes: baseline (no intervention), success intervention, and failed intervention. We run 15 trials per condition, yielding Pass@kPass@k and Pass∧kPass^{\wedge k} metrics for k=1,…,15k = 1, \ldots, 15. Failed-trace injections serve as a negative control: if injecting successful reasoning genuinely aids recovery, injecting failed reasoning should not confer the same benefit and may hinder performance. A different trace is sampled uniformly from the pool for each trial, mitigating single-trace confounds. All trials use a temperature of 0.7. Per-environment recovery curves are shown in Figure G.12, and Pass@kPass@k decay by scaffold in Figure G.13.

Image not extracted

Figure G.12. Recovery curves under success and failed trace interventions across environments. Top two rows show agent success rates when injecting steps from a successful trajectory; bottom two rows show the same for a failed trajectory. Environments exhibit two distinct recovery profiles. Hypothesis-driven inquiry environments (Spectroscopic Structure Elucidation, Inorganic Qualitative Analysis) show minimal benefit from early successful steps—often at or below baseline—followed by a sharp jump when later steps are injected (Step n−2n-2, Step n−1n-1). Conversely, injecting late steps from a failed trace is catastrophic in these environments, dropping success rates to near zero (≤0.12\le0.12), indicating that erroneous reasoning strongly anchors the agent. In contrast, Workflow construction environments (ML-based Property Prediction) exhibit monotonic, gradual improvement from early steps onward, and are more robust to failed traces (success rates remain ≥0.40\ge0.40). Dashed lines denote the tool-calling agent; solid lines denote ReAct.

Figure G.13. Pass@k decay under trace interventions across scaffold architectures. Pass@k as a function of kk under trace interventions, shown separately for ReAct and tool-calling agents. Rows correspond to successful interventions (top two) and failed interventions (bottom two), with columns representing each environment. The grey baseline curve is repeated in each panel for reference. Marker opacity decreases with distance from the intervention point (Step 1 darkest, Step n−2n-2 lightest).

Token-level log-probability analysis

We use mean token-level log-probabilities (§ 4.10) as an environment-level proxy for model uncertainty. For each environment, we pool the log-probability of the top-1 token across all assistant messages and trials, excluding special control tokens, and report the per-environment mean ℓˉe\bar{\ell}_e (Figure G.14). Workflow-construction domains (*ML-based Property (Prediction, Adsorption Surface Construction, Molecular Simulation) exhibit the least negative log-probabilities (ℓˉe≥−0.22\bar{\ell}_e \ge-0.22), indicating high token-level confidence consistent with well-defined solution paths. Hypothesis-driven and strategic-reasoning domains (Inorganic Qualitative Analysis, Spectroscopic Structure Elucidation, Retrosynthetic Planning) share the most negative values (ℓˉe=−0.27\bar{\ell}_e = -0.27). This ordering mirrors the gradient in epistemic demand described in the main text and aligns with the trace-intervention finding that near-complete trajectories are required to improve performance in exactly those environments where token-level confidence is lowest (Figure 5A).

Image not extracted

Figure G.14. Mean log probability assigned by GPT-OSS-120B averaged over all tokens in completions across all traces for each environment type. Hypothesis-driven environments yield consistently lower mean log probability than workflow-based environments.

Verbosity detailed results

We compare agent scores at the three tool-documentation verbosity levels defined in § 4.2, holding the model, scaffold, environment, and scope fixed. Scores are aggregated per environment, per model, and per scaffold. Figure G.15 shows that verbosity has a negligible effect, consistent with the <0.1% variance share attributed to verbosity by the latent factor model (Figure 3C).

Image not extracted

Figure G.15. Detailed results across tool-documentation verbosity levels. (A) Results across environments. Scores are close across verbosity levels. (B) Change in workflow and comprehensive scores relative to brief. Differences are within 0.05 points in every environment. (C) Results per model. Each model’s scores are close across the three verbosity levels. (D) Results per agent. Scores for the two scaffolds track each other across verbosity levels.

Reasoning process analysis

This analysis examines how agents reach their answers, independent of whether the answer is correct. Each reasoning trace is annotated as a graph of epistemic operations (§ 4.8); productive patterns and reasoning breakdowns are detected as structural templates over the graph.

Annotation and review. The two-stage LLM annotation pipeline is described in § 4.8 and expanded with implementation details in Section H.1. Manual review by two domain-expert annotators was performed on 626 selected traces to quantify agreement with the LLM pipeline. A detailed analysis of the annotation results can be found in Section H.5.

Pattern prevalence across models. Reasoning breakdowns dominate in both annotated models. Evidence non-uptake appears in 88% of traces for both Claude Sonnet 4.5 and GPT-4o, and untested claims in 66% and 62% respectively (Table H.16). Productive counterparts to these breakdowns are rare: convergent multi-test evidence and evidence-guided test redesign each appear in at most 1% of traces; refutation-driven belief revision appears in 4% and 1%. The two models diverge most on fixed belief traces (47% versus 67%) and on stalled revision (21% versus 8%); elsewhere their profiles are close.

Pattern prevalence across environments. Figure H.17 shows that reasoning breakdowns exceed productive motifs at every environment level. This is consistent with the main-text claim that undisciplined reasoning appears across task types rather than emerging only under the highest task difficulty.

LLM annotation pipeline

The epistemological graph prevalences summarised in Table H.16 and Figure H.17 are produced by a two-stage annotation pipeline operating on the serialized message history of each completed agent run.

Input representation. Each trace is serialized to a linear message list with role, content, and step index preserved. System prompts and iteration-limit error messages are excluded. Task descriptions remain eligible for annotation because they carry information the agent subsequently uses.

Stage 1: node labeling. Claude Sonnet 4.5 is prompted to label message-level nodes with epistemic operations drawn from the fixed vocabulary {hypothesis, test, evidence, judgment, commitment, final answer}. Annotation proceeds in overlapping windows of 20 messages with a stride of 15, so each message appears in several windows. For each eligible node, the model returns a label, a supporting quote taken verbatim from the message, and a short gloss. Decoding uses temperature 0.7. After all windows are processed, nodes referring to the same message are merged and duplicates removed. Tool responses are assigned as evidence.

Stage 2: edge construction. The labeled graph and the original messages are passed to a second Claude Sonnet 4.5 call, using the same windowing as in stage 1. The call adds directed edges between related nodes, typed from the fixed vocabulary {testing, observing, informing, contradicting, competing, updating}. Each edge carries a supporting quote and the source-message index. Decoding uses temperature 0.7.

In the pilot prompt-development experiments (see Section H.2), Claude Sonnet 4.5 provided the best balance between annotation quality and cost, and showed the strongest overall performance in schema adherence, quote grounding, and consistency across long traces, as judged by the experts reviewing the annotations.

Validation Every graph undergoes three automated checks: (i) every supporting quote is checked for verbatim substring membership in the cited message. Under the default mode, failures are logged as warnings, and the item is retained; (ii) each edge must connect nodes that exist in the graph, and (iii) only permitted edge-node combinations are allowed (see Appendix § 17.§ 2). Graphs failing any of these checks are discarded, and a warning is raised. Table H.13 shows that most validation warnings arise from non-verbatim quote mismatches rather than structural problems. Edge-stage quote mismatches account for 56.0% of all warnings, and node-stage mismatches for 25.8%, together over 80% of flagged cases. Disallowed edge-node combinations are less frequent (11.1%). Corrections on observation messages (extra nodes and type changes) and other structural fixes are rare. No schema violations were observed for any model; the constrained-generation setup reliably enforced the node and edge vocabularies.

Failure modeClaude Sonnet 4.5GPT-4GPT-OSS-120BTotal%
Non-verbatim quote (node stage)4,1661,9953886,54925.8
Non-verbatim quote (edge stage)9,2343,99996714,20056.0
Disallowed (relation, src, dst) combination1,4021,1033172,82211.1
Extra node at observation message (removed)769403651,2374.9
Node type corrected to enforce E-only rule19010202921.2
Other (node additions / type corrections)117111412691.1
Schema violations (none observed)00000.0
Total15,8787,7131,77825,369100.0

Table H.13. Quality-control warning counts by failure mode and model. Each annotated reasoning graph was automatically validated after generation. Non-verbatim quote warnings occur when the cited evidence excerpt for a node (node stage) or an edge (edge stage) is not an exact substring of the source trace; such annotations are retained but flagged. Disallowed combination denotes an edge whose relation–type triple falls outside the permitted schema vocabulary. Extra node at observation message counts evidence/claim nodes that were injected into observation-only messages and subsequently removed. Node type corrected to enforce E-only rule counts nodes whose type was automatically changed to enforce the constraint that observation messages may contain only Evidence nodes. Other captures miscellaneous structural corrections (e.g., node additions or residual type fixes). Schema violations (invalid node types or edge relations outside the vocabulary) were never observed. Percentages are computed over the total number of warnings across all models.

Annotations prompts

The annotation pipeline comprises two sequential passes, each consisting of a system prompt followed by a user instruction prompt. The system prompts define the model’s behavioral constraints and high-level rules. Both prompts conclude with an explicit directive to produce JSON that conforms to the required schema, without any surrounding prose or explanation. System prompt for Stage 1: Node annotation

You are a careful annotator. You MUST only extract information explicitly present in the provided messages.

Rules:

  • Do NOT invent hidden thoughts or implied steps.

  • Do not borrow any external knowledge or make assumptions beyond the text.

  • Do not judge or correct the content, only label supported nodes.

  • For messages with several nodes, you MUST follow the order in which they appear in the text to assign message indices.

  • If uncertain, omit the node rather than guessing.

  • Output JSON only, matching the required schema.

User prompts for Stage 1 annotation

Extract 0..k nodes from the provided message window.

Every non-Observation message must be assigned with at least one node. It is possible that some messages will have multiple nodes (e.g., a message that both states a hypothesis and describes a test). Avoid repeated nodes of the same type within a single message, unless the text explicitly supports multiple distinct instances.

Node types:

H = Hypothesis: a candidate explanation, or a working assumption about the system. It should be a revisable claim, proposal, suggestion, or the current best guess about the answer. Information in task definitions or environment descriptions do not count as hypotheses (H). Partial or full reiteration of the task description does not count as hypothesis (H). Correcting a typo in a tool argument does not count as a Hypothesis (H). We consider a Hypothesis (H) to be present if the agent states a claim similar to "I think the answer is X", "This suggests that X might be the solution", "The most likely explanation is X", "X could be the case if...", "It seems that the answer can be X", etc.

E = Evidence: all Observation messages must be assigned with and only with an Evidence (E) node. Only Observation messages can be Evidence (E) nodes, with the exception of the task description message.

N = Neutral: for boilerplate operations like writing or copying files, or non-scientific tool calls. Only the tool calls of a message can be assigned as Neutral (N).

T = Test: any information-seeking action, including experiments, evaluations, or lookups. Both the intention and the concrete tool call qualify as tests (T); what matters is that the system is seeking new scientific information to evaluate a hypothesis (H) or a judgment (J). If both the intention/plan to run a test and the actual tool call are present within the same message, count only the tool call as the test (T) node. Every tool call (with the tag <action>) must be assigned as either a Test (T) or a Neutral (N) node.

J = Judgment: an interpretation of test results (Observation) that goes beyond the literal repetition of the raw output. If the agent restates an observation while adding any evaluative, comparative, or inferential content, even brief it is a judgment (J).

F = Final Answer: Only the message that contains a final submission (with tag <final_answer>), must be assigned with a Final Answer (F) node.

Constraints for nodes:

  • Only label what is explicitly present in text.

  • Every node must have support quotes (exact substrings) and msg indices.

  • If you normalize text, still cite original quote(s).

Pseudo-nodes (not explicitly stated but can be inferred):

C = Commitment: if from the actions of the agents or system it can be inferred that they have reached an implicit commitment to an answer that is not yet fully supported by evidence, and that they are **refusing to revise it**, then create a pseudo-node labeled C. This is a special pseudo-node that captures the commitment even if it is not explicitly stated. It is possible that the Final Answer (F) node is also a Commitment (C) if the Final Answer (F) is not fully supported by evidence.

Constrains for pseudo-nodes:

  • Only Commitment (C) can be a pseudo-node.

  • For the quote support of a pseudo-node, you can cite the text that implies the commitment, even if it is not explicit. You are allowed to add a brief explanation to clarify the implication, but it must be concise and directly tied to the quote.

Return JSON with keys:

"nodes": [

"node_id": "N1",
"type": "H|T|E|J|C|F",
"time": <int message index of earliest support>,
"text": <normalized short node text>,
"support": ["msg_idx": <int>, "quote": <exact substring from that message>],
, ...
]

In the second pass (Stage 2), the model receives the previously extracted nodes alongside the same message window. Its task is to identify directed edges between those nodes. The user prompt enumerates all permissible relation types and, for each type, specifies the valid source–destination pair combinations. This ensures that only structurally grounded edges are extracted. Edges that are not permitted are deleted programmatically after annotation.

System prompt for Stage 2: Edge annotation

You are a careful annotator. You MUST only add edges supported by explicit text. Rules:

  • You may only connect nodes provided to you.

  • Do not borrow any external knowledge or make assumptions beyond the text.

  • Do not judge or correct the content, only label supported edges.

  • Every edge MUST include at least one support quote with message indices.

  • If uncertain, omit the edge rather than guessing.

  • Output JSON only, matching the required schema.

User prompts for Stage 2 annotation

Given the message window and a list of extracted nodes (with node_id and text), extract supported edges among these nodes.

Only the following edge types are allowed, any other combination is forbidden:

  • tests: Allowed only between: H→TH \to T, J→TJ \to T. H→TH \to T: the Test (T) directly addresses the Hypothesis’ claim (H), or attempts to falsify or verify it. J→TJ \to T: the Test (T) is designed in response to the Judgment (J), or the Judgment (J) motivates the test design.

  • observes: Allowed only between: T→ET \to E. T→ET \to E: the Evidence (E) is a direct result of the Test (T).

  • updates_to: Allowed only between: H→HH \to H. H→HH \to H: the later Hypothesis (H) is a revision of the earlier one, based on the nodes in between.

  • competes_with: Allowed only between: H→HH \to H. H→HH \to H: the two Hypotheses (H) are alternative explanations that are directly compared or evaluated against each other. Both hypotheses should be plausible and co-existing at the same time. If another hypothesis is introduced later as a revision of the first one, then it should be connected with updates_to instead of competes_with.

  • contradicts: Allowed only between: E→HE \to H, J→HJ \to H. E→HE \to H: the Evidence (E) contradicts the claim of the Hypothesis (H). J→HJ \to H: the Judgment (J) contradicts the claim of the Hypothesis (H).

- informs: Allowed only between: E -> H, E -> J, E -> C, J -> C, J -> H, J -> J.
E -> H: the Evidence (E) provides information relevant to the claim of the Hypothesis
(H).
E -> J: the Judgment (J) is an interpretation of the Evidence (E).
E -> C: the Commitment (C) is informed by the Evidence (E) but not necessarily in an
explicit way.
J -> H: the Judgment (J) provides information relevant to the claim of the Hypothesis
(H).
J -> J: the later Judgment (J) is a refinement, an extension, or a combination of one or
serveral earlier judgments (J).
J -> C: the Commitment (C) is informed by the Judgment (J) but not necessarily in an
explicit way.

Return JSON with keys:

"edges": [

"src": "<node_id>",

"dst": "<node_id>",

"relation": "<one of allowed>",

"time": <int message index of earliest support>,

"support": ["msg_idx": <int>, "quote": <exact substring from that message>]
, ...

]

Prompt optimization Two experts optimized the annotation prompts for both stages by defining node and edge definitions in an iterative process on a 50-trace training set. The experts manually reviewed the LLM’s annotations and suggested improvements. When both agreed that the annotations reliably captured the underlying reasoning patterns, they proceeded with the full annotation.

Effect of prompt optimization. Prompt optimization led to substantial improvements in annotation agreement (Figure H.16). Human-human agreement measured by PABAK increased from 85.8% to 92.6%, while human-LLM percent agreement rose from 89.3% to 95.7%. The gains were similar for node and edge annotations.

Figure H.16. Effect of prompt optimization on annotation agreement. Human-human (PABAK) and human-LLM (percent agreement) scores for node and edge annotations, comparing the initial round (old) and the optimized round (new).

Pattern taxonomies

Productive motifs and reasoning breakdowns are defined as explicit structural templates over the annotated graphs. Each template specifies the required node types, edge types, and their configuration; traces are scored by detecting instances of these templates. Templates are organized into three families. Hypothesis handling captures how candidate explanations are generated, weighed against evidence, and revised. Evidence handling captures how the results of tests are incorporated into belief. Inquiry control captures how the overall test strategy is planned and adjusted.

Productive motifs. Productive motifs are drawn from recognized practices of disciplined inquiry: Popperian falsification,[56] exploratory-to-confirmatory transitions,[57] triangulation by convergent multi-test evidence, and active-learning-style test redesign.[58, 59] Table H.14 lists every template with its short description and graph form.

GroupGraphDescription
Hypothesis handlingE →inf.\xrightarrow{\mathrm{inf.}} J ⋯\cdots H →tests\xrightarrow{\mathrm{tests}} TEvidence led hypothesis generation. Evidence is observed first; a hypothesis is formed afterward
H₁ →tests\xrightarrow{\mathrm{tests}} T; H₂ →tests\xrightarrow{\mathrm{tests}} T; H₁ ↔competes \xleftrightarrow[\mathrm{competes}]{\ } H₂Hypothesis reranking. Competing hypotheses are compared as new evidence arrives
H →tests\xrightarrow{\mathrm{tests}} T →obs.\xrightarrow{\mathrm{obs.}} E →inf.\xrightarrow{\mathrm{inf.}} J \xrightarrow{\mathrm{upd.}} H₂Refutation driven belief revision. Evidence triggers a belief update to a new hypothesis
T →obs.\xrightarrow{\mathrm{obs.}} E ⋯\cdots H →tests\xrightarrow{\mathrm{tests}} TExplore then test transition. Exploration precedes hypothesis formation, which then drives testing
Evidence handlingH →\to T₁, T₂, T₃ →\to EConvergent multi test evidence. One hypothesis is evaluated via multiple independent tests
Inquiry controlH →tests\xrightarrow{\mathrm{tests}} T →obs.\xrightarrow{\mathrm{obs.}} E →inf.\xrightarrow{\mathrm{inf.}} J$, with tests iteratively returning to T | Fixed hypothesis test tuning. Hypothesis is held fixed while tests are iteratively adjusted
J →tests\xrightarrow{\mathrm{tests}} T →obs.\xrightarrow{\mathrm{obs.}} EEvidence guided test redesign. A judgment motivates a new test, which then produces new evidence

Table H.14. Table H.14: Descriptions and graph showing the productive motifs in the epistemological analysis. Note that data-first hypothesis and iterative test refinement include two graphs each that refer to the same productive reasoning behavior.

Reasoning breakdowns. Reasoning breakdowns are drawn from established work on scientific-reasoning failure and from patterns observed during trace review. Hypothesis-handling breakdowns include untested claims, contradictions left without repair, and premature commitment. Evidence-handling breakdowns include evidence non-uptake, disconnected evidence, and unsupported judgment. Inquiry-control breakdowns include fixed belief traces, stalled revision, and fixed hypothesis test tuning (ad-hoc rescue in the Popperian sense). Both capture configurations in which the agent does not revise its plan in response to new information. Table H.15 lists every template with its short description and graph form.

Detailed annotation results

Table H.15. Definitions and graphs illustrating the different reasoning breakdowns.

Each pattern is detailed as a graph and a short description. Note that untested hypothesis, unused evidence, and absent/stalled revision include two different graphs per pattern, which are different cases of the same breakdown.

GroupGraphDescription
Hypothesis handlingH→tests∅H \xrightarrow{\text{tests}} \varnothingUntested claim. Hypothesis never linked to a test
Hypothesis handlingE→inf.H→inf.J→inf.CE \xrightarrow{\text{inf.}} H \xrightarrow{\text{inf.}} J \xrightarrow{\text{inf.}} C; E→contr.∅E \xrightarrow{\text{contr.}} \varnothingOne sided confirmation. Commitment (explicit or inferred) without contradicting evidence
Hypothesis handlingE→contr.H→upd.∅E \xrightarrow{\text{contr.}} H \xrightarrow{\text{upd.}} \varnothingContradiction without repair. Contradiction unresolved by any update or alternative
Hypothesis handlingJ→inf.H→tests∅J \xrightarrow{\text{inf.}} H \xrightarrow{\text{tests}} \varnothing; J→inf.CJ \xrightarrow{\text{inf.}} CPremature commitment. Commitment (explicit or inferred) to a hypothesis without testing it first
Evidence handlingE→inf.∅E \xrightarrow{\text{inf.}} \varnothingEvidence non uptake. Evidence collected but never used
Evidence handling∅→E→∅\varnothing\xrightarrow{} E \xrightarrow{} \varnothingDisconnected evidence. Evidence node with no edges
Evidence handling∅→inf.J\varnothing\xrightarrow{\text{inf.}} JUnsupported judgment. Judgment made without supporting evidence
Evidence handlingT→obs.∅T \xrightarrow{\text{obs.}} \varnothingUninformative test. Test produces no observed evidence
Inquiry controlH→T→E→JH \to T \to E \to J; C→upd.∅C \xrightarrow{\text{upd.}} \varnothingFixed belief trace. No hypothesis revision in the entire trace
Inquiry controlC…H→testsT→obs.EC \ldots H \xrightarrow{\text{tests}} T \xrightarrow{\text{obs.}} EPrecommitted test plan. Commitment (explicit or inferred) before evidence collection
Inquiry controlH→upd.H2→∅H \xrightarrow{\text{upd.}} H_2 \xrightarrow{} \varnothingStalled revision. Revised hypothesis never tested

:::

Detailed annotation results

Detailed per-model results. Table H.16 reports, for each model, the fraction of traces that exhibit each motif or breakdown, averaged across environments. The table is organized into three reasoning groups (hypothesis handling, evidence handling, and inquiry control), with productive motifs and breakdowns listed separately within each group. Breakdown prevalence consistently exceeds productive-motif prevalence across all models, confirming that the aggregate trend reported in Figure 4 is not driven by any single system.

PatternClaude-4.5-SonnetGPT-4oGPT-OSS-120B
Hypothesis handling
Evidence-led hypothesis generation0.630.780.51
Hypothesis reranking0.170.070.04
Refutation-driven belief revision0.410.210.13
Explore-then-test transition0.590.680.16
Untested claim0.480.440.72
One-sided confirmation0.060.130.09
Contradiction without repair0.230.240.08
Premature commitment0.050.100.11
Evidence handling
Convergent multi-test evidence0.130.020.06
Evidence non-uptake0.810.770.35
Disconnected evidence0.300.200.02
Unsupported judgment0.160.090.21
Uninformative test0.410.340.14
Inquiry control
Fixed hypothesis test tuning0.130.170.04
Evidence-guided test redesign0.470.620.16
Fixed belief trace0.570.780.82
Precommitted test plan0.020.000.01
Stalled revision0.190.080.10

Table H.16. Detailed pattern presence for the epistemological analysis. The table presents the percentages for each model included in our work, averaged across environments.

Environment-level prevalence. Figure H.17 breaks down the same prevalence estimates by environment and subtask level, aggregating across models. Each row represents one environment (e.g., Spectroscopic structure elucidation S1) and is color-coded according to its category among the three environment types: workflow execution, strategic reasoning, and hypothesis-driven inquiry. Across every environment level, reasoning breakdowns consistently outnumber productive motifs. Hypothesis-driven environments show the highest overall annotation density, which aligns with their greater demands for iterative belief revision.

Figure H.17. Motif mean prevalence detailed for each environment. Values are averaged across the corresponding annotated pattern sets and aggregated across models. Reasoning breakdowns are more prevalent than productive motifs across all environment scopes.

Scope invariance across subtask levels. Figure H.18 examines how epistemic operation prevalence changes as problem scope increases from S1 to S4, averaging across all multi-level environments. Despite the fourfold increase in task complexity, neither the absolute prevalence nor the relative ordering of these operations shifts systematically. This invariance suggests that agents do not adapt their epistemic behavior to broader or more complex problem settings.

Figure H.18. Epistemic operation prevalence remains invariant to increasing problem scope. Mean prevalence of productive motifs (solid lines) and reasoning breakdowns (dashed lines) across the three epistemic groups (hypothesis handling, evidence handling, and inquiry control), plotted against increasing problem scope (S1–S4), averaged over all multi-level environments. Error bars denote the standard error of the mean across environments. Despite a fourfold increase in task complexity, neither the absolute prevalence nor the relative ordering of epistemic operations shifts systematically, indicating that agents apply a fixed repertoire of reasoning strategies regardless of problem scope.

Annotation agreement analysis

To validate the automated LLM annotation pipeline, three domain-expert annotators independently reviewed a representative sample of 25 traces drawn from the full set of 626 traces, covering both annotation rounds (before and after prompt optimization). We report inter-annotator agreement at two levels: human–human and human–LLM. Because the label distribution is heavily skewed toward correct, which deflates Cohen’s κ\kappa, we additionally report the prevalence- and bias-adjusted kappa (PABAK) as a more informative chance-corrected measure.

Human–human agreement. Table H.17 summarizes pairwise agreement among the three annotators on the optimized prompt annotations. Overall percent agreement reaches 92.6%, with a mean PABAK of 0.853, indicating substantial agreement. Node-level annotations show higher concordance (PABAK up to 0.976) than edge-level ones (PABAK 0.737–0.913); this is expected, as judging relational edges requires more nuanced interpretation of the trace. Cohen’s κ\kappa values are low (overall 0.067) due to the extreme prevalence of the correct class. PABAK corrects for this imbalance and confirms that observed disagreements are rare.

PairCohen’s κ\kappaPABAK% Agreenn
Nodes
R1 vs R2-0.0140.84192.1%542
R1 vs R30.1970.88994.5%416
R2 vs R3-0.0060.97698.8%415
Edges
R1 vs R2-0.0050.73786.8%433
R1 vs R30.2330.76188.0%343
R2 vs R3-0.0050.91395.6%343
Overall0.0670.85392.6%2492

Table H.17. Human–human inter-annotator agreement on optimized annotations. Pairwise Cohen’s κ\kappa, PABAK, and percent agreement for nodes and edges across three domain experts. Low κ\kappa values reflect the high prevalence of the correct class; PABAK corrects for this imbalance.

Human–LLM agreement. Table H.18 reports the percent agreement between each annotator and the LLM pipeline on the optimized annotations. Overall human–LLM agreement is 95.7%, which exceeds the human–human baseline of 92.6%. For nodes, all three reviewers agree with the LLM on at least 92.8% of items, with two reviewers reaching 99.3%. Edge agreement is slightly lower (87.1%–99.8%), consistent with the pattern observed in the human–human analysis. These results indicate that the LLM pipeline produces annotations whose quality is comparable to, and in aggregate slightly above, the level of agreement among human experts.

Reviewer% Agreenn
Nodes
R192.8%543
R299.3%542
R399.3%416
Edges
R187.1%433
R299.8%433
R395.9%343
Overall95.7%2710

Table H.18. Human–LLM agreement on optimized annotations. Percent agreement between each of the three annotators and the LLM pipeline, reported separately for nodes and edges. Overall agreement (95.7%) exceeds the human–human baseline, showing that the automated pipeline operates within the range of human expert judgment.

Other agent behavior analysis

To complement the main performance results, we also analyze how agents interact with the environment at a behavioral level. Specifically, we examine the distribution of tool actions, the verbosity of model outputs, and the number of tool calls per trial. These analyses help clarify whether differences in task performance stem from distinct interaction strategies or from more subtle variations in how models use the same tool ecosystem. Figures H.19 and H.20 summarize these behavioral patterns across environments, agent scaffolds, and model families.

Figure H.19 shows the distribution of action categories across environments, split by agent type and by model. The two agent scaffolds exhibit similar action profiles, suggesting that ReAct and tool-calling agents rely on comparable tool-use patterns. In contrast, differences across models are more pronounced in specific environments, indicating that the model family has a stronger influence than the agent scaffold on which types of tools are invoked.

Image not extracted

Figure H.19. Detailed results of action distributions in Corral. Tools are classified according to the task type they perform. (A) Action distribution percentages by agent type. Differences between agents are minor, with both showing similar tool use patterns across environments. (B) Action distribution percentages by model type. Notable differences across models emerge in specific environments: only GPT-4o employs code execution tools in the Resistor environment. In AFM, both Claude and GPT-4o use these tools substantially more than GPT-Oss, whereas in MD, Claude shows the highest usage among the three models.

Figure H.20 provides a complementary view based on output length and tool-use intensity. The figure shows that verbosity varies substantially across environments (some settings elicit longer responses than others) and that models also differ in how many tokens they produce and how many tools they call during a trial. The behavioral differences in this figure are stronger than those in the action-mixture analysis, suggesting that models differ less in what types of actions they take than in how extensively they reason and interact with tools while solving the task.

Image not extracted

Figure H.20. Detailed results on tool calls and output tokens across environments and models. (A) Average output tokens per message across environments. Resistor yields markedly more tokens per message than the other environments. Environments with lower performance tend to produce a higher number of tokens. (B) Tokens per message for each agent. Despite ReAct explicitly incorporating a Thought category to prompt reasoning, the token distributions across agents are similar. (C) Tokens per message for each model. GPT-Oss is the most verbose model, most likely due to its use of reasoning tokens. (D) Tool calls per trial across environments. Average number of tool calls per trial, aggregated over models and agents. (E) Tool distribution for each agent. Both agents exhibit similar tool-call distributions. Although the tool-calling agent shows a slightly longer tail, the overall differences are minimal. (F) Number of tool calls per model. Claude, the best-performing model, also tends to make more tool calls than the other models.

Table H.19: Tools for the wetlab environment. The level of verbosity shown is [BRIEF]. Refer to https://lamalab-org.github.io/corral to inspect other verbosity levels, and all the tools for the other environments included in this work.

add_a_solution

Description: Adds a specific volume of ‘sol2_label‘ to all of ‘sol1_label‘ and returns observations about the changes of solution color and precipitation amount and color.

Arguments: test_label (str): label of the resulting solution &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sol1_label (str): label of the first solution &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sol2_label (str): label of the second solution &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sol2_vol (int): volume of the second solution to draw

Returns: a string containing the observations from the test

add_precipitate_to_solution

Description: Adds a precipitate to a specific volume of a solution and returns observations about the changes in the amount/color of the added precipitate or the solution color.

Arguments: test_label (str): label of the resulting mixture &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;prec_label (str): label of the precipitate &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sol_label (str): label of the solution &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sol_vol (int): volume of the solution

Returns: a string containing the observations from the test

check_inventory

Description: Returns the current contents of the Inventory

Arguments: None

Returns: a string containing the solutions and precipitates in the Inventory

checkout_color

Continued on next page

observe_colorDescription: Observe the color of a solution or precipitate.
Arguments: label (str): label of the target object
Returns: the type (reagent, solution, or precipitate) and the color of the object
filter_solution
Description: Separates the precipitate from the supernatant solution.
Arguments: label (str): label of the target solution
Returns: a string with a message about the success/failure of the filtration
get_available_reagents
Description: Returns the list of available reagent solutions.
Arguments: None
Returns: a string containing all available reagents
lookup_flame_colors
Description: Returns the characteristic flame colors of cations.
Arguments: None
Returns: the list of characteristic flame colors
lookup_precipitate_colors
Description: Returns the list of colored precipitates and their color.
Arguments: None
Returns: the list of precipitate colors
measure_pH
Description: Measures the pH of the solution using a pH paper.
Arguments: label (str): label of the target solution
Returns: the closest integer value to the actual pH of the solution
mix_two_solutions
Description: Mixes two solutions with the given volumes an returns observations about precipitation and color of the resulting solution
Arguments: test_label (str): label of the resulting solution
sol1_label (str): label of the first solution
sol1_vol (int): volume of the first solution to draw
sol2_label (str): label of the second solution
sol2_vol (int): volume of the second solution to draw
Returns: a string containing the observations from the test
perform_flame_test
Description: Performs a flame test on the solution. Cost = 1 mL
Arguments: label (str): label of the target solution
Returns: the resulting observation from the flame test
possible_anions
Description: Returns the list of possible anions.
Arguments: None
Returns: a string containing all the possible anions
possible_cations
Description:Returns the list of possible cations.
Arguments:None
Returns:a string containing all the possible cations
simulate_color_mixture
Description:Given a mixture of precipitate colors, it mixes them with the given fractions and returns the name of the resulting precipitate color.
Arguments:mixture (list[tuple[str, float]]): mixture components and their fractions as a list of tuples
Returns:name of the resulting color

Table H.19. continued from previous page

Illustrative traces

For each major breakdown category, we include one representative trace excerpt with the supporting quote the annotator used to assign the pattern. For every message in the excerpt, the epistemic nodes extracted by the annotator are listed, together with the edges that link them. A Ⓗ denotes a hypothesis, a Ⓣ a test, Ⓔ an evidence observation, Ⓙ a judgment, and Ⓒ a commitment (premature answer). Missing edges that should appear in a sound reasoning chain are marked with ×. The full annotated traces, with node and edge graphs, filtering by pattern, and alternative candidate annotations are browsable at https://lamalab-org.github.io/corral/#explainers.

Evidence non-uptake

Model: Claude Sonnet 4.5 Environment: NMR structure elucidation (spectra) Scope: 1 Trial: 20

Msg 11 (Observation) obtain_isomers_from_molecular_formula →\rightarrow 20 candidate SMILES returned for C18H14O2C_{18}H_{14}O_{2}, including COC(=O)/C=C\C1=CC2=CC3=CC=CC=C3C=C2C=C1 (the correct answer).

Ⓣ N13 Obtain isomers for molecular formula C18H14O2C_{18}H_{14}O_{2} →observes\xrightarrow{\text{observes}} Ⓔ N14 20 isomers returned, incl. several with methyl ester groups

Msg 12 (Agent) “I got 20 isomers with the molecular formula C18H14O2C_{18}H_{14}O_{2}. Looking at the list, I can see several candidates. Let me analyze which one matches our NMR data . . .”

→\rightarrow calls retrieve_dbe_formula

No node extracted—the agent begins a tangential calculation instead of using the isomer list.

Msg 14 (Agent) “Let me recalculate the DBE for C18H14O2C_{18}H_{14}O_{2} . . . Looking back at the isomers list, let me focus on those with methyl ester groups . . .”

→\rightarrow calls simulate_spectra for methyl 9-phenanthrenecarboxylate

Ⓣ N15 Simulate spectra for methyl 9-phenanthrenecarboxylate to compare with experimental

Ⓣ N13 →observes\xrightarrow{\text{observes}} Ⓔ N14 →informs\xrightarrow{\text{informs}} J/H

Annotator quote: “COC(=O)/C=C\C1=CC2=CC3=CC=CC=C3C=C2C=C1, COC(=O)C12C=CC(C3=CC=CC=C31)C4=CC=CC=C24”. The 20 candidate structures include the correct answer, yet node N14 has zero outgoing informs edges: the isomer list is never used to constrain any judgment or hypothesis. The agent instead guesses structures one by one (phenanthrenecarboxylate, anthracenecarboxylate, . . .) without consulting the evidence it already obtained.

Untested claim

Model: Claude Sonnet 4.5 Environment: NMR structure elucidation (spectra) Scope: 1 Trial: 22

Msg 3 (Observation) proton_nmr_spectra → “Deltas 2.23 (s, 3H), 2.81 (t, JJ = 7.1 Hz, 2H), 2.98 (t, JJ = 7.1 Hz, 2H), 7.17 (ddd, 7H), 7.90 (ddd, 2H).”

Ⓣ N1 Get proton NMR spectra to gather structural information —observes→ Ⓔ N2 Proton NMR: 2.23 (s, 3H), 2.81 (t, 2H), 2.98 (t, 2H), 7.17 (ddd, 7H), 7.90 (ddd, 2H)

Msg 4 (Agent) “The two triplets at 2.81 and 2.98 ppm suggest two CH₂ groups coupled to each other (likely -CH₂-CH₂-). The aromatic signals suggest a substituted benzene ring. The singlet at 2.23 ppm suggests a methyl group attached to a carbonyl (acetyl group).” → calls mass_spectrometry_spectra

Ⓔ N2 —informs→ Ⓗ N3 Two coupled CH₂ groups; singlet at 2.23 ppm suggests acetyl group; substituted benzene ring

Msg 5 (Observation) mass_spectrometry_spectra → “m/zm/z 224.12 (intensity 100) . . .”

The agent moves directly to mass spectrometry: no test is designed to evaluate the CH₂-CH₂ or acetyl-group hypothesis.

Ⓔ N2 —informs→ Ⓗ N3 —tests→ Ⓣ

Annotator quote: “The two triplets at 2.81 and 2.98 ppm suggest two CH₂ groups coupled to each other (likely -CH₂-CH₂-). The aromatic signals suggest a substituted benzene ring. The singlet at 2.23 ppm suggests a methyl group attached to a carbonyl (acetyl group).”

Node N3 carries a detailed structural interpretation but has zero outgoing tests edges.

The agent asserts the presence of an acetyl group and coupled CH₂ groups as fact, then immediately requests mass spectrometry to determine molecular weight rather than designing an experiment (e.g., simulating 13^{13}C NMR for a candidate) to verify the claim.

Fixed belief trace

Model: GPT-4o Environment: LAMMPS molecular dynamics (md) Scope: 2 Trial: 33

Msg 15 (Observation) list_files → “{‘files’: [‘/potentials/BKS’, ‘/potentials/EAM’, ‘/potentials/SW’, ‘/potentials/TERSOFF’]}”

Ⓣ N10 List files in /potentials/ to find correct SW potential path — observes → Ⓔ N11 Potentials directory contains BKS, EAM, SW, TERSOFF subdirectories

Msg 16 (Agent) “The /potentials/ directory contains a file named SW, which likely corresponds to the Stillinger-Weber potential. I will update the LAMMPS input script to use this file . . .”

→ calls write_file to set pair_coeff * * /potentials/SW Si

Ⓔ N11 — informs → Ⓗ N12 The SW potential file is located at /potentials/SW

∴ three error messages and re-writes later

Msg 27 (Observation) run_lammps → “ERROR: Lost atoms: original 8 current 0”

Ⓣ N33 Run LAMMPS simulation with updated script using /potentials/SW/Si.sw — observes → Ⓔ N34 LAMMPS simulation failed: Lost atoms

Msg 28 (Agent) “This might be due to the rapid heating rate or insufficient equilibration. I will adjust the heating rate and ensure proper equilibration . . .”

Ⓔ N34 (no informs to any existing H)

Ⓗ N35 Lost atoms error may be due to rapid heating rate or insufficient equilibration

Ⓗ N12 updates_to Ⓗ N19 ⋯ Ⓗ N46 updates_to Ⓗ N57

Annotator quotes:

N12 (msg 16): “The /potentials/ directory contains a file named SW, which likely corresponds to the Stillinger-Weber potential.”

N35 (msg 28): “This might be due to the rapid heating rate or insufficient equilibration.”

The trace contains 6 hypothesis nodes and zero updates_to edges. The first hypothesis (N12) mistakes the directory /potentials/SW for a file, producing an error that is never traced back to the belief. After two more failures, the agent introduces a second independent hypothesis (N35), attributing the lost atoms to heating rate, without revising the earlier structural error. Both beliefs persist unmodified through the remaining 15 messages.

Contradiction without repair

Model: Claude Sonnet 4.5 Environment: NMR structure elucidation (spectra) Scope: 2 Trial: 45

Msg 17 (Observation) simulate_spectra for CC(C)OC(=O)c1ccccc1N(C)C →\to 1H{}^{1}\mathrm{H} NMR: 1.46 (d, 6H), 2.97 (s, 6H), 4.75 (p, 1H), aromatic protons.

Ⓣ N31 Simulate spectra for CC(C)OC(=O)c1ccccc1N(C)C — observes — Ⓔ N25 Simulated: 1.46 ppm (d, 6H); experimental: 1.43 ppm (d, 3H)

Msg 18 (Agent) “H-NMR: 1.46 ppm (d, 6H) — This is for isopropyl CH3_3 groups, but experimental shows 1.43 ppm (d, 3H). The experimental data has only 3H in the doublet, not 6H. So this isn’t isopropyl ester. The experimental data clearly shows CH(CH3_3), not CH(CH3_3)2_2.”

→\to calls get_formula_from_smiles

Ⓔ N25 — informs — Ⓙ N33 Simulated isopropyl ester shows 6H doublet at 1.46 ppm, but experimental shows only 3H doublet at 1.43 ppm

Ⓙ N33 — contradicts — Ⓗ N30 The compound is 2-(dimethylamino)benzoic acid isopropyl ester

Msg 20 (Agent — final answer) “Simulations are approximations and can have errors . . . The key evidence: (1) quartet at 5.05 ppm, (2) singlet at 2.69 ppm (6H) = N(CH3_3)2_2, (3) 4 aromatic protons, (4) carbonyl at 168.30 ppm.”

<final_answer>CC(C)OC(=O)c1ccccc1N(C)C</final_answer>

Ⓙ N36 Despite the 3H vs. 6H discrepancy, the overall evidence supports the isopropyl ester

Ⓒ N37 Agent commits despite unresolved discrepancy

Reasoning graph showing N25 informing N33, N33 contradicting N30, and a missing updates_to edge to H′

Figure 7.

Annotator quote: “H-NMR: 1.46 ppm (d, 6H) — This is for isopropyl CH3_3 groups, but experimental shows 1.43 ppm (d, 3H) . . . The experimental data has only 3H in the doublet, not 6H. So this isn’t isopropyl ester.” — The agent explicitly notes that the simulated 1H{}^{1}\mathrm{H} NMR contradicts its own hypothesis (N30). Yet no revised hypothesis is generated: the contradicts edge from N33 to N30 has no accompanying updates_to edge. Instead, the agent dismisses the contradiction as a simulation artifact and submits the isopropyl ester (CC(C)OC(=O)c1ccccc1N(C)C) as the final answer (N38).

Scope and limitations

The results reported here are bounded by deliberate design choices and by practical constraints on model access and compute. We state both so that comparisons to future work, whether with richer scaffolds, broader model coverage, or larger compute budgets, can be made on equal terms.

Scope of the evaluation. Each task is treated as an independent episode: agents receive no carry-over of knowledge, strategies, or refined heuristics across tasks, in contrast with the iterative accumulation of intuition that characterizes real scientific work. We evaluate two simple scaffolds, ReAct and native tool calling, with a flat, append-only conversation history and a simple truncation heuristic (earlier non-assistant messages shortened to 100 characters) when the context window is exceeded. Advanced orchestration is outside the scope of this evaluation: plan-then-execute pipelines, hierarchical planners, explicit summarisation or retrieval, and multi-agent architectures such as debate or specialist delegation. This isolates the contribution of the base model and scaffold from engineering-intensive orchestration; the scores reported here, therefore, bound what minimal scaffolds achieve rather than what is achievable in general. All agents operate under a fixed budget of 20–40 LLM calls per task, calibrated in pilot runs to be sufficient for task completion, and at a fixed sampling temperature of 0.0. We do not vary this budget per task nor conduct a temperature sensitivity analysis.

Model coverage and compute. Our evaluation spans three models, Claude Sonnet 4.5, GPT-4o, and GPT-OSS (gpt-oss-120b), selected to span proprietary and open-weight systems at the current capability frontier; other prominent families are omitted. The full benchmark consumed approximately 3 billion tokens across 138 configuration-environment pairs at five trials each, with an estimated API cost of $7.5k for the two proprietary models (excluding infrastructure for the self-hosted open-weight model) for the main final runs shown in the main text (not including interventions and subtasks). Pass@kPass@k metrics account for stochastic variation, but scaling to additional models, trials, or environments carries a non-trivial cost.

Measurement caveats. Log-probability-based analyses of model confidence and calibration are restricted to GPT-OSS (gpt-oss-120b), as proprietary API providers do not expose per-token log-probabilities for tool-calling or for sufficiently long completions. We also observed higher rates of malformed responses from GPT-OSS-120B (41% of traces affected, with 1.26 errors per trace), and to a lesser extent GPT-4o (0.28 errors per trial, affecting 6.2% of traces), compared with Claude Sonnet 4.5 (0.5% of traces affected); the most common failure modes were invalid JSON in tool arguments and missing XML closing tags in ReAct traces. We did not apply engineering mitigations such as constrained decoding, output grammars, or retry-with-repair loops. The reported numbers thus reflect the behavior of each system under the same minimal scaffold, not their best-case performance with model-specific hardening.

References

  1. [1]Mirza, A. et al. A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists. Nature Chemistry 17, 1027–1034 (2025).DOI
  2. [2]Alampara, N. et al. Probing the limitations of multimodal language models for chemistry and materials research. Nature Computational Science 5, 952–961 (2025).DOI
  3. [3]Laurent, J. M. et al. Lab-bench: Measuring capabilities of language models for biology research. arXiv preprint arXiv:2407.10362 (2024).arxiv.org/abs/2407.10362
  4. [4]Chen, Z. et al. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. arXiv preprint arXiv:2410.05080 (2024).arxiv.org/abs/2410.05080
  5. [5]Shen, Y. et al. SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents. arXiv preprint arXiv:2602.12984 (2026).arxiv.org/abs/2602.12984
  6. [6]Cui, H. et al. Curie: Evaluating llms on multitask scientific long context understanding and reasoning. arXiv preprint arXiv:2503.13517 (2025).arxiv.org/abs/2503.13517
  7. [7]Alampara, N., Schilling-Wilhelmi, M. & Jablonka, K. M. Lessons from the trenches on evaluating machine learning systems in materials science. Computational Materials Science 259, 114041 (2025).arxiv.org/abs/2503.10837
  8. [8]Chen, H. et al. Mlr-bench: Evaluating ai agents on open-ended machine learning research. arXiv preprint arXiv:2505.19955 (2025).arxiv.org/abs/2505.19955
  9. [9]Liu, Y. et al. Researchbench: Benchmarking llms in scientific discovery via inspiration-based task decomposition. arXiv preprint arXiv:2503.21248 (2025).arxiv.org/abs/2503.21248
  10. [10]Du, M., Xu, B., Zhu, C., Wang, X. & Mao, Z. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763 (2025).arxiv.org/abs/2506.11763
  11. [11]Bragg, J. et al. Astabench: Rigorous benchmarking of ai agents with a scientific research suite. arXiv preprint arXiv:2510.21652 (2025).arxiv.org/abs/2510.21652
  12. [12]Mitchener, L. et al. BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology. arXiv preprint arXiv:2503.00096 (2025).arxiv.org/abs/2503.00096
  13. [13]Liu, X. et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688 (2023).arxiv.org/abs/2308.03688
  14. [14]Mialon, G., Fourrier, C., Wolf, T., LeCun, Y. & Scialom, T. Gaia: a benchmark for general ai assistants in The Twelfth International Conference on Learning Representations (2023).
  15. [15]Froger, R. et al. Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments. arXiv preprint arXiv:2602.11964 (2026).arxiv.org/abs/2602.11964
  16. [16]Zhu, K. et al. MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents. arXiv preprint arXiv:2503.01935 (2025).arxiv.org/abs/2503.01935
  17. [17]Lu, J. et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities in Findings of the Association for Computational Linguistics: NAACL 2025 (2025), 1160–1183.
  18. [18]Chollet, F. On the Measure of Intelligence. arXiv preprint arXiv:1911.01547 (2019).arxiv.org/abs/1911.01547
  19. [19]Lu, C. et al. Towards end-to-end automation of AI research. Nature 651, 914–919 (2026).
  20. [20]Mitchener, L. et al. Kosmos: An ai scientist for autonomous discovery. arXiv preprint arXiv:2511.02824 (2025).
  21. [21]M. Bran, A. et al. Augmenting large language models with chemistry tools. Nature Machine Intelligence 6, 525–535 (2024).
  22. [22]Boiko, D. A., MacKnight, R., Kline, B. & Gomes, G. Autonomous chemical research with large language models. Nature 624, 570–578 (2023).
  23. [23]Zou, Y. et al. El Agente: An autonomous agent for quantum chemistry. Matter 8 (2025).
  24. [24]Darvish, K. et al. ORGANA: A robotic assistant for automated chemistry experimenta- tion and characterization. Matter 8, 101897 (2025).
  25. [25]Novikov, A. et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131 (2025).
  26. [26]Ghafarollahi, A. & Buehler, M. J. SciAgents: automating scientific discovery through bioinspired multi-agent intelligent graph reasoning. Advanced Materials 37, 2413523 (2025).
  27. [27]Wang, F. Y. et al. Autonomous Agents Coordinating Distributed Discovery Through Emergent Artifact Exchange. arXiv preprint arXiv:2603.14312 (2026).
  28. [28]Swanson, K., Wu, W., Bulaong, N. L., Pak, J. E. & Zou, J. The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies. Nature 646, 716–723 (2025).
  29. [29]Ghafarollahi, A. & Buehler, M. J. ProtAgents: protein discovery via large language model multi-agent collaborations combining physics and machine learning. Digital Discovery 3, 1389–1409 (2024).
  30. [30]Ni, B. & Buehler, M. J. MechAgents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledge. Extreme Mechanics Letters 67, 102131 (2024).
  31. [31]Du, Y. et al. Accelerating Scientific Discovery with Autonomous Goal-evolving Agents. arXiv preprint arXiv:2512.21782 (2025).
  32. [32]Buehler, M. J. PRefLexOR: preference-based recursive language modeling for ex- ploratory optimization of reasoning and agentic thinking. npj Artificial Intelligence 1, 4 (2025).
  33. [33]Tang, X. et al. Risks of AI scientists: prioritizing safeguarding over autonomy. Nature Communications 16 (2025).DOI
  34. [34]Ma, Z. et al. SkillClaw: Let Skills Evolve Collectively with Agentic Evolver. arXiv preprint arXiv:2604.08377 (2026).arxiv.org/abs/2604.08377
  35. [35]Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, 24824–24837 (2022).arxiv.org/abs/2201.11903
  36. [36]Yao, S. et al. React: Synergizing reasoning and acting in language models in The eleventh international conference on learning representations (2022).arxiv.org/abs/2210.03629
  37. [37]Wang, X. et al. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022).arxiv.org/abs/2203.11171
  38. [38]Schick, T. et al. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems 36, 68539–68551 (2023).arxiv.org/abs/2302.04761
  39. [39]Wang, G. et al. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 (2023).arxiv.org/abs/2305.16291
  40. [40]Shinn, N. et al. Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv preprint arXiv:2303.11366 (2023).arxiv.org/abs/2303.11366
  41. [41]Meireles, M., Bhati, R., Lauffer, N. & Allen, C. The Influence of Scaffolds on Coordination Scaling Laws in LLM Agents in Workshop on Scaling Environments for Agents ().
  42. [42]Hori, K., Fukuhara, H. & Yamada, T. Item response theory and its applications in educational measurement Part I: Item response theory and its implementation in R. Wiley Interdisciplinary Reviews: Computational Statistics 14, e1531 (2022).
  43. [43]Toland, M. D. Practical guide to conducting an item response theory analysis. The Journal of Early Adolescence 34, 120–151 (2014).DOI
  44. [44]Lovelace, M. & Brickman, P. Best practices for measuring students’ attitudes toward learning science. CBE–Life Sciences Education 12, 606–617 (2013).DOI
  45. [45]Lalor, J. P., Wu, H. & Yu, H. Building an evaluation scale using item response theory in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016), 648–657.arxiv.org/abs/1605.08889
  46. [46]Lalor, J. P., Wu, H., Munkhdalai, T. & Yu, H. Understanding deep learning performance through an examination of test set difficulty: A psychometric case study in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (2018), 4711–4716.DOI
  47. [47]Rodriguez, P. et al. Evaluation examples are not equally informative: How should that change NLP leaderboards? in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (2021), 4486–4503.DOI
  48. [48]Schilling-Wilhelmi, M., Alampara, N. & Jablonka, K. M. Lifting the benchmark iceberg with item-response theory in AI for Accelerated Materials Design-ICLR 2025 (2025).
  49. [49]Zhou, H. et al. Lost in benchmarks? rethinking large language model benchmarking with item response theory in Proceedings of the AAAI Conference on Artificial Intelligence 40 (2026), 35085–35093.
  50. [50]Ndzomga, F. Efficient Benchmarking of AI Agents. arXiv preprint arXiv:2603.23749 (2026).arxiv.org/abs/2603.23749
  51. [51]Ioannidis, J. P., Fanelli, D., Dunne, D. D. & Goodman, S. N. Meta-research: evaluation and improvement of research methods and practices. PLoS biology 13, e1002264 (2015).
  52. [52]Hao, Q., Xu, F., Li, Y. & Evans, J. Artificial intelligence tools expand scientists’ impact but contract science’s focus. Nature, 1–7 (2026).DOI
  53. [53]Mitchell, M. & Krakauer, D. C. The debate over understanding in AI’s large language models. Proceedings of the National Academy of Sciences 120. http://dx.doi.org/10.1073/pnas.2215907120 (2023).
  54. [54]Thais, S. et al. AI for Science Needs Scientific Alignment. Philsci Archive preprint. https://philsci-archive.pitt.edu/28744/ (2026).
  55. [55]Ichikawa, J. & Steup, M. in The Stanford Encyclopedia of Philosophy (eds Zalta, E. N. & Nodelman, U.) Spring 2026 (Metaphysics Research Lab, Stanford University, 2026).
  56. [56]Popper, K. The logic of scientific discovery (Routledge, 2005).
  57. [57]Tukey, J. W. et al. Exploratory data analysis (Springer, 1977).
  58. [58]Lindley, D. V. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics 27, 986–1005 (1956).DOI
  59. [59]Fedorov, V. V. Theory of optimal experiments (Elsevier, 2013).
  60. [60]Priem, J., Piwowar, H. & Orr, R. OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts 2022. arXiv: 2205.01833 [cs.DL]. https://arxiv.org/abs/2205.01833.
  61. [61]Nanosurf. DriveAFM: Tip-Scanning AFM System https://www.nanosurf.com/en/products/driveafm. Product page, accessed 2026-04-19.
  62. [62]Jain, A. et al. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL materials 1 (2013).DOI
  63. [63]Kannas, C., Thakkar, A., Bjerrum, E. & Genheden, S. rxnutils – A Cheminformatics Python Library for Manipulating Chemical Reaction Data. chemrxiv preprint 10.26434/chemrxiv-2022-wt440-v2. http://dx.doi.org/10.26434/chemrxiv-2022-wt440-v2 (2022).
  64. [64]Binev, Y., Marques, M. M. B. & Aires-de-Sousa, J. Prediction of 1H NMR Coupling Constants with Associative Neural Networks Trained for Chemical Shifts. Journal of Chemical Information and Modeling 47, 2089–2097 (2007).pubmed.ncbi.nlm.nih.gov/17956083
  65. [65]Banfi, D. & Patiny, L. www.nmrdb.org: Resurrecting and Processing NMR Spectra On-line. CHIMIA 62, 280 (2008).DOI
  66. [66]Jablonka, K. M., Patiny, L. & Smit, B. Making Molecules Vibrate: Interactive Web Environment for the Teaching of Infrared Spectroscopy. Journal of Chemical Education 99, 561–569 (2022).DOI
  67. [67]Leal, A. M. M. Reaktoro: An open-source unified framework for modeling chemically reactive systems https://reaktoro.org. Accessed: 2026-04-19. 2015.
  68. [68]Alampara, N. et al. corral-QAs (Revision bb74c25) 2026. https://huggingface.co/datasets/jablonkagroup/corral-QAs.
  69. [69]Vehtari, A., Gelman, A. & Gabry, J. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and computing 27, 1413–1432 (2017).DOI
  70. [71]Kumar, R., Carroll, C., Hartikainen, A. & Martín, O. A. ArviZ: a unified library for exploratory analysis of Bayesian models in Python (2019).DOI

Paper details

vlnc:W_49reu8em
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents