PAPERBENCHX: Can AI Reproduce Science Across Disciplines?

🌐 Homepage: https://unipat.ai/blog/PaperBenchX

📝 Code: https://github.com/UniPat-AI/PaperBenchX

🏆 Leaderboard: https://unipat.ai/benchmarks/PaperBenchX

Introduction

Reproducibility is one of the defining features of science: a result is meaningful not only because it has been reported, but because the assumptions, procedures, and evidence behind it can be reconstructed and independently examined. For AI systems, this makes reproduction a demanding test of scientific competence. An agent must go beyond understanding a paper’s conclusion to recover how the result was obtained, translate incomplete descriptions into executable procedures, operate specialized tools, diagnose failures, and determine whether the resulting evidence actually supports the claim. Existing reproduction benchmarks largely focus on AI and machine learning, while cross-domain scientific benchmarks typically isolate only parts of the workflow ([22, 21, 7]). What remains underexplored is whether an agent can reproduce published experiments across disciplines, inside the environments native to the underlying science.

This distinction matters because, outside machine learning, the environment is not merely infrastructure. It is part of the scientific problem. Reproducing an antenna result may require constructing and solving an electromagnetic model in HFSS; reproducing a photonics result may depend on geometry, boundary conditions, mode extraction, and convergence behavior in Lumerical or Meep ([19]); reproducing a chemistry or materials result may require choosing basis sets, numerical tolerances, or electronic-structure settings; and reproducing a robotics result may require executing and validating behavior inside an embodied simulator. A workflow can run successfully, generate plausible numbers, and still represent the wrong physical system. Cross-disciplinary scientific reproduction therefore demands more than writing code that executes.

We identify three coupled challenges. First, scientific reconstruction: papers are not executable specifications. Agents must translate prose, equations, figures, and domain conventions into a concrete experiment, identify the target observable, and resolve modeling choices that are not fully specified by the task contract. Second, domain-native execution: agents must operate specialized scientific software, select appropriate physical and numerical settings, interpret diagnostics, and determine whether a computation is stable and converged. Third, scientific verification: matching a reported value is not sufficient. The result must arise from a valid experimental workflow, and the evidence supporting it must be independently regenerable. These challenges separate scientific reproduction from either rerunning released code or producing a plausible final answer.

To study this setting, we introduce PAPERBENCHX, a benchmark for paper-level scientific reproduction across disciplines. It contains 93 expert-constructed tasks spanning 12 research areas across electromagnetics, photonics, computational chemistry, computational materials science, computational biology and bioinformatics, and robotics and embodied AI. The tasks run in 10 domain-native scientific computing environments, including Ansys HFSS, Ansys Lumerical, Meep [19], PySCF/GPU4PySCF [23], ABACUS [29], Quantum ESPRESSO [8], AutoDock Vina [6], Habitat [20], SimplerEnv [14], and AI2-THOR [12]. Each task is centered on a verifiable scientific claim from a published paper. The agent receives the paper, a public task specification defining the reproduction scope and deliverables, and the required scientific environment, but does not receive the private reference workflow, reference evidence, or evaluation rubric. Its goal is to reconstruct the experiment and deliver a runnable workflow that regenerates evidence supporting the target claim.

PAPERBENCHX evaluates sustained, end-to-end scientific reproduction through replay-grounded verification: agents must maintain scientific intent, computational state, and evidence provenance while reconstructing experiments, running and diagnosing long-running simulations, and refining their analyses; after submission, each workflow is rerun from scratch in a clean environment, and only regenerated artifacts are scored against expert-validated criteria. This design distinguishes plausible implementations and successful software execution from actual reproduction of the target scientific result. Across 10 frontier agent configurations on the same 93 tasks, the strongest configuration exceeds the 90% full-reproduction threshold on only 13 tasks, while 71 tasks are not fully reproduced by any evaluated configuration; mean rubric scores range from 27.53% to 59.08%. Agents often make substantial progress in modeling and execution but fail at scientific validation, revealing a substantial gap between partial progress and reliably closing the loop from paper understanding to reproducible scientific evidence.

Our contributions are threefold:

• Cross-disciplinary paper-level reproduction. We introduce 93 expert-constructed tasks spanning 12 research areas and 10 scientific computing environments, each centered on a verifiable claim from a published paper.

• Domain-native scientific execution. Rather than reducing scientific problems to a common coding interface, PAPERBENCHX requires agents to operate inside the simulators, software stacks, numerical conventions, and resource constraints required by the underlying research.

• Replay-grounded scientific verification. We rerun every submitted workflow from scratch and score only regenerated evidence against expert-validated scientific criteria, distinguishing plausible outputs from reproducible scientific results.

Related Work

Machine-Learning Research and Paper Reproduction

MLAgentBench and RE-Bench evaluate long-horizon machine-learning experimentation and research engineering, but their objectives are specified as task-level metric improvement rather than reproduction of published claims [10, 27]. PaperBench and SA-Bench move toward paper-level evaluation through reproduction or semantic-alignment rubrics [22, 9]. They establish that a final metric alone cannot measure fidelity to a paper, but remain centered on relatively homogeneous machine-learning software stacks rather than heterogeneous domain-native scientific environments.

Cross-Domain Scientific Benchmarks

Cross-domain benchmarks broaden the scientific scope but generally isolate only part of the reproduction process. SciCode, DiscoveryBench, and ScienceAgentBench evaluate scientific coding, hypothesis discovery, or data analysis, while CORE-Bench recovers results by rerunning released code ([25, 15, 4, 21]). DISCOVERYWORLD evaluates interactive discovery in a unified virtual world, and NatureBench constructs paper-derived scientific-ML tasks ([11, 26]). ScienceIDE most directly evaluates domain-native scientific execution by placing agents inside production scientific software, but its tasks concern repository-level repair and implementation rather than reproduction of a published experiment ([7]).

PAPERBENCHX targets the intersection left open by these benchmarks. It requires paper-level reproduction of key experiments, executes tasks inside heterogeneous domain-native scientific environments, and uses replay-grounded scientific verification. Submitted workflows must run again from scratch and regenerate evidence that supports the target claim; neither a plausible final answer nor a completed repository is sufficient. Table 1 summarizes these distinctions.

Benchmark	Research scope	Primary evaluation target	Paper-level reproduction	Domain-native scientific execution	Replay-grounded scientific verification
Machine-learning research and paper-reproduction benchmarks
MLAgentBench ([10])	Machine learning	ML experimentation tasks	✗	✗	✗
RE-Bench ([27])	Machine learning	Research-engineering tasks	✗	✗	✗
PaperBench ([22])	Machine learning	Paper reproduction	✓	✗	✗
SA-Bench ([9])	Machine learning	Paper reproduction	✓	✗	✓
Cross-domain scientific benchmarks
DiscoveryBench ([15])	6 scientific fields	Data-driven discovery	✗	✗	✗
ScienceAgentBench ([4])	4 scientific fields	Scientific workflow tasks	✗	✗	✗
CORE-Bench ([21])	3 scientific fields	Repository-result reproduction	✗	✗	✗
DISCOVERYWORLD ([11])	8 scientific fields	Interactive scientific discovery	✗	✓	✓
NatureBench ([26])	6 scientific fields	Scientific experiment tasks	✗	✗	✓
ScienceIDE ([7])	Physics and geoscience	Repository-level scientific coding	✗	✓	✓
PAPERBENCHX	12 research areas	Reproduction of key paper experiments	✓	✓	✓

The PaperBenchX Benchmark

PAPERBENCHX evaluates whether an agent can turn a scientific claim into independently replayable evidence in the domain-native environment required by the underlying research. Figure 2 summarizes the complete lifecycle, from the materials provided to the agent through clean replay and evidence-based scoring. This section defines the task, describes the benchmark composition and construction pipeline, and specifies the evaluation protocol.

Task Definition

Each PAPERBENCHX task is centered on a verifiable scientific claim from a published scientific paper. As shown in Figure 2, the task input has three parts: the paper; a public task specification defining the scientific scope, target conditions and variants, observables, required evidence, and delivery interface; and a scientific computing environment containing the domain-native simulator, dependencies, data, and allocated resources. The environment is part of the task definition rather than incidental infrastructure, because model representations, numerical conventions, execution procedures, and convergence evidence differ across scientific software.

The agent must submit a runnable package containing all source, model-building, simulation, post-processing, and analysis code needed for reproduction, with reproduce.sh as the canonical entry point. During clean replay, the evaluator refreshes the task container, restores only the submitted package, runs this entry point, and collects the outputs regenerated by the workflow. Those outputs, rather than the

agent’s assertions, are scored against a hidden scientific rubric. The private reference workflow and reference artifacts are never exposed to the agent. Reference artifacts are expert-generated simulator logs, raw numerical outputs, convergence records, and derived metrics produced during task construction; they establish that the task is executable and ground its expected evidence and scoring criteria. Values reported in the source paper remain visible, but private tolerances and reference evidence are withheld. The task therefore tests whether an agent can construct a scientifically valid and independently replayable procedure, not whether it can recover a hidden number.

Task Statistics

Benchmark composition. PAPERBENCHX contains 93 paper-reproduction tasks spanning 12 research areas and 10 domain-native scientific computing environments, with 3,168 scorable criteria. The twelve research areas are radio-frequency antennas and arrays; radio-frequency metasurfaces and scattering; radio-frequency filters and frequency-selective surfaces; integrated photonic devices; photonic crystals and topological photonics; optical metasurfaces, gratings, and plasmonics; optical computing and signal processing; optical resonators, filters, and tunable devices; computational chemistry; computational materials science; computational biology and bioinformatics; and robotics and embodied AI. Figure 3A shows the distribution of tasks across these research areas.

The tasks run in ten domain-native scientific computing environments: Ansys HFSS, Ansys Lumerical, Meep, PySCF/GPU4PySCF, ABACUS, Quantum ESPRESSO, AutoDock Vina, Habitat, SimplerEnv, and AI2-THOR. The benchmark draws on 93 distinct source papers from 44 publication venues, as summarized in Figure 3B. Each task has an execution budget determined by its simulation scale and resource requirements; agent time limits range from 4 h to 72 h, with a median of 7 h. Figure 3C shows the distribution of scorable criteria across the same 93 tasks.

Release and benchmark integrity. We release one representative task from each of the twelve research areas. Nine tasks include complete, redistributable scientific computing environments; the remaining

three rely on commercially licensed software, for which we release the task specifications and scoring details but not the corresponding software environments. The remaining tasks are retained as a hidden evaluation set to reduce test-set contamination and preserve the benchmark’s validity for evaluating future agents.

Task Construction Pipeline

The pipeline proceeds in two stages (Figure 4): first building a runnable candidate task, and then validating whether it can be admitted to the benchmark.

Stage 1: Build. Steps 1–2: paper and claim selection. A domain expert selects a published paper, identifies a scientifically meaningful claim, and checks that it can be reproduced under the available software, licensing, compute, and time constraints. The expert then defines the scientific scope: the target figure or table, model configuration, experimental conditions and variants, observables, valid methodological alternatives, and evidence required to support the claim. Steps 3–4: environment and public contract. The expert provisions the domain-native scientific environment and writes the public task specification, which states the scope, required outputs, submission location, and replay entry point without disclosing the private rubric or reference solution. Step 5: initial execution check. The expert implements a private reference workflow and executes it in the assembled environment. This first execution check verifies that the simulator can be invoked, the scoped experiment can finish within the resource envelope, and the required reference artifacts can be generated. A task reaches the candidate stage only after this end-to-end workflow runs successfully.

Stage 2: Validate and review. Step 6: rubric construction. The expert converts the evidence requirements into independently scorable rubric leaves covering model construction, parameter settings, execution, convergence, post-processing, and claim validation. Each leaf specifies the evidence needed for credit and its scoring rule, tolerance, and weight. Step 7: clean reference replay. The private reference workflow is rerun from a clean copy of the final container through the same replay interface used for submissions. Unlike the initial check in Step 5, this stricter execution check verifies that the task does not depend on stale state, undeclared files, or hidden setup; it also confirms that the expected artifacts can be regenerated under the final resource limits and consumed by the evaluator. Step 8: calibration checks. Successful, partially successful, and invalid workflows are scored to confirm that the rubric awards supported progress, withholds credit for missing or contradictory evidence, and distinguishes a complete reproduction from a plausible but scientifically invalid result. Step 9: independent expert review. Two independent domain experts inspect the scientific scope, fairness, reproducibility, evidence requirements, and scoring behavior.

Tasks that fail because of their scope, environment, or reference workflow are rebuilt from Stage 1. Failures caused by the rubric, scoring rules, or evaluator return to Step 6. In either case, all affected downstream execution, calibration, and review checks are repeated. Only tasks that pass the full pipeline are frozen and admitted to the benchmark. Across the benchmark, task construction and validation required more than 1,000 hours of doctoral-level expert work, or approximately ten hours per task.

Evaluation

Scientific correctness cannot be established from an agent’s final report. PAPERBENCHX therefore evaluates each submission through clean replay, evidence-based grading, and hierarchical scoring. This evaluation of the agent’s workflow is distinct from the clean reference replay in Step 7 of task construction, which validates the benchmark task itself. Appendix B gives the full protocol.

Clean replay. After delivery, generated outputs and mutable execution state are removed. The evaluator refreshes the task container, restores only the agent’s submitted package, and invokes bash/home/submission/reproduce.sh with private evaluation resources inaccessible, outbound network disabled, and task-specific resource limits enforced. Exit status, timing, source hashes, and execution provenance are recorded independently of the agent’s report. Only artifacts regenerated by this replay are admissible evidence; a replay failure is recorded explicitly, and credit remains possible only for rubric requirements supported by admissible regenerated evidence.

Evidence-based grading. Replay outputs—including simulator logs, raw numerical arrays, trajectories, convergence records, derived metrics, and visualizations—are frozen as a read-only evidence set. Every scientific conclusion must be traceable to this set: a convergence claim requires a convergence record, and a reported observable requires both the raw output and the analysis that derives it. Deterministic checks evaluate directly testable properties such as file schema, units, numerical consistency, and task-specific tolerances. Requirements that need domain interpretation are assessed for every evaluated configuration by the same GPT-5.6-sol scientific rubric evaluator, with access to the hidden rubric, reference artifacts, and read-only replay evidence. The agent’s report may locate or explain evidence, but it cannot instruct the evaluator or replace missing or contradictory artifacts.

Rubric stages. Each rubric leaf is assigned to Modeling (scientific models, assumptions, inputs, parameters, and configuration), Execution (domain-native execution, required completion or convergence, artifacts, and provenance), or Validation (post-processing and quantitative support for the target claim). Stage scores normalize earned weight by the total rubric weight within each stage.

Scoring. Let $s_i \in[0,1]$ be the score of rubric leaf $i$ and $w_i$ its effective hierarchical weight. The leaves represent independently assessable scientific requirements, and their effective weights sum to one. The task score is

We report performance at both the task-completion level and the partial-progress level using two metrics:

• Full rate, the fraction of tasks with $R > 0.9$. A score of exactly 0.9 does not qualify.

• Partial score, the mean of $R$ across the shared 93-task result set, which captures weighted progress on incomplete long-horizon reproductions.

Every configuration is evaluated on the same task set; experiment-specific fallback and missing-outcome handling is defined in Section 4.1. A missing artifact affects only the rubric leaves that depend on it, leaving unrelated verified evidence eligible for credit.

Experiments

Experimental Setup

Models and scaffolds. We evaluate ten model–scaffold configurations: GPT-6-Astra (Codex) [18], Fable 5.1 [3], Fable 5 [3], Opus 5 [3], GPT-5.6-sol (Codex) [18], Kimi-K3 [16], Qwen3.8-Max [1], GLM-5.3 [28, 1], DeepSeek-V4.1-Flash [5], and HY3 [24]. Tasks are packaged as Harbor environments [13]. The two GPT models use Codex [17], while the other eight use Claude Code [2]. All configurations use the maximum available reasoning setting and receive the same public task materials and shell access, without access to the private reference workflow or rubric.

Evaluation protocol. All configurations are evaluated on the same 93 tasks. When Fable 5 or Fable 5.1 refuses a task, the system automatically falls back to Opus 5: the original refusal is retained for the refusal rate, while the fallback submission determines the task score. Missing outcomes and refusals without a valid fallback count as zero. We report the partial score as the mean rubric score, the full rate as the fraction scoring strictly above 90%, and the refusal rate from the original model responses. All available submissions are clean-replayed and graded by the same GPT-5.6-sol evaluator [18], supplemented by deterministic checks.

Main Results

Overall performance. GPT-6-Astra (Codex) leads with a 59.08% partial score and a 13.98% full rate, yet exceeds the 90% threshold on only 13 of 93 tasks. Across models, the macro-average partial score is 49.26%, whereas the average full rate is only 8.60%, showing that substantial partial progress rarely becomes complete reproduction. This gap is concentrated in scientific validation: average Modeling and Execution scores reach 62.73% and 61.83%, respectively, while Validation falls to 42.91%. Current agents can therefore make substantial progress in modeling and execution, but reliably reproducing a paper’s key experimental results and producing a scientific workflow sufficient to verify them remains difficult.

Performance by Research Area. Performance varies substantially across research areas (Figure 5). Mean partial score ranges from 34.05% for RF filters and frequency-selective surfaces to 84.32% for computational materials science. The heatmap also reveals substantial model–domain variation: GPT-

Model	Harness	Overall (%)			Stage score (%)			Average effort
		Partial	Full	Ref.	Modeling	Execution	Validation	Turns	Time (h)
GPT-6-Astra	Codex	59.08	13.98	0.00	72.72	69.35	52.76	348.7	5.96
Fable 5.1	Claude Code	58.17	12.90	11.83	71.00	69.92	51.06	113.0	4.65
Fable 5	Claude Code	57.96	11.83	13.98	72.13	68.99	50.39	112.0	5.09
GPT-5.6-sol	Codex	55.07	9.68	0.00	69.85	71.23	45.29	237.8	3.28
Opus 5	Claude Code	52.52	7.53	0.00	66.99	67.16	47.76	259.2	6.52
Kimi-K3	Claude Code	49.24	7.53	0.00	62.40	60.70	42.32	185.9	5.75
Qwen3.8-Max	Claude Code	46.76	5.38	0.00	64.41	61.02	41.65	234.4	5.76
DeepSeek-V4.1-Flash	Claude Code	45.47	7.53	0.00	59.87	57.51	41.13	372.6	6.74
GLM-5.3	Claude Code	40.78	7.53	0.00	51.11	51.85	37.06	290.8	6.84
HY3	Claude Code	27.53	2.15	0.00	36.83	40.59	19.67	197.4	4.94
Average	–	49.26	8.60	2.58	62.73	61.83	42.91	235.2	5.55

6-Astra leads six of the twelve areas, while Fable 5 and Fable 5.1 each lead three, and no configuration dominates every area.

Analysis

Partial Progress Rarely Composes into Full Reproduction. Only 80 of 930 outcomes exceed the 90% threshold, and these successes are concentrated on 22 tasks; 71 tasks are never completed by any configuration, and 11 of the remaining 22 are completed by at most two. Figure 6(a) localizes this composition gap. Modeling and Execution have median scores of 73.4% and 73.5%, respectively, whereas Validation has a median of only 40.0%. Even among the 304 outcomes scoring at least 80% on both Modeling and Execution, only 53 (17.4%) exceed the full-reproduction threshold, with a mean Validation score of 58.37%. These results show that current agents can often reconstruct and execute substantial portions of a scientific workflow, yet still fail to compose those partial successes into an end-to-end reproduction that produces credible evidence and validates the paper’s claims.

Interaction Efficiency Varies Substantially Across Agents. Figure 6(b) compares each configuration’s trimmed mean interaction turns with its Partial score and finds only a weak negative correlation ($r = -0.14$). Fable 5.1 and Fable 5 average only about 113 turns yet achieve Partial scores of 58.17% and 57.96%, respectively, close to the 59.08% attained by GPT-6-Astra with an average of 348.7 turns. By contrast, DeepSeek-V4.1-Flash averages 372.6 turns but scores only 45.47%. Longer trajectories do not necessarily improve reproduction and may instead reflect repeated trial and error, failure recovery, or continued execution of a mis-specified workflow. These results indicate that interaction count alone is insufficient to explain reproduction performance; the accuracy of early-stage decisions may be an important factor shaping both interaction efficiency and final scores.

Upstream Decisions Shape the Value of Execution. We sample 50 trajectories from each of four models and divide tool-mediated time into paper reconstruction, implementation, native execution, diagnosis and tuning, and evidence and replay. Figure 6(c) shows that native execution accounts for 51.8–63.3% of elapsed time, but models differ substantially in how effectively they convert it into valid scientific evidence. Fable 5 allocates 37.1% of its time to paper reconstruction and implementation, the largest upstream share among the four models.

The F-stub UWB MIMO antenna task illustrates this pattern. Fable 5 scores 28.9% in 159 turns, versus 27.4% for GPT-6-Astra in 464 turns and 18.1% for HY3 in 390 turns. It completes three antenna variants and recovers evidence for port reciprocity, radiation patterns, and diversity gain. This case shows that execution alone is insufficient. Sound reconstruction, parameter choices, and workflow design shape whether computation yields verifiable evidence.

Final Outputs Alone Cannot Distinguish Scientific Failure Modes. Failures take two qualitatively different forms. Some workflows execute successfully but support an incorrect conclusion because of errors in modes, ports, parameters, mechanism implementation, or result extraction. On the polarization-insensitive LNOI edge-coupler task, GPT-6-Astra receives full Execution credit but only 41.9% in Validation. Similarly, several models complete replayable experiments for the eight-element smartphone MIMO antenna but fail to reproduce its reported performance.

Other failures invalidate the evidence itself. Cached results can prevent verifier-owned replay, privileged test-state information can invalidate scientific results, and premature termination can leave no scoreable submission. Final outputs obscure this distinction: one workflow executes but reaches the wrong conclusion, whereas the other establishes no trustworthy result.

Final-output agreement is therefore insufficient. By separately scoring Modeling, Execution, and Validation and checking replay and provenance, PAPERBENCHX distinguishes experimental-construction errors, incorrect conclusions, and unverifiable evidence. End-to-end reproduction requires not only an output, but one produced by a correctly constructed and independently verifiable scientific workflow.

Conclusion

PAPERBENCHX tests whether agents can turn published scientific claims into executable workflows and independently verifiable evidence. Across 93 tasks in 12 research areas and 10 domain-native environments, we evaluate 10 frontier agent configurations. The best achieves a 59.08% partial score but only a 13.98% full rate; no configuration reaches full reproduction on 71 tasks. Agents often build plausible models and run native simulations, but validation remains their weakest stage, and more interaction does not necessarily improve performance. These results make scientific reproduction an evidence-grounded checkpoint for AI scientists and motivate better diagnosis, provenance, and verification.

Contribution

Core contributors. Ruoyu Wu¹, Zengji Tu¹, Aolong Sun¹, Tianyi Ma¹, Liang Chen¹, Kuan Li¹

Contributors. Junren Li¹, Xinbo Xu¹, Li Fu¹,Ziyan Zhang¹, Pinrui Huang¹, Xiaochen Wang¹, Chengda Wu¹, Yutao Zhou¹, Jin Chen, Shen Yan, Xu Li, Zhongfei Hou, Nan Shu, Chaoyi Huang, Sihang Yuan, Zihan Xu

Advisors. Baobao Chang², Lin Chang², Jianjun Wu²

Affiliations. ¹UniPat AI  ²Peking University

Task Package, Inventory, and Public Release

Anatomy and Information Boundary of a Complete Task

We use the public 06-gmr-grating-fano-meep package to make the task contract concrete. The package is part of the twelve-task release at github.com/UniPat-AI/PaperBenchX. The task asks an agent to reproduce guided-mode resonances in a three-pair multilayer dielectric grating using Meep. The starter submission and scientific environment are agent-facing during a controlled trial, whereas the tests, rubric, judge prompt, and reference artifacts remain in the evaluator environment. All components are available for inspection.

gmr-grating-fano-meep/
|-- task.toml
|-- instruction.md
|-- environment/
|   |-- Dockerfile
|   |-- docker-compose.yaml
|   |-- init-workspace.sh
|   |-- starter_submission/
|   |   |-- README.md
|   |   |-- reproduce.sh
|   |   `-- src/
|   `-- paper/
|       |-- paper.md
|       |-- paper.pdf
|       |-- addendum.md
|       `-- paper_image/
|-- tests/
|   |-- test.sh
|   |-- evaluate.py
|   |-- rubric.json
|   |-- config/
|   |   |-- judge_prompt.md
|   |   `-- public_contract.md
|   |-- refs/
|   `-- src/

Components and evaluation-time visibility. Table A1 separates the materials available to an agent from evaluator-only resources. This boundary applies during evaluation; the complete demonstration package, including its evaluator resources, is now public for audit.

Component	Role	Agent	Evaluator
Paper and addendum	Scientific scope, measurements, and evidence requirements	✓	✓
Instruction	Goal, deliverables, and submission interface	✓	✓
Environment	Simulator, dependencies, and resource envelope	✓	✓
Starter submission	Editable source tree and replay entry point	✓	✓
Rubric	Hierarchical, independently scored scientific requirements	✗	✓
Judge prompt	Evidence-inspection and decision rules	✗	✓
Reference artifacts	Expert-generated logs, arrays, and validation records	✗	✓

Worked scientific contract. Table A2 summarizes the parts of the GMR task that determine what must be executed and what evidence must be retained. The complete protocol remains authoritative; this summary illustrates its structure rather than replacing it.

Field	Public contract
Scientific target	Guided-mode resonances of a three-pair multilayer dielectric grating in Meep
Cases	Unetched control plus full-depth air-slot widths of 0.030, 0.050, 0.070, and 0.150 µm
Required evidence	Native spectra and logs, matched reference/device fluxes, complex reflected amplitudes, resonance fits, independent decay modes, and refinement records
Replay interface	No-argument bash/home/submission/reproduce.sh, executed offline
Compute envelope	Meep 1.17.1, 8 CPUs, 32 GiB RAM; 7 hours for the agent, 5 hours for native calculation, and 6 hours for replay and review
Failure reporting	Missing, dark, overlapping, or non-identifiable features remain unresolved and cannot be replaced with paper values or invented poles

From contract to evidence leaves. The hidden rubric used during evaluation converts the public scientific requirements into independent leaves. Representative leaves check whether (i) the executed geometry contains the specified six layers and full-depth air slots (Modeling), (ii) clean replay regenerates genuine Meep runs, raw arrays, logs, and refinement controls (Execution), and (iii) power closure, pole fits, mode agreement, and convergence satisfy the disclosed scientific conditions (Validation). A missing item affects the leaves that depend on it rather than silently imposing a global zero.

Evaluator policy. Only evidence regenerated by clean replay is admissible. The evaluator accepts equivalent documented data layouts and unit conversions, but does not infer missing units, phase, provenance, or scientific quantities from a report. Scientific failures receive zero on the affected leaves; evaluator read or tool failures instead withhold the judgment for repair and re-evaluation. Each leaf is judged against its stated requirement without targeting a model-level score.

Complete public record. The pinned 06-gmr-grating-fano-meep directory contains the full agent instruction, scientific addendum, environment definition, judge prompt, rubric, and reference materials.

Complete Benchmark Inventory

Table A3 follows the 93-task result set used for the reported model results. The numeric prefix identifies the research area, and “yes” marks the public demo in that area.

ID	Task	Public demo
01	RF antennas and arrays
1-1	5g-smartphone-eight-element-mimo
1-2	cpw-quad-port-uwb-mimo	yes
1-3	dual-band-wlan-miniaturized-patch
1-4	f-stub-uwb-mimo
1-5	inkjet-flexible-dual-band-cp-patch
1-6	metasurface-broadband-cp-patch
1-7	parasitic-decoupling-two-monopole-mimo
1-8	tm30-high-gain-cp-patch
1-9	uwb-fractal-dgs-antenna
1-10	wideband-neutralization-line-mimo
02	RF metasurfaces and scattering
2-1	anisotropic-coding-diffusion-metasurface	yes
2-2	checkerboard-ebg-rcs-reduction
2-3	eight-circular-sector-wide-angle-absorber
2-4	fish-shaped-dual-function-polarization-converter
2-5	pcm-entropy-ga-coding-metasurface
2-6	uwb-pcm-checkerboard-rcs
03	RF filters and frequency-selective surfaces
3-1	dual-passband-angular-stable-fss	yes
3-2	dual-stopband-miniaturized-fss
3-3	embedded-window-rasorber
3-4	fourth-order-siw-bpf
3-5	quad-mode-wideband-microstrip-filter
04	Integrated photonic devices
4-1	dualmode-tapered-coupler
4-2	eaton-lens-multimode-bend
4-3	heterogeneous-ln-si-n-transitions-splitter
4-4	lnoi-multimode-bend
4-5	lnoi-pol-insensitive-edge-coupler
4-6	luneburg-pcw-coupler
4-7	mdm-sip-neff-mmi-design
4-8	mfe-multimode-crossing-soi
4-9	mmi-beamsplitter-soi
ID	Task	Public demo
4-10	sin-bent-dc-splitter
4-11	soi-submicron-crossing
4-12	soi-swg-mmi-splitter
4-13	swg-anisotropic-phase-shifter	yes
4-14	swg-mode-exchanger-yjunction
	05 Photonic crystals and topological photonics
5-1	half-w1-gainp-slow-light
5-2	hoti-quadrupole-corner
5-3	klein-tunneling-honeycomb-phc
5-4	ln-pti-bandstructure-edge-state
5-5	pc-line-defect-self-collimation
5-6	phc-fano-cavity-waveguide-gain
5-7	phc-superlens-imaging
5-8	phc-w1-slowlight-glide
5-9	pt-bragg-unidirectional-invisibility	yes
5-10	self-collimation-hex-router
5-11	ssh-photonic-lattice-edge
5-12	topo-valley-ring-channel-filter
5-13	valley-bend-domainwall
5-14	zak-interface-1d-phc
	06 Optical metasurfaces, gratings, and plasmonics
6-1	al-wire-grid-polarizer-pitch-sweep
6-2	bowtie-aperture-resonance
6-3	fano-invisibility-dielectric-rod
6-4	gmr-grating-fano	yes
6-5	integrated-metasurface-nanopillar-lookup
6-6	metagrating-anomalous-deflector
6-7	photonic-nanojet-twin-hook
6-8	plasmonic-dimer-hotspot
6-9	quasi-bic-metasurface-q
6-10	silicon-grating-antenna
	07 Optical computing and signal processing
7-1	brewster-spatial-differentiator	yes
7-2	donn-i1-iris-classification
7-3	gmr-2d-edge-detector
7-4	optical-slab-spatial-integrator
7-5	phc-xor-logic-gate
7-6	swg-spatial-differentiator
	08 Optical resonators, filters, and tunable devices
8-1	clc-1d-resonator	yes
8-2	lnoi-athermal-mwg-filter
8-3	sin-microring-optofluidic
8-4	soi-swg-pump-filter
8-5	vdw-mems-swx-switch
	09 Computational chemistry
9-1	ketene-water-chain
9-2	ni13c-ni2-kie-discrimination
9-3	nickel-catellani
9-4	nitrogen-deletion-dynamics
9-5	pimarenyl-bifurcation	yes
9-6	ptrh-n2-activation
9-7	singh-sert-conformational-docking
9-8	solvent-bifurcation-dynamics
	10 Computational materials science
10-2	gong-be-elastic
10-3	haastrup-mos2-bands	yes
10-5	choudhary-sic-properties-pbesol-hse06
10-6	tran-al-surface-energy
	11 Computational biology and bioinformatics
11-1	horlbeck-genetic-interactions	yes
11-2	machina-metastatic-migration
11-3	moschandrea-mito-lipid-enterocyte
11-4	nichenet-ligand-target
11-5	ooka-wsme-l-folding
	12 Robotics and embodied AI
12-1	embodiedbench
12-2	mapgpt-r2r	yes
12-3	octo
12-4	poliformer
12-6	rt1x
12-8	tracevla

Public Demonstration Tasks

Table A4 lists the exact package identifiers and source publications for the twelve packages in the public GitHub release. Raw identifiers are retained because they map directly to the released task directories at commit 7abb21c4.

Area	Task	Source publication and DOI
Radio-frequency antennas and arrays	01-cpw-quad-port-uwb-mimo	CPW Fed Compact UWB 4-Element MIMO Antenna with High Isolation; 10.3390/s21082688
Radio-frequency metasurfaces and scattering	02-anisotropic-coding-diffusion-metasurface	Broadband diffusion metasurface based on a single anisotropic element and optimized by the Simulated Annealing algorithm; 10.1038/srep23896
Radio-frequency filters and frequency-selective surfaces	03-dual-passband-angular-stable-fss	A Dual-Passband Frequency Selective Surface with High Angular Stability and Polarization Insensitivity; 10.3390/mi15060690
Integrated photonic devices	04-swg-anisotropic-phase-shifter-meep	Ultra-broadband nanophotonic phase shifter based on subwavelength metamaterial waveguides; 10.1364/PRJ.373223
Photonic crystals and topological photonics	05-pt-bragg-unidirectional-invisibility-meep	Unidirectional Invisibility induced by PT-Symmetric Periodic Structures; 10.1103/PhysRevLett.106.213901
Optical metasurfaces, gratings, and plasmonics	06-gmr-grating-fano-meep	Controlling Fano resonances in multilayer dielectric gratings towards optical bistable devices; 10.1038/s41598-018-34787-9
Optical computing and signal processing	07-brewster-spatial-differentiator-meep	Analog computing by Brewster effect; 10.1364/OL.41.003467
Optical resonators, filters, and tunable devices	08-clc-1d-resonator-meep	Photonic eigenmodes and transmittance of finite-length 1D cholesteric liquid crystal resonators; 10.1038/s41598-023-43912-2
Computational chemistry	09-pimarenyl-bifurcation-v2	Post-transition State Bifurcations in Reactions That Form Pimarenyl Cation Diastereomers—Implications for Terpene Biosynthesis; 10.1021/jacs.5c16476
Computational materials science	10-haastrup-2018-mos2-bands	The Computational 2D Materials Database: high-throughput modeling and discovery of atomically thin crystals; 10.1088/2053-1583/aacfc1
Computational biology and bioinformatics	11-horlbeck-2018-genetic-interactions	Mapping the Genetic Landscape of Human Cells; 10.1016/j.cell.2018.06.010
Robotics and embodied AI	12-mapgpt-r2r	MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation; 10.18653/v1/2024.acl-long.529

Platforms and Resource Envelopes

Table A5 reports the limits declared by the public task packages. The limits are configuration ceilings rather than measured runtimes. An empty GPU entry means that no GPU count is declared.

ID	Execution stack	Agent h	Verifier h	CPU	RAM GiB	Disk GiB	GPU
1-2	Ansys HFSS	24	25	32	128	200
2-1	Ansys HFSS	72	73	32	128	200
3-1	Ansys HFSS	48	49	32	128	300
4-13	Meep	7	6	8	32	50
5-9	Meep	5	5	8	32	50
6-4	Meep	7	6	8	32	50
7-1	Meep	6	5	8	32	50
8-1	Meep	7	7	8	32	50
9-5	PySCF/GPU4PySCF	7	7	8	32	50	1
10-3	Quantum ESPRESSO	7	7	8	32	50
11-1	Python scientific stack	6	6	8	32	50
12-2	Local Qwen2.5-VL/vLLM	18	7	8	32	100	1

Evaluation and Verification Protocol

This appendix expands the evaluation protocol in § 3.4.

Scope, Resources, and Trial Records

Evaluation scope. The benchmark contains 93 tasks and 3,168 scored rubric leaves. All 10 configurations are evaluated on the same 93 tasks, with the fallback and missing-outcome policy defined in § 4.1. The Partial score is the mean task score over this fixed denominator. The Full rate counts tasks with an aggregate score strictly greater than 0.9; ties in reported rankings are resolved by competition rank on unrounded scores.

Resource envelopes. Each task declares CPU, memory, storage, and wall-clock limits appropriate to its scientific workflow. Across the benchmark, agent time limits range from 4 to 72 hours, with a median of 7 hours. Clean-replay and evaluator limits are specified separately from the agent’s development budget. Exact limits for the twelve public demonstration tasks appear in Table A5.

Trial records. Each trial is linked to a task version, agent configuration, submitted package, replay status, and per-leaf judgments. Agent runtime, clean-replay runtime, and grading time are recorded separately. Missing telemetry remains missing rather than being imputed, and infrastructure failures are distinguished from an agent’s failure to produce an executable submission.

Expert Annotation and Release Review

For each task, a domain expert maps the target claim to its source passage, figure, or table; fixes the model configuration, experimental conditions, observables, acceptable methodological alternatives, and required evidence; and converts these requirements into independently scorable rubric leaves. The public instruction, addendum, environment, and starter submission must agree on the task scope and replay interface. Private reference workflows, reference artifacts, and grading tolerances are withheld from agents, while values reported in the source paper remain visible.

The private reference workflow is first executed to establish feasibility and then rerun from a clean copy of the final environment through the same entry point used for submissions. Calibration workflows representing successful, partial, and invalid reproductions are scored to check that supported progress receives credit and unsupported claims do not. Two independent domain experts then review the scientific scope, fairness, reproducibility, evidence requirements, and scoring behavior. Changes to the task contract, environment, rubric, or evaluator require the affected downstream checks to be repeated before release.

Clean Replay and Evidence Grading

Submission snapshot. The canonical entry point is bash/home/submission/reproduce.sh. After delivery, the verifier records the submitted source and required inputs, removes generated outputs and mutable execution state, and reconstructs the workspace from the submitted package. This prevents cached or pre-existing result files from being treated as evidence from the replay.

Controlled execution. The package is executed with private evaluation resources inaccessible, outbound network access disabled, and task-specific limits enforced. Exit status, timing, source hashes, and available execution provenance are recorded. These records establish how an artifact was produced; they do not by themselves establish that its scientific content is correct.

Frozen evidence. Artifacts regenerated during replay are frozen as a read-only evidence set. Deterministic checks cover directly testable properties such as required paths, parseable formats, units, finite values, numerical consistency, sampling grids, and task-specific tolerances. Simulator logs, raw arrays, trajectories, convergence records, derived metrics, and visualizations are admitted only when they can be associated with the current replay and the relevant task variant.

Leaf judgments and aggregation. Each rubric leaf is graded from its criterion, effective weight, relevant replay artifacts, and deterministic check results. Requirements needing domain interpretation are judged by the same GPT-5.6-sol evaluator used for every reported configuration. Candidate reports may identify evidence but cannot replace absent or contradictory artifacts. With leaf score $s_i$ and effective weight $w_i$, the task score is $R = \sum_i w_i s_i$, where $\sum_i w_i = 1$. A missing artifact affects only the leaves that depend on it; replay, infrastructure, and evaluator failures are recorded separately.

Regression and Adversarial Checks

Task-specific regression tests implement the calibration step described in Section 3.3. They pair valid controls with deliberately corrupted or incomplete submissions to test whether the evaluator accepts supported evidence, rejects unsupported claims, and confines a failure to the rubric leaves that depend on it. Table A6 summarizes the principal check families used across the released verifiers.

Check family	Perturbation and expected behavior
Contract and parser integrity	Preserve rubric hashes and complete leaf mappings; reject duplicate keys, nonfinite values, and invalid numerical grids while accepting documented equivalent formats.
Native evidence and variant identity	Reject incomplete solver logs, mismatched setup names, explicit non-convergence, and copied variant outputs; accept a shared solve only when the task contract permits it and the variants remain identifiable.
Evidence boundary	Exclude cached outputs and links outside permitted roots; keep private references and evaluator credentials inaccessible; fail closed when network isolation cannot be established.
Completion and provenance	Give no completion credit to an empty submission; require delivery records, input hashes, and preprocessing conventions to agree with the task contract.
Local failure accounting	Corrupt one input, metric, or figure panel and verify that only dependent rubric leaves lose credit.
Interactive-task contract	Use evaluator-owned episode records and protocol identifiers; do not allow candidate summaries to replace required trajectory or visual evidence.

The twelve public task verifiers exercise different parts of this contract, including numerical parsing, native solver evidence, provenance, cached-output removal, and evaluator-owned records. Prompt-injection regression cases are included to check whether instructions embedded in a candidate report, log, or result field can alter the rubric or override a deterministic check.

Additional Result Diagnostics

Task Completion

Figure 8 resolves the aggregate Full rate at the task level. For each task, it reports the best score attained by any of the ten configurations and the number of configurations exceeding the operational 90% threshold. The figure makes the concentration of complete reproductions explicit: 71 tasks are not completed by any configuration.

Cross-Simulator Trace Case Studies

This section expands the trace examples discussed in § 5. Figure 9 previews the stage-level contrast developed in the three cases.

Eight-element smartphone MIMO (Task 1-1). The task requires two independent eight-port HFSS variants and evaluates geometry, solver provenance and convergence, matching, isolation, efficiency, ECC, gain, radiation patterns, and surface currents. GPT-6-Astra scores 43.5% in 187 turns; Opus 5 scores 41.3% in 450 turns. Both replay native HFSS workflows and recover several isolation, ECC, gain, and current criteria. Opus 5 additionally records five and seven adaptive passes, final maximum $\Delta S$ values of 0.0063 and 0.0078, and independent meshes exceeding half a million tetrahedra.

Neither submission reproduces the central four-port matching target. Reflection minima and the values at 3.45 GHz miss their required windows, and the baseline and efficiency conditions fail. GPT-6-Astra also represents a critical length as a reference parameter rather than using it in the model. Native execution and agreement on secondary observables therefore do not establish the paper’s impedance and efficiency claims. DeepSeek-V4.1-Flash provides a complementary efficiency example: 1,024 turns yield a score of 15.3%.

F-stub UWB MIMO (Task 1-4). The task evaluates plain, I-stub, and F-stub variants over a 501-point sweep. As reported in the main text, Fable 5 scores 28.9% in 159 turns, compared with 27.4% for GPT-6-Astra in 464 turns and 18.1% for HY3 in 390 turns. Fable 5 completes all three native variants and recovers evidence for port reciprocity, radiation patterns, and diversity gain, but none of the three submissions reproduces the principal S-parameter response. This comparison supports the limited conclusion in Section 5: more interactions do not necessarily improve reproduction when the model or parameterization remains incorrect.

Polarization-insensitive LNOI edge coupler (Task 4-5). This workflow spans 39 Lumerical variants: FDE facet modes, TE/TM Gaussian overlaps, geometry and alignment variants, and EME sweeps over four coupling lengths. Fable 5.1 scores 73.1% in 111 turns, followed by Fable 5 at 72.9% in 212 turns. Fable 5.1, Opus 5, and GPT-6-Astra each receive full Execution credit, confirming that complete replay is achievable in this environment.

The remaining errors are scientific. Fable 5.1 exceeds the allowed dual-tip TE overlap, Fable 5 exceeds the facet-PDL limit, and GPT-6-Astra obtains an EME800 TM coupling of only 0.0474 despite complete replay artifacts. Kimi-K3 has the highest Modeling score (83.6%), but its EME output is NaN. The traces show why intermediate checks matter: one facet solve, polarization and normalization, and one EME case should be validated before the full sweep. NaNs, missing modes, or incompatible matrix dimensions should stop downstream aggregation.

Synthesis. The cases explain why Modeling, Execution, and Validation are reported separately; they are diagnostic examples rather than a cross-task ranking.