betaFeedback

WorksComputer Science

Benchmarking LLM abilities on postdoc-level natural science problems

Abstract

Scientific problems have no answer key, so evaluating an agent means deciding what counts as a better solution before any agent is run. Existing benchmarks either match a symbolic answer against a key or, where they require executable work, score it by unit tests, by a rubric, or by position on a competition leaderboard. None of these is the criterion by which a field decides that one published method beats another. STEMBench adopts that criterion directly. Each of its twelve research problems in high-energy physics, astrophysics and neuroscience is scored by its own field's figure of merit, and each comes with two measured anchors: a submission that uses no domain knowledge, and an expert reference reproducing a published method, both obtained by running code shipped with the benchmark. The interval between them states how much of a problem is crossed by scientific insight rather than by engineering. Across eight frontier models, task-level wins and domain-level leadership dissociate: a model that wins five of the twelve tasks leads no domain mean. In the high-effort cross-model comparison, at least one model's mean score exceeds the expert anchor on seven of twelve tasks. The remaining five span uncertainty estimation, latent and causal inference, and cross-site clinical classification. Numerical scores and official-submission validity capture different aspects of performance and are reported separately. We release the tasks, graders, anchors and harness.

Introduction

Asking whether an agent solved a scientific problem requires deciding, before the agent is run, what makes one solution better than another. That decision is the hard part of building an evaluation for scientific work, and it is the part that existing benchmarks answer by substituting a criterion that is easier to automate: a reference answer, a test suite, a rubric, or a position in a competition field (§ 2). None of these is the criterion by which a field decides that one published method beats another.

We present STEMBENCH, twelve research problems in high-energy physics (HEP), astrophysics, and neuroscience that adopt that criterion directly. Each problem is scored by the figure of merit its own community already uses to compare published methods, from background rejection at fixed signal efficiency to coverage-penalised interval width. Each comes with two anchors obtained by running code shipped with the benchmark: a floor produced by a submission that uses no domain knowledge, and a ceiling produced by reproducing a published method. The interval between them states how much of the problem is crossed by a scientific insight rather than by engineering, and it is against that interval, not against a perfect score, that we read what an agent achieves. Agents work offline in a sandbox over staged data of 0.1 to 12 GB and are graded host-side on withheld splits.

Our contributions are:

  • A scoring design anchored at both ends. Every task ships a baseline and an expert reference submission, both runnable end to end on one workstation, so the interval a solution must cross is measured rather than assumed, and neither anchor is a number quoted from a paper (§ 3).

  • Twelve task contracts with community metrics. One uniform layout across three domains, with ground truth held host-side, retrieval restricted to a curated per-task corpus, and metrics carrying coverage penalties and geometric gates as explicit defences against exploitation (§ 4).

  • An evaluation of eight frontier models. Task-level wins and domain-level leadership dissociate. At least one model’s mean score exceeds the expert anchor on seven of twelve tasks in the high-effort cross-model comparison. The remaining five span uncertainty estimation, latent and causal inference, and cross-site clinical classification. Official-submission validity is reported separately from the numerical score (§ 5).

Related Work

Scientific knowledge and reasoning. Physics and mathematics benchmarks score an answer against a reference [44, 26, 6], and multimodal STEM evaluations and domain question-answering collections extend this setting to visual evidence and specialised knowledge [31, 45, 29]. Agentic systems now reach elite-human level here: Physics Supernova attains 23.5/30 on the IPhO 2025 theory problems, placing 14th of 406 contestants [25]. The criterion, however, is still a marking scheme for problems with known solutions. These evaluations measure important prerequisites for scientific work, but what they score is the quality of an answer, not of a method evaluated on withheld data under a community figure of merit.

Benchmarks that score executable work. SciCode scores scientific code by unit tests [36], ScienceAgentBench scores self-contained programs for 102 tasks drawn from 44 papers [9], and Terminal-Bench-Science verifies expert-curated research workflows in a terminal, with one oracle solution per task demonstrating feasibility [35]. MLE-bench scores 75 Kaggle competitions, and its multilingual and biomedical extensions and MLR-Bench carry the setting to other languages, to biomedical pipelines and to open-ended ML research [7, 37, 20, 8]. These are close to the classification and anomaly-detection tasks of STEMBENCH, but not to its inversion and latent-parameter tasks, where predictive accuracy alone does not describe scientific validity. PaperBench, CORE-Bench and PRBench, the last with 30 tasks across 11 physics subfields, score the reproduction of published results [33, 32, 27]. ScienceBoard and LabOSBench move agents into scientific software and instrument simulators, 169 tasks across four domains and 96 subtasks on eight instruments, and count a task as solved when the required operations are completed [34] (Zou et al. 2026). Reproduction is also how we build expert anchors, although the evaluated agents need not recover the reference method. A test suite, a terminal check or a completion check accepts or rejects a solution, whereas a field orders competing methods on a continuous figure of merit. STEMBENCH adopts the ordering and scores it on a withheld split.

Scoring relative to a reference. MLE-bench reads scores against a human leaderboard, InnovatorBench against a reference solution on 20 LLM-research tasks, and NatureBench against the reported state of the art of the 90 Nature-family papers its tasks come from [43, 38]. STEMBENCH ships two anchors per task, both measured by running code in the repository: a no-domain-knowledge floor and a reproduced published method. It reports raw scores beside them rather than a single normalised quantity (§ 3), and treats the expert anchor as an empirical reference, not an upper bound.

Autonomous discovery agents. A parallel line of work builds agents that run the discovery loop end to end; a recent survey organises the field into process-, autonomy- and mechanism-oriented views [40]. Kosmos iterates literature search, data analysis and hypothesis generation for up to twelve hours per run and is validated by experts auditing its final reports, with 79.4% of statements judged accurate [22]. Such assessment is made after the run, by readers, and cannot be re-executed. STEMBENCH fixes an executable acceptance criterion before any agent runs, bounded by two measured anchors, which makes it a regression test for the analysis stage of the discovery loop rather than a competitor to end-to-end evaluation.

Expert-rubric evaluations. COMPOSITE-STEM combines exact-match assessment with criterion-based rubrics and LLM-jury grading on 70 expert-written tasks, and LifeSciBench assesses 750 life-science workflow tasks against expert-authored rubrics [39, 19]. Rubrics capture aspects of scientific judgment that resist reduction to a number; STEMBENCH concentrates on computational problems where an explicit, executable scoring rule can order the submitted results. Table 1 summarises the comparison.

BenchmarkEvaluation targetAssessment or performance reference
ScienceAgentBenchData-driven scientific programsProgram and execution evaluation
Terminal-Bench-ScienceScientific workflowsTask-specific terminal verification
ScienceBoardScientific software workflowsTask-completion checks
LabOSBenchScientific instrument controlSubtask completion within a step budget
MLE-benchML competition submissionsCompetition metrics and human leaderboard results
BioML-benchBiomedical ML pipelinesTask performance and human baselines
PRBenchPhysics-paper reproductionReproduced quantitative results and scoring rubrics
MLR-BenchOpen-ended ML researchLLM reviewers and research-quality rubrics
COMPOSITE-STEMExpert STEM tasksExact-match and rubric-based jury grading
LifeSciBenchLife-science research responsesExpert-authored task-specific rubrics
InnovatorBenchRunnable LLM-research artifactsCorrectness, performance, and output quality
NatureBenchScientific ML solutionsSource-paper state-of-the-art performance
STEMBENCH (ours)Computational solutions in HEP, astrophysics, and neuroscienceScientific task metrics; executable baseline and expert anchors

Table 1. Selected related benchmarks and their evaluation targets. Rows summarise the main assessment criterion, not the only one. STEMBENCH reports its baseline and expert-reference scores alongside the task score.

Problem Formulation

We call a problem scientific, as opposed to an exercise, when it has the seven properties of Table 2. Each one breaks an assumption that current STEM benchmarks rely on, and each is exercised by at least one task in STEMBENCH.

Table 2: Seven properties that separate a scientific problem from an exercise, each with an example from STEMBENCH.

PropertyWhat holds instead of an answer key
P1 Underdetermined solution setdistinct methods are all admissible and are ordered by a continuous figure of merit, not accepted or rejected
P2 Model and code as deliverablean executable artefact that generalises to a withheld split; on two tasks the grader itself trains the submitted model
P3 Load-bearing domain priorsthe gap to the expert is closed by a domain insight, e.g. a decorrelation condition for weak supervision or a canonical basis for Stokes profiles
P4 Data beyond the context0.1–12 GB of staged data, reached through loaders and subsampling; deciding what to look at is part of the task
P5 Community figure of meritgated composites and non-monotone rules: a sharper interval that under-covers loses to a wider one that covers
P6 Silent failuresplausible errors, such as data leakage or a forward model that fits the spectrum but breaks the physics, which only withheld data exposes and an LLM judge reading the output cannot see
P7 Empirical ceilingthe literature ceiling for cross-scanner ASD classification is about 0.75 AUC, so progress is distance to the best published solution, not to 1.0

Table 2.

Tasks and scores. A task instance is a tuple τ=(P,Dtr,Dte,C,μ,b,e)\tau= (\mathcal{P}, \mathcal{D}_{\mathrm{tr}}, \mathcal{D}_{\mathrm{te}}, \mathcal{C}, \mu, b, e), where P\mathcal{P} is the natural-language specification (motivation, input/output contract, data description, metric description), $\mathcal{D}_{\mathrm{tr}} and DteD_{\mathrm{te}} are the visible and withheld splits, CC is the submission contract accepted by the scorer, μ:C→[0,1]\mu:C\to[0,1] is a deterministic scoring function, and bb, ee are two anchor scores defined below.

An agent is a policy π\pi that interacts with a sandboxed environment under a budget BB (wall-clock, tokens, messages) and returns a submission σπ∈C\sigma_\pi\in C. Two anchors are shipped with every task: a baseline b=μ(σbase)b=\mu(\sigma_{\mathrm{base}}), obtained by a submission that uses no domain knowledge, and an expert reference e=μ(σref)e=\mu(\sigma_{\mathrm{ref}}), obtained by reproducing a published method. Both are organiser-side artefacts. They are never staged into the agent sandbox and play no role in an episode beyond fixing the scale.

The harness reports three quantities per episode: the raw task score μ(σπ)\mu(\sigma_\pi) in the task’s native definition, the gap above the baseline

Δbase(π,τ)=100⋅(μ(σπ)−b)[percentage points],(1)\Delta_{\mathrm{base}}(\pi,\tau)=100\cdot\left(\mu(\sigma_\pi)-b\right)\quad\text{[percentage points],} \tag*{(1)}

and official_submission_valid, a reproducibility gate that is independent of the numeric score and requires a syntactically valid entry point, a README with a runnable command, and every declared reproducibility file. We report raw scores with both anchors alongside, because the quantity the benchmark is designed around is the headroom

h(τ)=e−b,(2)h(\tau)=e-b, \tag*{(2)}

the part of the task that is crossed by a scientific insight rather than by engineering. A normalised score μ^=clip⁡[0,1]((μ(σπ)−b)/h(τ))\hat{\mu}=\operatorname{clip}_{[0,1]}((\mu(\sigma_\pi)-b)/h(\tau)) can be derived post hoc from the reported triple, but we do not use it as the headline number: it discards information whenever h(τ)h(\tau) is narrow, and on this task set the narrow-headroom cases are exactly the scientifically hardest ones.

The STEMBench Benchmark

STEMBench consists of twelve research problems in three domains: four in high-energy physics, three in astrophysics and five in neuroscience. Every problem ships as a self-contained contract folder with a task description, a metric specification, a role-tagged data manifest, a declared submission contract, a host-side grader, and the two anchor submissions. Figure 1 shows the resulting evaluation loop and Table 3 lists the tasks with their anchors.

Figure 1. The STEMBENCH evaluation loop. The agent reads the task specification and the public split, works in an offline sandbox through six tools, and returns a submission in the task’s contract. Ground truth never enters the sandbox: the grader runs host-side on the withheld split and returns μ∈[0,1]\mu\in[0,1] with a reproducibility verdict, read against the anchors bb and ee.

DomainTaskEval. unitsMetric familyb→eb \rightarrow e
HEPNew-physics search (LHCO)135,261 eventsROC-AUC, SIC-AUC, rejection0.327 →\rightarrow 0.912
HEPHiggs uncertainty (FAIR Univ.)100 pseudo-exps.coverage-penalised interval0.400 →\rightarrow 0.687
HEPTop tagging (TopLandscape)403,999 jetsROC-AUC, acc., rejection at ϵS\epsilon_S0.063 →\rightarrow 0.884
HEPDark Machines34 signal cellsϵS\epsilon_S at fixed ϵB\epsilon_B, AUC0.003 →\rightarrow 0.221
AstroStokes inversion, simulated10,000 spectrapinball loss + forward gate0.009 →\rightarrow 0.624
AstroStokes inversion, real (Hinode)2 rasters+ spatial-structure term0.000 →\rightarrow 0.598
AstroSN Ia classification (PLAsTiCC)317 objectsROC-AUC, PR-AUC, log-loss0.421 →\rightarrow 0.849
NeuroASD classification (ABIDE)332 subjectsheld-out-site AUC, Brier, age0.000 →\rightarrow 0.312
NeuroNMM family id.45 serieschance-corrected accuracy0.000 →\rightarrow 0.900
NeuroNMM digital twins12 interventionssigned causal-response composite0.100 →\rightarrow 0.715
NeuroNMM parameter recovery20 subjectsmedian relative parameter error0.125 →\rightarrow 0.515
NeuroBOLD deconvolution21 casesneural drive + two kernel banks0.012 →\rightarrow 0.353

Table 3. The twelve STEMBENCH tasks. “Eval. units” is what the metric is computed over once. Anchors b→eb \rightarrow e are the reference_scores declared in each task’s config.yaml, obtained by running the shipped baseline and expert submissions end to end on a single workstation.

The three domains were selected because they stress different parts of the capability: high-energy physics contributes large-scale data and hypothesis-free search, astrophysics contributes ill-posed inversion from remote observation with uncertainty quantification, and neuroscience contributes multi-scale forward models and cross-site generalisation on real clinical data.

Task Selection

Each of the twelve problems satisfies five criteria, applied before any solution was written. The problem falls within the scientific scope of the domain; a solution by classical or machine-learning methods is known to exist, so the task is solvable rather than open; a typical solution requires several non-trivial reasoning steps; a clear mathematical metric is definable on the submission; and the problem exercises a metric family that the others do not. The last criterion is what prevents a single agent from topping the aggregate by specialising in one scoring rule.

The neuroscience block is the one place where we revised the selection after the first round of runs. A single composite neural-mass-model task bundled three distinct inferential operations, model-family identification from generated dynamics, causal digital-twin response prediction, and latent parameter inference, into one weighted score, so a high score on one operation could mask a failure on another. The three now stand as independent contracts with their own hidden-target boundaries, submission schemas and anchors, and the legacy composite is retired.

Table 3 lists the twelve problems with the unit each metric is computed over, the family its scoring rule belongs to, and the two anchors. Appendix A states every task in the terms the agent receives it, and Appendix B reproduces one complete agent-facing prompt.

The Uniform Task Contract

Heterogeneity across scientific domains is real, but it should not be part of what a benchmark measures. Every task is therefore organised in one layout. A single config.yaml is the authoritative contract: it declares the data manifest with a role per file (training, test_input, auxiliary, ground_truth), the required submission files with the format checks applied to them, the host-side scorer entry point, the two anchors, the package requirements and the compute requirements. Alongside it, description.md carries the scientific specification handed to the agent, metric_description.md pins down the scoring rule, grade.py implements it, checksums.yaml fixes the data, and baseline_solution/ and expert_solution/ hold the two anchor submissions with their own instructions for reproducibility. Eight tasks, all of astrophysics and neuroscience, additionally ship a curated references/ corpus of papers and domain notes.

Only files tagged with a public role are staged into the sandbox. Ground-truth files never leave the host and are read only by the grader after the episode ends. Every active task exposes the same grading signature, grade(submission_dir, answers_path), which is what makes twelve heterogeneous scientific workflows scoreable through one harness.

For the high-energy-physics family the upstream datasets are hosted externally, so the repository bundles loaders, manifests and checksums while the bulk data is fetched from the cited sources and preprocessed into the staged splits, which range from 0.1 to 12 GB per task. All datasets used are openly accessible with redistribution rights compatible with a public benchmark: Zenodo releases for the LHC Olympics R&D dataset and PLAsTiCC, public challenge releases for TopLandscape and the FAIR Universe Higgs data, a public data-use agreement for ABIDE, open solar data for the Hinode/SOT-SP rasters, and in-project generation for the synthetic neuroscience tasks.

Metrics and Defences Against Gaming

Each of the twelve metrics instantiates the figure of merit by which the corresponding community already compares published methods: SIC-AUC as used to rank the LHC Olympics teams, the official coverage-and-width scoring rule of the NeurIPS 2024 FAIR Universe competition, background rejection at fixed signal efficiency as used in every modern jet-tagging paper, signal efficiency at fixed background efficiency per (channel, signal) cell for Dark Machines, the PLAsTiCC ranking metric, the axes along which solar inversion codes are compared, the validation metrics of the Virtual Brain and Balloon model papers, the rsHRF benchmark metrics, and macro AUC over held-out acquisition sites for ABIDE. Each metric therefore inherits a literature floor and ceiling, which is what makes the anchors in Table 3 interpretable.

Three failure modes are addressed by construction. Against metric exploitation, composite scores bound every component to [0,1][0,1] before convex combination, so no submission can inflate the headline by driving a single component to an extreme, and the Higgs task applies an asymmetric coverage penalty so that an under-covering interval cannot win on width alone. Against component substitution, the two inversion tasks add a gate of our own on top of the community metric: the forward-consistency and spatial-structure terms enter as RF\sqrt{R_F} and RS\sqrt{R_S}, so that a submission cannot win on spectral reconstruction while failing to recover parameters, and a declared forward model that is inconsistent with the physical law it claims to implement is caught by a dedicated validity battery and zeroes the corresponding term. Against reward hacking on the reference, every expert submission was produced by reproducing a published method without consulting the scorer implementation, so the metric and the reference were authored independently; a leakage attack would surface as a discrepancy between the reference score and the published score of the same method on the same task.

Anchors

The baseline anchor is deliberately weak but not degenerate: it is what a capable engineer produces without domain knowledge, for example a constant train-marginal quantile predictor on the inversion tasks, a range-midpoint parameter estimate on neural-mass parameter recovery, a constant-μ\mu interval on the Higgs task, or a literal zero-response prediction on the digital-twin task. Several metrics normalise against exactly that predictor, which is why six baselines sit at or near 0.000 by construction rather than by accident.

The expert anchor reproduces a published method end to end. The resulting gaps are wide and scientifically meaningful, from 0.000 →\rightarrow 0.900 on neural-mass family identification to 0.009 →\rightarrow 0.624 on simulated Stokes inversion, with the deliberately narrow 0.000 →\rightarrow 0.312 on ABIDE reflecting the hard ceiling that real cross-scanner clinical data imposes (P7). In every case the gap was crossed by a domain-specific insight rather than by scaling compute or data, which is the property that makes the headroom a measurement of scientific reasoning rather than of engineering throughput.

Environment and Toolchain

The harness is built on Inspect-AI (AI Security Institute). Publication runs use per-domain container images that mount only the public files declared in the manifest, read-only, with networking disabled and every package pre-installed; Appendix § 9 lists the images, the hardware and the per-episode limits.

The agent-facing surface is six tools, fixed across tasks. bash and python execute code in the sandbox. data_explorer returns a structural summary of a public data file (shape, dtype, statistics) without loading it into the model context, over eight formats, and refuses pickle-like extensions rather than implicitly unpickling them. local_library_search and document_reader expose the task-local references / corpus, including PDF extraction, on the eight tasks that ship one. validation_feedback is a pre-submission checker that reports missing files and schema violations and never reveals the numeric score. There is no internet access, which would turn problems with published solutions into retrieval exercises.

All results reported here use the basic agent; the basic+references variant is released with the harness and its evaluation is left to future work. The basic agent runs a validation-gated ReAct loop in which a submission is accepted only after the structural validator passes, with rejections returned as JSON feedback that the model can act on. The basic+references agent is identical except that the two retrieval tools are enabled where a corpus exists. Both are benchmark-pure: no hidden score, anchor or ground truth is revealed at any point. Each episode runs under one hour of wall-clock time, 10610^{6} model tokens and 200 messages.

The system prompt is identical across tasks and models and is reproduced in Appendix § 8. Two of its instructions affect how a score should be read. The agent is told to produce a valid submission early and keep it valid, so that a budget limit truncates improvement rather than erasing all work; an episode that still ends with no valid submission is therefore a substantive failure and not a formatting accident. The agent is also warned away from constant, group-midpoint and globally broadcast predictions. Several baseline anchors are exactly such predictors, so the scaffold tells the agent that the trivial solution is known to be trivial, without disclosing its score.

Evaluation Protocol

Holding the harness fixed and varying the model isolates the core LLM; holding the model fixed and varying the harness isolates the tooling. Both axes are needed, since a gain reported on one is routinely produced by the other. The harness passes a functional test suite and every contract passes checksum and anchor-grading smoke tests (Appendix § 9). A blinded expert-agreement study, comparing the automatic ranking of intermediate submissions between bb and ee with rankings by domain reviewers against targets of τ≥0.85\tau\ge0.85 and ≥90%\ge90\% pairwise agreement, is the remaining validation step.

Results

Figure 2 reports eight frontier models under the basic scaffold at high reasoning effort (exact values and domain means are in Appendix § 10). A retained run is one whose model dialogue completed without infrastructure interruption. All retained runs enter every mean, including runs scoring zero.

Raw task scores of eight frontier models using the basic agent at high reasoning effort

Figure 2. Raw task scores of eight frontier models using the basic agent at high reasoning effort. Each marker is the mean of the three retained scored runs, including zeros. Vertical bars show the minimum and maximum of these three scores, not standard deviations or confidence intervals. Horizontal lines mark the baseline bb (dashed) and expert reference ee (solid). A numerical zero does not by itself identify an invalid submission. The digital-twin expert anchor is e=0.715233e = 0.715233.

Three observations follow. First, task-level wins and domain-level leadership differ. DeepSeek V4 Pro has the highest HEP mean (0.60), while Gemini 3.1 Pro leads astrophysics (0.84) and neuroscience (0.32). Opus 5 and Gemini 3.1 Pro each lead five of the twelve tasks; GLM 5.2 leads the remaining two. Opus nevertheless leads no domain mean. In HEP, its wins on Higgs uncertainty and top tagging coexist with a lower new-physics-search score, yielding a domain mean of 0.55, below DeepSeek’s 0.60. In neuroscience, its wins on digital twins and parameter recovery coexist with means displayed as 0.00 on the other three tasks.

Gemini’s domain means range from 0.32 to 0.84, a spread of 0.52. The cross-model spreads of domain means are 0.37 in HEP, 0.64 in astrophysics, and 0.26 in neuroscience, using the displayed values. Thus, this within-model spread exceeds the cross-model spread in HEP and neuroscience, but not in astrophysics. A descriptive two-way sum-of-squares decomposition of the 96 unrounded task–model means assigns 60.5% of the total variation to task main effects, 11.5% to model main effects, and the remaining 28.0% to departures from additivity. This residual includes both task-model interaction and sampling variability in the three-run cell means.

Second, at least one model’s mean score exceeds the expert anchor on seven of the twelve tasks: new-physics search, top tagging, Dark Machines, both Stokes inversions, SN Ia classification, and neural-mass family identification. All model means remain below the expert anchor on the other five: Higgs uncertainty (best mean 0.48, e=0.687e = 0.687), ASD classification on ABIDE (0.29, e=0.312e = 0.312), neural-mass digital twins (0.31, e=0.715233e = 0.715233), neural-mass parameter recovery (0.42, e=0.515e = 0.515), and BOLD deconvolution (0.19, e=0.353e = 0.353). On digital twins, the best mean remains closer to the baseline (b=0.100b = 0.100) than to the expert anchor. Aggregate scores alone do not establish a single cause for these gaps. Counting individual runs rather than means, nine tasks exceed the expert anchor: ABIDE reaches 0.314 and parameter recovery 0.583.

Third, numerical scores and official-submission validity are distinct outcomes. Of the 288 retained runs, 168 have an affirmative official-submission flag, 64 a negative flag, and 56 an unavailable flag. Among the 88 runs scoring exactly zero, these counts are 15, 17, and 56, respectively. Zero scores therefore cannot be interpreted automatically as failures to produce an official submission.

Four model means on real Stokes are exactly zero, yet all twelve underlying flags are unavailable; two ABIDE means round to 0.00 but are positive, with negative flags for Opus and affirmative ones for Gemini 3.1 Pro. Exposure to the wall-clock cap varies across models in the retained campaign. It is recorded for 14 of 36 MiMo V2.5 runs (38.9%), 15 of 36 DeepSeek V4 Pro runs (41.7%), and 2 of 36 Gemini 3.7 Flash runs (5.6%). These frequencies describe observed stopping conditions; they do not quantify the score improvement that a larger budget would produce. Because the cap is wall-clock, it also absorbs provider latency, so these runs measure capability under a fixed time budget rather than per-token capability.

Pilot effort sweep. Before the campaign, a pilot sweep ran Gemini 3.1 Pro at low, medium and high reasoning effort on all twelve tasks (Appendix § 11). It used an earlier task revision and predates a fix to GPU access in the containers, so its absolute scores are not comparable with Figure 2. Qualitatively, most of the gain came between low and medium, several tasks were bimodal, with runs landing either near the best score or at zero, and the effect of effort was not monotone on every task. These observations motivated fixing the campaign at high effort and repeating every cell three times.

Conclusion

STEMBENCH scores scientific problem solving using community metrics and two measured anchors: a baseline and a reproduced expert reference.

In the high-effort comparison of eight frontier models, at least one model’s mean score exceeds the expert anchor on seven of twelve tasks, while on others, such as digital-twin response prediction and BOLD deconvolution, the best model remains far below it. The five remaining gaps span uncertainty estimation, latent and causal inference, and cross-site clinical generalisation. Task wins and domain leadership differ, and numerical scores must be distinguished from official-submission validity. This range of difficulty lets STEMBENCH differentiate models by domain and task type rather than by a single aggregate score.

These conclusions have limits. Stable model rankings will require more runs per cell and confidence intervals; a pilot effort sweep (Appendix § 11) covers one model on an earlier task revision; and agreement between the automatic metrics and blinded expert rankings remains to be measured.

We release the tasks, graders, anchors and harness [3 b].

Reproducibility Statement

Every claim in this paper is tied to an artefact in the benchmark repository. The twelve task contracts, their graders, both anchor submissions per task and the harness are released; § 4 describes the contract layout and Appendix § 8 reproduces the agent-facing prompt surface verbatim. Anchors quoted in Table 3 are the reference_scores in each task’s config.yaml and are reproduced by running the shipped baseline and expert submissions. Appendix § 10 reports aggregate per-model scores for the cross-model comparison. Appendix § 11 reports a pilot effort and budget sweep on an earlier task revision. Data provenance and licences are given in § 4, and per-domain container images pin the software environment. The task contracts, graders, both anchor submissions and staged data are archived at [3], and the evaluation harness at [3].

Ethics Statement

The ABIDE task uses de-identified human neuroimaging data obtained under the public ABIDE data-use agreement; no new human-subject data was collected and no participant is identifiable from the staged files. The task predicts a clinical label from resting-state fMRI, and we state explicitly in the task description and in Appendix § 7 that cross-scanner transfer of such a marker is a known hard limit: the benchmark measures methodological competence on a public research dataset and the scores reported here carry no clinical validity and must not be read as evidence that any method is fit for diagnostic use. The neural-mass digital-twin task is a controlled synthetic benchmark and is likewise not a patient-specific model. All other datasets are simulated or openly released physical-science data.

AI Use Disclosure

We used generative AI tools in the preparation of this paper. Specifically: to draft § 3 and § 4 from internal project reports authored by us; to develop the conceptual framing of those sections, including the taxonomy of properties P1–P7 and the formalisation of a task instance with its two anchors; to search for and verify related benchmarks; to aggregate per-run scores into Table 7 and Table 2 and to draft the accompanying interpretation; and to edit prose for readability throughout. We did not use generative AI tools to formulate or prove mathematical claims, to generate any benchmark data, or to write any task, grader, baseline or expert solution in the benchmark itself. The twelve problems, their metrics, the reference implementations and all experimental runs are the authors’ own work. All AI-assisted output was reviewed by the authors. The cross-model task and domain means in Table 5 and Figure 2 were recomputed from the 288 retained per-run scores in the campaign report. We take responsibility for the final content of this work, including text, claims and artifacts produced with the aid of generative AI. The language models evaluated in § 5 and Appendix § 11 are the object of study, not tools used to produce this paper.

TaskbbeeGemini 3.7 FlashGemini 3.1 ProOpus 5DeepSeek V4 ProQwen3.8 2.4TGLM 5.2MiMo V2.5MiniMax M3
New-physics search0.330.910.310.930.530.830.540.620.000.38
Higgs uncertainty0.400.690.410.430.480.390.440.280.140.13
Top tagging0.060.880.560.650.940.890.930.600.590.56
Dark Machines0.000.220.220.290.250.280.220.320.180.20
HEP mean——0.370.570.550.600.530.460.230.32
Stokes, simulated0.010.620.010.780.570.250.000.000.430.11
Stokes, real0.000.600.120.820.090.000.000.000.140.00
SN Ia classification0.420.850.500.910.940.900.930.600.930.93
Astro mean——0.210.840.530.380.310.200.500.34
ASD classification0.000.310.090.000.000.100.100.290.080.19
NMM family id.0.000.900.001.000.000.510.560.000.370.02
NMM digital twins†0.100.720.100.240.310.140.100.030.000.10
NMM param. recovery0.120.510.120.180.420.120.120.080.020.12
BOLD deconvolution0.010.350.000.190.000.000.020.000.000.06
Neuro mean——0.060.320.150.180.180.080.090.10

Table 5. STEMBENCH results, basic agent, raw task scores. Each cell is the mean over three scored high-effort runs retained in the current report (288 runs in total), including zeros. Domain means are unweighted means of the task means, computed before rounding. Bold denotes the highest unrounded mean in each row. b/eb/e are the baseline and expert anchors. †The digital-twin expert anchor is e=0.715233e = 0.715233 (displayed as 0.72).

TasklowmediumhighBest0-runs
HEP
New-physics search0.3±0.30.3 \pm0.30.3±0.50.3 \pm0.50.6±0.50.6 \pm0.50.9324/9
Higgs uncertainty0.44±0.050.44 \pm0.050.46±0.160.46 \pm0.160.5±0.10.5 \pm0.10.6400/9
Top tagging0.806±0.0060.806 \pm0.0060.86±0.030.86 \pm0.030.8±0.20.8 \pm0.2†0.9410/20
Dark Machines0.22±0.190.22 \pm0.190.31±0.030.31 \pm0.030.32±0.050.32 \pm0.050.3590/9
Astro
SN Ia classification0.73±0.020.73 \pm0.020.87±0.030.87 \pm0.030.91±0.040.91 \pm0.040.9350/9
Stokes, simulated0.07±0.110.07 \pm0.110.74±0.070.74 \pm0.070.5±0.40.5 \pm0.40.8150/9
Stokes, real0.000±0.0000.000 \pm0.0000.3±0.30.3 \pm0.30.81±0.030.81 \pm0.030.8314/9
Neuro
NMM family id.0.000±0.0000.000 \pm0.0001.000±0.0001.000 \pm0.0001.000±0.0001.000 \pm0.0001.0003/9
NMM digital twins0.27±0.150.27 \pm0.150.16±0.090.16 \pm0.090.18±0.050.18 \pm0.050.3810/9
NMM param. recovery0.125±0.0000.125 \pm0.0000.2±0.30.2 \pm0.30.10±0.040.10 \pm0.040.4682/9
BOLD deconvolution0.16±0.140.16 \pm0.140.28±0.130.28 \pm0.130.06±0.090.06 \pm0.090.4101/9
ASD classification0.13±0.160.13 \pm0.160.327±0.0000.327 \pm0.0000.000±0.0000.000 \pm0.0000.3274/9
Mean of task means0.2720.4840.484
Median of task means0.1900.3330.499

Table 7. Agent scores by task and reasoning effort, google/gemini-3.1-pro-preview on the basic scaffold. Cells give mean ±\pm standard deviation over three runs, except top_quark_tagging at high effort (n=14n = 14, marked †). “Best” is the maximum over all runs of the task; “0-runs” counts runs scoring exactly zero: a missing submission, a submission that failed validation, or a valid submission that did not outperform the no-skill baseline; the count does not distinguish these cases. The last rows aggregate the twelve per-task means.

Task Descriptions

This appendix states each of the twelve problems as the agent receives it: what is visible in the sandbox, what the submission contract requires, and the property that keeps a competent generic pipeline well below the expert anchor. Evaluation units, metric families and the two anchors are given once, in Table 3, and are not repeated here. Identifiers are the directory names in the release.

High-energy physics

New-physics search (lhco_anomaly). Half a million unlabelled dijet events, seven observables per jet; submit one anomaly score per test event. No labels exist at any point, so the discriminant must come from the unlabelled mixture itself at S/B∼10−2S/B \sim10^{-2}. The resonance masses are disclosed, as in the public LHC Olympics release, so the difficulty is statistical rather than one of guessing the signal model [18].

Higgs uncertainty (higgsml_uncertainty). Ten million weighted simulated events; submit a point estimate of the signal strength and a central interval per pseudo-experiment. Each pseudo-experiment is drawn at its own hidden values of six nuisance parameters, so the test distribution is shifted. Only coverage and width are scored: a sharp interval that under-covers loses to a wider one that covers [5].

Top tagging (top_quark_tagging). Jets as unordered variable-length sets of constituent four-momenta plus global kinematics; submit a top-jet probability. Two of the four score components are background rejections on a logarithmic scale, so a percent-level gain in discrimination moves the headline far more than the same gain in accuracy. Rewards permutation-invariant pooling, penalises flattening [28].

Dark Machines (darkmachines_anomaly). Four channels of background-only events, up to 7.7 million in a ragged text format. The submission is a model class with fit and predict rather than a prediction file: the grader instantiates a fresh model per channel, trains it on that channel’s background and applies it to hidden files holding every BSM signal. No signal is visible to the agent and the metric cannot be computed locally [1].

Astrophysics

Stokes inversion, simulated (stokes_inversion). Simulated Stokes spectra around one Fe I line; submit three quantile arrays over eleven Milne–Eddington parameters and your own Unno–Rachkovsky forward model [11, 21]. Azimuth is identifiable only modulo 180∘180^\circ, exact singular points make some parameters unobservable, and filling factor trades against field strength. The grader resynthesises the spectrum from the submitted parameters in an isolated container.

Stokes inversion, real (hinode_inversion). Real Hinode/SOT-SP rasters at a two-line pair; the same submission plus complete parameter maps [21] (del Toro Iniesta & Ruiz Cobo 2016). The reference is a Merlin inversion that is itself noisy, real instrument effects are present, and the source-function convention needs a gain the forward model must apply. The spatial term is scored on the parameter maps rather than on synthesised spectra, so smoothing the output cannot buy it.

SN Ia classification (lightcurve_classification). Irregularly sampled, heteroscedastic six-band light curves with fewer than five hundred labelled objects; submit a per-object probability [12]. The work lies in representing a sparse, gappy, multi-band series rather than in model capacity. The log-loss component requires calibration, so a good ranking with overconfident scores is penalised.

Neuroscience

ASD classification (abide_classification). Resting-state fMRI on a 200-region atlas from sixteen training sites; submit an autism probability and a chronological age for every subject of four entirely held-out sites [17, 24]. Scored on discrimination plus calibration against the training prevalence, with an auxiliary age term. The ceiling is set by cross-scanner transfer, not by method quality.

NMM family identification (nmm_forward_modelling). Forty-five anonymous three-node recordings; name the generating neural-mass family for each [30, 10]. Equations, constants and parameter ranges are withheld, only the recorded observable per family is disclosed, and several series sit in corners of one family’s parameter space that mimic another. The score is chance-corrected, so hedging earns nothing.

NMM digital twins (nmm_digital_twins). Spontaneous baseline activity of a 30-region delayed Wilson–Cowan system [41], its connectivity and delays, and a future intervention; submit the signed, time-resolved causal difference between activity with and without it. Matching resting means, spectra or connectivity does not determine the response [23], and the shared self-excitation gain and per-node drive parameters that set the operating regime are withheld; the model’s remaining coefficients, numerics and noise scale are published in a manifest.

NMM parameter recovery (nmm_parameter_recovery). z-scored BOLD per subject; submit global coupling and two receptor-weighted gain parameters [10, 16]. The generator is fully specified in the task description, so the whole difficulty is the inverse problem: inference over a stochastic delayed mean-field model seen through a slow, noisy, standardised channel. Standardisation removes amplitude, so only the regional pattern carries the parameters.

BOLD deconvolution (rs_bold_deconvolution). Two paired hemodynamic views of one neural realisation, with no stimulus timing [15, 42]; submit the shared latent neural drive, a double-gamma kernel bank and a Balloon–Windkessel parameter bank [14, 13]. The two kernel terms dominate the score and are gated, so a regionally constant template scores zero. Balloon–Windkessel parameters are credited only up to functional equivalence, through a frozen ODE on four probes.

Agent-Facing Prompt Surface

Two texts fully determine what an agent sees before it begins: a system prompt shared by all twelve tasks, and the task’s own description.md. Neither reveals the grader implementation, the anchors, or any ground truth. We reproduce the system prompt in full, followed by one complete task prompt. The remaining eleven are released with the benchmark.

System prompt

You are an autonomous scientific problem-solving agent.

Your primary objective is to maximize the scientific score using only the visible workspace data and the tools available in the sandbox. Structural validity and reproducibility are acceptance requirements, not substitutes for solving the scientific problem.

Before writing solution files, read the complete task brief and relevant public metadata. Identify the scored scientific target, the variation the metric rewards, and every stated numerical or modelling convention. Inspect the actual data, select a scientifically appropriate method, implement it, and test the scored outputs. A submission can pass structural validation and still score zero. Do not rely on a constant, group-midpoint, globally broadcast, or data-ignoring predictions when the metric rewards continuous, sample-specific, regional, or temporal variation. Do not search outside the workspace for existing solutions, hidden answers, grader code, or private data.

After inspecting the task and data, create a valid initial submission before starting long training, simulation, or optimization jobs. Keep the required files valid as you improve the scientific method so a budget limit cannot erase all completed work.

After that checkpoint is valid, use available budget for substantive scientific improvement. Diagnose the current method from public data, identify its largest unresolved uncertainty, and test a materially different method, modelling choice, or numerical setting. Keep an improvement only when public-data diagnostics support it. Do not spend budget on repeated summaries, unchanged tests, or speculative prose; do not repeat an experiment unless its hypothesis, implementation choice, or diagnostic changes.

Once the scientific method produces working scored outputs, create the smallest reproducibility package required by the task. Reuse the tested implementation; do not spend budget polishing documentation or package structure. Produce every scored output and required artifact at its exact relative path, preserving required subdirectories. The final run.py must read only visible task data and regenerate the scored outputs in the requested output directory, and README.md must give concise usage instructions.

Before finishing:

  1. Run the solution entry point and fix execution errors.

  2. Confirm that every required scored output and artifact exists at its exact path.

  3. Call validation_feedback(), fix every reported structural problem, and validate again.

  4. Call submit() only after validation succeeds. submit() independently rechecks the submission and rejects incomplete or malformed output.

If the full method cannot be completed within the budget, submit the strongest scientifically defensible working solution available. Do not replace the scientific method with an empty or deliberately invalid stub.

Example task prompt: stokes_inversion

We reproduce this task because it exercises four of the seven properties of Section 3 at once: an underdetermined solution set with exact structural non-identifiability (P1), a submission that includes the agent’s own forward model rather than predictions alone (P2), a composite geometrically gated metric (P5), and a failure mode in which a spectrum is reproduced while the physics is violated (P6).

# Simulated Stokes Inversion

Given a simulated Stokes spectrum on a fixed 56-point wavelength grid around the Fe I 6302.5 Å line, predict the 11 Milne--Eddington atmospheric parameters for each pixel independently, together with an 80% central prediction interval (the 10th / 50th / 90th percentiles).

## Scientific motivation

The Sun’s photospheric magnetic field governs every form of solar activity that matters at human timescales -- sunspots, the 11-year cycle, flares, coronal mass ejections, and the space-weather drivers that disrupt satellites and power grids. Its only direct remote diagnostic is **spectropolarimetry**: in the presence of a magnetic field, spectral lines split into polarized components through the Zeeman effect, and the four Stokes parameters $(I, Q, U, V)$ encode the full magnetic vector together with the thermodynamic and velocity state of the line-forming layer. Recovering atmospheric parameters from observed Stokes profiles -- the **inversion problem** -- is therefore the central numerical task of modern solar physics.

In the Milne--Eddington (ME) approximation, the line-forming atmosphere is summarised by 11 parameters: magnetic-field strength $B$, inclination $\theta$, azimuth $\chi$, Doppler width $\Delta_D$, damping $a$, line-strength ratio $\eta_0$, the source-function pair $(S_0, S_1)$, line-of-sight velocity $v_\mathrm{los}$, magnetic filling factor $\alpha$, and stray-light velocity $v_\mathrm{text}$. The forward map from these eleven scalars to the four Stokes profiles is the analytical Unno--Rachkovsky solution of the polarized radiative-transfer equation. Participants are expected to implement their own forward map from the published solar-physics literature (e.g. Landi Degl’Innocenti & Landolfi 2004).

This benchmark isolates this inverse problem in a **fully simulated, fully controlled** setting. The training and test data are produced from the validated ME forward model with channel-dependent Gaussian noise; generating parameters are exactly known to the organizer, and test targets remain private. The challenge is pixelwise: each spectrum is processed independently, with no spatial context. This isolates the per-pixel inversion competence as a clean probe of a method’s ability to invert a polarised radiative-transfer model.

Physical difficulties of ME inversion

Even with perfect training data, the inverse problem has structural difficulties that any method must address:

  1. Non-linearity. The Unno--Rachkovsky forward map composes Voigt profiles, Faraday--Voigt anomalous-dispersion profiles, and trigonometric functions of the field angles. Small parameter changes can produce discontinuous-looking changes in the spectrum (e.g. when an absorption profile transitions from un-split to split as BB crosses a Doppler-width-dependent threshold), and large parameter changes can produce visually identical spectra.

  1. The 180-degree azimuth ambiguity. The azimuth χ\chi enters Stokes QQ and UU only through sin⁡(2χ)\sin(2\chi) and cos⁡(2χ)\cos(2\chi). The pair χ\chi and χ+180∘\chi+ 180^\circ produce identical Stokes profiles, so there is no information in the spectrum that distinguishes them. The benchmark treats χ\chi as a circular variable with period 180^\circ everywhere in scoring (submission validation, pinball loss, interval coverage, interval width).

  1. Structural non-identifiability at singular points. A few exact generator-defined sentinels make specific parameters genuinely unobservable for a given pixel, as a matter of physics rather than noise:

  • α=0\alpha= 0 (no magnetic component): BB, θ\theta, χ\chi, vLOSv_\text{LOS} are unobservable.

  • B=0B = 0 (zero field): θ\theta, χ\chi are unobservable (no transverse or longitudinal direction).

  • α=1\alpha= 1 (no stray component): vstrayv_\text{stray} is unobservable. These exact-singularity samples are masked out of the score for the affected parameters only; near-singular cases remain scored and an honest method can protect itself through appropriately wide conditional intervals.

  1. Polarization signal-to-noise varies with BB, geometry, α\alpha, and noise. Weak-field, near-longitudinal, or low-α\alpha pixels have faint polarization and the recovered field direction is noisy --- but this is governed by the polarization SNR actually present in the spectrum, not by a fixed BB threshold. The dataset contains weak-field pixels with detectable polarization; do not assume a blanket cutoff.

  1. Filling-factor / field-strength degeneracy. A weak field filling the entire pixel (α=1\alpha= 1, small BB) can produce a circular-polarization amplitude very similar to a strong field filling a fraction of the pixel (α<1\alpha< 1, large BB). When α→1\alpha\to1, vstrayv_\text{stray} becomes mathematically undefined.

Dataset

The published dataset lives under dataset/:

FileShapedtypeRole
spectra_train.npy(100000, 56, 4)float32absolute-scale Stokes [I,Q,U,V][I, Q, U, V], with noise
params_train.npy(100000, 11)float64ground-truth training parameters
spectra_test.npy(10000, 56, 4)float32test spectra (the only test input you receive)
wavelength_grid.npy(56,)float64mÅ offsets from Fe I 6302.5 Å line center

params_test.npy (shape (10000, 11)) is the withheld ground truth used for scoring; it is not an input. Load the published arrays directly with numpy:

import numpy as np
spectra_train = np.load("spectra_train.npy")    # (100000, 56, 4)  float32
params_train = np.load("params_train.npy")      # (100000, 11)      float64
spectra_test = np.load("spectra_test.npy")      # (10000, 56, 4)   float32
wavelength = np.load("wavelength_grid.npy")     # (56,) mÅ offsets from Fe I 6302.5 Å

The harness stages these four arrays plus the public_manifest.json dataset manifest --- and nothing else --- into the agent workspace; the exact mount path is supplied by the STEMBench instruction wrapper. Ground-truth files marked role: ground_truth in config.yaml (params_test.npy, test_metadata.npz) are never mounted into the agent sandbox.

The 11 physical parameters (canonical order --- never re-order)

IdxSymbolNameUnitRange
0BMagnetic field strengthG[0,5000][0, 5000]
1thetaField inclination from line of sightdeg[0,180][0, 180]
2chiField azimuth in plane of sky (period 180°)deg[0,180][0, 180]
3Delta_DDoppler widthmÅ[20,90][20, 90]
4aDamping (Voigt parameter)---[0,1.5][0, 1.5]
5eta_0Line-strength ratio---[0.01,100][0.01, 100]
6S_0Source function at τ=0\tau= 0---[0,38,603][0, 38, 603]
7S_1Source function gradientintensity[0,60,464][0, 60, 464]
8v_LOSLOS velocity (magnetic component)km/s[−10,10][-10, 10]
| 9 | `alpha` | Magnetic filling factor | --- | $[0, 1]$ |
| 10 | `v_stray` | LOS velocity (field-free component) | km/s | $[-10, 10]$ |

### Train/test split composition

The split is **globally group-disjoint**: no exact 11-parameter vector and no source row is
shared between training, the organizer-only calibration set, and the test set. Both
train and test are **balanced across four magnetic-field regimes** of equal test weight:

| Regime | $B$ range (G) |
|---|---|
| weak | $[0, 200)$ |
| intermediate | $[200, 500)$ |
| moderate | $[500, 1500)$ |
| strong | $[1500, 5000]$ |

Each regime contributes an equal share of train and of test, so changes in sample composition
cannot silently reweight the scientific question. The four regimes are averaged equally
in the score.

The test set additionally contains a **stress** subset (20% of test, 2,000 rows). The stress
designation and per-row core/stress labels are **private** (organizer-only); only
aggregate core/stress results are reported after evaluation. The stress split preserves
the $B$-regime × filling-factor balance and the one-dimensional parameter marginals (
audited against the core split) but is a **stratified random diagnostic**, not an
interpolation probe: the audit documents that stress joint cells do recur in training
within each $B$ regime. The headline score evaluates probabilistic recovery ($R$) and
forward consistency ($F$) over the four equally weighted magnetic regimes; core vs
stress is reported as a stratified random split diagnostic, not a prediction about which
method classes should succeed or fail.

### Spectra, noise, and units

Spectra are stored in **absolute intensity units** (not continuum-normalised), so the
continuum level directly anchors the source-function parameters $S_0$ and $S_1$. Channel
-dependent additive Gaussian noise is already baked into the released spectra with per-
channel sigma

$$\sigma_{I,Q,U,V} = [109,\,2,\,2,\,44]\,/\,20\,000 \;\cdot\; (S_0 + \mu S_1), \qquad \mu =
1,$$

in continuum-intensity units. This is the benchmark noise model --- a frozen, reproducible
channel-dependent model; it is not an exact Hinode/SOT-SP calibration.

## Submission contract

A submission is a directory containing **three numpy arrays** of shape $(10000, 11)$ **plus a
forward-model entry point**:

- `<submission_dir>/pred_q10.npy` --- 10th-percentile prediction
- `<submission_dir>/pred_q50.npy` --- median (point prediction)
- `<submission_dir>/pred_q90.npy` --- 90th-percentile prediction
- `<submission_dir>/forward_model/entry_point.txt` --- text file with one line of the form `<module>:<attr>` (e.g. `me:model`), naming a callable inside `<submission_dir>/forward_model/`
- `<submission_dir>/forward_model/<module>.py` --- your implementation(s) of the callable

The grader **rejects** (scores 0, before any metric is computed) any submission with **structural** violations:

- A missing or extra-shaped array; an object, boolean, integer, NaN, or infinity value.
- Wrong test length or parameter count.

**Parameter-level violations are scored, not rejected** --- the pinball metric penalizes them
naturally, and the grader report lists them as warnings:

- Any non-circular parameter outside its physical bounds in any of the three quantile files.
- Reversed quantiles: `q10 > q50` or `q50 > q90` for any non-circular entry.
- For `chi`: `q50_chi` outside $[0, 180^\circ]$; `q10_chi` or `q50_chi` or `q50_chi` or `q90_chi`,
  either side extending more than $90^\circ$ from `q50_chi`.

`validation_feedback` reports these parameter issues as warnings so you can fix them before
submitting.

### Forward-model contract (for the $F$ term)

The forward-consistency term $F$ synthesizes a noiseless spectrum from your submitted `q50`
parameters using **your** forward model --- not an organizer-supplied one --- and
compares it to the observed noisy test spectrum. `forward_model/entry_point.txt` is a **
required submission file**; a submission missing it is rejected at validation. To
compute $F$, the grader runs your forward code inside an **isolated Docker container** (

non-root, read-only filesystem, no network, no access to private ground truth or the task root):

  1. Reads forward_model/entry_point.txt, parses <module>:<attr>.

  2. Imports <module> from your forward_model/ directory inside the container.

  3. Calls <attr>(params) where params is (N, 11) float64 --- your q50.

  4. Expects the return to be (N, 56, 4) float64 absolute-scale noiseless Stokes [I,Q,U,V][I,Q,U,V].

If the callable is present but fails to import, raises, or returns the wrong shape, F=0F = 0. RR (recovery) is unaffected --- you keep the credit for calibrated parameter quantiles even with a non-working forward. Line constants (λ0=6302.5\lambda_0 = 6302.5 Å, geff=2.5g_\text{eff} = 2.5, μ=1\mu= 1; the 56-point wavelength grid in mÅ spans 6302.069--6303.254 Å) must be embedded in your forward model code --- either hard-coded or imported from a constants module shipped inside your forward_model/ directory. At grade time the grader’s own grader/constants.py is available on the import path inside the synthesis container; the canonical values are LINE_VEC = (6302.5, 2.5, 1) and LINE_ARG = 1000 * (np.linspace(6302.0692255, 6303.2544205, 56) - 6302.5) (in mÅ).

Provide a physical synthesis model that preserves the meanings and units of the 11 inputs. The data were generated with a Milne--Eddington / Unno--Rachkovsky atmosphere, but your implementation may use a richer, stratified or multicomponent atmosphere. Supply any additional atmospheric quantities deterministically within your forward model and document their relationship to the 11 inputs. The grader does not compare your implementation with a prescribed solver.

Submitted forward models also pass physical-validity probes (the PphysP_\text{phys} factor of the score); their specification is in metric_description.md. They check the physical meanings of the declared interface --- not agreement with any particular ME solver.

Chi (circular) interval convention

χ\chi keeps the three-file contract but its endpoints live on a local unwrapped coordinate:

  • q50_chi is the circular center, in [0,180∘)[0, 180^\circ).

  • q10_chi \le q50_chi \le q90_chi, and each side may extend at most 90∘90^\circ from q50_chi.

  • q10_chi and q90_chi may fall outside [0,180∘)[0, 180^\circ) so an interval can cross the 0/180∘0/180^\circ boundary.

During scoring, the true χ\chi is lifted to the equivalent value nearest q50 before the pinball loss, interval coverage, and interval width are computed. This makes every χ\chi operation circular while preserving the three-file submission format. Predictions that differ only by a 180∘180^\circ shift in χ\chi score identically.

If you only produce a point estimate, set q10 = q50 = q90 (a zero-width interval). This is valid but earns an uncertainty penalty under the pinball metric --- calibrated intervals score better.

Allowed Tools and Libraries

The sandbox has no network access, so only the packages pre-installed in the sandbox image can be used. Packages relevant to this task:

  • numpy --- numerical arrays and core math for the spectral cubes.

  • scipy --- scientific routines (optimization, interpolation, statistics).

  • scikit-learn --- classical machine-learning models and preprocessing.

  • xgboost --- gradient-boosted decision trees.

  • catboost --- gradient boosting on tabular features.

  • torch --- neural network models.

  • numba --- JIT compilation for fast pixelwise numerical loops.

Scoring

A submission is scored on the test split as a single scalar in [0,1][0, 1], higher is better, perfect = 1:

FinalScore  =  0.80⋅R  +  0.20⋅R⋅Feff\text{FinalScore} \;=\; 0.80 \cdot R \;+\; 0.20 \cdot\sqrt{R \cdot F_\text{eff}}

where RR is a baseline-normalized, null-gated probabilistic recovery score and FF is a forward-consistency score. A constant train-marginal predictor scores at the floor (its raw skill is ~0 by construction and the null gate floors the residual); an oracle scores 1.

RR --- probabilistic recovery (primary)

For each magnetic-field regime rr and parameter kk, the grader computes the mean pinball loss across τ∈{0.1,0.5,0.9}\tau\in\{0.1, 0.5, 0.9\} over the observable sample--parameter pairs (exact singularity pairs are masked; χ\chi is handled circularly):

L_{r,k} = \frac{1}{3}\sum_{\tau\in\{0.1, 0.5, 0.9\}} \overline{\rho_\tau(y_{i,k}, \hat}
ρτ(y,q^)=max⁡ ⁣(τ(y−q^), (τ−1)(y−q^)).\rho_\tau(y, \hat q) = \max\!\bigl(\tau(y-\hat q),\, (\tau-1)(y-\hat q)\bigr).

Let L0,r,kL_{0,r,k} be the pinball loss of the constant train-marginal quantile predictor (the no-skill baseline) on the test split. The raw skill is baseline-normalized:

Sr,kraw=clip⁡ ⁣(1−Lr,k/L0,r,k, 0, 1),S^\text{raw}_{r,k} = \operatorname{clip}\!\left(1 - L_{r,k} / L_{0,r,k},\ 0,\ 1\right),

then null-gated and rescaled above a frozen null threshold tr,kt_{r,k} (the 95th percentile of raw skill across ≥1000\geq1000 null submissions: label permutations, shuffled good predictions, random bounded predictions):

Sr,k={0Sr,kraw≤tr,k(Sr,kraw−tr,k)/(1−tr,k)otherwise.S_{r,k} = \begin{cases} 0 & S^\text{raw}_{r,k} \leq t_{r,k} \\ (S^\text{raw}_{r,k} - t_{r,k})/(1 - t_{r,k}) & \text{otherwise.} \end{cases}

Within each regime, skills are grouped by physical role and weighted:

Smag=mean⁡(SB,Sθ,Sχ,Sα),Sthermo=mean⁡(SΔD,Sa,Sη0,SS0,SS1),Svel=mean⁡(SvLOS,Svstray),S_\text{mag} = \operatorname{mean}(S_B, S_\theta, S_\chi, S_\alpha), \quad S_\text{thermo} = \operatorname{mean}(S_{\Delta_D}, S_a, S_{\eta_0}, S_{S_0}, S_{S_1}), \quad S_\text{vel} = \operatorname{mean}(S_{v_\text{LOS}}, S_{v_\text{stray}}),
Rr=0.50 Smag+0.30 Sthermo+0.20 Svel,R=mean⁡(Rweak,Rintermediate,Rmoderate,Rstrong).R_r = 0.50\,S_\text{mag} + 0.30\,S_\text{thermo} + 0.20\,S_\text{vel}, \qquad R = \operatorname{mean}(R_\text{weak}, R_\text{intermediate}, R_\text{moderate}, R_\text{strong}).

The grader applies frozen monotone transforms internally before computing pinball loss --- log⁡1p\log1p on BB and η0\eta_0, identity on the rest --- so quantiles are equivariant and the score is invariant to test-composition changes. Pearson correlation is not a component of the score.

FF --- forward consistency (physical-equivalence gate)

For each test row, the grader synthesizes a noiseless spectrum from the submitted ‘q50‘ parameters using the participant’s submitted forward model (declared in ‘forward_model/entry_point.txt‘), and compares it to the observed noisy test spectrum with the per-sample channel noise:

Di=mean⁡λ,c Huber⁡ ⁣(xobs−xpred(q50)σ, δ ⁣= ⁣3).D_i = \operatorname{mean}_{\lambda, c}\ \operatorname{Huber}\!\left(\frac{x_\text{obs} - x_\text{pred}(q_{50})}{\sigma},\ \delta\!=\!3\right).

Per regime, Fr=clip⁡ ⁣((D0−Dr)/(D0−Doracle), 0, 1)F_r = \operatorname{clip}\!\left((D_0 - D_r)/(D_0 - D_\text{oracle}),\ 0,\ 1\right), where D0D_0 is the constant-baseline residual and DoracleD_\text{oracle} the hidden-truth residual; then F=mean⁡rFrF = \operatorname{mean}_r F_r. The task is to provide a genuine physical synthesis model, and forward consistency is scaled by physical validity Pphys∈[0,1]P_\text{phys} \in[0, 1] (Feff=F⋅PphysF_\text{eff} = F \cdot P_\text{phys}). FF rewards a physically equivalent inversion (good spectral fit) but the geometric gate R⋅Feff\sqrt{R \cdot F_\text{eff}} ensures a pure forward fit with arbitrary latent parameters cannot win on its own --- only helps when recovery is already non-trivial.

Reported diagnostics (not additional score terms)

The grader report includes, beyond ‘FinalScore‘: RR and FF; per-parameter pinball skill; the four SS-regime scores; core vs stress scores; q10q_{10}–q90q_{90} coverage and width; circular-χ\chi coverage and width; median MAE in physical units; polarization-SNR strata; exact-singularity counts and masks; forward residual by Stokes channel.

Evaluation

The STEMbench harness scores a submission automatically by calling ‘grade(submission_dir, answers_path)‘ from ‘grade.py‘. No scoring command is issued by the participant. To produce a scoreable submission, write exactly three files to your solution directory:

  • ‘pred_q10.npy‘ --- shape ‘(10000, 11)‘, float32/float64

  • ‘pred_q50.npy‘ --- shape ‘(10000, 11)‘, float32/float64

  • ‘pred_q90.npy‘ --- shape ‘(10000, 11)‘, float32/float64

Ground truth is never mounted into the agent sandbox; the harness reads it host-side.

Run Configuration

Table 4 specifies the per-episode configuration used for every run reported in Sections 5 and E, unless explicitly stated otherwise (the extended-limits ablation of Table 8 is the only exception). All fields are harness-level run policy, declared at launch and logged in every episode’s metadata; none of them is part of a task’s config.yaml; the wall-clock, token and message budgets are restated to the agent in its instructions.

FieldValueSemantics
sandboxdocker-cudaOffline per-domain container (network disabled, public data mounted read-only, writable solution directory).
time_limit_sec3600Wall-clock budget per episode. The agent loop is cancelled at the deadline and whatever submission is present at that moment is graded; a single bash/python execution may run for at most two thirds of the budget.
token_limit10610^{6}Cumulative model tokens (input plus output) across all model calls of the episode; the next model call is refused once the budget is spent.
max_messages200Total conversation messages (user, assistant and tool results) per episode; the episode ends when the limit is reached.
max_tool_calls200Declared tool-call budget, recorded in the run plan and episode metadata. It is not an independent stop condition: every tool call adds messages, so the message limit binds first.
effortlow / medium / highReasoning-effort tier forwarded to the model API as reasoning.effort through the OpenRouter gateway; the experimental variable of Section E, with high the harness default.

Table 4. Per-episode run configuration. “Value” is the setting used for all reported runs; “Semantics” states what the field controls and how the harness enforces it.

EffortMainExtended
low0.806 ± 0.006 [0.802, 0.813]0.802 ± 0.000 [0.801, 0.802]
medium0.86 ± 0.03 [0.829, 0.892]0.85 ± 0.05 [0.802, 0.901]
high0.84 ± 0.16 [0.652, 0.931]0.941 ± 0.004 [0.937, 0.946]

Table 8. Budget ablation on top_quark_tagging. “Main” is the primary grid (1 h, 10610^{6} tokens, 200 messages, 200 tool calls); “Extended” raises these to 2.5 h, 2×1062 \times10^{6} tokens and 500/500, executed through Inspect’s native OpenRouter provider. Mean ± standard deviation, with the observed range in brackets; n=3n = 3 per cell. The fourteen-run variance study of the Main/high cell is reported in Table 7.

All campaign episodes executed on single-GPU instances: one Tesla T4 (16 GB) on an Intel Ice Lake platform with 8 vCPU and 32 GB RAM, Ubuntu 24.04, with the per-domain CUDA container image holding the only software the agent can touch.

Model access uses an OpenAI-compatible protocol through the OpenRouter gateway; sampling parameters (temperature, top-pp) are not exposed on this route for reasoning models and were not set, which is why run-to-run variability is measured by repetition.

Harness and contract tests. The harness is covered by a functional test suite over the tool layer, which passes 17/17 checks: the six-tool registry, the eight file formats that data_explorer advertises, the refusal of unsafe deserialisation, valid and invalid submission checks (including split-keyed JSON and per-key .npz constraints), and the mapping from the submission checks in config.yaml to validator specifications. All twelve task contracts additionally pass dataset-checksum verification and smoke tests that grade the shipped baseline and expert submissions and reproduce the anchors in Table 3.

Per-Model Results

Table 5 reports eight frontier models under the basic scaffold at high reasoning effort. Scores are the task’s native headline metric in [0, 1] under the current task revisions, and each cell is the mean over three retained scored runs, including zeros.

Table 6 complements Table 5 with the maximum of the same three retained runs per cell. The two views separate ability from reliability: on the HEP tasks six of the eight models reach a best-run mean of 0.63–0.71, so the spread of the means in Table 5 is driven largely by how often a model fails, not by whether it can solve the task at all. The individual-run records quoted in § 5 appear here as cell maxima: ABIDE at 0.31 (0.314 unrounded, DeepSeek V4 Pro) and parameter recovery at 0.58 (0.583, Opus 5).

TaskbbeeGemini 3.7 FlashGemini 3.1 ProOpus 5DeepSeek V4 ProQwen3.8 2.4TGLM 5.2MiMo V2.5MiniMax M3
New-physics search0.330.910.500.940.930.940.820.940.000.94
Higgs uncertainty0.400.690.410.490.530.470.470.440.410.38
Top tagging0.060.880.800.950.950.930.940.930.910.87
Dark Machines0.000.220.330.330.380.480.350.340.320.32
HEP mean——0.510.680.700.710.640.660.410.63
Stokes, simulated0.010.620.010.800.850.740.010.000.740.32
Stokes, real0.000.600.180.860.150.000.000.000.410.00
SN Ia classification0.420.850.500.930.950.930.940.950.940.93
Astro mean——0.230.870.650.560.310.320.700.42
ASD classification0.000.310.280.000.000.310.310.320.240.24
NMM family id.0.000.900.001.000.000.870.870.000.870.07
NMM digital twins†0.100.720.100.260.560.230.100.100.000.10
NMM param. recovery0.120.510.120.300.580.120.120.120.070.12
BOLD deconvolution0.010.350.000.270.000.000.070.000.000.18
Neuro mean——0.100.370.230.310.300.110.240.14

Table 6. STEMBENCH results, basic agent, maximum raw task scores. Each cell is the maximum over the same three retained high-effort runs as in Table 5, including zeros. Domain rows are unweighted means of the task maxima, computed before rounding. Bold denotes the highest unrounded maximum in each row. b/eb/e are the baseline and expert anchors. †The digital-twin expert anchor is e=0.715233e = 0.715233 (displayed as 0.72).

Pilot Reasoning-Effort Sweep

This appendix reports a pilot sweep run before the cross-model campaign of § 5. Its absolute scores are not comparable with that campaign for two reasons: several task contracts were revised afterwards, and the runs predate a fix to GPU access in the task containers. We report it for what it shows about run-to-run variance and the non-monotone effect of reasoning effort within one model, not as a measurement of the current benchmark. After the pilot, all campaign runs were fixed at high effort.

We evaluate google/gemini-3.1-pro-preview with the basic scaffold and the limits of Appendix C, varying only the reasoning effort (low, medium, high): 12 tasks × 3 levels × 3 runs, with the top_quark_tagging high-effort cell extended by 11 runs to estimate single-run variance, for 119 graded runs. Runs interrupted by infrastructure (proxy) outages were flagged automatically and excluded; zeros in retained runs are genuine episode outcomes.

Table 7 aggregates the grid. Three patterns are visible. First, reasoning effort buys most of its improvement between low and medium (mean of task means 0.272 →\to 0.484) and nothing on average beyond it (0.484 at high), although the median continues to rise (0.190 →\to 0.333 →\to 0.499), so high effort helps the typical task while losing ground on a few. Second, the dispersion is not noise around a single mode: several tasks are bimodal, with runs landing either near the agent’s best score or exactly at zero, which is what drives the large standard deviations on lhco_anomaly (±0.53\pm0.53 at both medium and high) and stokes_inversion (±0.44\pm0.44 at high). Four of nine runs on each of lhco_anomaly, hinode_inversion and abide_classification produced nothing that cleared validation or the no-skill baseline. Third, the effort–score relation is not monotone per task: abide_classification scores 0.327±\pm0.000 at medium and 0.000 in all three high-effort runs, and rs_bold_deconvolution degrades from 0.276 at medium to 0.055 at high. These aggregate scores do not establish whether the declines arise from invalid submissions, valid low-scoring predictions, or truncation; distinguishing these explanations requires run traces and validity flags.

Table 8 compares the main and extended-budget runs. Under extended budgets all three high-effort runs on top_quark_tagging land in [0.937, 0.946] (mean 0.941 ±\pm 0.004), above the main-budget mean, and the lower mode visible in the fourteen-run variance study of Table 7 (runs at 0.80 and 0.063) does not appear. At low effort the extended limits change nothing (0.802 ±\pm 0.000), since low-effort agents stop early on their own. This is consistent with truncation contributing to high-effort variance, but the extended series also switched to Inspect’s native OpenRouter provider, and the missing GPU acceleration may itself have caused part of the truncation; the ablation therefore does not isolate the effect of the budget.

References

  1. [1]Thea Aarrestad, Melissa van Beekveld, Marcella Bona, Antonio Boveia, Sascha Caron, Joe Davies, Andrea De Simone, Caterina Doglioni, Javier M Duarte, Amir Farbin, et al. The dark machines anomaly score challenge: benchmark data and model independent event classification for the large hadron collider. arXiv preprint arXiv:2105.14027, 2021.DOI
  2. [3]Anonymous. Stembench: an inspect-ai benchmark harness for research-grade stem agent tasks, September 2026a. URL https://doi.org/10.5281/zenodo.22959834.
  3. [5]Wahid Bhimji, Ragansu Chakkappai, Po-Wen Chang, Yuan-Tang Chou, Sascha Diefenbacher, Jordan Dudley, Ibrahim Elsharkawy, Steven Farrell, Aishik Ghosh, Cristina Giordano, et al. Fair universe hggsml uncertainty dataset and competition. Advances in Neural Information Processing Systems, 38, 2026.arxiv.org/abs/2410.02867
  4. [6]Bartosz Bieganowski, Daniel Strzelecki, Robert Skiba, and Mateusz Topolewski. Putnam-like dataset summary: Llms as mathematical competition contestants. arXiv preprint arXiv:2509.24827, 2025.arxiv.org/abs/2509.24827
  5. [7]Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. In International Conference on Learning Representations, volume 2025, pp. 50466–50494, 2025.arxiv.org/abs/2410.07095
  6. [8]Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, and Bryan Hoo. MLR-Bench: Evaluating AI agents on open-ended machine learning research, 2025a. URL https://arxiv.org/abs/2505.19955.
  7. [9]Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. In International Conference on Learning Representations, volume 2025, pp. 96934–96990, 2025b.arxiv.org/abs/2410.05080
  8. [10]Gustavo Deco, Adrián Ponce-Alvarez, Patric Hagmann, Gian Luca Romani, Dante Mantini, and Maurizio Corbetta. How local excitation–inhibition ratio impacts the whole brain dynamics. Journal of Neuroscience, 34(23):7886–7898, 2014. doi: 10.1523/JNEUROSCI.5068-13.2014.DOI
  9. [11]Jose Carlos del Toro Iniesta and Basilio Ruiz Cobo. Inversion of the radiative transfer equation for polarized light. Living Reviews in Solar Physics, 13:4, 2016. doi: 10.1007/s41116-016-0005-2.
  10. [12]Mariia Demianenko, Konstantin Malanchev, Ekaterina Samorodova, Mikhail Sysak, Aleksandr Shiriaev, Denis Derkach, and Mikhail Hushchyn. Understanding of the properties of neural network approaches for transient light curve approximations. Astronomy & Astrophysics, 677:A16, 2023.
  11. [13]Karl J. Friston, Andrea Mechelli, Robert Turner, and Cathy J. Price. Nonlinear responses in fMRI: The Balloon model, Volterra kernels, and other hemodynamics. NeuroImage, 12(4):466–477, 2000. doi: 10.1006/nimg.2000.0630.DOI
  12. [14]Gary H. Glover. Deconvolution of impulse response in event-related BOLD fMRI. NeuroImage, 9(5):416–429, 1999. doi: 10.1006/nimg.1998.0419.
  13. [15]Daniel A. Handwerker, John M. Ollinger, and Mark D’Esposito. Variation of BOLD hemodynamic responses across subjects and brain regions and their effects on statistical analyses. NeuroImage, 21(3):1639–1651, 2004. doi: 10.1016/j.neuroimage.2003.11.031.pubmed.ncbi.nlm.nih.gov/15050587
  14. [16]Justine Y. Hansen, Golia Shafei, Ross D. Markello, et al. Mapping neurotransmitter systems to the structural and functional organization of the human neocortex. Nature Neuroscience, 25: 1569–1581, 2022. doi: 10.1038/s41593-022-01186-3.DOI
  15. [17]Xuan Kan, Wei Dai, Hejie Cui, Zilong Zhang, Ying Guo, and Carl Yang. Brain network transformer. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2210.06681
  16. [18]Gregor Kasieczka, Benjamin Nachman, David Shih, Oz Amram, Anders Andreassen, Kees Benkendorfer, Blaz Bortolato, Gustaaf Brooijmans, Florencia Canelli, Jack H Collins, et al. The lhc olympics 2020 a community challenge for anomaly detection in high energy physics. Reports on progress in physics, 84(12):124201, 2021.
  17. [19]Amelia Liu, Andrew Ho, Anne Marie Droste, David Martin, Edmund Wong, Edward Zhou, Isabelle Zhou, Joshua Park, Joy Jiao, Katie-Rose Skelly, Kenny Kim, Kevin Rao, Masatoshi Uehara, Max Marion, Nicole Fitzgerald, Rachel Dias, Suyash Shringar­pure, Yuan Yuan, and Yunyun Wang. LifeSciBench: Evaluating language models on realistic, expert-level tasks in the life sciences. Technical report, OpenAI and Tacit Labs, 2026. URL https://cdn.openai.com/pdf/b4299379-0a97-4ffa-8b9b-c3fbb299caa9/lifescibench_preprint.pdf.DOI
  18. [20]Henry E. Miller, Matthew Greenig, Benjamin Tenmann, and Bo Wang. BioML-bench: Evaluation of AI agents for end-to-end biomedical ML. bioRxiv, 2025. doi: 10.1101/2025.09.01.673319. URL https://doi.org/10.1101/2025.09.01.673319.
  19. [21]Lukia Mistryukova, Andrey Plotnikov, Aleksandr Khizhik, Irina Knyazeva, Mikhail Hushchyn, and Denis Derkach. Stokes inversion techniques with neural networks: Analysis of uncertainty in parameter estimation. Solar Physics, 298:98, 2023. doi: 10.1007/s11207-023-02189-4.DOI
  20. [22]Ludovico Mitchener et al. Kosmos: An AI scientist for autonomous discovery. arXiv preprint arXiv:2511.02824, 2025.arxiv.org/abs/2511.02824
  21. [23]Lia Papadopoulos, Christopher W. Lynn, Demian Battaglia, and Danielle S. Bassett. Relations between large-scale brain connectivity and effects of regional stimulation depend on collective dynamical state. PLoS Computational Biology, 16(9):e1008144, 2020. doi: 10.1371/journal.pcbi.1008144.DOI
  22. [24]Pavel Popov, Usman Mahmood, Zening Fu, Carl Yang, Vince Calhoun, and Sergey Plis. A simple but tough-to-beat baseline for fMRI time-series classification. NeuroImage, 303:120909, 2024. doi: 10.1016/j.neuroimage.2024.120909.DOI
  23. [25]Jiahao Qiu et al. Physics supernova: AI agent matches elite gold medalists at IPhO 2025. arXiv preprint arXiv:2509.01659, 2025a.arxiv.org/abs/2509.01659
  24. [26]Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, et al. Phybench: Holistic evaluation of physical perception and reasoning in large language models. arXiv preprint arXiv:2504.16074, 2025b.arxiv.org/abs/2504.16074
  25. [27]Shi Qiu, Junyi Deng, Yiwei Deng, Haoran Dong, et al. PRBench: End-to-end paper reproduction in physics research, 2026. URL https://arxiv.org/abs/2603.27646.
  26. [28]Huilin Qu, Congqiao Li, and Sitian Qian. Particle transformer for jet tagging. In International Conference on Machine Learning, pp. 18281–18292. PMLR, 2022.arxiv.org/abs/2202.03772
  27. [29]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023.arxiv.org/abs/2311.12022
  28. [30]Paula Sanz-Leon, Stuart A. Knock, Andreas Spiegler, and Viktor K. Jirsa. Mathematical framework for large-scale brain network modeling in The Virtual Brain. NeuroImage, 111:385–430, 2015. doi: 10.1016/j.neuroimage.2015.06.016.
  29. [31]Jianhao Shen, Ye Yuan, Srbuhı Mirzoyan, Ming Zhang, and Chenguang Wang. Measuring vision-language stem skills of neural models. arXiv preprint arXiv:2402.17205, 2024.arxiv.org/abs/2402.17205
  30. [32]Zachary S Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. arXiv preprint arXiv:2409.11363, 2024.arxiv.org/abs/2409.11363
  31. [33]Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating AI’s ability to replicate AI research. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=xF5PuTLPbn.arxiv.org/abs/2504.01848
  32. [34]Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, et al. Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows. In International Conference on Learning Representations, volume 2026, pp. 75694–75731, 2026.arxiv.org/abs/2505.19897
  33. [35]Terminal-Bench-Science Team. Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains, 2026. URL https://github.com/harbor-framework/terminal-bench-science.
  34. [36]Minyang Tian, Luyu Gao, Shizhuo D Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. Scicode: A research coding benchmark curated by scientists. Advances in Neural Information Processing Systems, 37:30624–30650, 2024.arxiv.org/abs/2407.13168
  35. [37]Ekaterina Trofimova, Zosya Shamina, Maria Selifanova, Artem Zaitsev, Remi Savchuk, Maxim Minets, Daria Ozerova, Emil Sataev, Denis Zuenko, and Andrey Ustyuzhanin. Ml2b: Benchmarking llms on cross-lingual ml pipeline generation. 2026.arxiv.org/abs/2509.22768
  36. [38]Yuru Wang, Lejun Cheng, Yuxin Zuo, Sihang Zeng, Bingxiang He, Che Jiang, Junlin Yang, Yuchong Wang, Kaikai Zhao, Weifeng Huang, et al. Naturebench: Can coding agents match the published sota of nature-family papers? arXiv preprint arXiv:2606.24530, 2026.arxiv.org/abs/2606.24530
  37. [39]Kyle Waters, Lucas Nuzzi, Tadhg Looram, Alessandro Tomasiello, Ariel Ghislain Kemogne Kamdoum, Bikun Li, Damien Sileo, Egor Kretov, Francesco Fournier-Facio, Georgios Soloupis, Haile Kassahun, Hew Wolff, Jiaqi Cai, Lianghui Li, Marc Roth, Mohinder Naiya, Naixu Guo, Qicheng Tang, Richard Wheeler, Samuele Sala, Serguei Popov, Steven Dillmann, and Yuqi Li. COMPOSITE-Stem, 2026. URL https://arxiv.org/abs/2604.09836.
  38. [40]Jiaqi Wei et al. From AI for science to agentic science: A survey on autonomous scientific discovery. arXiv preprint arXiv:2508.14111, 2025.arxiv.org/abs/2508.14111
  39. [41]Hugh R. Wilson and Jack D. Cowan. Excitatory and inhibitory interactions in localized populations of model neurons. Biophysical Journal, 12(1):1–24, 1972. doi: 10.1016/S0006-3495(72)86068-5.
  40. [42]Guo-Rong Wu, Wei Liao, Sebastiano Stramaglia, Ju-Rong Ding, Huafu Chen, and Daniele Marinazzo. A blind deconvolution approach to recover effective connectivity brain networks from resting state fMRI data. Medical Image Analysis, 17(3):365–374, 2013. doi: 10.1016/j.media.2013.01.003.
  41. [43]Yunze Wu, Dayuan Fu, Weiye Si, Zhen Huang, Mohan Jiang, Keyu Li, Shijie Xia, Jie Sun, Tianze Xu, Yang Xiao, et al. Innovatorbench: evaluating agents’ ability to conduct innovative ai research. In International Conference on Learning Representations, volume 2026, pp. 36144–36185, 2026.
  42. [44]Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, and Yang Wang. Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models. arXiv preprint arXiv:2502.00334, 2025.
  43. [45]Mohd Zaki, NM Krishnan, et al. Mascqa: A question answering dataset for investigating materials science knowledge of large language models. arXiv preprint arXiv:2308.09115, 2023.

Paper details

vlnc:W_2v83tte6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents