A semi-autonomous multi-agent pipeline for high-energy astrophysics research with enforced provenance and verifiable self-extension

Introduction

Scientists increasingly use large language models (LLMs) in their research, for example to search the literature and to write analysis code. An LLM agent is an LLM that calls tools, such as a code interpreter, and acts on their output. Agent systems can retrieve the literature, write and run analysis code, and draft the text that reports the results. Some assist a scientist at each step, while others take a research question and return a manuscript. Such systems have been demonstrated in machine-learning research [1], in collider physics [2, 3], and for data-driven discovery across fields [4, 5]. Tests of LLM agents, however, have revealed three recurring failure modes.

First, agents fabricate results. Yang et al. [6] gave seven models tasks with empty or insufficient data and forbade them to report a failure. The models produced synthetic data in 57.1% of runs and did not disclose it in 20.6%. When the agents could report a failure, undisclosed synthetic data fell to 3.2% of runs, but synthetic data still appeared in 55.6%. Benchmarks in collider physics [7] and gravitational-wave modelling [8] found similar fabrications. In Lu et al. [1], the agent occasionally skipped required experiments and reported invented results. Second, agents game the checks that are meant to constrain them. Zhong et al. [9] gave models software-engineering tasks whose unit tests contradicted the task description, so that a correct solution failed the tests. The strongest model they tested still passed the tests in 54.0% of these tasks, for example by modifying the tests. Zhang et al. [10] scored a self-modifying coding agent on how rarely its model faked a tool call in its text. The agent’s code logged a marker at each real tool call. When the model faked a call, it copied the marker into its text, and a script counted these copies. The agent removed the logging. It then received a perfect score but still faked tool calls. Third, agents do not act on their own reviews. In 800 agent trajectories on 100 tasks from published papers, the most frequent failure, in 82.5% of the analyses, was that the agent identified a flaw during its self-review and nevertheless reported the unrevised conclusion [11].

Because of these failure modes, a scientist needs to verify the results of an agent. Most systems write new code for each task rather than add it to a reviewed, reusable library [4, 12, 1, 13]. Thus the scientist has to check new code for each result. Even when a system links each statement to its source, the scientist still has to check the statement. The Kosmos system [5] links each statement in its reports to the analysis notebook or the paper it came from, yet independent scientists judged only 79.4% of 102 such statements to be accurate.

In astronomy, agent systems have been applied to cosmological data [12], to galaxy images and cosmological simulations [13], and to gamma-ray data analysis [14]. This project grew as a separate branch from the participation of the author in a group working on the development of an agentic AI pipeline for the analysis of the data from the Imaging X-ray Polarimetry Explorer (IXPE) satellite [15]. That pipeline is described in [16], called Paper I hereafter. The author is also a co-author of Paper I and developed a complementary analysis pipeline, called auto_research in the following. The auto_research pipeline differs in several design principles from the one of Paper I: (i) auto_research takes project analysis ideas, but does not develop its own ideas [17]; (ii) the pipeline focuses on the development of a toolkit. A new tool is added after automatic checks and an LLM code review. A scientist then decides whether to promote the tool to the trusted branch. The pipeline uses an architecture in which each tool returns values through a well-defined interface through pre-defined return values. Only those values are used as results. (iii) The automatic checks include quantitative tests of new analysis tools. The pipeline of Paper I reviews the code with a critic, but runs no quantitative tests. (iv) The auto_research pipeline writes reports about the results.

Many safeguards act on the reasoning of the agents, for example instructions in the prompt, debate between agents, and critique by a reviewing agent [1, 16]. The third failure mode shows their limit, because an agent can identify a flaw and still report the unrevised conclusion. We therefore check what the agents produce, at two points: when the pipeline adds a new tool, and when an agent reports a number. For new tools, earlier tool libraries let the model that wrote a tool grade it [18], or relied on tests and human review [19]. Checks on the results of a task can miss a defective tool: in one benchmark, 215 of 222 tools written by an agent gave wrong results on every held-out input [20]. Dong and Chen [21] applied scripted checks followed by an audit by a model from a different provider. auto_research uses the same combination. Each new tool passes automatic checks (the “gate”) and then a review by a model from a provider other than its author’s. A human then decides whether to promote the tool to the trusted branch of the toolkit.

At the second point, we enforce provenance. The provenance of an output is the record of the tool and the data that produced it. Provenance enforcement means that a result (a number, a figure, a table) passes from one agent to another, or to the human, only if its provenance links it to a toolkit tool, the research request, or a location in a cited paper. A toolkit tool is either written by a human or committed after the checks on new tools. In auto_research, each agent ends its task with a report, and a validator admits a number in the report only if it appears in a recorded output of a toolkit tool or in the request, or if the report gives its location in a cited paper. Earlier systems link each number to a labelled output of the analysis code [22] or to logged tool calls [23], or require that the named run has finished [24], but they do so for a fixed set of tools or for code written for a single task. Because new tools enter the toolkit only through the checks, auto_research can extend its toolset and still enforce provenance.

Section 2 describes auto_research, which retrieves and analyses raw data from the High Energy Astrophysics Science Archive Research Center (HEASARC) and writes the analysis tools that it needs. Section 3 describes its checks on reported numbers and on new tools. Section 4 applies it to IXPE data, including a reproduction of the Cyg X-1 results of Krawczynski et al. [25]. Section 5 measures how well the checks perform, and Sect. 6 tests whether the pipeline still works with weaker models in the main roles. Section 7 discusses the results and their limits, and Sect. 8 summarizes them.

The pipeline

Design principles

Verify rather than trust. We require that the results of code written by an agent can be verified in three ways. First, the code is tested. For a new research question the correct answer is usually not known. The brief that the lead writes for the coder (Sect. 2.2) therefore asks for tests of properties of the results. These tests check that the results are deterministic and do not change when the input is split or combined. They also check that the results agree with those of trusted routines and fall within stated sanity bounds. The brief can also ask for a test that recovers a known signal from synthetic data. An analyst worker (Sect. 2.2) generates these data beforehand with existing tools and gives them an observation identifier that marks them as synthetic. The co-lead (Sect. 2.2) judges whether the tests are adequate. Second, the code runs on real data and writes real output files, so that a human can reproduce its results. Third, every output carries its provenance (Sect. 1). The provenance of any output also names the earlier outputs that it was computed from, so that a human can check that a tool received the right input.

Two repositories. The code is split into two repositories. The repository auto_research holds the orchestration code (agent graph, prompts, message schemas, sandbox, report generation). The running pipeline has no tool that writes to it, and an operating-system sandbox keeps the agents out of it (Sect. 2.2). The repository research_toolkit holds the analysis tools. The running pipeline can write to it only through the checked process of Sect. 3.2, which commits each new tool to a working branch. The running pipeline has no access to the trusted branch. Each tool is committed as one atomic commit, so that promoting or reverting a tool is a single operation.

Two halves per tool. Each tool consists of two halves. The lib/ half holds the analysis code, which a human can import and run. The skills/ half holds a manifest and short wrappers that tell the agents how to call the lib/ functions. Choices that are easy to get wrong are hard-coded in the lib/ functions, so that neither a human nor an agent has to make them. For example, the IXPE reduction selects the response files of the epoch closest to the observation date.

Reporting a failure is an acceptable outcome. The agent retries an error that it can fix, within a fixed budget. When the agent cannot complete the task, for example because the task is ill-posed or the data are missing, it reports this to the human, or it ends the run in a FAILED or NEEDS-HUMAN state. We call this a declared failure. Following Yang et al. [6] (Sect. 1), the prompts tell the agents that a declared failure is not penalized. A tool-writing session likewise ends with a commit that passed all checks or with a declared failure. Its code enters an analysis only as a committed tool.

Agent team and infrastructure

auto_research is built on LangGraph [26], and the agents exchange typed messages defined with Pydantic [27]. The team has six roles, which follow the flow of an analysis (Figure 1). The reviewer, a model that can read figures, receives the research request and dispatches literature and analysis tasks. Librarian agents query arXiv, NASA ADS, and the web. The analyst workers (two by default) use models from different providers. Each analyses the data, and then they debate their results. An analyst lead adjudicates between the workers and decides when the team needs a new tool. A coding agent (the coder) then writes the tool from a brief by the lead. A code reviewer (the co-lead) reviews the code of each new tool; it uses a model from a different provider than the coder. The results return to the reviewer, which evaluates them against the retrieved literature and either accepts them or requests rework. In addition, a status monitor summarizes the progress of a run for the human and takes no part in the analysis. The assignment of models to roles is set in a configuration file (Appendix § 18).

The agents work in restricted environments. Each analyst worker runs its tool calls in a separate, restricted Python process. The science software (HEASoft, XSPEC [28], ixpeobssim [29]) is available to these processes, to the shell commands of the agents, and to the tests run by the gate. A file-system allow-list, enforced by an operating-system sandbox, excludes the orchestration repository, which contains the code of the checks. The agents therefore cannot change the gate or the review.

In keeping with the ethics of scientific research, a human remains responsible for the analysis. The agents also often need expert input. Through a messaging connector, the agents ask the human questions, and the human answers them and can steer a running analysis.

The report system and provenance enforcement

Provenance of reported results

The reviewer, the librarians and the analysts can end a task only by submitting a report through the tool turn_end. Before the task ends, the validator checks that the report fills its required fields, that each cited figure file exists, and that each cited output matches its recorded hash. The validator admits a number only if one of the following holds:

• it appears in the cited output of a toolkit tool;

• it appears in the cited request;

• the report gives its location in a cited paper.²

The following do not support a number:

• shell output, and files and web pages that an agent reads;

• the output of a failed call or of a run on synthetic data, and any output computed from such an output.

A rejected report returns to the agent with the reasons.

Checks on new tools

Each new tool passes the gate and the review by the co-lead (Figure 2). The gate applies seven checks:

1. Determinism: a science tool comes with a test that runs it twice on real data and checks that the two outputs agree within a tolerance. The gate runs each of the tool’s tests twice and requires the same pass or fail outcome both times.

2. Tool-local tests: the changed test files pass, with at least one passing test.

3. Banned vocabulary: the gate scans the comments, docstrings and strings of the changed code, and changed text files, for 35 phrases that typically accompany invented results (listed in Appendix § 18).

4. Data-fabricating code: the gate rejects placeholder function bodies and binary data added to the toolkit, and passes random-number generators and hard-coded arrays to the co-lead as findings.

5. Parseability: the changed Python files parse.

6. Structure: the tool carries a description for the agents, the changes include a test file, and the changes lie on allowed paths.

7. Provenance: when the tests call a new or changed tool, the figures it returns exist, the pipeline records the files it writes, and the numbers it returns appear in the output that the pipeline records.

The gate checks mechanical properties. The co-lead then judges whether the code follows the brief and whether the tests are adequate, and it stops most defects that pass the gate (Sect. 5.2). The co-lead receives the brief, the code changes and the gate report, and it can read the working copy of the toolkit. The co-lead comes from a different provider than the coder, because LLM evaluators favour their own output [30] and a reviewer from the coder’s model family may share the coder’s blind spots, as conjectured by Dong and Chen [21]. The co-lead keeps no memory between submissions, because a claim of prior approval steers review agents towards approving vulnerable code [31]. After a rejection, the lead usually has the coder fix the code and resubmit, within a budget of five submissions by default. If the lead considers a rejection unfounded, it can dispute it. The co-lead then reconsiders the same submission, with its verdict and the lead’s reply as input. The pipeline commits the code only if both the gate and the co-lead approve it, with one atomic commit per tool. After a failed attempt, it resets the working branch to its base revision.

Example investigations

Analyses and their outcomes

We ran the pipeline on four research requests. Each analysis wrote new tools or extended existing ones before it produced results. The pipeline recorded each run in an event log. We quote agent output verbatim from these logs. Table 1 lists the analyses roughly by the strength of their external check: first the reproduction of a published result, then two analyses checked against the literature, then one with no external check. We use the first analysis to test the pipeline against a published result. The other three show how the pipeline writes tools under review, reports a null result and states its limits. Appendix § 16 records the results of the first three analyses and Appendix § 17 those of the fourth.

Analysis	Data	Tools	Cost (US$) | Time | External check
Cyg X-1 polarization reproduction (Sect. 4.2)	2 ObsIDs	7	55	7.0 h	published values [25]
Polarization stability catalogue (Sect. 4.3)	6 ObsIDs	3	35	2.4 h	internal, literature
Polarization–hardness survey (Sect. 4.4)	29 ObsIDs	4	45	—	Paper I, literature
$q$–$u$ time-lag search (Sect. 4.5)	45 ObsIDs	3	377	12.0 h	none

Reproduction of a published result

We first test the pipeline on a known result. We asked the agents to reproduce the IXPE polarization degree (PD) and polarization angle (PA) of Cyg X-1 in the hard state reported by Krawczynski et al. [25]. The request told the agents to use the published values as a benchmark, not to tune the analysis toward them, and to report any discrepancy above $3\sigma$. The pipeline wrote six tools and extended the data loader to several observations. The new tools are a polarization-cube tool, a plotter of the energy-resolved polarization cubes, two spectropolarimetric fits of the Stokes $I$, $Q$, $U$ spectra with XSPEC, a tool that compares results with a benchmark, and a plotter of that comparison. The pipeline performed a background-subtracted polarization-cube analysis with event weights (the NEFF weighting of ixpeobssim) and a cross-check with XSPEC, and compared 14 quantities with the published values (Table 2).

Quantity	Unit	Pipeline	Krawczynski et al. [25]	$z$
May PD (2–8 keV, PCUBE)	%	$3.37 \pm0.20$	$4.01 \pm0.20$	$-2.29$
May PA (2–8 keV, PCUBE)	deg	$-20.5 \pm1.7$	$-20.7 \pm1.4$	$+0.11$
June PD (2–8 keV, PCUBE)	%	$3.73 \pm0.30$	$3.84 \pm0.31$	$-0.25$
June PA (2–8 keV, PCUBE)	deg	$-24.9 \pm2.3$	$-25.7 \pm2.3$	$+0.24$
Combined PD (2–8 keV)	%	$3.46 \pm0.17$	$3.95 \pm0.17$	$-2.09$
Combined PA (2–8 keV)	deg	$-21.7 \pm1.4$	$-22.2 \pm1.2$	$+0.26$
May PD (XSPEC)	%	$3.36 \pm0.14$	$3.9 \pm0.2$	$-2.21$
May PA (XSPEC)	deg	$-20.3 \pm1.2$	$-21 \pm1$	$+0.45$
PD($E$) intercept $A$ (1 keV)	%	$1.72 \pm0.34$	$2.9 \pm0.4$	$-2.24$
PD($E$) slope $B$	%	$0.53 \pm0.11$	$0.58 \pm0.15$	$-0.24$
May PD (2–3 keV)	%	$2.58 \pm0.25$	$3.5 \pm0.3$	$-2.36$
May PD (3–4 keV)	%	$2.87 \pm0.24$	$3.5 \pm0.3$	$-1.63$
May PD (4–6 keV)	%	$3.76 \pm0.28$	$4.7 \pm0.3$	$-2.31$
May PD (6–8 keV)	%	$4.99 \pm0.64$	$5.8 \pm0.8$	$-0.80$

All 14 quantities agree within $3\sigma$. The PAs agree to better than $0.5\sigma$ and the slope of PD versus energy to $0.24\sigma$, so the pipeline reproduces the geometrical conclusions of the original paper. However, all 10 PD quantities, including the slope, lie below the published values, by $0.2$–$2.4\sigma$, for both analysis methods and in all energy bands (Table 2). The pipeline reported the offset as systematic but did not trace its cause. Paper I traced it to the event weighting.

Polarization stability within an observation

The second analysis tests whether the polarization of a black hole X-ray binary (BHXRB) stays constant during a long IXPE observation (see the similar analysis in Paper I). The pipeline split six observations of four sources (Cyg X-1, Swift J1727.8−1613, 4U 1630−47, LMC X-1) into 5 ks segments (effective exposures of 1.1 to 4.9 ks). For each segment it derived a weighted 2–8 keV polarization cube from the combined data of the three detector units, which gives the PD, the PA and the normalized Stokes parameters $q = Q/I$ and $u = U/I$. The

pipeline did not subtract the background. It noted that this is acceptable for these bright sources but should be revisited for the faintest one, LMC X-1. It then tested with a $\chi^2$ test the null hypothesis that all $N$ segments of an observation have the same $(q,u)$.

For five of the six observations, $(q,u)$ is consistent with being constant (Figure 3), with $p$ between 0.75 and 0.98. The exception is the Swift J1727.8−1613 observation 03006001 ($\chi^2 = 155.9$ for 116 degrees of freedom, $p = 0.0079$). For this data set, no significant polarization was detected (PD = $(0.05 \pm0.33)\%$). After accounting for the six trials, the $p$-value is 0.046, so the variability is marginal. No observation shows significant PD variability. Fig. C.2 shows the PA and PD time series.

Polarization–hardness survey

In the third analysis (see also Paper I), the pipeline correlated the 2–8 keV PD with the spectral hardness for 29 IXPE observations of six BHXRBs: Cyg X-1 (12 observations), Swift J1727.8−1613 (8), LMC X-3 (4), 4U 1630−47 (2), GX 339−4 (2) and LMC X-1 (1). The pipeline computed the hardness ratio $\mathrm{HR} = (H-S)/(H+S)$ from the Stokes $I$ count rates in the 2–4 keV ($S$) and 4–8 keV ($H$) bands, summed over the three detector units, after subtracting an annulus background. It subtracted the background from the PD of LMC X-1, LMC X-3 and Swift J1727.8−1613, but not from that of the bright sources Cyg X-1 and 4U 1630−47 [32]. The PD exceeds the MDP99 in 24 of the 29 observations, from four sources. The five observations without a detection lie at the soft end of the HR range.

For the 24 detections, PD and HR are strongly correlated (Spearman $\rho= 0.947$, $p = 2.5 \times10^{-12}$, Fig. C.4). However, the hardness ratios are not corrected for absorption. The two observations of the heavily absorbed source 4U 1630−47 therefore appear harder than the hard state of Cyg X-1, and they lie at the top right of Fig. C.4. They contribute to the combined correlation, and the pipeline did not recognize this problem. The correlations within single sources are thus more meaningful. The pipeline tested each source with at least four detections. The correlation holds within Cyg X-1 and Swift J1727.8−1613, and is not significant within LMC X-3 (Fig. C.3). The trend agrees with the published decrease of the PD as Swift J1727.8−1613 softened [34] [33] and with the correlations within these sources in Figure 3 of Paper I. Paper I analysed the same 29 observations. Its reviewer agent suspected that the combined correlation was an artifact of mixing sources.

The q–u time lag

In the fourth analysis, the pipeline searched for a time lag between the normalized Stokes parameters $q(t) = Q/I$ and $u(t) = U/I$ within single observations. A non-zero lag would indicate a rotation of the polarization vector. We know of no published measurement. The pipeline read 45 of the 47 requested observations. The other two are still proprietary and encrypted, and the pipeline reported them as not measured.

The pipeline divided each observation into time slices with enough events to reach a minimum detectable polarization at 99% confidence (MDP99) of 6%, the value set in the request. It assigned each slice the mean arrival time of its events. The pipeline then computed the discrete correlation function (DCF) of Edelson and Krolik [35] on the resulting irregular time grid. To estimate the significance of the largest $|\mathrm{DCF}|$ over all lags, it shuffled the time order of the $u$ values 1,000 times, with $q$ and the time stamps fixed. It then compared the observed maximum with the maxima of the shuffled data. This method can underestimate the chance probability of a DCF peak, because it ignores the time coherence of the $q$ and $u$ light curves. Spectral methods that preserve the autocorrelation of the $q$ and $u$ variations would be more reliable (see [36, 37], and references therein).

Eleven of the 45 observations have enough events for at least eight slices, and the pipeline excluded the other 34. The 11 observations yield 12 segments from six sources, because one observation was split into two. Figure 4 shows the DCFs. Eleven of the 12 segments show no significant lagged correlation. Cyg X-3 observation 02250301 has a DCF peak at a lag of −57.7 ks with $p = 0.0060$, below the threshold of 0.01. After accounting for the 12 segments tested, $p = 0.07$ [38], so the pipeline reported this lag as a candidate, not a detection. Appendix D describes the tool writing for this analysis.

Benchmarking the tool-writing pipeline

In this section, we test the tool-writing pipeline (Figure 2). Section 6 tests the analysis pipeline (Figure 1). We first list the integrity problems that occurred during tool writing (Sect. 5.1). We then plant defects in tools and count how many of the checks stop. We repeat the count with one check disabled at a time (Sect. 5.2). Finally, we measure the reliability and cost of writing the same tool repeatedly with four different coders (Sect. 5.3).

Integrity problems in the tool-writing sessions

Table 3 lists the problems that occurred during tool writing and how each was detected. As in Yang et al. [6], most problems arose under pressure to finish, here a nearly used-up submission budget, or when input data were missing. For example, the synthetic data that one test required did not exist, and the coder asked the lead how to proceed (Appendix B).

Class	Problem	Caught by	Outcome
Fabrication	Test files hard-coding science results, while claiming they come from real data	co-lead	Rejected and reworked
Fabrication	Documentation claiming test items that do not exist	co-lead	Rejected and reworked
Fabrication	Synthetic data for a test of the tool’s capability did not exist	coder → lead escalation	The lead declined to generate data; the committed test writes hand-picked tables (Appendix A)
Defect in code written by the pipeline	Plotter silently failing when inputs missing	co-lead	Rejected, reworked, and fixed
Defect in code written by the pipeline	Tests not fault-detecting	lead → coder; co-lead	Rejected and fixed
Defect in code written by the pipeline	Accepting the null hypothesis where significance was undefined	co-lead	Rejected and fixed; unresolved in one trial
Defect in code written by the pipeline	A spectral fit operating on every loaded data group while documented as single-mission	co-lead	Rejected before commit
Defect in trusted code	Indexing error in a human-written reduction helper that dropped IXPE detector unit 3, and uncertainties added instead of variances	reported by a human	Fixed by the coder, with a regression test
Failure of a check	False rejections on banned vocabulary	gate: banned vocabulary	Reworded

The checks themselves can be wrong. The most frequent error was a false rejection by the banned-vocabulary scan. We accept these false rejections because they are cheap. A false rejection uses one submission, and the coder rewords the text and submits again. In the trials of Sect. 5.3, the gate rejected fewer submissions than the co-lead, and every failed trial ended with an objection of the co-lead still open (Table 7).

Coder	Co-lead	$N$	Committed	Gate rej./trial	Co-lead rej./trial	Cost US$ (median) | Minutes (median)
deepseek-v4-flash	gpt-5.6-sol	5	5	0.6	1.6	6.5	52
glm-5.3	gpt-5.6-sol	5	4	0.2	2.8	16.0	90
claude-sonnet-5	gpt-5.6-sol	10	7	1.4	2.2	82.0	116
gpt-5.6-terra	gpt-5.6-sol (same provider)	5	5	0.0	2.8	10.8	43

Catch rate of planted defects

We know of no established benchmark for how well checks catch defects in AI-written scientific code, so we built one. The benchmark inserts each defect into a copy of the toolkit and passes the modified toolkit through the gate and the code review (Figure 5). It contains 65 planted defects and 69 defect-free control changes (Table 4).

Table 4: Planted defects stopped by each check, by defect class, with gpt-5.6-sol as the co-lead. The “scan” column covers gate checks 3–5 of Sect. 3.2, and the “tests, structure” column covers checks 1, 2, 6 and 7. The co-lead column counts rejections among the changes that passed the gate. Tests that check nothing have no assertions, cannot fail, or are skipped. A hidden limit is a plausible fixed constant or an upstream product accepted as complete. Missing structure is a missing test file or manifest entry. Defects invisible to the gate exploit known weaknesses of the gate checks. The controls are defect-free changes, so each co-lead rejection of a control is a false rejection. Figure 6 compares eight models as the co-lead.

Defect class	Defects	Gate scan	Gate tests, structure	Co-lead	Reaching a commit
Fabrication, with a banned word	8	6	1	1/1	0
Fabrication, without a banned word	6	0	1	5/5	0
Wrong result, test exposes it	4	0	4	—	0
Wrong result, test misses it	15	0	0	15/15	0
Tests that check nothing	9	0	2	6/7	1
Hidden limit	10	0	0	10/10	0
Missing structure or manifest	7	0	4	3/3	0
Invisible to the gate	6	0	0	6/6	0
All planted defects	65	6	12	46/47	1
Control (defect-free change)	69	0	0	40/69	29

The defects fall into six classes. Fabrications invent values, with or without a banned word in the text. Wrong results include

a PA sign flip, a degree/radian confusion, an inverted calibration epoch, bin edges shifted by half a bin, and 11 errors taken from objections of the co-lead during our analyses. We plant each of the first four twice, once with a test that exposes the error and once with a test that misses it. Tests that check nothing have no assertions, cannot fail, or are skipped. Hidden limits are a plausible fixed constant or an upstream product accepted as complete. Missing structure is a missing test file or manifest entry. Six further defects are invisible to the gate, so only the co-lead can stop them.

Table 4 gives the results for the default configuration. The checks stopped 64 of the 65 planted defects. All controls passed the gate, but the co-lead rejected 40 of the 69, mostly because it judged their tests incomplete (see below). An LLM read the stated reason for each rejection of a defect and checked whether it named the planted defect. Of the 46 defects that the co-lead rejected, 44 were rejected for the planted defect. In the eight-model comparison below, 295 of 308 rejections named the defect in the specific code.

We re-ran the benchmark with one check disabled at a time (Table 5). In each run with a gate check disabled, the co-lead caught every defect that this check stopped in the default configuration. Without the co-lead review, most defects reached a commit.

Configuration	Reaching a commit
Default	1/65
No scan	0/65
No tool-local tests	0/65
No co-lead review	46/65

We also compared eight models from five providers as the co-lead (Figure 6). The models detect between 31 and 46 of the 47 defects that reach the review and falsely reject between 0 and 40 of the 69 controls. The three GPT models detect the most defects and also falsely reject the most controls. In all 113 false rejections, the GPT models judge the tests inadequate, and some also object to the code. The controls contain no planted defect, but most of these objections name real weaknesses in their tests or code. The GPT verdicts also vary between repeated submissions of the same defect-free package (Figure 6). When a provider returned no verdict, we counted a defect as not detected and a control as not falsely rejected.³

The choice of co-lead is a trade-off between detection and false rejection. Each false rejection costs one of the five submissions and one more review. We chose gpt-5.6-sol because it detects the most defects, at the cost of the most false rejections (Figure 6).

Reliability and cost of tool writing

To measure the reliability and cost of tool writing, we had the pipeline write the same tool in repeated trials, following Gendreau-Distler et al. [3]. Each trial started from a fresh copy of the toolkit. Every trial used the brief that an earlier analyst lead (claude-opus-4-8) wrote during the stability analysis.

(Appendix A). No human took part. In all trials, we fixed the lead (claude-opus-5) and the co-lead (gpt-5.6-sol), varying the coder. We ran 10 trials with claude-sonnet-5 as the coder and five with each other coder. Table 7 summarizes the trials, and Figure 7 shows their cost over time. With claude-sonnet-5, the pipeline committed a tool in 7 of the 10 trials. Each of the three failed trials ran out of submissions with an objection of the co-lead still open.

With claude-sonnet-5, the coder accounted for a median of 89% of the cost of a trial. With deepseek-v4-flash, all five trials committed a tool, at a median total cost of US$6.5 per trial and a median coder cost of US$0.68. The median total cost with claude-sonnet-5 was about 13 times higher. With so few trials, the difference in commit rate (5 of 5 against 7 of 10) is consistent with chance. The coders also made changes of different scope. No deepseek-v4-flash commit changed the reduction module of the toolkit, whereas nearly all commits by the other coders did, so its higher commit rate may reflect a smaller change.

The tool-writing benchmark has logged 252 coder sessions since it began, including the 25 trials of Table 7. Over these sessions the gate rejected 68 of 386 submissions, mostly for banned vocabulary (Table 6). In the 25 trials of Table 7, banned vocabulary caused all 18 gate rejections. Some of these rejections were false positives (Sect. 5.1).

Failing check	Rejections	Of gate runs
Banned vocabulary	52	13.5%
Tool-local tests	10	2.6%
Structure	6	1.6%
Determinism	1	0.3%
Parseability	1	0.3%
All rejections	68	17.6%
Gate runs		386

Generality across model tiers

We test whether the analysis pipeline still produces a verifiable result, or reports that it cannot, when weaker models fill its main roles. We keep the coder (deepseek-v4-flash) and the co-lead (gpt-5.6-sol) of Sect. 5, and change the models of the analyst lead, the reviewer and one of the two analyst workers together. The default models form the “anchor” assignment.

Models that score lower on published benchmarks for the tasks of each role form the “contrast” assignment (Table 8). We ran the Cyg X-1 request of Sect. 4.2 three times with each assignment, with the same brief, toolkit snapshot and observations, and with tool writing enabled.

Table 8: The three roles varied in the model-tier test (Sect. 6). For each role, the models are compared on published benchmark scores for the abilities that the role requires. “Anchor” is the default assignment. The coder and the co-lead are the same in all runs (coder deepseek-v4-flash and co-lead gpt-5.6-sol). The scores were read on 2026-09-08, when the contrast assignment was chosen, from Artificial Analysis (Intelligence Index and CritPt), the Terminal-Bench 4.0 leaderboard and the LMArena vision leaderboard. ∗AutomationBench-AA was read on 2026-09-22 from Artificial Analysis.

Role	Abilities	Scores consulted	Anchor	Contrast	Anchor → contrast
Analyst lead	planning<br>adjudication over<br>long agent loops	AA composite<br>Terminal-Bench 4.0	claude-opus-5	gpt-5.6-terra	AA: 51 → 42<br>TB4: 51.8% → 21.5%
Reviewer	scientific critique<br>figure reading<br>general capability	AA composite<br>CritPt<br>LMArena vision	claude-fable-5	qwen3.8-max	AA: 53 → 40<br>CritPt: 30 → 20<br>LMArena vision: 1st → 3rd
Analyst worker	planning tool<br>chains<br>long agent loops	AA composite<br>AutomationBench-AA	claude-sonnet-5<br>+ gpt-5.6-terra	claude-sonnet-5<br>+ qwen3.7-plus	AA: 42 → 26<br>AutomationBench-AA: 59.6% → 17.4%∗<br>(the Anthropic worker is unchanged)

Three runs per assignment are too few to measure run-to-run consistency [39]. We therefore describe how each run handled the same task, which required tools missing from the toolkit. Two runs wrote these tools and recovered all 14 quantities. The other four reported the rest as not measured. Each report thus gives a verifiable value or states that the value is missing.

Table 9 lists the six runs, and Figure 8 compares the recovered quantities. All runs reproduced the systematic PD offset of Sect. 4.2. All runs started from the same toolkit snapshot and data, and five of them wrote their own tools on top of it. The two complete anchor runs and the three contrast runs agree closely on the 12 quantities that all five recovered (Figure 8).

Run	Assignment	Tools	Hours	US$ | $k/14$ | mean $\lvert z\rvert$ | Gate | Review
1	anchor	0	3.5	38	8/14	1.12	0/5	5/5
2	anchor	6	21.5	121	14/14	1.24	1/20	13/19
3	anchor	7	7.2	95	14/14	1.29	1/18	11/18
4	contrast	8	9.5	60	12/14	1.23	3/25	14/22
5	contrast	3	6.6	102	12/14	1.23	42/49	4/7
6	contrast	3	3.1	51	12/14	1.23	2/8	3/6

In the first anchor run, the coding session ended in a declared failure. The co-lead rejected all five submissions. The lead then reported that the required capability was unavailable, and the reviewer accepted this report as a partial reproduction. The run recovered eight of the 14 quantities with the tools already in the toolkit (Table 9). All rejections concerned the robustness of the code, for example the reuse of stale output files and exposure keywords taken from the first input only. The number of objections per rejection fell from 4 in the first to 2 in the last.

None of the contrast runs produced a working XSPEC polconst tool, and the toolkit held no substitute. In run 5, one analyst worker computed the May PD and PA of this fit with PyXspec in a shell session during the first round. The reviewer later told the analysts to record the two quantities as not measured if no toolkit tool could fit the model, and to write no further tools in the next round. The values reached the lead in the worker’s first-round report. The lead did not carry the values of the shell fit forward, because the fit lacked background subtraction. This exclusion agrees with the rule of Sect. 3.1 that shell output supports no number. The final report records both quantities as not measured.

Discussion

Unlike systems that write new code for each task [4, 12, 1, 13], auto_research keeps the tools that it writes. A tool passes the checks once and is then available to later analyses. A human can check it by running its library functions directly. The gate, the report validator, the two-repository boundary and the rule for promoting tools contain no code specific to an instrument or to astronomy. They could therefore serve other quantitative analyses.

Our results are based on four analyses of similar data. We have not yet measured how often later runs revise the conclusions of earlier ones, how often later analyses reuse committed tools, or how much it costs a human to verify a tool. The design has three limitations. First, provenance enforcement establishes the origin of a number, not its correctness. A defective tool that passes the gate and the review yields numbers that the validator admits. Second, the co-lead is a language model and therefore fallible. In the benchmark of Sect. 5.2, it let one of the 47 defects that passed the gate reach a commit and rejected 40 of the 69 defect-free controls. Third, the validator verifies that a number appears in a cited output, not that it was taken from the correct place in that output.

Summary and outlook

We have presented auto_research, a semi-autonomous multi-agent pipeline that analyses X-ray polarimetry data and writes the analysis tools that it needs. It checks its work at two points. A new tool passes automatic checks and a review by a model from a different provider before an analysis uses it, and a human decides whether to promote it to the trusted branch. The validator accepts a number in a report only if a cited output of a toolkit tool or the request contains it, or if the report gives its location in a cited paper.

In four analyses of IXPE data, the pipeline wrote 13 tools and extended four. It reproduced the Cyg X-1 results of Krawczynski et al. [25] with all 14 quantities within $3\sigma$ and found a systematic offset of the PD values. The checks stopped 64 of the 65 planted defects. With weaker models in the main roles, the pipeline recovered 12 of the 14 quantities. It reported the other two as not measured, because no toolkit tool could fit the required model, and stated no value for them, even in a run in which an agent had computed them in a shell. In contrast, the models of Yang et al. [6] produced synthetic data in 55.6% of runs with insufficient data, even when they could report a failure.

In auto_research, the reviewer, the librarians and the analysts each work with their own tools and can end a task only through a report that the validator checks. We plan to build these agents, and new ones, from one reusable agent design. A team of any structure could then be assembled from such agents, and an agent could start new agents during a run, so that the team adapts to the task while the validator still checks every report. The code of the pipeline and of the toolkit is public (see Data availability).

CRediT authorship contribution statement

Fang Zhou: Methodology, Software, Investigation, Writing – original draft, Visualization.

Declaration of competing interest

The author declares that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

The author thanks Kun Hu for discussions and comments on the manuscript, and Henric Krawczynski and the members of the AI group of the Physics Department at Washington University in St. Louis for joint discussions and the sharing of resources (research project ideas, downloaded data). This research used data and software provided by the High Energy Astrophysics Science Archive Research Center (HEASARC), a service of the Astrophysics Science Division at NASA/GSFC. The author acknowledges support from the Physics Department at Washington University in St. Louis through its Hufferd Seed Funding program.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The pipeline is built from LLM agents (Appendix § 18). During the preparation of this work the author used Claude (Anthropic) in order to write the pipeline’s code and working notes for the manuscript. The author reorganised these notes into the first draft and revised it over several rounds. After using this tool, the author reviewed and edited the content as needed and takes full responsibility for the content of the published article.

Data availability

The IXPE data are public at HEASARC. The code of the pipeline and of the toolkit, including the tools that the pipeline wrote, is available at https://github.com/tomthestuka/auto_research and https://github.com/tomthestuka/research_toolkit. The event logs of the runs and the planted defects and controls of Sect. 5.2, with the review verdicts, are available from the corresponding author on request. The reviewer’s reports of the six model-tier runs of Sect. 6 are provided as ancillary files with the arXiv version.

Brief and verification records of the stability tool

During the stability analysis (Sect. 4.3), the analyst lead wrote the following brief for the tool ixpe_pa_pd_stability. The tool-writing trials of Sect. 5.3 use the same brief unchanged. We quote it from the run log up to its third computation step.

“Extend xspec_utils with a new pure-numpy/scipy human-half module lib/pa_pd_stability.py (must NOT import xspec or ixpeobssim), one skills/session.py wrapper ixpe_pa_pd_stability, and one TOOL_MANIFEST entry […].
Signature: pa_pd_stability(obs_db_path, obsids, output_dir, sources=None, variability_p_threshold=0.01, sf_lags_ks=None (default [5,10,20] ks), min_seg_for_sf=10).
Per obsid it reads {obs_db_path}/{obsid}/pcube_timeseg_{obsid}.csv (reuse red_ixpe_read_existing_timeseg_csv to obtain provenance + rows_by_segid + integrated; […]).
Compute: (1) weighted mean by bar=sum(w_i x_i)/sum(w_i) with w_i=1/sigma_i^2 and chi2=sum(w_i(x_i−x_bar)^2), dof=N−1, for PD, QI, UI, and PA; PA uses mod-180 handling (doubled-angle circular weighted mean, residuals wrapped to (−90,+90] via ((d+90)%180)−90); (2) AUTHORITATIVE 2D Stokes chi2_2D=chi2_QI+chi2_UI, dof_2D=(N−1), p_2D=scipy.stats.chi2.sf; variable flag = p_2D<variability_p_threshold; also per-quantity p-values; (3) moment-estimator intrinsic scatter sigma_int=sqrt(max(0,sigma_obs^2−mean_var)) with upper_limit flag when the term <=0, for PA (wrapped residuals) and PD; […]

The run log records the following gate, review and commit events for this tool and for its plotting tool, ixpe_pa_pd_stability_plot. We abbreviate the report and commit hashes.

gate.result: rejected: no_
fabrication.vocabulary:
    xspec_utils/tests/test_pa_pd_stability.py:12 -
banned phrase ‘synthetic’

gate.result: passed (report 1df76135... → cf4e402e...)

review.result: approved. “The implementation matches the requested PA/PD/Stokes stability capability [...] the committed tests are adequate for the core science capability: they include a named reproducibility test on real input, real-data and marked synthetic capability discrimination, independent chi2 recomputation, CSV-reader differential coverage [...]”

commit: ixpe_pa_pd_stability → a6e427c5

review.result: rejected. “‘stability_plot’ treats a nonexistent ‘catalog_csv’ as an empty catalog [...] the code silently accepts a missing required catalog CSV and still reports success [...]”

review.result: rejected. “‘_load_catalog_rows()’ does not validate the CSV header before returning [...] The catalog is not checked up front for all fields the plotter later requires [...]”

review.result: approved (third submission)

commit: ixpe_pa_pd_stability_plot → 727fbace

The committed tests write two fixture CSVs with hand-chosen values, synth_pa_pd_variable and synth_pa_pd_constant, into a temporary directory. One test checks that the tool flags the first as variable and the second as constant. The lead wrote this test and told the coder to add it, although it had earlier told the coder that “a test never generates data” (question B.1 in Appendix B). The gate rejected the word ‘synthetic’ in the test file, and the coder reworded the text. The co-lead then approved the tests, citing “marked synthetic capability discrimination”. Neither check stopped the tests from writing these fixture CSVs.

Questions from the coder to the lead

While writing ixpe_pa_pd_stability, the coder asked the lead two questions. We summarize each question and quote the lead’s answer from the run log.

B.1 Missing synthetic test data. One check in the brief required two synthetic observations, one with variable and one with constant polarization. The coder found no such data on disk and offered two options. In option 1, the lead or the human would point it to pre-generated files. In option 2, the coder would drop the check and test the discriminating power with real data and arithmetic tests. The lead answered:

“Go with option 2 — do NOT fabricate synthetic fixtures. My checks of disk state confirm your finding: data_catalog lists 16 real local IXPE ObsIDs and no synth_pa_pd_* entry exists [. . . ] No synthetic fixture was ever pre-generated, and the policy is firm: a test never generates data, and the toolkit contains no data … Since the fixtures don’t exist, drop the ‘variable vs constant’ synthetic capability check from this pass and cover discriminating power with real, local data plus lib-level arithmetic — exactly as the existing test already does [. . . ]”

B.2 Conflicting instructions in the brief. The brief asked for a module that imports neither xspec nor ixpeobssim. It also asked the module to reuse a CSV reader from red_ixpe, which imports both at module level. The coder noticed this conflict and proposed its own CSV reader. The lead agreed, because importing red_ixpe “would pull the entire science stack into your pure-numpy/scipy lib — exactly what the brief forbids.”

Result records of the analyses

The q–u time-lag analysis

This appendix gives the details of the fourth analysis (Sect. 4.5). The analysis has no published reference and no expected detection. A false detection is therefore its main risk. The request stated that “a null result (no significant lag), or per-observation upper limits, is the expected, correct, and publishable outcome”, and it instructed the agents not to “tune the null, hand-roll the correlation, or back-fill a detection”.

Tool writing. Three tools were already in the toolkit: the slicer ixpe_qu_slice_pubes, the DCF tool ixpe_qu_lag_dcf, and the plotting tool ixpe_qu_lag_dcf_plot. One analyst worker ran all three before the first coding session. Seven coding sessions then changed these tools or added a new one, and five of them committed. Two commits fixed the handling of numerical arguments in the slicer and the DCF tool. One stopped the slicer from splitting a merged 386 ks observation (04002899) at each of its orbital gaps. The other two added the absolute lag of the DCF peak, $|\tau_{\mathrm{peak}}|$, to the Spearman tests across the segments and to the plotting tool. Each commit passed the gate and the review on its first submission. The sixth session worked on a further change to the slicer. The gate rejected it once for banned vocabulary, and the session ended before review. The seventh session wrote a new tool, ixpe_ pol_report, for a table of the polarization of each observation. The gate rejected it once for banned vocabulary, and the review rejected it four times. In total, the gate ran 11 times and rejected two submissions, and the review ran nine times and rejected four.

Shortfalls. The final report of the run states two shortfalls. First, the report has no measurement for the two proprietary observations of Sect. 4.5, SS 433 (04004001, 745 ks) and GS 1354−64 (05250399, 452 ks). They are two of the long observations that the request expected to anchor the analysis. Second, the report lacks the separate polarization table that the request asked for. The last review of ixpe_pol_report found that “the resumable checkpoint path can silently return an invalid successful science row after an interrupted write”, and the lead started no further session for this tool. The report therefore uses the polarization significance computed inside the DCF tool, and it says so.

Result. The permutation probabilities of the 11 segments without a significant correlation range from 0.065 to 0.933. The per-test criterion of the request requires $p < 0.01$ and a 68% interval of the peak lag $\tau_{\mathrm{peak}}$ that excludes zero. The candidate in Cyg X-3 observation 02250301 meets this criterion. Its interval, from flux randomization alone [40], is [−100.94, −57.68] ks. However, Peterson et al. [40] combine flux randomization with random-subset selection, which tests the sensitivity to the sampling, and find that either method alone generally gives smaller uncertainties. The interval of the candidate is therefore probably too narrow.

The pipeline also ran four Spearman tests across the segments, and two of them have $p < 0.01$. The signed $\tau_{\mathrm{peak}}$ correlates with the polarization significance ($\rho= +0.776$, $p = 0.0030$), and $|\tau_{\mathrm{peak}}|$ anticorrelates with a brightness proxy ($\rho= -0.734$, $p = 0.0065$). The report claims neither result. It attributes the first correlation to the three segments with the weakest polarization detections, which all have large negative $\tau_{\mathrm{peak}}$. Their peaks lie at lags of 0.4 to 1.6 Ms, longer than a single visit. The DCF tool searches lags up to half the time span of a segment, so for a segment with several visits it also searches lags between visits. The report recommends restricting the peak search in future runs. It attributes the second correlation to the sampling. Fainter sources need wider slices and were observed over several visits, so both the shortest and the longest lag that can be measured grow as brightness falls.

What the checks did. The reviewer read the DCF code, compared it with the noise-corrected normalization in Eq. (3) of Edelson and Krolik [35], and accepted the report. The reviewer also wrote that this reference “explains why few-pair lag bins yield $|\mathrm{DCF}| > 1$”. The reference does not support this. It gives the expected scatter of the DCF of uncorrelated series, about $\pm M^{-1/2}$ in a lag bin of $M$ pairs, and does not discuss $|\mathrm{DCF}| > 1$. For the peaks of $|\mathrm{DCF}| = 2.2$–$3.4$, the report and the reviewer based the significance on the permutation probability alone. One analyst worker computed all results, and the second worker only plotted them again. The analyst lead contacted the human 11 times during the run. Ten contacts discussed the design of the tools to be written, and one asked whether to accept the 45 readable observations as the sample.

Cost. The run made 613 model calls and 1,592 tool calls. The coding agent accounts for 78% of the cost in Table 1, in seven tool-writing sessions.

Implementation and cost

Table E.1 lists the models of each analysis. auto_research has 34,289 lines of Python in 121 modules. The typed messages between agents (Sect. 2.2) are 35 Pydantic models. The deterministic test suite has 1,824 tests in 27,444 lines. The benchmark harnesses of Sect. 5 have 16,716 lines and 499 tests.

Role	Sect. 4.2	Sects. 4.3–4.4	Sect. 4.5
Reviewer	claude-fable-5	claude-opus-4-8	claude-fable-5
Analyst lead	claude-opus-4-8	claude-opus-4-8	claude-opus-5
Co-lead	gpt-5.5	gpt-5.5	gpt-5.6-terra
Analyst workers	claude-sonnet-4-6<br>gpt-5.5	claude-sonnet-4-6<br>gpt-5.5	claude-sonnet-5<br>gpt-5.6-terra
Coder	claude-sonnet-5	claude-sonnet-5	claude-sonnet-5
Librarian	claude-sonnet-4-6	claude-sonnet-4-6	claude-sonnet-5
Status monitor	—	—	gpt-5.6-luna

On the working branch, research_toolkit holds three tool packages with 25 tools. The lib/ halves of these tools (Sect. 2.1) have 8,247 lines, the skills/ halves 3,444 lines, and the tests of the toolkit 11,130 lines. No file in the toolkit imports auto_research. Thus a human can reproduce a tool with only the science software and the toolkit.

The 35 phrases of the banned-vocabulary scan (Sect. 3.2) are example data, sample data, fake data, fake value, fake result, mock data, dummy data, made up, made-up, fabricated, fabricate, fictional, fictitious, hardcoded result, hard-coded result, canned, pretend, bogus, toy data, representative data, representative value, representative result, for now, in operational mode this would, in production this would, in practice this would, would actually, would normally, for demonstration, for illustration, placeholder, TODO, FIXME, XXX, HACK.

auto_research records the tokens and the cost of each model call. The experiments reported here cost about US$2,200 at list prices. This total comprises the analyses of Table 1, US$259 for the benchmark of Sect. 5.2, US$984 for the tool-writing trials of Sect. 5.3 and US$466 for the runs of Sect. 6.