betaFeedback

WorksComputer Science

Collaborative AI agents for autonomous experimental research with solid-state quantum spins

Abstract

An autonomous research laboratory could conduct an entire study, from formulating questions to preparing manuscripts. Artificial intelligence (AI) offers one route towards this goal. Recent AI systems take on several research roles, but may draw conclusions that the evidence does not support. Here, we present an autonomous research laboratory in which agents based on large language models assess one another's work and can stop or redirect investigations. They can generate and reject research questions, require separate theoretical checks and conduct experiments designed to challenge their claims. We demonstrate these capabilities by investigating ensembles of quantum spins in diamond. The agents derived the minimum number of calibration settings needed to predict responses to a defined family of microwave pulse sequences. They checked the result computationally and designed experiments to distinguish competing predictions. An initial experiment agreed with the theoretical predictions, but repeated reference measurements showed that the signal was drifting during data collection. The agents therefore treated the result as inconclusive and revised the measurement protocol without relaxing the acceptance criteria. The revised experiment met those criteria and supported the theoretical predictions. This study demonstrates that collaborating AI agents can select research questions, derive theoretical results and test their predictions experimentally.

To conduct research autonomously, a system must decide what to investigate, how to test its ideas and what to conclude from the results. Robot scientists have generated hypotheses and tested them in the laboratory [1, 2], while autonomous laboratories have used experimental feedback to select and carry out further experiments [3, 4]. Systems based on large language models (LLMs) can also search the literature, propose ideas, plan experiments and analyse data [5, 6, 7].

In recent studies, LLMs have challenged hypotheses, checked physical reasoning and revised interpretations in response to experimental evidence [6, 8, 9, 10]. A demanding test is whether agents can use this criticism to guide a study in which they develop a theoretical claim and design experiments to test it. Agents may complete every step of a study yet still draw a conclusion that the evidence does not support. In one evaluation, an AI Scientist reported performance improvements that were based on flawed experiments [11]. We therefore ask whether agents can revise their experiments or conclusions when criticism reveals a flaw.

Scientific judgement becomes most consequential when it prevents an apparently promising investigation from proceeding. A plausible research question may already have an answer, and an elegant mathematical argument may rest on an unchecked assumption. For a proposed physical law, even a rigorous mathematical derivation cannot establish its empirical validity and relevance: its predictions must also withstand experiments designed to distinguish them from competing explanations. Organized scepticism requires these uncertainties to be confronted rather than merely acknowledged [12]. Independent verification, precommitted predictions and acceptance criteria, and experimental controls provide safeguards against unjustified conclusions [13, 14, 15]. The critical test is whether AI agents can enforce these safeguards, even when doing so means rejecting their own work.

Ensembles of nitrogen-vacancy (NV) centres in diamond provide a suitable setting for this test. Their optically readable spins can be manipulated through precisely controlled microwave-pulse sequences, while variations in spin properties create a nontrivial problem of predicting ensemble responses [16, 17, 18]. This combination of experimental control and physical complexity allows mathematical predictions to be independently examined and tested against simpler calibration methods. The platform thus provides an opportunity to evaluate whether AI agents can formulate a meaningful question, establish a theoretical result and design an experiment capable of challenging that result rather than merely confirming it.

Here we present a collaborative AI laboratory that uses scientific criticism to guide an investigation from question selection to experimental assessment. Our objective is to determine whether collaborating agents can develop a substantive theoretical result and establish what experimental evidence supports its predictions. The agents pursued this objective by identifying the minimum calibration information needed to predict unmeasured responses of an NV spin ensemble, without reconstructing the unknown distribution of spin errors. The transition from theory to experiment required further collaboration: an experimentalist agent developed the control software and performed measurements, while a manager agent proposed redesigning the experiment rather than stopping after an agent serving as principal investigator rejected an initial design. During the research an experiment failed, the agents investigated the discrepancy and revised the acquisition procedure. The final measurements supported the calibration law developed by the proposer and PI. Together, the mathematical result and its experimental test demonstrate a scientific investigation in which AI agents collectively selected, challenged and substantiated a new physical claim.

Collaborative AI laboratory for scientific discovery

To investigate whether scientific judgement can emerge through collaboration among AI agents, we developed a research system that distributes scientific responsibilities across specialized agents (Figure 1). The system connects research-question generation, theoretical analysis, computational verification and physical experimentation through exchanges of research outputs and critical feedback. Rather than automatically advancing results from one stage to the next, agents evaluate the work of their collaborators and can reject or request revisions when scientific criteria are not satisfied. By separating the generation and evaluation of scientific claims across agents with distinct contexts, this design aims to reduce self-confirmation bias and the reinforcement of erroneous assumptions across successive reasoning steps [19, 20].

Figure 1. Scientific workflow and coordination of the collaborative AI laboratory. (a) Five-stage research cycle with checkpoints overseen by the principal investigator (PI). Dotted lines indicate PI involvement; dashed arrows indicate rejection and revision. (b) Specialized AI agents and their exchanges of research outputs, with safety-agent approval required before experimentation. (c) Document flow during one pass through the API-orchestrated workflow, from left to right (not to scale). The orchestrator routes documents to the persistent PI conversation (blue). Dots mark document deliveries, circled ticks and crosses indicate PI decisions, and dashed links trace rejected work. Upper branches represent separate model calls; lower blocks represent coding-agent jobs.

The laboratory comprised six principal agent roles: proposers, a principal investigator (PI), a computational researcher, an experimentalist, a manager and independent reviewers (Figure 1). The proposers independently generated candidate research questions, which the PI evaluated for scientific novelty, logical consistency, significance and experimental feasibility. After selecting a research question, the PI developed its theoretical foundations, while the computational researcher independently verified the mathematical results and examined the robustness of their underlying assumptions on a supercomputer. The experimentalist developed the instrument-control software and performed measurements on an NV-diamond platform. The manager coordinated exchanges between the agents and contributed to experimental planning, while independent reviewers assessed the resulting manuscript.

Scientific evaluation was integrated into the communication between agents rather than restricted to a final review. Research outputs were exchanged through written documents, and key decisions required explicit assessments before the investigation could proceed (Figure 1). Rejected proposals could be revised in response to criticism, mathematical claims were subjected to independent verification, and experimental designs could be reconsidered when they failed to distinguish the proposed theory from alternative explanations. Experimental predictions and acceptance criteria were fixed before the target measurements were analysed. These mechanisms allowed criticism to influence not only the interpretation of results, but also the direction and execution of the research itself.

Human researchers assembled the apparatus and configured the safety agent with preset operating limits. Within these boundaries, the AI agents carried out the calibration investigation through their respective roles, from research-question selection to experimental assessment.

Critical evaluation and selection of research questions

We first examined how the laboratory selected a research question from independently generated proposals (Figure 2). Each proposer received a description of the experimental platform and was tasked with identifying a mathematically substantive research question in machine learning or information science in which physics was essential, without access to the other proposals. The principal-investigator (PI) agent evaluated the proposals in batches, assessing their scientific novelty, logical validity, significance and experimental feasibility. Rather than selecting the most promising candidate by default, the PI could reject an entire batch and provide written criticism to guide subsequent proposals.

Figure 2. How the accepted proposal emerged. (a) Every proposal at its delivery time, marked by the PI’s decision. The dashed line marks when the repeatedly rejected idea families were listed in the proposers’ instructions. The ringed proposal was rejected but named the best seed, and its revision was accepted. (b) Reasons for rejecting 75 proposals, in four categories: logically invalid (false or invalid claims, overclaims), already known, not significant (no core theorem, trivial, renamed mathematics, too narrow) and no feasible test. Dark bars count the decisive reason and light bars every reason cited.

This selectivity proved consequential. Over approximately 30 hours, the PI rejected 75 proposals, accepting none during the first 17 batches (Figure 2a,Figure 2b). The rejections reflected recurring problems: previously established results, logical inconsistencies, insufficient scientific significance and the absence of feasible experimental tests. Moreover, independent proposers repeatedly converged on similar unsuccessful research directions. To overcome this limitation, the manager agent independently identified recurring rejected ideas and excluded them from subsequent proposal instructions. These ideas had accounted for 52 of the 56 earlier fresh proposals; following their exclusion, nine new proposals explored seven previously unrepresented research directions. The feedback therefore changed not only which proposals were rejected, but also the directions explored by the agents.

The eventual selection emerged through revision rather than acceptance of an initial proposal. The PI identified a previously rejected idea as a promising starting point and provided a critique that a new group of proposers used to reformulate the problem. The revised proposal replaced an approximation based on leading-order effects with a formulation retaining the full dependence on pulse sequences. This led the PI to select and refine a research question concerning the predictive information contained in measurements of an inhomogeneous spin ensemble: what is the minimum number of calibration measurements required to predict the ensemble response to arbitrary phase-controlled pulse sequences?

Discovery and verification of an ensemble-calibration law

The selected research problem concerned predicting the collective response of an inhomogeneous spin ensemble to phase-controlled microwave-pulse sequences. Individual spins experience different detunings and microwave-drive strengths, so the measured signal averages over an unknown distribution of pulse errors. The agents considered sequences of pulses with fixed duration, amplitude and shape but independently variable phases, assuming that each spin retained the same error parameters throughout a sequence. The selected proposal established a polynomial upper bound of 8L2+6L+18L^2 + 6L + 1 independent response functions for sequences of up to LL pulses, but did not determine the exact number of calibrations required.

During the subsequent theoretical investigation, the principal-investigator (PI) agent reformulated the problem and derived an exact result (Figure 3a). It proved that L2+1L^2 + 1 calibration measurements are both necessary and sufficient to predict the ensemble response to every phase-controlled sequence of at most LL pulses, independently of the distribution of persistent spin errors. The proof established that these measurements span the space of possible ensemble responses, even with an unknown common fluorescence background and contrast. Conversely, fewer calibrations cannot guarantee reconstruction, because distinct physically admissible ensembles can produce identical calibration measurements while differing on unmeasured sequences. The PI thus transformed the proposal’s sufficient upper bound into an exact minimum, reducing the required count from the proposed bound of 91 to 10 for L=3L = 3. More generally, the calibration requirement grows quadratically with sequence length, whereas the number of possible pulse sequences grows exponentially for a fixed discrete set of phase choices (Figure 3b).

Figure 3. Discovery and verification of a minimal ensemble-calibration law. (a) The revised proposal supplied an upper bound of 8L2+6L+18L^2+6L+1 calibrations, which the principal-investigator agent refined to the exact necessary-and-sufficient count L2+1L^2+1 under persistent pulse errors. (b) Calibration requirements compared with the number of possible sequences for five phase choices per pulse, with the first phase fixed. (c) Certified prediction-error and calibration-ambiguity bounds under hidden phase rotations, relative to photon-counting noise.

This result makes it possible to predict an entire family of control responses without reconstructing the unknown distribution of spin errors. With sufficient calibration, different microscopic ensembles can remain compatible with the measurements while giving identical predictions for every sequence in the specified family. The conceptual advance is the exact identification of the response information needed for this task and its equivalence to physically necessary calibration measurements. The resulting predictor takes the calibrated mean responses as inputs and returns mean responses for unmeasured phase-controlled sequences. The matching lower bound establishes a fundamental limitation: if calibration leaves a response direction unresolved, a more flexible predictor cannot guarantee its recovery for every admissible ensemble. The law therefore provides both a constructive way to select sufficient measurements and a criterion for when additional calibration is needed. The minimum L2+1L^2 + 1 refers to calibration settings with linearly independent responses; repeated acquisitions determine the precision of the measured means.

The mathematical result was then subjected to independent computational verification. A computational-researcher agent reproduced the symbolic checks and independently reconstructed the response-space structure, confirming dimensions of 26 and 37 for sequences of up to five and six pulses, respectively. The PI further challenged the assumption that pulse errors remain constant throughout a sequence by specifying a numerical stress test involving hidden phase rotations between pulses. Using the TSUBAME 4.0 supercomputer, the computational researcher generated 252,620 exact certificates bounding calibration ambiguity and prediction errors under these perturbations. Here, calibration ambiguity is the largest difference in target means between ensembles that agree on the short-sequence calibrations and remain statistically indistinguishable under the full calibration. For the specified six-pulse test, the certified prediction error remained at the photon-noise level for hidden phase rotations of approximately 1.2∘1.2^\circ or less (Figure 3c). These calculations examined the robustness of the proposed law beyond its ideal assumptions, while its empirical validity remained to be established through measurements capable of distinguishing its predictions from simpler calibration methods.

Collaborative design of a discriminating experiment

The experimental objective was to test the predictive value of the calibration law using withheld sequence responses. The agents therefore sought a design in which the law and specified interpolation methods made different predictions, allowing the measurements to distinguish between them. The initial design combined a complete grid of nine three-pulse calibration sequences with five shorter sequences to predict four withheld targets (Figure 4a). However, the nine-point grid already contained sufficient information to determine those targets through Fourier interpolation. Agreement between the predicted and measured signals would therefore not distinguish the proposed law from an ordinary interpolation method. The PI identified this limitation and rejected the design as an inadequate test of the theoretical result.

Figure 4. Experimental design to distinguish the calibration law from interpolation. (a) Rejected design using a complete grid of three-pulse calibrations, from which the withheld targets can already be determined by Fourier interpolation. (b) Revised design combining five three-pulse calibrations with five shorter sequences to predict four withheld targets (T1T_1–T4T_4). Filled symbols indicate calibration sequences and open symbols indicate withheld targets. In both maps, rows give the phase of pulse 2 and columns the phase of pulse 3. Both designs also include the same five shorter calibrations of zero to two pulses, which are not drawn. (c) Measured signals of the final run along each row of (b), relative to the mean of the five three-pulse calibrations. Dots are measured calibrations, whose standard errors are smaller than the dots. The calibration law (blue) predicts each row from all ten calibrations, whereas the two interpolation-based alternatives (grey, solid and dashed) use only the five three-pulse calibrations. Rings mark the withheld targets at the law’s predictions, fixed before the target measurements were analysed; the measured targets are shown in Figure 5c.

The rejection prompted further collaboration between the agents. The PI proposed a different approach that combined measurements from pulse sequences of different lengths, using five three-pulse calibrations together with five shorter sequences. The shorter sequences supply the information missing from the five three-pulse settings. The PI refined the phase settings and verified that these ten calibrations spanned the response space required by the law, while leaving four three-pulse targets that could not be determined by the competing interpolation methods (Figure 4b). This cross-length design produced distinct predictions from the discovered law and simpler interpolation rules (Figure 4c), providing an experimental test capable of distinguishing the competing explanations.

To ensure that the comparison could not be influenced by the observed targets, the PI established the prediction rules and acceptance criteria before data acquisition. The experimentalist was required to fix the numerical predictions obtained from the calibration measurements before analysing the withheld target data. The protocol also incorporated independent experimental controls and predefined conditions for classifying an outcome as supporting, contradicting or inconclusive, the last including any acquisition that failed a control. Before measurement, the experimental design was approved by a safety agent. Consequently, agreement with the theoretical prediction alone would not be sufficient to establish experimental support for the law.

Rejecting an apparently successful run and establishing experimental support

The first main acquisition met all ten precommitted accuracy criteria, which assessed agreement for the four withheld targets and a contrast formed from those responses (Methods). However, the global-phase and reference-pulse controls failed (Figure 5b). These controls tested whether applying the same phase shift to every pulse left the measured response unchanged. The principal-investigator (PI) agent therefore classified the run as inconclusive.

Figure 5. Critical rejection and experimental support for the AI-derived calibration predictions. (a) Signal of the first run in each 10-min sub-run, in weighted counts per shot, for the two acquisition programs; the dashed line marks 11:48, when the room light came on. (b) The global-phase and reference-pulse checks of each acquisition, as a ratio to their preset limits; checks in the shaded region fail. (c) The four withheld targets, relative to the mean of the five three-pulse calibrations: locked predictions of the calibration law (blue) and of the two interpolation-based rival rules (grey), and the final measurements (orange), whose standard errors are smaller than the symbols. (d) The target contrast SS formed from the four targets in 10−310^{-3} weighted counts per shot; the shaded band marks the predefined tolerance of ±20%\pm20\% around the law’s locked prediction.

The experimentalist traced the failed controls to changes in the signal during acquisition. The signal rose by 7.6% over the morning and showed a further 2.9% step at 11:48 a.m., coinciding with a laboratory member switching on the room lights (Figure 5a). The experiment alternated between two programs containing different global-phase replicas. Because these programs sampled the drifting signal at different times, measurements expected to be equivalent acquired systematic offsets.

The agents revised the acquisition protocol without relaxing the acceptance criteria. The experimentalist shortened the alternation between programs and tested different sequence orders in pilot acquisitions (short runs used to test adjustments to the acquisition procedure). The first pilot still failed the phase controls; a second, with phase switching within a single program, passed. The PI retained the inconclusive verdict for the original run and specified a new six-hour acquisition with randomized sequence ordering and additional diagnostics of the null pair. The manager conveyed the revised protocol to the experimentalist, who implemented it.

The final acquisition yielded 21,600 valid one-second records and passed all ten accuracy criteria and experimental controls (Figure 5b). Predictions from the calibration measurements were fixed before the withheld targets were analysed. The four withheld responses were summarized by the target contrast S=(−T1+T2+T3−T4)/4S = (-T_1 + T_2 + T_3 - T_4)/4, which cancels the common fluorescence background. The measured target contrast was (40.0±0.4)×10−3(40.0 \pm0.4) \times10^{-3} weighted counts per shot, consistent with the law’s prediction of (39.7±1.3)×10−3(39.7 \pm1.3) \times10^{-3}, whereas the interpolation alternatives predicted zero (Figure 5c,Figure 5d). These measurements support the transfer of information from the shorter calibration sequences to the four withheld three-pulse responses within the predefined accuracy tolerances.

Discussion

Recent debate about AI scientists concerns whether automated research can produce conceptual insight and reliable scientific judgement [5, 21]. Our results demonstrate both in a physical investigation. Critical evaluation shaped the research question, strengthened the mathematical theory and determined which experimental evidence could support a conclusion. The principal-investigator agent rejected an experimental design because conventional interpolation could predict the same targets. It subsequently retained an inconclusive verdict for an acquisition that met all ten precommitted accuracy criteria but failed independent experimental controls. The agents revised the acquisition protocol without relaxing the acceptance criteria. These decisions demonstrate that collaborating AI agents can distinguish predictive agreement from sufficient scientific support and enforce that distinction when evaluating their own work.

The calibration law provides a concrete example of conceptual progress during the investigation. The PI reformulated calibration in terms of the dimension of the ensemble-response space and proved that L2+1L^2+1 measurements are both necessary and sufficient for sequences of at most LL pulses under persistent pulse errors. Establishing necessity required identifying physically admissible ensembles that agree on an insufficient set of calibrations yet differ on unmeasured sequences. This reasoning transformed a sufficient construction into an exact characterization of the information required for prediction. The experimental measurements supported this predictive capability on the four withheld targets within the predefined tolerances, distinguishing the law’s predictions from those of the two specified interpolation methods. Together, the theoretical result and its experimental test show how an AI laboratory can identify a precise information requirement, turn it into a measurable prediction and assess the evidence supporting that prediction.

Solid-state spin defects provide a particularly promising platform for extending this approach. Nitrogen-vacancy centres combine optical spin initialization and readout with programmable microwave control, allowing many experimental protocols to be executed without manual intervention once the apparatus is operational [8, 16, 17, 18]. Their sensitivity to control imperfections and environmental fluctuations also provides demanding tests of quantum-control theories. Such systems could enable AI agents to investigate new pulse sequences, calibration strategies and sensing protocols through repeated cycles of theoretical prediction and physical measurement. Similar approaches may be applicable to other quantum platforms.

Extending the laboratory towards greater autonomy will require improvements in both the agent architecture and the experimental infrastructure. Larger groups of parallel proposers could broaden the search for research questions, while more specialized agents could extend safety assessment or coordinate work across laboratories. Limited diversity in generated research ideas has been identified in evaluations of LLM research agents [22]. Here, the manager used the record of rejected proposals to revise the search strategy, after which the proposers explored previously unrepresented research directions. This adaptation demonstrates that an agent-led investigation can use accumulated criticism to change the scientific questions it pursues. Connecting multiple experimental setups, including cryogenic platforms, would further expand the range of testable physical phenomena. Such extensions would also require more reliable instrument diagnostics, coordination and safety mechanisms as agents gain additional experimental degrees of freedom.

A broader challenge is therefore to develop experimental testbeds in other branches of science that allow AI-generated hypotheses to be tested with comparable independence and rigour without human involvement. Greater scientific autonomy will depend not only on improvements in AI reasoning, but also on instruments that make theoretical predictions experimentally accessible and competing explanations distinguishable. As these capabilities develop, the role of human researchers may shift towards identifying consequential questions, expanding experimental possibilities and critically assessing the significance of automated discoveries. The long-term opportunity is not simply to automate scientific work, but to extend the range of questions that can be investigated and answered with reliable evidence.

Methods

Agent implementation and human contributions

We implemented the laboratory using model conversations and coding-agent sessions configured for different research roles. The principal investigator (PI) used GPT-6 at the Pro reasoning setting, with Extra High for reviews. Proposal generation used GPT-5.6 extra high. The computational researcher ran GPT-6 Astra at maximum reasoning effort through Codex, using CPUs and GPUs on the TSUBAME 4.0 supercomputer. The experimentalist used Claude Fable 5 through Claude Code on the instrument computer, and the manager used Claude Opus 5 on a separate computer with browser and SSH access to the experimentalist and the computational researcher on the supercomputer.

Mechanisms explored in earlier projects were incorporated into the exclusion instructions for subsequent proposal searches to prevent new proposals from overlapping with previously explored research directions. Each role received instructions specifying its scientific task, expected output and evaluation criteria.

For the NV investigation, human researchers assembled the apparatus and supplied a description of its capabilities. During setup, they configured and checked the safety agent using preset operating limits, including maximum voltage and power.

Context allocation, document exchange and review

Agents exchanged proposals, manuscripts and research reports as written documents relayed by the manager. Information was supplied according to each agent’s task. Fresh proposers received a short description of the experimental platform and were not shown one another’s proposals. Each proposal batch was assessed in a new PI conversation. After selection, the PI continued that conversation through theoretical development, manuscript preparation and computational test design. A separate feasibility reviewer received the manuscript and full apparatus specification without the PI’s preceding exchanges.

An internal manuscript carried the research question, assumptions and mathematical derivations between agents. The computational researcher received its source files together with the PI’s test specification, and returned verification results and a revised manuscript. Its working directory persisted across tasks. The experimentalist received the experimental protocol and task instructions, maintained local notes and logs, and returned laboratory reports. Raw photon counts remained on the instrument computer; the PI and computational researcher assessed the measurements from these reports.

Assessments at decision points ended with an explicit verdict and written reasons. Requests for revision specified numbered changes for the agent responsible for the task. The decisive computational experiment underwent one PI review followed by one correction pass by the computational researcher. Final manuscript reviewers started in new conversations and received only the manuscript, without the preceding agent exchanges, source code or raw data. They returned a referee report, recommendation and confidence score. After receiving the reviews, the PI judged the experimental evidence sufficient and instructed the computational researcher to add more information in the manuscript as requested by the reviewers.

Proposal generation and adaptive selection

Each proposer was instructed to formulate one theoretical contribution to information science in which physics was essential and that could be applicable to NV centres. The selected proposal met this brief as a learning problem: it asked how many calibration measurements are necessary and sufficient to predict the responses of an exponentially large family of control sequences, a question of identifiability and sample complexity whose answer is set by the physics of persistent pulse errors. Proposals had to address a significant problem, identify a feasible theoretical approach and include at least one central prediction specific to the idea and testable on the available NV platform. Numerical validation and empirical benchmarks were excluded at this stage. Proposers were instructed to develop candidates before checking the literature, assess novelty by the underlying mechanism, and exclude directions adopted from published future-work or open-problem discussions.

Fresh batches launched ten proposer conversations to avoid overloading the context of the PI. Each fresh batch was forwarded to the PI once five proposals had arrived, and the remaining proposers were stopped. A separate deduplication stage grouped proposals by central mechanism and retained one representative from each group. The PI required proposers to return the strongest idea reached, with remaining uncertainties stated. The PI assessed novelty, logical validity, significance, theoretical feasibility and experimental testability, and could reject an entire batch.

Proposal instructions excluded mechanisms explored in earlier projects. After twelve consecutive assessments without a winner, the manager added exclusions derived from repeatedly rejected mechanisms in the PI’s verdicts. Revision batches launched six proposers, each receiving a rejected proposal verbatim and the PI’s full assessment of its batch. They were instructed to address decisive objections and reassess novelty, with permission to change the mechanism if the original could not be sustained. Returned revisions were not used as seeds for further revision batches.

The assessment criteria and literature checks were retained, and the selection instruction was to choose and strengthen the best proposal while stating remaining uncertainties. Selecting none was permitted only if no candidate was novel, feasible or impactful. The PI was instructed to distinguish justified claims from plausible statements and outstanding proof obligations. Before manuscript preparation, it compared the contribution with the five closest works, specifying what each established and what remained new.

Spin-response model and calibration construction

The PI and the proposer who originated the research considered pulses with fixed duration, amplitude and shape and independently programmable phases. In the two-level model, a pulse of phase ϕ\phi was represented by

Uϕ(a)=(aqe−iϕ−qeiϕaˉ),q=1−∣a∣2,U_{\phi}(a) = \begin{pmatrix} a & \sqrt{q}e^{-i\phi} \\ -\sqrt{q}e^{i\phi} & \bar{a} \end{pmatrix}, \qquad q = 1 - |a|^2,

where aa encodes the pulse action on an individual spin and aˉ\bar{a} denotes complex conjugation. Each spin retained the same aa throughout a sequence. The ensemble was described by an arbitrary unknown probability distribution ν\nu on an admissible domain K⊆{a:∣a∣≤1}K \subseteq\{a : |a| \le1\} with nonempty two-dimensional interior.

For a sequence w=(ϕ1,…,ϕℓ)w = (\phi_1,\ldots,\phi_\ell) in time order, with ℓ≤L\ell\le L, the return probability and mean fluorescence signal were

pw(a)=∣⟨0∣Uϕℓ(a)⋯Uϕ1(a)∣0⟩∣2,μ(w)=B+κ∫Kpw(a) dν(a).p_w(a) = \left|\langle0|U_{\phi_\ell}(a)\cdots U_{\phi_1}(a)|0\rangle\right|^2, \qquad\mu(w) = B + \kappa\int_K p_w(a)\,\mathrm{d}\nu(a).

The empty sequence had p∅=1p_{\varnothing}=1. The distribution ν\nu, unknown offset BB and nonzero contrast κ\kappa were common to all sequence settings.

The PI and the proposer defined FL\mathcal{F}_L as the real span of the response functions for sequences of at most LL pulses. Uniform identification of every target in this class required calibration responses spanning FL\mathcal{F}_L, whose dimension was dL=L2+1d_L = L^2 + 1. The general construction used the five phases 2πk/52\pi k/5, k=0,…,4k = 0,\ldots,4, to resolve the phase Fourier components and selected dLd_L linearly independent sequence responses as a basis. For each target, expansion in this basis determined real coefficients hj(w)h_j(w) satisfying

pw(a)=∑j=1dLhj(w)pbj(a),∑j=1dLhj(w)=1.p_w(a) = \sum_{j=1}^{d_L} h_j(w)p_{b_j}(a), \qquad\sum_{j=1}^{d_L} h_j(w) = 1.

Target predictions were therefore computed as

μ^(w)=∑j=1dLhj(w)μ^(bj),\hat{\mu}(w) = \sum_{j=1}^{d_L} h_j(w)\hat{\mu}(b_j),

where μ^(bj)\hat{\mu}(b_j) denotes the measured mean for calibration sequence bjb_j. The count dLd_L refers to distinct calibration settings; repeated acquisitions determine their precision. The specific laboratory calibration design is described below.

For the theoretical photon-noise analysis, acquisitions were modelled as independent Poisson counts with means bounded by MM. A determinant-maximizing acquisition design with N≥dLN \ge d_L observations, combined with least-squares estimation, gave worst-case mean-squared prediction error at most MdL/(N−dL+1)Md_L/(N-d_L+1).

Independent computational verification and robustness analysis

The computational researcher reproduced the symbolic checks and independently reconstructed response-space coordinates for L=5L = 5 and L=6L = 6. A complementary stress test, specified by the PI, examined whether departures from persistent pulse errors could remain statistically hidden in calibration while changing target predictions. This test fixed

L=6,K=[3/5,4/5]+i[−1/5,1/5],B=1,κ=1/100.L = 6, \qquad K = [3/5,4/5] + i[-1/5,1/5], \qquad B = 1, \qquad\kappa= 1/100.

Five phase settings with rational sine and cosine values were used, with the first pulse phase fixed. The design comprised 192 calibration sequences, including every sequence through three pulses, and 3,715 targets. A hypothetical budget of 8.8×10108.8 \times10^{10} Poisson acquisitions, with approximately one photon per acquisition, set the statistical scale.

Perturbations consisted of at most one hidden rotation about each spin’s zz axis, with angles +δ+\delta and −δ-\delta occurring with equal probability. Rotations after pulse three, four or five were considered separately, together with an unperturbed case. Calibration ambiguity was defined as the largest target-mean separation between ensembles that agreed on all short-sequence means and were statistically indistinguishable under the full calibration. The indistinguishability criterion required an optimal discrimination success probability of at most 55%.

Candidate ensemble pairs were sought numerically, and bounds were certified using exact rational arithmetic. The domain KK was subdivided into rectangles, on each of which polynomial inequalities were checked using the minimum and maximum Bernstein coefficients. This procedure certified the inequalities across the continuous parameter domain for each specified test case. A Berry–Esseen bound on sums of photon counts supplied a necessary condition for the indistinguishability criterion and was used to tighten the ambiguity bounds.

Separate certificates bounded the calibrated predictor’s squared bias and variance. Their sum defined the mean-squared prediction error, whose square root was expressed relative to σM\sigma_M, the predictor’s photon-noise standard deviation. Certification covered every target, the three rotation positions and the unperturbed case at twelve evaluated rotation strengths. Seven exact interval halvings refined the crossing of the photon-noise reference. A control retained only the 32 calibration means through three pulses and sought ensembles agreeing on those means but differing on a six-pulse target.

Experimental design and prediction locking

Experimental proposals underwent separate assessments of apparatus feasibility and scientific discrimination. The feasibility reviewer evaluated whether the available apparatus could test a central prediction, and the PI assessed whether the proposed measurements distinguished that prediction from specified alternative models.

The final design combined five three-pulse calibration sequences with five sequences of at most two pulses, including the empty sequence. Four additional three-pulse settings served as withheld targets. Phases were selected in multiples of 60°, and an exact rank calculation verified that the ten calibration responses spanned the L=3L=3 response space. The PI derived a 4×104 \times10 transfer matrix HH, with each row summing to one, to obtain target predictions as T^=HC\hat{T}=HC. Denoting calibration and target means in weighted counts per shot by C0,…,C9C_0,\ldots,C_9 and T1,…,T4T_1,\ldots,T_4, respectively, the test compared the measured contrast

S=−T1+T2+T3−T44S = \frac{-T_1+T_2+T_3-T_4}{4}

with the calibration prediction

P=34(C0−C1+C6)−14(C2+C3+C4).P = \frac{3}{4}(C_0-C_1+C_6)-\frac{1}{4}(C_2+C_3+C_4).

Both combinations cancel the common fluorescence background.

Two interpolation baselines were specified. A nine-term Fourier interpolant fitted the five three-pulse calibration means by minimizing the squared coefficient norm. A six-term alternative incorporated the symmetry of the repeated pulse. Both predicted S=0S=0 for any calibration record. The protocol additionally required a nine-parameter model of coherent ensemble evolution to be fitted to all ten calibration means.

A digital twin created by the computational researcher evaluated the decision rule using 20,000 simulated runs per case, including intended operation, a target phase misprogrammed by 60°, and cases at the accuracy-tolerance boundaries. The laboratory packet contained the protocol, data format and analysis code, and underwent a separate review before execution. Its feasibility assessment used an approximately 12-hour budget on the HPC, including calibration, without relaxing the prescribed thresholds.

Before acquisition, the PI fixed HH, the definition of PP, the accuracy tolerances, controls and outcome criteria. Initial analysis was restricted to the ten calibration means and their covariance. The protocol required the numerical law predictions, PP, both interpolation predictions and the physical fit to be locked and hashed before target analysis. For the final acquisition, the documented prediction lock occurred after data collection and before any target mean was computed. The statistical outcome rules are specified below.

NV apparatus and fluorescence readout

Measurements were performed at room temperature using a confocal microscope and a single-crystal diamond (Element Six DNV-B14) with dimensions of 3.0 × 3.0 × 0.5 mm, {100} faces, approximately 4.5 ppm NV centres and natural-abundance 13C{}^{13}\mathrm{C} (1.1%). Continuous-wave 532-nm excitation was gated by an acousto-optic modulator and focused through a 100× objective with numerical aperture 0.90. A three-axis piezoelectric stage positioned the sample. Collected fluorescence passed through a 30-µm pinhole and a filter centred at 650 nm with a 10-nm bandwidth before detection by a single-photon counter. The detected rate was typically 2.8–3.1 × 10610^{6} counts s−1^{-1}, with an imposed operating limit of 3.4 × 10610^{6} counts s−1^{-1}.

The permanent-magnet field was determined by fitting the resonance spectrum of all four NV orientations, giving a field magnitude of 4.18 mT directed 3.8° from one NV axis. Microwave excitation addressed the ms=0→ms=−1m_s = 0 \to m_s = -1 transition of this orientation near 2.7503 GHz. The carrier was tuned to the central line of the 14N{}^{14}\mathrm{N} hyperfine triplet, whose adjacent lines were separated by 2.16 MHz.

An FPGA controller synchronized optical and microwave switching and recorded photon arrivals. Programmed in-phase and quadrature waveforms controlled microwave phase and frequency offset through a mixer. A fast switch blocked microwave transmission between pulses, and an amplifier drove a gold-wire antenna beside the diamond. Waveform offsets were adjusted to suppress carrier leakage, and gain and phase imbalance were corrected using a 2 × 2 matrix, with parameters adjusted individually against spectrum-analyser readings. Qualification measurements compared Rabi oscillations at phases of 0°, 60° and 120°, and checked residual drive using a zero-amplitude waveform with the microwave switch open. Each new controller program was simulated before execution on the instrument. Microwave pulses in the final calibration and target sequences lasted 64 ns, with 152-ns intervals between pulses.

Fluorescence was recorded during a 50-µs laser pulse following microwave manipulation. Counts from three successive readout windows were combined with fixed weights of 1, 0.45 and 0.10, respectively, to obtain the weighted photon count per shot. Sequence means were expressed in these units. The same laser pulse initialized the spin ensemble for the next shot. A 1-µs dark interval completed an approximately 52-µs shot cycle, and all sequence settings used the same total shot duration.

Acquisition protocol and amendments

A preliminary measurement of the predicted identity used 20-ns inter-pulse gaps and 16 five-minute sub-runs. The first main acquisition assigned equal exposure to each sequence and comprised 36 alternating ten-minute sub-runs. Two controller programs contained complementary sets of global-phase replicas: program A applied common phase shifts of 0°, 120° and 240°, and program B applied 60°, 180° and 300°, to all pulses in each nonempty sequence. Following discussion between the manager and PI, this schedule replaced the initially proposed randomized 66-s cycles.

Failures of the global-phase and reference-pulse controls, associated with signal changes sampled at different times by the two programs, prompted amendments to the acquisition. The first run retained its inconclusive classification. A first pilot shortened the alternation to 30-s blocks and used three sequence orders. A second pilot implemented phase alternation within a single controller program, adding 60° to every pulse phase on alternate sweeps through the sequences. The pilots were classified as acquisition development, and the original decision criteria were retained.

The amended protocol adopted the within-program phase toggle and specified 720 blocks of 30 s, giving six hours of integration. Three prespecified sequence orders were arranged in 120 randomized six-block groups. Before the final acquisition, the protocol required a 20-minute diagnostic comparing the null pair: the empty sequence and a shot with microwaves disabled. These two settings exchanged positions during the diagnostic to assess their sensitivity to acquisition order. The protocol also required at least 20 min of warm-up and 240 s of operation before recording, while maintaining the laser duty cycle. A signal-level step was to be flagged if it exceeded both 1% and six standard errors.

A 32-bit integer overflow in the reference-diagnostic calculation produced an undefined standard error. The experimentalist corrected the calculation using floating-point arithmetic and reanalysed the same diagnostic data against the unchanged cutoff. The final acquisition accumulated 21,600 valid one-second records across eight segments, completing the prescribed six hours of integration.

Statistical analysis and outcome classification

For each withheld target, the prediction residual was defined as ri=Ti−(HC)ir_i = T_i - (HC)_i, where CC contains the ten calibration means and HH is the fixed transfer matrix. The primary accuracy test used ten margins,

S−0.8P,1.2P−S,0.3P+ri,0.3P−ri(i=1,…,4).S - 0.8P,\qquad1.2P - S,\qquad0.3P + r_i,\qquad0.3P - r_i\qquad(i = 1,\ldots,4).

Positive margins corresponded to agreement within 20% of PP for the measured contrast and residual magnitudes below 30% of PP for each target.

Standard errors were propagated using the complete coefficients of the underlying weighted-count means. This propagation included uncertainty in the calibration-derived predictions and the covariance between PP and each residual arising from their shared calibration data. Poisson-based variance contributions were multiplied by an excess-variance factor F∗F^*, with a floor of 1.1. Values of F∗>1.25F^* > 1.25 failed the variance-validity requirement.

Control measurements assessed global-phase invariance and consistency of the null signal. A joint χ2\chi^2 statistic compared the six global-phase replica means for each of the 13 nonempty sequences, giving 65 degrees of freedom. A separate phase-invariance check used six replicas of a single 128 ns reference pulse. The null control tested equality between the mean signal of the empty sequence and that of a shot with microwave emission disabled. Each control used a rejection threshold corresponding to a tail probability of 10−410^{-4}; the null comparison was two-sided.

Acquisitions were classified using the fixed decision rule. Support required all control and variance-validity criteria to pass, positive individual one-sided 99% lower confidence bounds for all ten margins, and a positive measured contrast SS at one-sided p≤0.001p \le0.001. Contradiction required the same validity criteria, a predicted contrast PP more than 3.09 standard errors above zero, and at least one margin with a negative one-sided upper confidence bound at tail probability 10−410^{-4}. All remaining cases were classified as inconclusive, including any acquisition that failed a control or the variance-validity requirement.

The primary classification therefore assessed agreement within the predefined tolerances. Later diagnostic analyses examined joint consistency of the residuals with zero and the dependence of SS and PP on acquisition order. These diagnostics were separate from the fixed outcome-classification rule.

Acknowledgements

This work was supported by JST FOREST Program (Grant No. JPMJFR2547). E.K. was supported by JST SPRING (Grant No. JPMJSP2180).

Author Contributions

E.M.M. conceived the study, developed the AI-agent framework and the experimental platform, integrated the framework with the experimental platform, analysed the data, and drafted the manuscript. E.K. constructed and developed the experimental platform and contributed to the structure and writing of the manuscript. M.K. constructed and configured the experimental apparatus and reviewed the manuscript. K.A. secured funding, supervised the project, and revised the manuscript. All authors reviewed and approved the final manuscript.

Competing Interests

The authors declare no competing interests.

AI Use Disclosure

The authors used generative AI agents for the research methodology, including scientific reasoning, computational analysis, experimental planning and execution, and manuscript drafting. The human authors take full responsibility for the scientific claims, data integrity, and final manuscript.

Data / Code Availability

The experimental data and analysis code supporting the findings of this study are available from the corresponding author upon reasonable request.

Open-Source Release

To make the full pipeline openly available, we release Auctopus, an open-source implementation of the multi-agent laboratory described in this work, at https://github.com/E-zClap/auctopus. It runs every stage of the workflow, from proposal generation to laboratory measurement and review, through the OpenAI and Anthropic APIs. The prompts used in this study are not included; the release provides templates to adapt to other research questions and instruments. Auctopus is distributed under the MIT licence, and we welcome its use, adaptation and contributions to its further development.

References

  1. [1]Ross D. King, Kenneth E. Whelan, Ffion M. Jones, Philip G. K. Reiser, Christopher H. Bryant, Stephen H. Muggleton, Douglas B. Kell, and Stephen G. Oliver. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature, 427(6971):247–252, 2004. doi: 10.1038/nature02236.DOI
  2. [2]Ross D. King, Jem Rowland, Stephen G. Oliver, Michael Young, Wayne Aubrey, Emma Byrne, Maria Liakata, Magdalena Markham, Pinar Pir, Larisa N. Soldatova, et al. The automation of science. Science, 324(5923):85–89, 2009. doi: 10.1126/science.1165620.DOI
  3. [3]Nathan J. Szymanski, Bernardus Rendy, Yuxing Fei, Rishi E. Kumar, Tanjin He, David Milsted, Matthew J. McDermott, Max Gallant, Ekin Dogus Cubuk, Amil Merchant, et al. An autonomous laboratory for the accelerated synthesis of inorganic materials. Nature, 624(7990):86–91, 2023. doi: 10.1038/s41586-023-06734-w.DOI
  4. [4]Benjamin Burger, Phillip M. Maffettone, Vladimir V. Gusev, Catherine M. Aitchison, Yang Bai, Xiaoyan Wang, Xiaobo Li, Ben M. Alston, Buyi Li, Rob Clowes, et al. A mobile robotic chemist. Nature, 583(7815):237–241, 2020. doi: 10.1038/s41586-020-2442-2.DOI
  5. [5]Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune. Towards end-to-end automation of AI research. Nature, 651(8107):914–919, 2026. doi: 10.1038/s41586-026-10265-5.DOI
  6. [6]Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Petar Sirkovic, Artiom Myaskovsky, Grzegorz Glowaty, Felix Weissenberger, Alessio Orlandi, Dan Popovici, et al. Accelerating scientific discovery with Co-Scientist. Nature, 655(8122):487–496, 2026. doi: 10.1038/s41586-026-10644-y.DOI
  7. [7]Ali E. Ghareeb, Benjamin Chang, Ludovico Mitchener, Angela Yiu, Caralyn J. Szostkiewicz, Dmytro Shved, Gavin J. Gyimesi, Jon M. Laurent, Samantha M. Wright, Muhammed T. Razzak, et al. A multi-agent system for automating scientific discovery. Nature, 655(8122):497–505, 2026. doi: 10.1038/s41586-026-10652-y.DOI
  8. [8]Takuya Isogawa, Ryotaro Okabe, Nutdech Phadetsuwannukun, Mingda Li, and Paola Cappellaro. Agentic AI for scientific reasoning in autonomous quantum sensing experiments. arXiv:2607.25145v1, 2026. URL https://arxiv.org/abs/2607.25145.
  9. [9]Shuxing Yang, Fujia Chen, Rui Zhao, Junyao Wu, Yize Wang, Haiyao Luo, Ning Han, Qiaolu Chen, Yuze Hu, Wenhao Li, et al. End-to-end autonomous scientific discovery on a real optical platform. arXiv:2604.27092v1, 2026. URL https://arxiv.org/abs/2604.27092.
  10. [10]Xianrui Zeng, Pengfei Liu, Yirui Zang, Yang Shen, Fei Yu, Chunlei Yu, Minghao Liu, and Yang Du. Socratic agents for autonomous scientific discovery in high-dimensional physical systems. arXiv:2606.26722v1, 2026. URL https://arxiv.org/abs/2606.26722.
  11. [11]Joeran Beel, Min-Yen Kan, and Moritz Baumgart. Evaluating Sakana’s AI Scientist: Bold claims, mixed results, and a promising future? ACM SIGIR Forum, 59(1):1–20, 2025. doi: 10.1145/3769733.3769747.DOI
  12. [12]Robert K. Merton. The normative structure of science. In Norman W. Storer, editor, The Sociology of Science: Theoretical and Empirical Investigations, pages 267–278. University of Chicago Press, Chicago, 1973. Originally published as “A Note on Science and Democracy”, Journal of Legal and Political Sociology 1, 115–126 (1942).
  13. [13]Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11):2600–2606, 2018. doi: 10.1073/pnas.1708274114.DOI
  14. [14]Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11):1359–1366, 2011. doi: 10.1177/0956797611417632.DOI
  15. [15]Andrew Gelman and Eric Loken. The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Unpublished manuscript, November 2013. URL https://sites.stat.columbia.edu/gelman/research/unpublished/p_hacking.pdf.
  16. [16]Marcus W. Doherty, Neil B. Manson, Paul Delaney, Fedor Jelezko, Jörg Wrachtrup, and Lloyd C. L. Hollenberg. The nitrogen-vacancy colour centre in diamond. Physics Reports, 528(1):1–45, 2013. doi: 10.1016/j.physrep.2013.02.001.DOI
  17. [17]C. L. Degen, F. Reinhard, and P. Cappellaro. Quantum sensing. Reviews of Modern Physics, 89(3):035002, 2017. doi: 10.1103/RevModPhys.89.035002.DOI
  18. [18]John F. Barry, Jennifer M. Schloss, Erik Bauch, Matthew J. Turner, Connor A. Hart, Linh M. Pham, and Ronald L. Walsworth. Sensitivity optimization for NV-diamond magnetometry. Reviews of Modern Physics, 92(1):015004, 2020. doi: 10.1103/RevModPhys.92.015004.DOI
  19. [19]Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations (ICLR 2024), 2024. URL https://openreview.net/forum?id=IkmD3fKBPQ. arXiv:2310.01798.arxiv.org/abs/2310.01798
  20. [20]Zhuo Li, Yupeng Zhang, Pengyu Cheng, Jiajun Song, Mengyu Zhou, Hao Li, Shujie Hu, Yu Qin, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. MARCH: Multi-agent reinforced check for hallucination. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 39389–39415, 2026. doi: 10.18653/v1/2026.acl-long.1828.DOI
  21. [21]Why AI cannot do good science without humans. Nature, 653(8115):650, 2026. doi: 10.1038/d41586-026-01551-3.DOI
  22. [22]Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can LLMs generate novel research ideas? A large-scale human study with 100+ NLP researchers. In International Conference on Learning Representations (ICLR 2025), 2025. URL https://openreview.net/forum?id=M23dTGWCzy. arXiv:2409.04109.arxiv.org/abs/2409.04109

Paper details

vlnc:W_ny6664us
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI agents

View statement
Looking for related papers…
Contents