WorksComputer Science
Verifiable pXRD: Hill-climbing pXRD structure determination with LLM agents
Abstract
Autonomous crystal-structure determination from powder X-ray diffraction (pXRD) remains a key bottleneck for self-driving materials laboratories. We formulate the task of pXRD structure determination as a verifiable agentic search problem: candidate structures can be assessed during inference through forward-simulated profile agreement and during evaluation through structural matching to a known reference. To benchmark progress, we introduce pXRD-Bench, a suite of 200 simulated single-phase problems spanning in-distribution structure determination, challenging and polymorphic structures, and controlled compositional and topological distribution shifts. Agent adaptation raises selected structural recovery from 27% to 69%, with further gains from repeated sampling and verifier-guided inference-time scaling. We extend this approach to experimental multiphase pXRD through a reflexive evolution framework that learns from self-generated synthetic problems. A replaceable laboratory specification defines the sample family and measurement conditions. An evolver agent uses the known generating structures and measurement parameters to diagnose solver failures and develop persistent guidance and executable tools, which are then provided to agents on measured patterns. This supplies verifiable training experience without requiring a collection of solved experimental measurements. On 15 measured patterns from solid-state synthesis, the evolved package improves average profile agreement and transfers an analogue-substitution procedure that recovers a phase whose structure is absent from the supplied reference library. These results demonstrate how learning from verifiable synthetic experience can improve agentic characterization of experimental samples whose structures are not known in advance.
Introduction
Materials discovery and innovation hold promise for advancing many of the world’s major challenges, including energy, compute, and sustainability [55, 16]. Certain materials have properties that enable functionalities like energy storage, electrical state control, and carbon sequestration. The distinctive properties of materials are largely determined by their atomic structure. Indeed, structure determination from diffraction and spectroscopy have been central to materials characterization and the broader materials research process, allowing materials researchers to connect synthesis, structure, and function. Insights derived from this connection have driven the improvements in solar cells, batteries, semiconductors, catalysts, and more in recent decades [6, 23].
Accelerated materials discovery through self-driving labs therefore requires autonomous structural characterization. Powder X-ray diffraction (pXRD) is a primary technique for structure determination
An earlier version of this manuscript was submitted to NeurIPS AI4Mat in August 2026. that measures how crystalline materials diffract X-rays, producing patterns that encode their phase composition, crystal symmetry, lattice dimensions, and atomic arrangement. Efforts to automate its analysis have included crystallographic software packages [15, 57]; database-based identification of known phases, including ML-assisted retrieval [39, 19]; physics-guided and global structure-search algorithms [41, 3, 54]; and, of particular recent interest, generative models in both multi-step pipelines and one-shot structure generation [46, 11, 26, 32, 37, 68] (Li et al. 2025). Despite this progress, reliable and accurate autonomous powder-diffraction structure determination remains elusive.
The recent development and rapid advancement of LLM agents have opened up a promising approach for autonomous characterization interpretation. Structure determination has historically been an adaptive investigation where scientists generate hypotheses, use established scientific tools, perform forward calculations, verify results, and form conclusions [15, 40]. Rather than relying on algorithmic search techniques or generative models to complete the task end-to-end, AI agents have the potential to perform this adaptive investigation and tackle diverse, realistic problems with greater accuracy. Recent scientific-agent systems illustrate this approach by coupling LLM reasoning with specialized models, simulation codes, and laboratory interfaces [7, 8, 74]. Related work has explored modular, reusable skills as a way to provide domain knowledge and computational capabilities without imposing rigid task-specific workflows [44, 17]. Within materials characterization, agentic systems have begun to automate XRD phase identification [39, 14], X-ray Rietveld refinement through AgentBuild [51], neutron Rietveld refinement [37], microscopy-to-theory workflows [65], and molecular structure elucidation [18]. Here, we formulate pXRD structure determination as a verifiable task and demonstrate its suitability for modern agent adaptation and inference-time scaling techniques, as well as provide comprehensive benchmarking against existing baselines and strong general-purpose coding agents.
Following our initial submission to NeurIPS AI4Mat 2026, parallel works AutoXRD [62] and Gan Jiang [10] also demonstrated rigorous benchmarking and designed/learned skills for agents on pXRD, and Periodic Labs reported improvements in experimental multiphase interpretation through model post-training and increased inference-time computation [45]. Woo et al. [2026] also introduced PXRD-Bench for evaluating generative structure models.
We identify two ways in which pXRD structure determination is verifiable—at inference-time and at evaluation-time—and apply multiple techniques to leverage this verifiability to improve performance. While verifiability makes this problem suitable for reinforcement learning [34, 25, 69], we demonstrate meaningful performance improvements through agent adaptation [31] and inference-time scaling techniques that can be applied easily to frontier closed-weight models. We benchmark our methods on a primary 200-problem benchmark of single-phase patterns and a 20-problem representative subset for the inference-time scaling techniques. Our adapted agent achieves the highest structure-recovery accuracy among the methods evaluated on pXRD-Bench, and inference-time compute scaling yields further gains on the 20-problem panel. We extend the applicability of our findings by introducing a reflexive evolution framework that learns from self-generated synthetic training problems that achieves superior refinement and phase recovery capabilities on a 15-problem multiphase panel from the Precursor Genome [58], as well as providing ablations that give useful insights about the value of evaluation-time verifiability, the limits of learned skills/guidance, and best practices for using agents to evaluate pXRD patterns. Code, benchmark data, and per-experiment result artifacts are available at: https://github.com/sache07/verifiable-pxrd.
pXRD structure determination as a verifiable task
Structural characterization problems can be posed as inverse problems. Given characterization measurements and potentially some experimental parameters, the goal is to determine the underlying atomic structure. For pXRD structure determination, we consider the following inverse problem.
Given a chemical composition , an X-ray wavelength , and a measured intensity profile , infer a periodic crystal structure , described using a CIF.
This problem is a verifiable task, making it particularly amenable for performance hill-climbing through both agent adaptation and inference-time scaling. A reliable, inexpensive verifier provides a stable objective for hill-climbing through updates to agent model policy, tools, or scaffolding [52, 64]. This generate–verify–update loop has been central to the rapid advancement in LLM agent capabilities in coding and mathematics, where checking a candidate solution is often substantially easier than producing one [38, 29]. During inference, it can rank parallel rollouts, guide targeted revisions, and allocate additional compute to insufficient solutions[9, 53, 50]. Figure 1 summarizes the inverse problem and its inference- and evaluation-time verification pathways.
Figure 1. pXRD structure determination as a verifiable inverse problem. Ground truth: pXRD-Bench case pxrd100-rep-010. Proposed: pxrd-codex selected.cif. Structures rendered with Crystal Toolkit [28].
In particular, pXRD structure determination is verifiable in two distinct senses:
Inference-time verifiability: When a diffraction measurement is taken of a sample, the sample’s true structure diffracts X-rays in such a way as to produce the measured profile. This ground truth structure is unknown, so success can be verified by calculating the forward model , where is the intensity at , denotes experimental and profile nuisance parameters, and captures measurement noise and physical effects not represented by the forward model. This forward calculation is compared with the measured profile, often by computing the weighted profile residual [47]. A low residual establishes that a candidate is consistent with the measurement, a strong verifier of solution quality. It cannot, however, establish the absence of equally or more diffraction-consistent structures.
Evaluation-time verifiability: When evaluating the success of a pXRD structure solver, there is often a ground truth structure solution. Generative models are trained on pairs of diffraction profiles and the true structures that generated them, and benchmarking often evaluates the ability of a solver/solution-generator to generate a known reference structure given its diffraction profile. Success is verified by the similarity of the proposed structure and the ground truth structure. pymatgen’s StructureMatcher is commonly used for this comparison, accepting structures as equivalent if, after applying a coordinate transformation that accounts for equivalent cell choices, basis transformations, and origin shifts, lattice-length mismatch is below , the lattice-angle mismatch is below , and the maximum normalized corresponding atomic site displacement is below .
The agent-adaptation and inference-time scaling techniques we utilize and benchmark in our results take advantage of both types of verifiability in various ways to improve performance. We utilize the evaluation-time structure-matcher verification as our primary metric of success in our benchmarking, following precedent in the literature and to compare against baseline methods.
| Corpus | Category | Construction and intended test | |
| pXRD-100 | Representative | 30 | Distribution-matched sample of eligible MP-20 structures, balancing crystal system, primitive-cell size, chemical complexity, stability, density, and elemental prevalence. |
| pXRD-100 | Challenge | 30 | Six structures selected along each of five predefined difficulty axes: rare symmetry, site and chemical complexity, peak congestion, scattering ambiguity, and large primitive cells. |
| pXRD-100 | Polymorph | 40 | Twenty structurally distinct pairs with identical reduced compositions, stratified by diffraction profile similarity |
| pXRD-ood | Substitution | 25 | Composition-novel structures obtained by full-orbit isovalent substitution of an exposed structural parent. |
| pXRD-ood | Ordered decoration | 25 | Composition-novel structures obtained by ordered decoration of symmetry orbits in an exposed structural parent. |
| pXRD-ood | Wyckoff enumeration | 25 | Topology-novel structures with exposed compositions generated by symmetry-constrained Wyckoff enumeration. |
| pXRD-ood | Crystal search | 25 | Topology-novel structures with exposed compositions obtained from MACE-MH-1 crystal-search |
Table 1. Composition of pXRD-Bench. pXRD-100 measures recovery of structures represented in the training distributions of existing generative methods, whereas pXRD-ood separates compositional extrapolation from structural-topology extrapolation.
Results and Discussion
Agents for single-phase pXRD: pXRD-Bench
pXRD-Bench
To establish a uniform measure of solver performance and enable comparison between existing methods, our presented agent frameworks, and future approaches, we present pXRD-Bench. pXRD-Bench comprises 200 single-phase structure-determination problems divided equally between an in-distribution corpus, pXRD-100, and an out-of-distribution corpus, pXRD-ood. This suite is distinct from the independently developed PXRD-Bench of Woo et al. [ [61], which evaluates generative models under controlled training splits and scores candidates before downstream refinement. For each problem, the solver receives a reduced chemical composition, the X-ray wavelength, and a simulated intensity profile , and must return a complete periodic structure as a CIF. The reference structure, source identifier, lattice parameters, space group, formula-unit count, and benchmark category remain hidden during inference. All case-selection criteria, summarized below and detailed in Appendix § 7.§ 4, were fixed before running any evaluated solver.
The pXRD-100 corpus contains 100 ordered Materials Project structures drawn from the frozen MP-20 training split [30, 63]. Every structure contains at most 20 sites in its primitive cell and is also an audited strict training-set match for Crystalze, PXRDDGen, deCIFer, and DiffractGPT, and plausibly exposed to the other frameworks. [46, 32, 11] (Li et al. 2025) pXRD-100 is therefore intentionally an in-distribution benchmark rather than a held-out test of novelty. However, the challenge and polymorph panels were designed to stress-test the solvers, as recent work has shown that peak overlap, weak elemental contrast, and light-atom localization remain difficult for learned pXRD methods [Li et al., 2025], while evaluation on same-composition polymorphs helps establish that predictions are conditioned on the diffraction signal rather than primarily on a composition–structure prior [49].
The pXRD-ood corpus instead tests two operationally distinct forms of extrapolation. Novelty is evaluated relative to a frozen exposure boundary containing the Materials Project, the Crystallography Open Database (COD), JARVIS, and the released training corpora or retrieval indices associated with the evaluated structure-determination systems [30, 24, 12, 46, 32, 11, 19] (Li et al. 2025). Table 1 summarizes the benchmark composition by category.
When benchmarking other solvers for this comparison, the aim was to allow them to perform to the best of their ability. The diffraction inputs given to each system were calculated from the same reference CIFs in their respective native representations using inputs that reproduce each method’s † deCIFer receives the known conventional-cell formula-unit count , matching its native conditioning. ‡ PXRDGen and RealPXRD-Solver receive the known primitive-cell ; these are oracle-input comparisons.
| Method | Rep. | Chall. | Poly. | Sub. | Ord. | Wyck. | CSS | pXRD-100 | pXRD-ood | Overall |
| Crystalyze | 5 | 9 | 2 | 4 | 4 | 6 | 2 | 4 | 0 | 1 | 0 | 0 | 0 | 1 | 11 | 19 | 2 | 6 | 6.5% | 12.5% |
| DiffractGPT | 5 | – | 0 | – | 4 | – | 5 | – | 1 | – | 0 | – | 0 | – | 9 | – | 6 | – | 7.5% | – |
| Ab-PXRD-Solver | 21 | – | 5 | – | 16 | – | 9 | – | 9 | – | 1 | – | 1 | – | 42 | – | 20 | – | 31.0% | – |
| deCIFer† | 9 | 12 | 3 | 9 | 6 | 11 | 4 | 5 | 3 | 5 | 0 | 0 | 2 | 2 | 18 | 32 | 9 | 12 | 13.5% | 22.0% |
| PXRDGen‡ | 21 | 30 | 11 | 22 | 15 | 32 | 10 | 12 | 7 | 13 | 0 | 0 | 0 | 1 | 47 | 84 | 17 | 26 | 32.0% | 55.0% |
| RealPXRD-Solver‡ | 20 | 26 | 8 | 15 | 13 | 21 | 12 | 16 | 7 | 10 | 0 | 0 | 1 | 2 | 41 | 62 | 20 | 28 | 30.5% | 45.0% |
| GPT-5.6 Sol (high) | 15 | – | 6 | – | 9 | – | 11 | – | 6 | – | 2 | – | 5 | – | 30 | – | 24 | – | 27.0% | – |
Table 2. Exact structural recovery of existing pXRD structure-determination methods on pXRD-Bench. Category and corpus entries report pass@1 | pass@8 as the number of recovered structures.
documented training and/or evaluation preprocessing (see Appendix § 7.§ 5). This includes wavelength convention, angular or reciprocal-space grid, profile model, broadening, noise model, and discretization pulled from the papers or source code. When benchmarking our agent frameworks, we forward-simulated the diffraction inputs with GSAS-II using a fixed, benchmark-specific profile intended to represent an idealized, background-subtracted laboratory measurement [57]. The simulation uses monochromatic Cu Kα1 radiation with Å and span to in at intervals, yielding 8,501 intensity values. The constant-wavelength pseudo-Voigt profile uses , , , , polarization 0.7, and , together with isotropic broadening from a 1 µm crystallite size and 1000 ppm microstrain (parameter conventions are defined in Appendix § 7.§ 4). Preferred orientation, sample displacement, and background are disabled. Each clean profile is scaled to a maximum of 5,000 counts, sampled with Poisson noise, augmented with Gaussian noise of standard deviation five counts, and finally normalized to a maximum intensity of 100.
We evaluate submitted structures using pymatgen’s StructureMatcher [43]. We use pymatgen’s default geometric tolerances, , , and , while disabling lattice rescaling to require recovery of the unit-cell dimensions.
As expected, exact recovery was consistently higher on pXRD-100 than on pXRD-ood. On pass@1, PXRDGen, Ab-PXRD-Solver, and RealPXRD-Solver form a narrow leading cluster at 30.5–32.0% overall, although PXRDGen and RealPXRD-Solver receive the oracle primitive-cell . Ab-PXRD-Solver and DiffractGPT do not have pass@8 results because Ab-PXRD-Solver carries out substantial search and comparison producing one final candidate, while DiffractGPT uses a greedy decoding algorithm that results in identical samples. However, these methods show significant gains from best-of-8 sampling. Notably, GPT-5.6 Sol remains competitive at 27.0% overall and 24% pXRD-ood on pass@1 with a simple prompt, no access to code/tools, and no oracle knowledge. This suggests that frontier LLMs possess strong crystallographic knowledge and reasoning, and in the rest of the paper we explore techniques utilizing them to reach even higher performance.
Agent adaptation
Agent adaptation improves a fixed language model by changing the harness around it—its action interface, accessible tools and data, and reusable task knowledge—rather than its parameters. Prior work shows that interleaving reasoning with environment actions and designing agent-facing computer interfaces can materially change what an LLM can accomplish [67, 64]. Complementary approaches use task feedback to distill experience into persistent linguistic memory without weight updates [52, 71]. pXRD structure determination is especially amenable to this form of adaptation: established crystallographic programs provide precise operations for indexing, simulation, and refinement, while exact structure matching turns completed attempts into unambiguous supervision. We therefore construct a controlled adaptation ladder around the same frontier model. Starting from tool-free prediction relying solely on LLM reasoning, we add a coding-agent harness with a general materials-science Python environment and crystallographic database access (matsci-codex), then access to GSAS-II and DICVOL14 (pxrd-codex), and finally a compact frozen capability synthesized from externally verified attempts on a disjoint 50-problem curriculum (adapted-codex). This design separates improvements due to agent scaffolding, domain tools, and learned nonparametric capability from changes to the underlying model.
| Condition | Rep. | Chall. | Poly. | Sub. | Ord. | Wyck. | CSS | pXRD-100 | pXRD-ood | Overall |
| GPT-5.6 Sol (high) | 15 | 16 | 6 | 6 | 9 | 11 | 11 | 12 | 6 | 6 | 2 | 2 | 5 | 5 | 30 | 33 | 24 | 25 | 27.0% | 29.0% |
| matsci-codex | 25 | 26 | 14 | 14 | 21 | 21 | 15 | 15 | 12 | 12 | 7 | 7 | 10 | 10 | 60 | 61 | 44 | 44 | 52.0% | 52.5% |
| pxrd-codex | 22 | 23 | 12 | 12 | 24 | 24 | 20 | 20 | 16 | 18 | 8 | 8 | 12 | 12 | 58 | 59 | 56 | 58 | 57.0% | 58.5% |
| adapted-codex | 27 | 27 | 22 | 22 | 28 | 28 | 19 | 19 | 21 | 21 | 10 | 12 | 11 | 12 | 77 | 77 | 61 | 64 | 69.0% | 70.5% |
Table 3. Exact structural recovery across the agent-adaptation ladder on pXRD-Bench. Entries report selected exact | any exact, where any exact permits a match from either selected.cif or alternative-<n>.cif.
All conditions received the same simple prompt, with the exception of GPT-5.6 Sol (high) receiving “from the below pattern” instead of “from pattern.csv” and the complete diffraction profile as prompt text because the no-tools Responses API condition truncates .csv files, and adapted-codex receiving a prefix saying to use its capability package.
Benchmark prompt
Solve the crystal structure from pattern.csv. The composition is reduced composition. The X-ray wavelength is wavelength angstrom (radiation). Save your primary structure as selected.cif and, if necessary, up to a few surviving alternatives as alternative-<n>.cif.
Moving from tool-free GPT-5.6 Sol (high) to matsci-codex increases selected-structure recovery from 27.0% to 52.0%, a gain of 25 percentage points. Because the model weights are fixed, this comparison measures the joint value of a stateful coding-agent substrate, a scientific Python environment, and crystallographic database access. In place of a single model response, the agent can maintain workspace state, inspect intermediate results, and synthesize case-specific programs that compose established materials-science libraries. This improvement requires substantially more test-time computation: the mean reported input-plus-output usage was 6.09 million tokens over a mean wall-clock duration of 25.3 minutes among completed matsci-codex rollouts that didn’t hit the 1-hour limit (), compared with 161,293 tokens and 6.94 minutes across the 200 single-call, tool-free GPT-5.6 Sol (high) condition solves. Adding crystallography-specific software in pxrd-codex yields a further five-point increase in overall recovery, from 52.0% to 57.0%. This gain is concentrated in pXRD-ood, where recovery rises from 44% to 56%, while pXRD-100 changes from 60% to 58%, within the range of stochastic variance. This moderate gain demonstrates how access pXRD structure determination software is beneficial for agent performance, but also how coding agents are already able to consistently carry out complex crystallographic tasks like cell-indexing, charge flipping, and Rietveld refinement using custom scripts.
A modern paradigm in agent adaptation is experience-driven self-evolution: an agent executes a curriculum of tasks, receives external feedback on its rollouts, and distills recurring successes and failures into persistent prompts, memories, skills, or scaffold code that changes its future behavior without updating the base model. Reflexion and ExpeL [52, 71] established this pattern for linguistic memory. Voyager [59] accumulated reusable executable skills through environment interaction, and more recent systems such as GEPA, Agentic Context Engineering (ACE), and the Self-Improving Coding Agent (SICA) [1, 70, 48] optimize increasingly general forms of agent state—from prompts and evolving playbooks to the agent implementation itself—using evaluated trajectories. Verifiability allows for easier learning during these processes by allowing richer and more direct feedback from trajectories.
We instantiate this experience-driven loop to create adapted-codex using a trainer–trainee framework and a disjoint 50-problem curriculum sampled from the same distribution as pXRD-Bench. The trainer launches a blind pxrd-codex trainee on each curriculum problem, analyzes their trajectories and outcomes, designs a capability package comprising natural-language skills and optional executable tool adapters, and iteratively reruns problems and revises the capability pack to optimize performance. The trainer had access to the ground-truth structures as well as the structure-matcher verifier, meaning evaluation-time verifiability was utilized as a learning signal. The resulting adapted-codex capability pack contained just (1) a 139-word skill and (2) a DICVOL adapter that improves the agent’s understanding of raw DICVOL input—a minimal surface as intended by our trainer prompt. More details on the training process and resulting package are provided in Appendix A.3.1.
adapted-codex achieved 69.0% exact matches from selected.cif on pXRD-Bench, significantly higher than pxrd-codex and the other conditions. Our framework is a relatively simple and efficient implementation of self-evolution, in which we find limiting the package surface achieved better performance than allowing the package to sprawl and attempt to correct for each observed mistake. In practice, a significant portion performance gain can be attributed to trainer-written guidance in the skills about the nature of the task and verifier; this guidance leads behavior to be better aligned with verifier scoring, analogous to verifier-guided prompt optimization and reinforcement learning with verifiable rewards [1, 34, 25]. These results with our proof-of-concept approach establish pXRD structure determination as a suitable domain for experience-driven agent adaptation and motivate more involved reflective optimizers and post-training techniques.
Inference-time scaling
Inference-time scaling improves a fixed agent by allocating additional computation to each problem rather than modifying its parameters or persistent capabilities. The simplest approach samples independent solution trajectories and aggregates or ranks them after generation [13, 60, 9]. More structured methods make this computation adaptive: they extend or revise active trajectories, branch and backtrack over partial solutions, and direct search using learned value functions, executable feedback, or formal verifiers [42, 66, 27, 73, 53, 50]. We demonstrate the potential of inference-time scaling on agentic pXRD structure determination through repeated sampling on a 20 problem representative subset of pXRD-Bench, which we denote pXRD-Bench-20 (stored as inference-scaling-20), as well as results demonstrating the value of LLMs as verifiers/judges for adjudication and coverage-directed continuation. Full experimental details including panel construction, rollout, verifier and token-accounting protocols are given in Appendix A.4.
Panels (a), (b), and (c) in Figure 2 show the results of sampling 8 solution trajectories for each of the 20 pXRD-Bench-20 problems. Some problems are consistently solved or consistently missed, but many are occasionally solved, which allows repeated sampling to consistently recover a correct solution. Our results in Figure 2(d) shows expected candidate-set match rate scaling with repeated sampling across three initial-performance tiers, with adapted-codex Sol (high) reaching 90% at pass@8 vs 68.8% pass@1. These gains require substantial but highly parallelizable test-time compute. Across 160 pxrd-codex attempts per model, Sol averaged 18.5 minutes per rollout (49.4 aggregate agent-hours; 2 timeouts), whereas Luna averaged 29.8 minutes (79.5 agent-hours; 29 timeouts). Each benchmark worker was allocated 2 vCPUs and 5 GB of memory and capped at one hour. Notably, the expected GPT-5.6 Luna (max) pass@5 performance exceeded GPT-5.6 Sol (high) expected pass@1 performance at an estimated 30% lower expected token cost.

Figure 2. Repeated-sampling results on pXRD-Bench-20. Panels (a)–(c) show candidate-set exact StructureMatcher recovery for pxrd-codex Sol, pxrd-codex Luna, and adapted-codex Sol, respectively. A colored square denotes a candidate-structure match; the rightmost Exact column counts these matches. Panel (d) shows order-invariant expected oracle pass@k, computed by averaging cumulative candidate-set coverage over all subsets of the observed rollouts.
pXRD structure determination is particularly amenable to inference-time scaling because of its inference-time verifiability—the measured pattern provides a reference-blind selection signal. Repeated sampling alone improves only oracle coverage—the probability that at least one trajectory produces the correct structure. Realizing this gain requires an inference-time verifier that can recognize the correct candidate without access to the hidden reference. In domains without reliable verifiers, majority-vote and learned-reward selection can saturate even while oracle coverage continues to increase, and scaling best-of- selection against an imperfect learned reward can eventually improve the proxy score while degrading ground-truth performance [9, 21]. Here, every proposed structure induces a physical prediction that can be compared directly with the observed diffraction profile. Across eight independent adapted-codex Sol trajectories on our 20-problem panel, the candidate pool contained an exact structure in 18 cases, and a lowest- selector with bounded nuisance-profile fitting ranked an exact structure first in all 18. Thus, on this panel, -based verification converted the complete oracle pass@8 gain into top-1 performance.
We also utilize an LLM verifier to scale inference-time compute more efficiently. Despite being a powerful discriminator between solutions, there is no clear cutoff that determines if a solution is correct [56]. We use an LLM agent to review the provided structures and traces of an initial solve attempt and determine if the structures are likely to be valid solutions, determining as well as performing more involved, qualitative analysis. The verifier was tuned using GEPA-style prompt evolution [1], in which a seed policy given in the prompt that emphasized fit quality, physical plausibility, and unresolved structural ambiguity was iteratively revised through reflections on labeled training errors, with the optimization retaining a Pareto frontier of candidate prompts across case-level performance. The final policy was selected on held-out validation trajectories and frozen before benchmark evaluation; full details are provided in Appendix § 7.4.
On the 200 adapted-codex pXRD-Bench solves, the evolved-prompt verifier correctly decided stop on 103/141 correct solves and continue on 52/59 missed solves. Compared with the seed prompt, the evolved verifier reduced the false-continue rate from 51.8% to 27.0%, while increasing the false-stop rate from 1.7% to 11.9% (Figure 3a). Retroactively-picked cutoffs on the same bounded-fit , chosen to match the false stop rate of the LLM verifiers, produce more false continues, despite being generous comparisons rather than realistic de novo verifiers, demonstrating the LLM verifiers make decisions more nuanced than what only considering scalar can afford. Figure 11 in Appendix A.5 shows the presence of structure misses with lower shared fixed-parameter than many successful matches.

Figure 3. Inference-time verifier accuracy and token-cost-scaled recovery. (a) Decisions on 200 adapted-codex pXRD-Bench trajectories. False stops are incorrect structures judged complete; false continues are exact structures allocated another solve. The thresholds were selected retrospectively to match the two LLM policies’ false-stop counts. (b) Verifier-gated continuation framework plotted against the 8 independent adapted-codex Sol reruns. The eight independent reruns points are placed at expected token cost for rollouts. The verifier-gated curve shows cumulative recovery after cumulative token cost following the initial blind pass, a coverage-directed continuation, and two verifier-authorized fresh solves.
Figure 11. Candidate-level fixed-parameter distributions under the shared fixedparams-v1 GSAS-II forward model. Blue and red distributions denote candidates accepted and rejected, respectively, by the benchmark structure matcher. Solid lines indicate means and dashed lines indicate medians. Both reference and candidate profiles are clean, independently maximum-normalized simulations; no fitting, alignment, or refinement is performed. The horizontal scale and forward model are shared across all panels.
Additionally, we tested verifier-gated continuation as a more efficient way to leverage inference-time compute than naively repeating solves. Starting from the first 20 adapted-codex solutions on pXRD-Bench-20, cumulative recovery rose from 13/20 to 17/20 after a verifier-authorized coverage-directed solve (given the reason field of the verifier output) and to 18/20 after one verifier-authorized fresh solve; a second fresh solve added no further exact candidate. Including the initial 20 solves, their reviews, and 12 coverage-directed continuations, the first two gated stages cost $28.97 more than two complete independent passes. As correctly solved problems were subsequently stopped, the third and fourth stages cost $12.16 and $23.29 less, respectively, than allocating one additional solve to all 20 problems.
Agents for experimental multiphase pXRD: learning from self-generated synthetic problems
In addition to the simulated single-phase problems in pXRD-Bench, we also evaluate agents on experimental multiphase patterns. These patterns arise during powder synthesis experiments and are relevant to materials discovery efforts. In a powder synthesis experiment, the compositions of the starting materials (precursors) are known, but the product of the solid-state reaction can contain contain multiple phases, including the intended product (but not necessarily), residual precursors, intermediates, competing phases, etc. Its characterization must establish which crystalline phases are present, construct defensible atomic models, and estimate their proportions. This information determine whether a synthesis succeeded and what should change in the next experiment, an important piece of feedback in the materials discovery loop.
We study this setting using 15 measured patterns from the Precursor Genome, a collection of solid-state reactions with associated experimental metadata and deposited refinements [58]. The selected family comprises reactions between and sources of Mg, Ca, Sr, Ba, Pb, Zn, Cu, Co, Ni, or Mn, fired at 200–900 °C. Sources include oxides, carbonates, and a peroxide. The measurements span largely unreacted powders, partially reacted mixtures, and product-dominated samples; 13 of the 15 deposited interpretations are multiphase. We refer to this panel as divalent–tungstates–PG15.
The solving agent receives the measured counts, the two precursor formulas, the firing temperature, and the radiation and instrument description, but not the product composition, the number of phases, a calibrated peak-profile function, or the deposited interpretation. This is intended to mimic the information known to a materials scientist / crystallographer when analyzing a pXRD measurement, a substantial change from the single-phase benchmark: the agent must infer the phase inventory while estimating instrument and specimen effects that influence which phases appear to explain the pattern. Its deliverables are phase CIFs, crystalline weight fractions, a calculated profile, a report, and the final GSAS-II project [57]. Appendix A.6 gives the baseline solver configuration and task prompt.
A reflexive evolution framework that learns from self-generated synthetic problems
Similar to the agent adaptation to create adapted-codex on pXRD-Bench, we ask whether reflexive evolution can improve agent performance. However, unlike pXRD-Bench problems where representative training problems could be sampled from the same distribution as the evaluation problems, experimental pXRD datasets do not necessarily come with additional representative patterns nor would those patterns come with ground-truth solutions. Real samples don’t have ground truth CIFs—an expert-determined structural model, if available, is the closest substitute and may still be imperfect. We develop a framework for this setting, allowing agents to improve performance beyond their initial baselines and the deposited expert-determined solutions. We ask whether synthetic problems, for which the generating structures and measurement parameters are known, allow an evolver/trainer to utilize its access to the ground-truth solutions to learn prose guidance and create executable tools that improve analysis of experimental mixtures when provided to agents. Our framework learns persistent guidance and optional code from these problems while keeping the language-model weights fixed. The framework separates the adaptation procedure from a replaceable laboratory specification (Figure 4). Groups of samples synthesized and characterized in a (potentially autonomous) materials research lab often form a well-defined family, linked by a shared structural prototype, a common synthesis route, or a target chemical class—as with the divalent metal tungstates in divalent-tungstates-PG15. The laboratory specification in our framework allows a researcher to define the sample family, precursors, chemically plausible outcomes, measurement geometry, ranges for instrument / specimen effects, and information available to a solver. For the tungstate family, it describes nominally 1:1 metal-to-tungsten precursor mixtures, a single firing step, and laboratory Cu K Bragg–Brentano measurements. Paired with a synthetic generator, an evolver can curate and create synthetic training structural models within the laboratory scope. In the case of our experiments, the synthetic generator combines COD structures [24] into chemically balanced mixtures and simulates patterns with the researcher-defined counting noise and varied instrument and specimen effects. By changing the laboratory specification and sample generator, agents can undergo a reflexive learning process for a new experimental setting and sample family without rewriting the evolution mechanisms.
Figure 4. Synthetic experience for experimental characterization. A laboratory specification defines the scientific setting and permitted synthetic variation. Fresh solver workspaces receive only public measurements and metadata. A persistent evolver can inspect trajectories and use synthetic reference information to diagnose failures, revise a capability package, and test it on further solves. The frozen package is subsequently applied to measured patterns. Ground-truth access is confined to synthetic research and evaluation; experimental fits retain the solving agent’s own model and calibration.
A persistent evolver agent curates six initial problems and runs three independent baseline attempts on each. It reads the resulting scripts, projects, trajectories, and verifier diagnostics, then proposes a small package of guidance and, when useful, executable helpers. Its role extends beyond rewriting a prompt: it can inspect failed numerical procedures, perform controlled tests on completed projects, and use the known generating structural model and scorer (exact-content verifiability) to identify what the solver missed. The solver workspaces remain reference-blind. Both roles use GPT-5.6 Sol at maximum reasoning effort; a stronger teacher model is not introduced, meaning our results should apply to the frontier of released models.
The adaptation protocol screens evolved packages with one solve per active problem. Cases that succeed can be set aside temporarily while unresolved cases receive further revisions, followed by regression checks. A promising package is evaluated with three solves per case, a choice made due to observed rollout variability. The initial run allows 30 development solves per cycle, with additional baseline, transfer, and evaluation repeats accounted for separately. At budget exhaustion, a package is promoted based on its improvement on the scores compared to the baseline or incumbent package, and new synthetic problems are curated for further evolution and learning. Details about the evolution/algorithm can be found in the repo.
What the synthetic verifier measures. The quality of a solution in the setting of pXRD can be assessed both in terms of profile agreement—how well the structural solution’s predicted diffraction pattern reproduces the input—and structural agreement—how close the structures and phase weights are to the generating ground-truth. These criteria are complementary. Weak sensitivity to light-atom positions, overlapping reflections, and weak minor-phase contributions can leave multiple models consistent with the available diffraction information. Evaluation based solely on correspondence with the generating model may therefore penalize plausible alternatives that the measurement cannot distinguish. Conversely, a lower profile residual does not necessarily indicate a more accurate description of the sample: additional phases or flexible background, texture, and structural parameters can absorb noise or compensate for an incorrect model. Our scorer design assesses profile agreement alongside recovery of the known synthetic solution and the physical plausibility of the inferred sample model.
The verifier evaluates the submitted GSAS-II project without further refinement, retaining the solver’s structures, background, and instrument and specimen parameters. Profile agreement is measured against the hidden noise-free generating pattern using mean Poisson deviance ; a structural score assesses phase identities, atomic structures, and weight fractions. The combined score is , subject to physical-validity checks. While the solver receives only noisy observations, the evolver can use these reference-based diagnostics to distinguish missing phases and structural errors from inadequate profile modeling.
Evolution of phase-recovery and refinement procedures
The initial evolver-curated curriculum comprised six synthetic problems spanning single- and multiphase samples, low counts, weak residual phases, and deliberately withheld reference structures. Across the full run, the evolver completed 135 solver attempts on ten synthetic problems and produced 15 package revisions.
The first revision addressed a phase-identification failure: the baseline missed the withheld intermediate in all three attempts. The evolver introduced a helper that searches for chemically analogous structural prototypes, allowing the solver to construct candidates when an exact reference is unavailable. Subsequent revisions addressed coupled errors in phase selection and profile modeling. Ground-truth-assisted tests identified misassigned broadening as a major source of error, but translating this diagnosis into reliable solver guidance required several revisions and initially caused regressions. The retained package, v6, combined the prototype-search helper with 331 words of guidance on source-structure selection, reconsideration of weak phases, instrument and specimen effects, and validation of saved solutions.
On the six development problems, v6 increased the mean composite score from 50.04 to 74.07 across three attempts per problem, improving every case mean and recovering the complete generating phase inventory in 17 of 18 attempts (Figure 5). Its initial single-attempt screen had scored 82.19, illustrating the variability obscured by evaluating each revision only once. v6 was promoted at the 30-solve development-budget limit.
Figure 5. Evolution improved the initial curriculum but was neither monotonic nor uniformly reliable. (a) Case-level scores for the baseline, single-attempt revision screens, and the retained v6 package. Gray cells were not evaluated under that revision. The v6 mean includes its initial screen and two additional attempts per case; it is a repeated development evaluation, not an independent held-out test. (b) Individual baseline and v6 scores, with larger dots denoting case means. (c) Initial-panel means and prospective comparisons on two subsequent pairs of new synthetic cases, each with three attempts per condition and no package edits before that comparison. Scores are the composite .
The next 2 batches each introduced two new problems, evaluated with the unchanged Baseline and Evolved v6 conditions. Mean scores increased from 58.79 to 72.44 and from 43.02 to 68.97, providing evidence of transfer beyond the initial curriculum. Further revisions produced local gains and regressions, but none replaced v6. Details on the evolution framework including scoring formula, generator implementation, and package promotion rules are provided in Appendix § 7.6; evolver-written notes, execution logs, and the entire implementation are available in the repository.
Evolved agent performance on divalent-tungstates-PG15
We evaluated the frozen v6 package on the 15 measured patterns, with three independent attempts per measurement. The Baseline and Evolved v6 conditions use the same GPT-5.6 Sol model alias at maximum reasoning effort, baseline task prompt, Codex harness, crystallographic software, and reference library; Evolved v6 additionally receives the v6 guidance and prototype-search helper. Each condition therefore contributes 45 attempts. We compare their results with one deposited PG solution per measurement, withheld from both agents.
Uniform evaluation. For each attempt, the evaluator reopens the final GSAS-II project and calculates its profile without further refinement, retaining the submitted structures, background, calibration, and specimen parameters. We refer to the independently recomputed residual as “verified” ; this verifies the reported profile agreement, not the structural interpretation. We score these replayed profiles and the saved PG calculated profiles against the same measured counts over – in , using with . The comparison measures profile agreement under each analysis’s fitted model. The deposited models provide a useful baseline and comparison but not a structural ground truth. For each agent condition, we average verified attempts within a measurement before calculating the mean and median across the 15 measurements. Appendix § 7.7 gives the per-attempt scores and evaluation details.
The evolved package lowers mean from 7.90% to 6.71%, a relative reduction of 15.1%, and improves the measurement-level mean on 10 of 15 scans (Table 4, Figure 6). The largest improvements occur on Sr and Zn. Mean duration increases from 27.0 to 50.2 minutes, so the package changes both the analysis and the amount of computation the agent devotes to it. The deposited profiles have a mean residual of 12.59% and a median of 10.22%; their mean is particularly affected by the Cu deposit’s 45.05% residual. v6 has a lower mean residual than the deposit on 13 of the 15 measurements, although the phase inventories and fitted parameters also differ.
| Condition | Mean (%) | Median (%) | Mean best-of-3 (%) | Within-scan SD (pp) | Mean time (min) |
| Deposited PG | 12.59 | 10.22 | — | — | — |
| Baseline | 7.90 | 7.20 | 6.92 | 1.04 | 27.0 |
| Evolved v6 | 6.71 | 5.86 | 6.04 | 0.80 | 50.2 |
Table 4. Transfer from synthetic experience to measured patterns. Mean and median summarize the 15 measurement-level mean residuals. Best-of-three averages the lowest among three attempts on each measurement. Within-scan SD is the mean sample standard deviation across measurements, in percentage points (pp). Time is mean wall time per attempt. Lower residuals indicate better profile agreement.
Figure 6. Experimental transfer of the v6 package. (a) Agent points show the mean verified for each measurement; horizontal lines span the observed attempts, not a confidence interval. Each PG point scores the saved deposited profile. The horizontal axis is logarithmic. Labels give the case and second precursor; every reaction also contains . (b) Change in the measurement-level mean from baseline to v6; negative values indicate improved profile agreement. Only the initial three attempts per measurement are included.
Refinement of an established phase inventory. The Zn measurement (015) illustrates how v6 changes the refinement procedure (Figure 7). Both agents identify , , and residual . The baseline’s three residuals span 5.49–13.64%, whereas v6’s span 4.82–5.89%. In the illustrated v6 attempt, the agent replaces a coarse source containing a short O–O contact with a neutron-derived record, uses an eight-coefficient background, and tests crystallite size and microstrain separately. It retains a microstrain term and removes an unstable size contribution. These changes follow the package’s guidance to screen source structures and distinguish common profile errors from phase-specific effects. The trajectory supports a more systematic refinement procedure and better profile agreement.
Figure 7. Same phase inventory, improved profile agreement: the Zn measurement. Baseline and evolved attempt 3 on scan 015, shown as an illustrative trajectory comparison rather than a best-of-three selection. All curves are taken from saved independent project replays. Observations are black points, calculations are colored lines, and differences are gray with a vertical offset. The right panels resolve the -rich region. Both fits contain , , and ; the evolved attempt changes the source structure and profile treatment. Intensities share the observed maximum as a display normalization, while reported uses raw counts over the full evaluation window.
Phase recovery and disagreements with the deposited interpretations. The Sr measurement (014) provides a different kind of gain. The deposited structure is absent from the supplied reference library. In two of its three attempts, v6 constructs a matching model by substituting into a prototype retrieved with the learned helper. These attempts assign 24.9–25.9 wt% to the phase, close to the deposited 24.8 wt%, and achieve residuals of 7.12% and 7.88%. The baseline recovers it in none of its three attempts.
Table 5 summarizes selected differences in phase inventory or fraction among the baseline, v6, and PG deposits. On Cu, both agents consistently fit two phase models, dominated by , whereas the deposit contains seven. Appendix Figure 13 compares their calculated profiles. On Mg and Zn, both agents repeatedly include residual precursor oxides absent from the deposits; on Ca, all three interpretations include carbonate, but the agents assign about twice its deposited fraction. Conversely, neither baseline nor v6 recovers the deposited structure. Differences between recovered phase inventories don’t indicate for certain which (or if any) of the solutions are correct, but a lower provides a useful signal.
| Case | Deposited PG | Baseline | Evolved v6 |
| 003 (Ca) | : 13.5 wt%<br> | : 3/3<br>27.7–33.8 wt%<br> | : 3/3<br>28.4–30.2 wt%<br> |
| 006 (Cu) | 7 phase models; : 68.1 wt%<br> | 2 phase models in 3/3<br>: 97.7–97.9 wt%<br> | 2 phase models in 3/3<br>: 96.8–97.5 wt%<br> |
| 007 (Mg) | : absent<br> | : 3/3<br>7.0–7.6 wt%<br> | : 3/3<br>7.1–7.4 wt%<br> |
| 008 (Mn) | : absent<br> | : 3/3<br>2.0–2.1 wt%<br> | : 3/3<br>2.0–3.3 wt%<br> |
| 013 (Pb) | : 15.3 wt%<br> | : 0/3<br> | : 0/3<br> |
| 014 (Sr) | : 24.8 wt%<br> | : 0/3<br> | : 2/3<br>24.9–25.9 wt%<br> |
| 015 (Zn) | : absent<br> | : 3/3<br>6.9–7.6 wt%<br> | : 3/3<br>7.1–7.5 wt%<br> |
Table 5. Selected differences in phase assignment and quantity. Agent entries give the number of initial attempts containing the indicated phase and the range of its reported crystalline weight fraction where present. averages all three attempts, including those without the phase. The Cu row instead gives the total number of phase models and the dominant-phase fraction. Sr and Pb presence requires a structural match to the deposited reference structure; other rows use the submitted phase formula. The table highlights substantial disagreements rather than listing every trace phase.
Framework access and disentangling the value of reflexive evolution
Despite the encouraging gains resulting from our reflexive evolution on self-generated problems framework, we run additional experiments to understand what gains can be attributed to the learning process and what gains can be attributed to the additional information included in the evolution framework but unavailable to the baseline solvers. The Framework access condition gives the baseline read-only access to the framework’s starting directory: the laboratory description, generator and scorer code, and other process specification / infrastructure. Two further conditions separate combining these resources from evolving the framework-access agent itself. Framework + v6 supplies framework access together with the original v6 package. Evolved framework v5 supplies framework access with a 205-word v5 package from a separate evolution run that began from the framework-access baseline. That package contains no additional helper and was not promoted by its development gate; its experimental evaluation is exploratory. All five conditions share the model alias, reasoning effort, baseline task prompt, harness, scientific software, measurements, and reference library.
Framework access lowers mean to 5.98%; it improves on the baseline on all 15 measurements and on v6 alone on 12 (Table 6, Figure 8). Adding the v6 guidance/tools to framework access raises the mean to 6.49%, and the separately evolved v5 yields 6.44%. Both improve on framework access on only four measurements. Thus the best average fit in this comparison comes from direct access to the scientific context and code, without an evolved package. We first examine the framework-access gains and their origin. Section 2.4.1 then highlights a phase-recovery capability elicited by evolution despite poorer average profile agreement, and Section 2.4.2 discusses the value of reflexive evolution for pXRD and in general, considering the relative results of each evaluated condition.
| Condition | Mean (%) | Median (%) | Mean best-of-3 (%) | Mean time (min) |
| Baseline | 7.90 | 7.20 | 6.92 | 27.0 |
| Evolved v6 | 6.71 | 5.86 | 6.04 | 50.2 |
| Framework access | 5.98 | 5.49 | 5.47 | 66.1 |
| Framework + v6 | 6.49 | 6.36 | 5.68 | 67.0 |
| Evolved framework v51 | 6.44 | 6.15 | 5.65 | 43.8 |
Table 6. Scientific context and evolved guidance. Summary conventions follow Table 4. Best-of-three averages the lowest among three attempts on each of the 15 measurements; the later Pb/Sr repeats are excluded.
Figure 8. The effect of framework access varies across measurements. (a) Points show the mean verified ; horizontal lines span the observed attempts. Labels give the second precursor and firing temperature; every reaction also contains . (b) Differences from the baseline’s measurement-level mean. The framework-access condition improves on baseline on every scan, but is not the best condition on every scan. Only the initial three attempts per measurement are included.
The framework-access trajectories show active use of the supplied material, including inspection of the simulator and evaluator and execution of reference-blind replay utilities. On several measurements the final phase inventory is much the same as in the baseline, but the agent tests a more adequate profile or structural model. For the Cu measurement (006), mean falls from 6.80% for Baseline to 5.40% for Evolved v6 and 3.76% with framework access. The dominant assignment remains CuWO with a small WO contribution. On Mn measurements 008 and 009, framework access similarly reaches means of 3.92% and 3.95%, with changes involving atomic parameters, broadening, and low-angle intensity corrections.
The final GSAS-II projects provide a second view of these behavioral changes. The median number of background coefficients is 15 for Baseline, 8 for Evolved v6, and 16 for framework access. The axial-asymmetry parameter differs from its standard 0.002 starting value in 12/45, 17/45, and 44/45 projects, respectively. These are observations about submitted models, not isolated causal effects: a parameter can be fitted and then fixed, and a more flexible background can reduce without resolving a missing phase. Nevertheless, the two interventions lead to visibly different refinement choices. v6 consistently favors a restrained background and explicit tests of physical contributions, possibly due to the synthetic patterns being constrained to what the generator can create, leading to tendencies that don’t transfer positively to real patterns; framework access more often encourages broad exploration of the available model.
The framework-access results suggest that much of the improvement over the baseline comes from giving the solver access to scientific knowledge already encoded in the experimental setup, rather than learning new procedures through evolution. The generator exposes implementations of background, peak broadening, and specimen effects, while the replay utilities provide a way to check the resulting
fits. The trajectories show agents inspecting and using this code to test changes to their structural and profile models. v6 distills some of the same information, together with experience from synthetic rollouts, into a short set of recommendations. This improves the baseline, but may also favor refinement choices suited to the synthetic curriculum rather than the measured patterns. Direct framework access leaves the solver more freedom to decide which assumptions and operations are useful for each measurement while still providing useful context about the samples and success measures. Having access to this information induces longer solves and more persistence/care, even more so than the guidance learned by the reflexive evolution process.
Transfer of a missing-phase recovery procedure
Average profile agreement does not capture the clearest additional contribution of evolution. The deposited PG solution for the Sr measurement (014) includes a phase whose exact structure is absent from the supplied reference library. Neither the original baseline nor framework-access attempts recover a structurally matching model, but solvers with packages learned from the evolution process were able to discover this phase and achieve significantly lower compared to even the best solutions without the phase. In a successful evolved attempt, coherent residual reflections motivate a search across Sr-rich tungstate stoichiometries. The agent uses the learned helper to retrieve orthorhombic (COD 1529756), substitutes with and with , and tests the resulting model. Its scale is positive before cell refinement. The final three-phase interpretation contains approximately 66.1 wt% , 24.9 wt% , and 9.0 wt% , with verified of 7.12% (Figure 8). The submitted structure matches the deposited reference structure under the evaluator’s structural criterion.
This is a transfer of a procedure, not memorization of the experimental phase. The synthetic failure that motivated the helper involved a intermediate and a parent; the experimental solve requires a different stoichiometry and a different substituted prototype. The helper makes a composition-led route through the structural library explicit. It neither supplies the target CIF nor resolves the mixture without further hypothesis testing and refinement. The agent’s own report also distinguishes the strongly supported phase and cell from prototype-inherited coordinates and reports that its fraction uncertainty exceeds the formal least-squares error.
Figure 9. The Sr measurement exposes a difference in the hypotheses explored. The illustrated baseline r1 and framework-access r2 are their lowest- attempts in the original three-repeat panels; evolved r2 is the successful prototype-transfer example discussed in the text. Framework access fits several weak features using a larger phase set but does not recover SrWO. The evolved attempt constructs SrWO from a BaReO prototype and fits a three-phase mixture. Dashed lines mark diagnostic reflection positions identified in that trajectory, not a separate reference-assisted fit. All plotted profiles are frozen project replays; display and residual conventions follow Figure 7. The repeat frequencies in Table 7 provide context for these selected examples.
Targeted repeats. After observing recovery differences in the initial panels, we collected six additional attempts for every condition on the Sr measurement, reported on Table 7. The repeats demonstrate how trends in the initial 3 attempts are not fully prescriptive—v6 recovers the phase in 3/9 attempts rather than the initial 2/3, the combined framework-plus-v6 condition recovers it in 5/9, the independently evolved framework-access v5 recovers it in 1/9 despite the initial 1/3. However, a clear behavioral trend persists: the baseline and framework access agents both recover the SrWO phase in 0/9 attempts, with no attempts reaching below 10 in (besides an outlier 8.98 that fit 8 phases), while each of conditions that included evolved packages were able to discern the SrWO phase at varying rates and produced solutions with SrWO clustered around 6-8 (besides an outlier at 13.40 that got trapped in a poor lattice/mixture during refinement).
| Condition | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Total |
| Baseline | 12.89 | 16.48 | 16.72 | 13.99 | 13.53 | 12.54 | 18.68 | 16.71 | 15.27 | 0/9 |
| Evolved v6 | 14.66 | 7.12 | 7.88 | 12.97 | 12.90 | 14.67 | 7.32 | 15.77 | 15.34 | 3/9 |
| Framework access | 14.65 | 8.98 | 11.73 | 11.131 | 11.14 | 11.36 | 13.28 | 10.21 | 12.90 | 0/9 |
| Framework + v6 | 15.85 | 13.04 | 7.53 | 13.44 | 7.19 | 6.82 | 6.93 | 13.40 | 13.08 | 5/9 |
| Evolved framework v5 | 6.64 | 15.23 | 15.59 | 15.22 | 14.38 | 12.97 | 13.07 | 13.18 | 12.93 | 1/9 |
Table 7. Profile agreement and phase recovery on the Sr measurement (014). Entries are verified values (%). Bold entries indicate a submitted structure matching the deposited phase; Total counts matches among the nine completed solves. Initial attempts belong to the full-panel experiment; added attempts were collected after inspecting those results. All repeats concern the same specimen.
This points to the reflexive learning from self-generated synthetic problems process being able to elicit helpful behaviors that extend beyond the additional context/knowledge embedded in the evolution framework. Oracle-knowledge of answers to the synthetic problems allows the same model as the solvers (equal intelligence) to reason about why the solvers fail and design guidance and tools to push them towards the approaches that would lead to the right solution. However, the worse overall profile-agreement performance of the conditions with evolved packages available to them compared to the framework-access condition demonstrates important limitations of reflexive evolution.
What reflexive evolution adds to an informed agent for pXRD
Framework + v6 evaluates whether the evolved package can combine its phase-recovery capability with the stronger refinement performance of framework access. Recovery of increases from zero to five of nine completed attempts, but mean across the experimental panel rises from 5.98% to 6.49% (Table 6 and Table 7). The package therefore encourages a productive phase-search route while also changing the analysis in ways that worsen average profile agreement. Interpreting this tradeoff requires examining what the package contributes beyond the information already available in the framework.
The two framework-evolution campaigns assess whether guidance developed specifically for this better-informed framework-access agent can improve its performance. Neither produced a promoted package (Figure 10). The earlier campaign used the same screening and promotion criterion as the evolution process as evolved v6, revising guidance on unresolved cases while setting aside successful cases. It produced revisions through v8 but never completed a successful validation of one package across the panel; v9 remained unscored when the run was stopped. The later campaign with a slightly modified screening/promotion process in the evolution framework evaluated every revision three times on each active case, completing 156 solves across baseline evaluations and seven revisions. Every revision exceeded the permitted case-level regression. Local improvements were repeatedly accompanied by losses elsewhere.
Figure 10. Two framework-access evolution campaigns retained their starting agent. (a) Early selective screening: baseline cells average three solves; revision cells show single solves. Gray cells were not scored. Passing cases were set aside; combining different revisions does not validate a single package. (b) Later 3-repeat evaluation: each revision has three solves on six cases. Changes are relative to the baseline on the same cases. Mean gains must reach the plotted threshold and every case must remain above the dashed regression limit. Vertical separators mark panel changes; the boxed v5 was also tested experimentally. Appendix A.8 gives the protocols and scores; further details can be found in the verifiable-pxrd repo.
The unpromoted v5 from the second campaign illustrates this difficulty in generalizing improvements across cases. Its synthetic mean increased by 2.96 score points, but a 38.50-point gain on a missing- problem was offset by a 37.62-point regression on a difficult Zn refinement. Its exploratory experimental evaluation on divalent-tungstates-PG15 reproduced the broader tradeoff: mean increased to 6.44% relative to framework access, although one of nine Sr attempts recovered the missing phase. Thus, both adding the original v6 package and evolving a new package from framework access yielded useful phase-recovery behavior without improving overall fitting performance.
The contrasting outcomes of baseline and framework-access evolution suggest that a substantial part of the initial opportunity lies in making existing scientific knowledge and task requirements usable by the solver. The framework contains descriptions of the sample family and expected challenges, implementations of relevant physical effects, and explicit evaluation criteria. Framework-access trajectories show agents inspecting and using this material to improve their analyses. Evolution can distill some of that information into guidance, whereas direct access allows the agent to consult the underlying context as the analysis develops. A related pattern appears in the single-phase experiments: the compact adapted-codex package significantly improved performance by clarifying the submission contract and improving the DICVOL14 interface, without providing guidance on solution pathways. Failures arising from missing task context or tool-interface difficulties can admit compact corrections that apply across many problems.
Improving an agent that already has this context presents a different optimization problem. Decisions about phase search, background flexibility, atomic coordinates, and specimen effects depend on the current fit and on one another. Guidance derived from a particular failure must remain useful when those circumstances change. A restrained background can preserve evidence of a weak phase in one pattern and underfit the baseline in another; further prototype search can resolve a missing phase or divert effort from refining an adequate inventory. The evolved packages impose preferences on these decisions. Evolved v6 consistently favors a lower-dimensional background, while the framework-evolved v5 limits optional search and explicitly discourages inspection of the generator and scorer. These prescriptions provide a plausible explanation for why guidance that improves the original baseline can constrain the more informed agent.
These results suggest that the benefit of reflective adaptation depends on which failures remain amenable to instruction changes, a result reflected in the reflexive evolution literature. Akbar et al. [ [2] report substantially smaller gains from GEPA for a stronger model with higher initial performance, attributing this pattern to reduced opportunity for prompt-level correction. Analyses of learned agent skills also identify gains from restoring workflow instructions and supplying task-specific knowledge absent from the initial configuration [33]. Once this information is available, further improvement requires guidance that generalizes across heterogeneous decisions. Gao et al. [ [22] show why this can be difficult: rollout variability can obscure differences between prompts, while changes beneficial to some problems harm others. Our framework-access evolution campaigns exhibit these patterns, managing to improve performance with 3-rollout evaluations during evolution, but struggling to find generalizable instructions when the low-hanging fruit of task-alignment and providing context disappear. Together, these observations support distinguishing adaptation that clarifies how to perform and submit the task from adaptation that improves the scientific procedures themselves.
For agentic pXRD analysis in real laboratories, our work suggests providing informative scientific context and evaluating multiple independent solves before investing in extensive prompt evolution (just selecting the lowest- baseline solve from three independent attempts reduces mean from 7.90% to 6.92% on divalent-tungstates-PG15 Table 4). Additional reasoning effort and model choice also influence performance: mean decreases from 8.98% for the earlier Sol-high baseline to 7.90% for Sol-max, while the Opus comparison reaches 6.32%. Appendix § 7.7 reports the differing runtimes, harnesses, and repeat counts. Framework access and evolution provide additional means of improving a fixed model when a higher reasoning level or a stronger model is unavailable.
The distinctive contribution of synthetic experience is the ability to investigate failures using information unavailable during experimental analysis. Knowledge of the generating phases allows the evolver to establish that a solution omitted a component, identify why its search failed, and test an alternative procedure. In this study, that process produced an analogue-substitution procedure that transferred from a synthetic calcium intermediate to the measured Sr mixture. An expert could also have suggested that route when presented with the baseline solve traces and high s. The broader opportunity arises when promising routes are not known in advance: self-generated, verified, reproducible problems can provide experience from which to discover and validate them, a motivation shared with Absolute Zero [72].
Conclusion
LLM agents can perform end-to-end pXRD structure determination by combining scientific reasoning, executable analysis, and feedback from physical models. On the 200-problem pXRD-Bench, adding a coding harness, crystallographic software, and verifier-informed adaptation raises selected exact recovery from 27% to 52%, 57%, and 69%, respectively. On the 20-problem inference-scaling panel, repeated adapted-agent solves supply an exact candidate for 18 problems, and profile-based selection recovers all 18. These results demonstrate the complementary roles of evaluation-time structural verification in adaptation and inference-time profile verification in search and selection.
Experimental multiphase characterization requires a broader notion of success: identifying phases, constructing plausible structures, estimating their fractions, and explaining instrument and specimen effects. Our laboratory-conditioned synthetic framework supplies verifiable experience for this setting without requiring ground-truth experimental solutions. The evolved v6 package improves mean profile agreement over its baseline and transfers an analogue-substitution procedure to a measured mixture whose SrWO structure is absent from the supplied library. Direct access to the framework produces still better average fits, while further evolution exposes a tradeoff between phase recovery and routine refinement. Together, these results distinguish two sources of improvement: scientific context helps an agent use its existing capabilities more effectively, while verifiable synthetic experience can develop specific procedures that context alone does not reliably elicit.
The multiphase experiments extend our central argument beyond problems with known reference solutions. Although experimental measurements rarely provide definitive ground truth, synthetic problems supply the information needed to diagnose failures, test alternative strategies, and learn procedures that can transfer to measured samples. The recovery of an absent-library phase in the Sr mixture demonstrates this possibility. For laboratory deployment, our results support providing scientific context and comparing independent solves, then directing adaptation toward persistent failures. A replaceable laboratory specification makes this approach applicable to recurring challenges within a sample family without requiring a large collection of solved experimental measurements. Its scientific value ultimately rests on improving phase identification and structural interpretation together with agreement to the measured diffraction pattern.
Additional experimental details
pXRD-Bench construction
pXRD-100: controlled training exposure. pXRD-100 controls for documented training exposure. Candidates came from the MP-20 training split at frozen CDVAE commit f857f598d6f6cca5dc1ea0582d228f12dcc2c2ea and were required to have a numeric Materials Project identifier, full ordered occupancy, 1–20 primitive-cell sites, energy above hull at most 0.08 eV atom, and formation energy below 2 eV atom [63]. Each selected structure also matched a released Crystalyze training record, a public deCIFer training record, and the later official DiffractGPT training split, yielding 3,649 structures jointly exposed to these systems and MP-20-trained PXRDGen [46, 32, 11] (Li et al. 2025). DiffractGPT exposure is a proxy because its paper-era split is unavailable. RealPXRD-Solver was trained on structures from the Materials Project; however, because its exact training split was unavailable, we could not assess case-level overlap with pXRD-Bench. Ab-PXRD-Solver instead conducts crystallographic search rather than learning a structure generator, so exact membership in a structure-training split is not defined for its core solver, although its learned auxiliary models may still be sensitive to distribution shift.
After retaining the lowest-energy eligible structure per formula, the representative panel was moment-matched to MP-20 over symmetry, cell size, chemical complexity, stability, density, volume per atom, and elemental prevalence. The challenge panel contains six cases from the top decile of each of five prespecified difficulty scores: rare symmetry, site and chemical complexity, peak congestion, scattering ambiguity, and primitive-cell size. The polymorph panel contains 20 same-composition pairs stratified by diffraction-profile similarity into five easy, ten medium, and five hard pairs. Eligibility and selection were fixed without using evaluated-model outputs.
A frozen public-exposure boundary for pXRD-ood. Out-of-distribution status is certified relative to a specified public boundary, not asserted universally. Before selection, we hash-locked a 3,279,483-record SQLite index comprising the released Crystalyze, CDVAE/PXRDGen, deCIFer, DiffractGPT, and DARA corpora together with frozen JARVIS-DFT, COD, and Materials Project releases [30, 24, 12, 46, 32, 11, 19] (Li et al. 2025). When checkpoint provenance was incomplete, we conservatively indexed a public superset, including complete NOMA, JARVIS, and COD snapshots. Source versions, counts, and hashes are retained in the benchmark manifests. Inaccessible data, notably DARA’s licensed ICSD collection, remain outside this certificate.
Separating compositional and topological extrapolation. Every pXRD-ood candidate first had to be absent as a species-aware structure from the exposure index under StructureMatcher with , , and , primitive-cell reduction, and lattice rescaling. Rescaling prevents a volume-only perturbation from establishing novelty. For the 50 composition-novel cases, the reduced composition is absent from every source while the species-erased geometry matches an exposed parent under stricter tolerances (0.15, 3°, 0.20). For the 50 topology-novel cases, the composition occurs in all five learned-model sources and at least one retrieval source, but StructureMatcher.fit_anonymous finds no match among indexed structures with the same anonymized formula. These certificates respectively isolate unfamiliar chemistry on an exposed scaffold and unfamiliar topology at familiar chemistry.
Generation, filtering, and final selection. We generated 496 candidates by isovalent substitution (100), ordered decoration (100), PyXtal Wyckoff enumeration (140), and MACE-MH-1-relaxed crystal search (156); 403 passed structural-validity, diffraction-quality, and novelty checks [20, 4, 5]. A deterministic mixed-integer program selected 25 cases per stream, balanced each stream across five primitive-cell-size bins, enforced unique compositions and parents, and limited repeated crystal systems, space groups, diffraction profiles, and species-erased geometries. Formulas are disjoint from pXRD-100, and neither selection nor filtering used baseline performance. An independent audit then recomputed all certificates from the released CIFs and frozen index. The certificates establish operational novelty and well-formed targets, not stability, synthesizability, or absence from every inaccessible database; simulator hashes and noise seeds are retained in the manifests.
GSAS-II profile-parameter conventions. For constant-wavelength profiles, GSAS-II stores the Gaussian variance coefficients , , in squared centidegrees and the Lorentzian width coefficients , (and constant term ) in centidegrees, where one centidegree is in . is the dimensionless Finger–Cox–Jephcoat axial-divergence parameter controlling peak asymmetry. Crystallite size is specified in , and microstrain in parts per million of relative reciprocal-spacing broadening; thus 1000 ppm corresponds to 0.1% [57].
Recovery evaluation. Unlike the novelty check, final recovery scoring disables lattice rescaling because cell dimensions are part of the solution. It otherwise uses , , and , with primitive-cell reduction, supercell matching, no subset matching, ElementComparator, and evaluation in both comparison directions [43]. Exact@1 evaluates the declared primary structure; exact@ is an oracle match among the first candidates. Missing or unparseable CIFs are failures.
Specialist pXRD baseline protocols
The specialist comparison holds the reference structures and exact-recovery evaluator fixed, but preserves each framework’s native diffraction representation, conditioning variables, and generation procedure. Recalculating the same reference CIF under each released preprocessing pipeline avoids evaluating a trained checkpoint on an artificial common input distribution. Consequently, the rows in Table 8 are matched evaluation experiments, not equal-information ablations.
| Method | Native diffraction input | Structural side information | Candidates | Generation and reporting protocol |
| Crystalyze | Cu K Caglioti–Gaussian encoder profile | Element set; stoichiometric counts not enforced | 8 | Released diffusion sampler with allowed-element constraints; samples are reported in generation order. |
| DiffractGPT | Released 180-bin nearest-bin vector | Reduced composition | 1 | One greedy decode with a 1,024-token generation limit; no stochastic best-of-eight policy is introduced. |
| Ab-PXRD-Solver | Released noisy GSAS-II profile | Reduced composition; space group inferred | 1 | Deterministic search with Halton quasi-random sampling and a four-hour CPU limit per case; a candidate emitted before timeout remains eligible. |
| deCIFer | Clean 1,000-bin -space pseudo-Voigt profile | Full conventional-cell composition (oracle ) | 8 | Stochastic autoregressive sampling at temperature 1 and top- = 50; raw samples are reported in generation order. |
| PXRDGen | Frozen GSAS-II Cu K profile | Exact primitive-cell composition and atom count (oracle ) | 8 | 200 diffusion steps with step size 5.0; the reported structures are raw generated samples without the paper’s subsequent Rietveld stage. |
| RealPXRD-Solver | Variable-length native sparse peak list | Exact primitive-cell composition and atom count (oracle ); no lattice | 20 (first 8 reported) | 200 diffusion steps with step size 5.0; candidate order is released sampler order because the public GSAS-II ranking/refinement stage is unavailable. |
Table 8. Reproduction protocol for the specialist pXRD baselines. Oracle inputs are derived from the hidden reference structure and therefore provide more information than the reduced-composition benchmark prompt.
Candidate generation and scoring. For Crystalyze, deCIFer, and PXRDGen, exact@1 is the first generated sample and exact@8 is oracle candidate-set recall over samples 1–8. RealPXRD-Solver generates 20 structures per case, but the main comparison uses the same first-eight cutoff; its best-of-20 result is retained only as an auxiliary oracle diagnostic. DiffractGPT follows its released deterministic greedy helper, so repeated calls under the frozen input produce the same output and no exact@8 value is reported. Ab-PXRD-Solver performs an internal search and emits at most one final structure. No method is reranked with the shared verifier, and missing, invalid, or unparseable outputs count as failures on the original 200-case denominator.
Output completeness and validity. Table 9 distinguishes missing or invalid outputs from structures with the wrong composition. Basic validity requires a parseable CIF, cell volume at least Å, and minimum periodic interatomic distance at least Å; these checks do not establish physical plausibility. Composition matching compares reduced elemental proportions. Crystalyze’s element-set conditioning does not enforce stoichiometric counts: all 1,600 candidates pass the validity checks, but only 351 match the required composition. Conversion or parsing fails for 41 DiffractGPT outputs and 41 deCIFer samples. PXRDGen and RealPXRD-Solver return valid, composition-matching structures in nearly all evaluated slots, so their exact-recovery shortfall predominantly reflects incorrect structures rather than missing or invalid outputs.
| Method | Candidate slots | Valid structures | Valid and composition-matched |
| Crystalyze | 1,600 | 1,600 | 351 |
| DiffractGPT | 200 | 146 | 138 |
| Ab-PXRD-Solver | 200 | 160 | 160 |
| deCIFer | 1,600 | 1,500 | 1,406 |
| PXRDGen | 1,600 | 1,592 | 1,592 |
| RealPXRD-Solver | 1,600 | 1,587 | 1,587 |
Table 9. Output completeness and validity before additional refinement, pooled across all 200 problems. Counts refer to candidate slots, not recovered problems. We use one slot per problem for DiffractGPT and Ab-PXRD-Solver, and the first eight for the other methods, including RealPXRD-Solver. Missing outputs remain in the slot denominator.
Ab-PXRD-Solver produces structures for 160 problems (77 pXRD-100 and 83 pXRD-ood). Of the remaining 40, 23 reach the four-hour limit without a candidate (15 pXRD-100 and eight pXRD-ood), and 17 terminate without a viable cell–space-group solution (eight and nine, respectively). Another 56 runs reach the limit after producing a candidate; these structures remain eligible and are included among the 160. The 40 missing outputs remain failures in the 200-problem recovery denominator; only the 160 available structures can enter the additional refinement experiment below.
Reproduction-specific caveats. DiffractGPT states that Gaussian broadening was used but does not specify its width or release the paper-era preprocessing code; we therefore use the later released 180-bin nearest-bin representation, spanning – in increments. Ab-PXRD-Solver’s regex-only composition parser required parenthesized formulas to be flattened to chemically equivalent integer element counts in 17 cases from each corpus; neither composition nor stoichiometry changed. For deCIFer, a reduced-composition diagnostic recovered only 2/200 cases at exact@8, whereas the paper-native full-cell conditioning reported in the main text supplies the hidden conventional-cell . PXRDGen and RealPXRD-Solver likewise receive the hidden primitive-cell formula and atom count; 54/100 pXRD-100 targets have a primitive-cell formula that differs from the reduced formula, making these materially oracle-assisted conditions. Exact code revisions, checkpoint hashes, random seeds, per-case inputs, generated CIFs, and verifier outputs are retained in the released experiment bundles.
Method-specific refinement of specialist predictions
We refine the saved specialist predictions to measure changes in exact structural recovery and profile agreement. Each method uses its original diffraction input and its own refinement procedure. The completed experiments include eight candidates per problem for Crystalyze and PXRDGen, 20 for RealPXRD-Solver, and the 160 available Ab-PXRD-Solver predictions described above. Reference structures are used only for post-hoc scoring.
Refinement procedures. All three CPU procedures use GSAS-II revision 5806 [57]. For PXRDGen, we execute the released two-pass refinement functions, fitting cell, scale, and coordinates, with conventional-cell conversion between passes and symmetry detection at symprec=0.5 [Li et al., 2025]. We adapt the file paths and supply the benchmark observation, excluding the example’s reference-structure analysis. Each refinement call allows 100 cycles, with a 30-minute limit per candidate. For Ab-PXRD-Solver, we use its released staged optimizer with 20 cycles per cell and a one-minute limit [54]. Its early-exit rule skips atomic and microstrain refinement if the profile/cell fit remains above . We add an export of the fitted phase because the released routine returns the input CIF path. Both methods retain the default GSAS-II uncertainty-based CIF formatting, including export failures caused by rounding.
For RealPXRD-Solver, we implement the sequence described in Supplementary Figure S4 of Li et al. [2026a]: background, scale, profile, cell, zero shift, coordinates, then joint refinement. This is our adapter; the inspected release did not include the authors’ refinement implementation. Saved peak lists are broadened with 0.10°-FWHM Gaussians on a 5°–80° grid at 0.01° spacing, using Å. We use unit weights, P1, fixed occupancies and Å, and one fixed highest-atomic-number atom to set the origin. Each stage allows 20 cycles, with 15 minutes per candidate. A stage is rolled back if the recalculated residual worsens or the geometry/profile bounds are violated.2
Crystalyze / StructSnap. The GPU experiment refines eight saved candidates per problem using the first stage of their released differentiable routine [46]. It fits cell lengths and peak width but not coordinates, minimizing negative cosine similarity against the unmasked 8,500-point target. This stage permits 30,001 iterations with native early stopping; each candidate has a one-hour CUDA limit. We repair an obsolete pymatgen import, save the best peak-width tensor with the best structure, and run one candidate per batch. Since we report only the first refinement stage and omit the subsequent fallback schedules with alternative initializations, this should be considered an exploratory experiment, not a reproduction of their released StructSnap routine. Including a second coordinate-refinement stage lowered exact matches from 60 to 23 among 1,067 candidates, but we did not evaluate the additional lattice-refinement schedules alluded to in the paper because of their computational cost.
Recovery and failure handling. We apply the benchmark’s composition, validity, and symmetric StructureMatcher checks to the exported CIFs. A completed refinement replaces its original candidate even if it loses an exact match. On an execution or export failure, the original is retained. Table 10 reports this refinement-with-fallback procedure using the original candidate order and all 100 problems per corpus. Pass@8 is oracle candidate-set coverage, not -ranked selection.
Table 10: Exact recovery before → after method-specific refinement, out of 100 problems per corpus. Execution/export failures retain the original candidate. Crystalyze results use the first StructSnap stage.
| Method | Corpus | Pass@1 | Pass@8 |
| Crystalyze / StructSnap | pXRD-100 | 11 → 12 | 19 → 20 |
| Crystalyze / StructSnap | pXRD-ood | 2 → 2 | 6 → 6 |
| PXRDGen | pXRD-100 | 47 → 32 | 84 → 67 |
| PXRDGen | pXRD-ood | 17 → 6 | 26 → 17 |
| Ab-PXRD-Solver | pXRD-100 | 42 → 41 | — |
| Ab-PXRD-Solver | pXRD-ood | 20 → 23 | — |
| RealPXRD-Solver adapter | pXRD-100 | 41 → 29 | 62 → 62 |
| RealPXRD-Solver adapter | pXRD-ood | 20 → 18 | 28 → 27 |
Table 10.
First-candidate recovery increases from 13/200 to 14/200 for Crystalyze, decreases from 64/200 to 38/200 for PXRDGen and from 61/200 to 47/200 for the RealPXRD-Solver adapter, and increases from 62/200 to 64/200 for Ab-PXRD-Solver. Among all 20 RealPXRD-Solver candidates, oracle recovery changes from 100/200 to 97/200.
Execution failures and retained matches. Table 11 separates matching refined outputs from matches retained only because refinement failed. PXRDGen’s 38 first-candidate successes comprise 27 refined matches and 11 retained originals (nine pXRD-100 and two pXRD-ood). Neither Ab-PXRD-Solver nor the RealPXRD-Solver adapter gains a first-candidate match through fallback.
| Method | Exported/ available | Failures | Refined | Retained original |
| Crystalyze / StructSnap | 1,600/1,600 | 0 | 14 | 0 |
| PXRDGen | 1,225/1,600 | 375 | 27 | 11 |
| Ab-PXRD-Solver | 159/160 | 1 | 64 | 0 |
| RealPXRD-Solver adapter | 3,975/4,000 | 25 | 47 | 0 |
Table 11. Execution and fallback accounting across both corpora. Export/failure counts cover all available candidate slots. The final two columns count exact first candidates out of 200 problems and sum to post-refinement pass@1. Export completion does not establish structural correctness or optimizer convergence.
PXRDGen has 273 unreadable exports, 87 inter-pass symmetry-conversion failures, 13 CIF-import failures, one numerical failure, and one composition-changing export. Ab-PXRD-Solver has one unreadable export. The RealPXRD-Solver adapter has 24 unreadable inputs and one structure with no reflections in the observation range. Rejected individual RealPXRD-Solver stages restore the state accepted state and are recorded separately from full failures. A readable Ab-PXRD-Solver export also loses one match because rounding changes its parsed site multiplicity; this is scored as a completed but invalid structure rather than replaced by the original.
Paired profile agreement. We independently simulate each original/refined CIF pair against the method-specific observation using identical fixed profile and displacement parameters before and after. Only a nonnegative scale and four Chebyshev background coefficients are fitted, with unit weights. For Crystalyze, PXRDGen, and Ab-PXRD-Solver the profile width is initialized from observed peaks; the RealPXRD-Solver adapter uses the fixed width. These residuals measure exported-structure fit separately from the optimizer’s reported objective. Absolute values should be compared within a method, since observations and forward models differ. Unlike the clean-reference distributions in Appendix § 7.5, this diagnostic compares structures against the observation and fits scale/background.
| Method | Corpus | Mean (%) | Median (%) | |
| Crystalyze / StructSnap | pXRD-100 | 100 | 93.35 → 80.93 | 95.66 → 86.23 |
| pXRD-ood | 100 | 94.57 → 87.68 | 95.40 → 92.66 | |
| PXRDGen | pXRD-100 | 76 | 89.47 → 51.10 | 93.79 → 51.60 |
| pXRD-ood | 71 | 93.10 → 82.10 | 96.17 → 92.09 | |
| Ab-PXRD-Solver | pXRD-100 | 76 | 41.47 → 30.50 | 36.54 → 17.70 |
| pXRD-ood | 83 | 54.66 → 46.12 | 53.08 → 30.67 | |
| RealPXRD-Solver adapter | pXRD-100 | 100 | 94.71 → 85.52 | 97.68 → 96.34 |
| pXRD-ood | 99 | 96.98 → 94.02 | 98.06 → 98.04 |
Table 12. Profile agreement before → after refinement. Here counts successfully exported first candidates with both residuals available; the same candidates contribute on each side. Failed refinements and missing originals are excluded from this paired table, but remain in recovery denominators.
Improved profile fit does not consistently improve structural recovery. For PXRDGen on pXRD-100, the median residual falls from 93.79% to 51.60%, while first-candidate recovery falls from 47 to 32. These outcomes reflect the specified refinement procedures, including their numerical and export behavior. Across all 1,600 Crystalyze candidates, the first StructSnap stage increases median native cosine similarity from 0.235 to 0.427, but first-candidate recovery increases by only one problem and oracle pass@8 changes from 25/200 to 26/200.
Reproducibility. Source versions, dependency pins, runner snapshots, original/refined CIFs, per-candidate scores, and failure logs are retained under experiments/appendix/native-refinement and results/appendix/native-refinement. For 508 RealPXRD-Solver candidates, the independent profile diagnostic failed after refinement; recalculating it in a fresh process recovered the residuals without changing the CIFs or rerunning refinement. Fourteen successful-diagnostic controls agreed within percentage points. The earlier precision-patched pilot is retained separately and is not pooled with these results.
Agent-adaptation experimental details
All four adaptation-ladder conditions use GPT-5.6 Sol at high reasoning effort and the same reduced composition, wavelength, diffraction profile, and output contract. The tool-free condition receives the complete 8,501-row profile inline in one user message. Coding-agent conditions receive it as pattern.csv in an isolated persistent workspace and may return selected.cif plus a small number of explicitly named alternatives. Network and web access are disabled for every coding-agent rollout. The shared coding environment used Python 3.11.15 with NumPy 2.4.6, SciPy 1.17.1, pymatgen 2026.5.4, ASE 3.29.0, spglib 2.7.0, and PyXtal 1.1.4; the pXRD-specific conditions additionally exposed GSAS-II revision 5837 and a hash-locked DICVOL14 executable. All coding conditions received read-only access to the 62,098-entry COD subset defined by DARA’s cod_filtered_info_2024 index [24, 19]. Under the benchmark’s structural-equivalence criterion, this COD subset overlaps with 7 of the 100 pXRD-100 reference structures and none of the 100 pXRD-ood references.
| Condition | Interface | Scientific environment | Persistent pXRD capability | Per-case limit |
| GPT-5.6 Sol (high) | Single Responses API call; no workspace or tools | None | None | 65,536 output tokens |
matsci-codex | Stateful Codex coding agent | Scientific Python, including NumPy, SciPy, pymatgen, ASE, spglib, and PyXtal; read-only local COD | None | 2 vCPU, 5 GiB, 1 hour |
pxrd-codex | Stateful Codex coding agent | matsci-codex environment plus GSAS-II revision 5837 and bare DICVOL14 | None | 2 vCPU, 5 GiB, 1 hour |
adapted-codex | Stateful Codex coding agent | Same pXRD environment as pxrd-codex | Frozen 139-word skill and neutral DICVOL14 adapter | 2 vCPU, 5 GiB, 1 hour |
Table 13. Interfaces and scientific environments in the agent-adaptation ladder. The capability column describes persistent task-specific material supplied before a rollout; scripts written by an agent during its own rollout are not persistent across cases.
| Condition | Attempts | Timeouts / other incomplete | Mean / median wall (min) | Total wall-hours | Usage records | Mean input / cached / output tokens |
| GPT-5.6 Sol (high) | 200 | 0 / 0 | 6.94 / 6.95 | 23.1 | 200 | 0.140M / 0 / 0.021M |
matsci-codex | 200 | 10 / 3 infrastructure | 25.28 / 21.32 | 84.3 | 187 | 6.488M / 6.294M / 0.041M |
pxrd-codex | 200 | 1 / 7 nonzero exits | 16.30 / 12.09 | 54.3 | 192 | 7.112M / 6.976M / 0.039M |
adapted-codex | 200 | 3 / 0 | 21.21 / 16.16 | 70.7 | 197 | 4.203M / 4.045M / 0.036M |
Table 14. Compute and token accounting for the 200-case agent-adaptation evaluations. Wall time includes model-service latency, agent reasoning, and local command execution. For attempts without a final usage record, token counts are estimated from wall-clock duration using the mean tokens per minute of recorded attempts under the same condition; cached input is a subset of input rather than an additional token category.
The coding agents completed a mean of 43.8, 48.9, and 43.2 local command executions per attempt for matsci-codex, pxrd-codex, and adapted-codex, respectively. Their archived event streams preserve command boundaries but not timestamps for individual model and command events, so the total wall time cannot be decomposed reliably into LLM-service and local CPU components. The three matsci-codex infrastructure failures were memory-limit exits and remain denominator misses. The seven non-timeout pxrd-codex nonzero exits terminated before a final usage event; any CIF already written remained eligible for evaluation. The matsci-codex and adapted-codex evaluations used machine-local ChatGPT authentication, whereas the tool-free and pxrd-codex evaluations used API authentication. The corresponding Codex standalone versions were 0.145.0, 0.147.0, and 0.149.0 for matsci-codex, pxrd-codex, and adapted-codex; the ladder should therefore be interpreted as a systems comparison rather than a strict single-variable ablation.
adapted-codex experience-driven evolution
adapted-codex was constructed through a bounded, experience-driven self-evolution procedure. The trainer operated over a fixed curriculum of 50 development problems comprising 25 pXRD-100-style and 25 pXRD-ood-style cases, with structural categories and primitive-cell sizes chosen to approximate the benchmark distribution. The curriculum is disjoint from pXRD-Bench by reduced composition, structure, source material, and OOD generation parent. Its composition and ordering were fixed from pre-run metadata, without access to solver outcomes or verifier scores.
The trainer first launched a blind pxrd-codex rollout on each of the 50 curriculum problems. Trainees received only the reduced composition and diffraction profile; after each trajectory terminated, the trainer gained access to the submitted structures, reference structure, and structure-matcher outcome. The initial system recovered 30/50 reference structures, yielding an experience corpus containing both successful and unsuccessful trajectories together with evaluation-time verifier feedback.
The trainer then synthesized a persistent capability update from recurring trajectory-level patterns. Adaptation was restricted to the agent’s capability layer: the underlying model, scientific environment, task interface, and compute allocation remained unchanged. The resulting package consisted of a compact 139-word skill and a neutral adapter exposing DICVOL14 through a reliable programmatic interface. Together, these components encode transferable task and verifier priors while improving access to existing crystallographic computation, without introducing reference structures, case-specific solution policies, or an additional scientific ranking procedure.
The revised capability was behaviorally validated in a single blind wave over all 20 initial misses and three predeclared successful cases chosen as regression controls. It recovered 6/20 previous misses while preserving exact recovery on all three controls; two of the 23 attempts reached the one-hour limit. The capability package was then frozen and evaluated once on the 200 held-out pXRD-Bench problems. This protocol separates experience collection and capability synthesis from the final benchmark evaluation, allowing the resulting performance gain to measure transfer of the learned agent behavior rather than direct optimization against pXRD-Bench.
Our trainer protocol prescribed a minimal capability package surface, with no prescribed workflow or case-specific corrections to preserve agent autonomy. Thus, the trainer agent accepted the 6-case capability gain and did not attempt to further optimize performance against the difficult remaining cases. In our experience, we found that allowing self-evolution to continue unbounded led to sprawling packages with stalled performance gains as the tools and skills over-corrected for observed mistakes and struggled to generalize. pXRD structure determination is a complex task requiring adaptable strategies and crystallographic reasoning. Discovering policies that improve general crystallographic capabilities and lead to improvements beyond verifier alignment may require substantially more rollouts than we allotted and potentially updates to the weights rather then only the harness.
Frozen pXRD skill. The complete 139-word pxrd-solve instruction body bundled with the frozen adapted-codex capability is reproduced below verbatim apart from Markdown-to-LaTeX formatting.
pXRD submission contract
Only selected .cif and alternative-<n>.cif are graded. Keep a parseable selected.cif current once a concrete model exists; it must be the best-supported model.
A candidate is an element-specific periodic realization with explicit cell scale, coordinates, occupancies, and ordering. A prototype, indexed cell, unmaterialized site permutation, or averaged/disordered site is a different candidate. If a few distinct element-to-site orderings, supercells, or periodic models genuinely survive, write each as a separate declared CIF instead of collapsing them into mixed occupancy or leaving them in scripts. Alternatives provide coverage; they do not replace the primary.
Round-trip every declared CIF and confirm its intended reduced formula, occupancies, cell, and coordinates.
dicvol14-adapter is a neutral interface to the installed DICVOL14 binary. Its --help describes explicit peak-list, input-kind, crystal system, bound, provenance, and timeout options; it performs no peak picking or solution ranking.
Inference-time scaling protocols
Panel construction. All inference-time scaling experiments use the same frozen pXRD-Bench-20 panel. The panel was selected without consulting solver outcomes, fit values, or trajectory contents: three cases lacking a completed neutral pilot for infrastructure reasons were first excluded, and a seeded mixed-integer selection then chose 10 pXRD-100 cases (three representative, three challenge, and four polymorph) and 10 pXRD-ood cases (two substitution, two ordered-decoration, three Wyckoff-enumeration, and three crystal-search). Constraints balanced primitive-cell size and crystal system, retained two complete polymorph pairs and three challenge axes, and prevented repeated OOD parents. The panel is disjoint from the 50-case adaptation curriculum by reduced composition, structure, source material, and OOD generation parent. Selection seed 20260817 controlled only deterministic tie-breaking; the model API exposed no deterministic sampling seed. The manifest and selection record were hash-locked before outcomes were opened. Because this panel was subsequently inspected while developing the inference-time procedures, it should be regarded as a development panel rather than an untouched estimate of final-test performance.
Independent repeated sampling. Each condition in Figure 2 comprises eight independent invocations for each of the 20 problems. Every invocation began with a fresh model session and isolated workspace, received the same task prompt and scientific environment, and shared no state or candidates with another invocation. Workers were limited to 2 vCPUs, 5 GiB of memory, and one hour; timeouts, process failures, and missing or unparsable submissions remained failures. The pxrd-codex Sol and Luna campaigns used Codex 0.147.0, while the later adapted-codex Sol campaign used Codex 0.149.0. Replicate numbers identify independent API calls rather than reproducible pseudorandom streams.
For a problem with candidate-set successes among observed rollouts, we report the order-invariant estimator
Here a candidate-set success means that either selected.cif or a declared alternative-<n>.cif passes the benchmark matcher. Thus, the curves measure expected oracle coverage of a size- subset of the eight observed rollouts; they do not describe a deployable selector or constitute confidence intervals over new problems.
| Condition | Usage records | Input tokens | Cached-input subset | Output tokens | Wall hours | Timeouts |
| pxrd-codex, Sol (high) | 156/160 | 1,110.30M | 1,089.35M | 6.28M | 49.4 | 2 |
| pxrd-codex, Luna (max) | 112/160 | 1,827.73M | 1,792.60M | 12.02M | 79.5 | 29 |
| adapted-codex, Sol (high) | 156/160 | 621.02M | 605.17M | 5.56M | 44.9 | 3 |
Table 15. Compute and token accounting for the repeated-sampling campaigns. Each row contains 160 attempted rollouts. Token totals sum attempts with a final usage event and are therefore lower bounds; cached input is included in, rather than additional to, total input.
Reference-blind profile selection. The experiment pooled every selected.cif and declared alternative from the eight adapted-codex Sol rollouts. Of 160 attempts, 157 emitted at least one CIF, giving 213 candidates; all 213 completed the same bounded GSAS-II fit. Candidate structures were expanded to an equivalent P1 representation for backend robustness, while their lattices, coordinates, species, and occupancies remained fixed. We fitted only phase scale, four Chebyshev background coefficients, a zero shift bounded to , and Gaussian Caglioti and , with ; all remaining profile terms were shared and fixed. Because the supplied profiles contain no counting uncertainties, all profile weights were set to one. The candidate with the lowest post-fit residual was selected using only the observed pattern, wavelength, and submitted CIFs. Reference structures and matcher labels were opened only after these selections were sealed. This is a controlled nuisance-profile comparison, not a structural Rietveld refinement or a general proof of structural identity.
Blind stop/continue verifier. The LLM verifier receives the composition, wavelength, measured pattern, chronological solver trace, final response, and submitted CIFs. The reference CIF and StructureMatcher result are excluded from its workspace, and the reviewer can only write a JSON stop/continue decision and reason. We evolved its policy with GEPA using 50 nonbenchmark adapted-codex trajectories, split by problem into 30 training, 10 validation, and 10 sealed development cases; each 10-case holdout contained seven exact candidate sets and three misses. GPT-5.6 Sol (high) served as reviewer and reflector. Under a 180-call budget and optimizer seed 0, GEPA generated 10 policies with utility 1 for a correct decision, 0.5 for an unnecessary continuation, and 0 for a false stop or invalid output. Candidate 3 was selected using two validation repeats before the development set was opened, then evaluated with three development repeats. Oracle evidence became available only to the post-decision reflection step during optimization. The selected policy was frozen before its evaluation on the 200 pXRD-Bench first-pass trajectories shown in Figure 3(a).
Verifier-gated continuation. Figure 3(b) starts from replicate 1 of the adapted-codex Sol campaign. The judge first reviews each blind solve. A continue decision launches an evidence-conditioned solve in a fresh workspace supplied with the original problem, first-pass trace and CIFs, and the judge’s reason; a stop decision preserves the existing submission without another solver call. Surviving cases are reviewed again, then receive at most two ordinary fresh solves that see only the original problem. Candidates accumulate across completed stages, but the hidden reference is used only for retrospective scoring. Judges and solvers were capped at 30 and 60 minutes, respectively, with at most 16 concurrent calls. The USD axis reconstructs per-call token charges and calibrates them to the contemporaneous $446.15 API bill for the source eight-replicate campaign: the independent baseline assigns of that bill to complete passes, while the gated curve sums the initial solves, all reviews, and only the solver calls authorized at each stage. Later model-price changes are not applied retroactively.
Table 16: Case-level execution of verifier-gated inference-time scaling on pXRD-Bench-20. Columns follow the fixed panel order. Solve rows report when selected.cif is exact, when only a declared alternative is exact, and when no submitted CIF is exact; – denotes a case already terminated by the gate. Review rows report (stop) or (continue), with ✓ or × indicating retrospective agreement or disagreement, respectively, with candidate-set exactness at that stage. is the initial blind solve, the evidence-conditioned continuation, and fresh solves, and the intervening reviews.
| Stage | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 |
| E | E | M | E | M | M | E | E | E | E | M | E | M | M | E | E | E | E | M | E | |
| E | E | E | – | A | M | – | – | – | – | M | – | E | M | A | – | – | M | E | E | |
| – | – | – | – | – | – | – | – | |||||||||||||
| E | E | E | – | – | M | – | – | – | – | E | – | – | M | – | – | – | M | M | E | |
| – | – | – | – | – | – | – | – | – | – | – | ||||||||||
| A | – | M | – | – | M | – | – | – | – | M | – | – | – | – | – | – | M | M | – |
Table 16. Case-level execution of verifier-gated inference-time scaling on pXRD-Bench-20.
The initial solve covered 13/20 problems. The evidence-conditioned continuation recovered four additional cases, and the first verifier-authorized fresh solve recovered one more; the final fresh-solve stage added no further match. All 20 execution chains completed, yielding cumulative candidate-set coverage of 18/20.
Stop/continue policy prompts
The 102-word seed and 415-word selected policies compared in Figure 3(a) are reproduced below verbatim apart from Markdown-to-LaTeX formatting. A fixed 51-word wrapper supplied the evidence paths and decision.json schema. The wrapper, reflection and evaluator instructions, continuation prompt, and hashes of all frozen prompt artifacts accompany the released experiment.
Seed policy.
Act as a inference-time verifier for exact crystal-structure recovery. Decide whether the submitted candidate set is sufficiently established or whether another solution attempt is warranted.
Consider the diffraction fit, physical plausibility, and what the trajectory actually instantiated and compared. Distinguish support for the cell, , and average framework from support for the submitted symmetry, coordinates, occupancies, and element ordering. A small unresolved discrete ambiguity should not be considered resolved merely because an averaged or convenient realization fits well. Decide continue when the diffraction fit is dubious or a materially different, scientifically motivated structure that could plausibly improve the candidate set seems likely.
Selected policy (judge-v1).
Act as an inference-time verifier for exact crystal-structure recovery. Use only the blind evidence. Decide whether the submitted CIF set already contains a sufficiently established realization or whether another solution attempt is likely to add a materially different viable candidate. Audit claims against the actual trace and CIFs:
Confirm that the primary CIF parses and symmetry-expands to the stated composition, atom count, occupancies, cell, and ordering. Reduce or primitive-standardize cells before treating different settings, supercells, or reported Z values as different structures.
Require strong indexing coverage and no important unexplained reflections. Then require atom-coordinate evidence: a whole-profile or integrated-intensity comparison of the instantiated CIF, not peak positions alone. Consider fit relative to noise, a Pawley/profile lower bound, or competing models when available. Absence of a separately supplied comparable score is not itself a reason to continue if the trace contains auditable equivalent evidence.
Determine what was actually solved. Favor convergence from independent maps/searches, chemically assigned scattering peaks, coordinate refinement, extinction or symmetry tests, and constrained-versus-unconstrained comparisons. A familiar prototype name alone is weak evidence.
Check periodic distances, composition/charge balance, bond-valence consistency, and coordination topology. Use chemistry as a veto for demonstrable failures such as overlaps, pervasive impossible bonds, wrong stoichiometry, or incompatible occupancies—not as a preference for conventional density, packing, coordination number, or literature form. Inspect complete neighbor shells; a cutoff can mislabel long or distorted coordination as undercoordination.
Track diffraction-relevant discrete ambiguities such as element ordering, symmetry lowering, superstructures, occupancies, and distinct frameworks. An ambiguity is adequately handled when alternatives are explicitly submitted or when data, multiplicities, maps, refinement, or physical impossibility quantitatively reject it.
Do not continue merely because an untested polymorph, integer-multiple Z, denser packing, or more conventional coordination can be imagined. Such an alternative must be concrete: compatible with indexed reflections and systematic absences, composition and site multiplicities, and supported by a plausible periodic realization or a genuinely comparable fit. Conversely, a low residual cannot rescue a flexible profile fit whose atomic model was never instantiated or whose decisive ordering/symmetry choices remain unsupported.
Decide continue for an invalid CIF, dubious line coverage or atomic-intensity fit, a concrete surviving incompatible realization absent from the set, or a material exact-structure choice supported only by assumption. Decide stop when at least one explicit ordered realization has convergent diffraction support and passes validity/chemistry checks, while credible remaining ambiguities are covered or rejected. Base the reason on the decisive evidence and set proposed_branch only when a specific viable branch is identifiable.
Profile-residual diagnostics
We analyze candidate-level profile agreement across both benchmark corpora, labeling each parseable, forward-simulable structure as an exact match or miss using the benchmark StructureMatcher; omitted candidates remain failures in the primary recovery metric. For a shared cross-method diagnostic, the hidden reference and candidate are independently forward-simulated and maximum-normalized under the clean GSAS-II fixedparams--v1 model [57]. We then compute the unit-weight, fixed-parameter profile residual
which is the standard expression with all profile weights set to one. Unit weights are used because these simulated profiles provide no counting uncertainties. No noise is added, and no nuisance, lattice, or structural parameters are fitted. Consequently, this shared fixed-parameter is an evaluation-time oracle diagnostic rather than a conventional refined Rietveld result; unlike the bounded-fit used for candidate selection, it requires the hidden reference and is unavailable during blind inference. Throughout this section, “fixed-parameter,” “bounded-fit,” and “native-profile” identify the scoring protocol rather than distinct residual definitions.
Under the shared fixed-parameter protocol, strongly separates matches from misses for AbXRD-Solver (median 24.5% versus 70.8%) and the agent conditions (match medians 2.4%–8.8%, versus 12.6%–99.4% for misses), but less consistently for the other generative systems. Because StructureMatcher tolerates finite geometric deviations while this diagnostic uses narrow, unaligned peaks, an accepted structure can still have a large residual; profiles with comparable squared intensity but little peak overlap approach . This diagnostic therefore probes parameter precision but does not replace exact structural recovery. We also compute framework-native
using each method’s frozen input representation listed in Table 8 [46, 11, 54, 32] (Li et al. 2025). RealPXRD-Solver is excluded only from this native-profile analysis because its released variable-length sparse-peak representation has no fixed-grid without an additional binning or broadening convention; its candidate CIFs remain included in the shared analysis above.
Native-profile likewise separates matches from misses for Ab-PXRD-Solver (median 25.3% versus 73.3%), deCIFer (28.7% versus 94.0%), and the agent conditions, but overlaps substantially for Crystalyze and PXRDGen. It can therefore support candidate ranking within some frameworks, but is neither calibrated across frameworks nor guaranteed to track structural correctness; its usefulness depends on the profile representation and structural-verifier tolerances.
![Framework-native by structural recovery. All finite generated candidates from pXRD-100 and pXRD-ood; each panel uses the input representation native to that condition. Legend: Exact match, Miss, Mean, Median. Panels: Crystalyze, exact n=99, miss n=1,501; DiffractGPT, exact n=15, miss n=184; Ab-PXRD-Solver, exact n=62, miss n=98; deCIFer (known Z), exact n=225, miss n=1,333; PXRDGen (known Z), exact n=536, miss n=1,064; RealPXRD-Solver (known Z), Native-profile undefined, The released model consumes a variable-length discrete peak list rather than a profile grid; GPT-5.6 Sol (high), exact n=62, miss n=163; matsci-codex, exact n=112, miss n=137; pxrd-codex, exact n=129, miss n=104; adapted-codex, exact n=151, miss n=120. Axes: Framework-native (%), Density. Agent conditions use the shared fixed-parameter .]
Figure 12. Candidate-level framework-native distributions. Colors and vertical-line styles follow Figure 11. Because the angular or reciprocal-space grids, peak shapes, broadening models, noise, and discretization vary between panels, absolute values should be compared only within a panel and not used to rank frameworks. For matsci-codex, pxrd-codex, and adapted-codex, the displayed range ends at the pooled candidate 95th percentile; the upper 5% tails remain included in the reported means and medians. The RealPXRD-Solver panel records that a fixed-grid native residual is undefined for its released sparse-peak input representation.
Baseline condition, synthetic curriculum, and evolution process for experimental multiphase pXRD
This section specifies the baseline solver, synthetic generator, scoring objective, and evolution protocol used in § 5.1 and § 5.2.
Baseline condition and task specification. The baseline uses GPT-5.6 Sol at maximum reasoning effort in Codex CLI 0.154.0, with a terminal, a scientific Python environment, GSAS-II, DICVOL14, and the frozen filtered COD 2024 reference library. Each attempt starts in a fresh workspace. Measured-pattern attempts receive two CPU cores and 5 GiB of memory, with no wall-time limit. Synthetic training attempts have a four-hour limit, with up to ten solvers running concurrently. The baseline receives neither learned guidance nor access to the evolution framework’s source code.
The solver inputs are inputs/pattern.csv, containing and observed counts, and inputs/metadata.json, containing the precursor formulas, firing temperature, and radiation and instrument information. For measured patterns, the counts are supplied without smoothing, background subtraction, normalization, or resampling. Product composition, phase count, and a calibrated instrument profile are unspecified. Synthetic ground truth and deposited experimental interpretations remain unavailable to the solver. The same baseline task prompt is used for synthetic and experimental solves:
Determine the crystalline phases and their weight fractions from inputs/pattern.csv. Read inputs/metadata.json for the chemical context and measurement information. The product composition and number of phases are unknown. Use the available scientific tools to determine and refine a structural model for each phase. Save one CIF per phase, the combined observed/calculated/difference profile as fit.csv, and result.json listing each phase, its CIF path, its weight fraction normalized over the modeled crystalline phases, and the fit statistics. Explain the evidence and remaining uncertainty briefly in report.md; state when a structure or fraction cannot be determined reliably. Save the final GSAS-II project used to produce your submitted CIFs, phase fractions and fit.csv as final.gpx.
The evolved v6 condition appends its learned guidance to this prompt and provides read-only access to its prototype-search helper.
Synthetic problem construction. The evolver selects source structures, positive crystalline mass fractions summing to one, precursor metadata, count level, and a random seed for each case. It records a scientific rationale and the intended reference availability: the full library, all entries of one reduced composition withheld, or specified COD records withheld. The initial six cases are listed in Table 17. The next batch adds an Mg partial reaction and a four-phase Ba mixture; the third adds a Ni source-quality problem and a Cu mixture without the nominal CuWO product. Unresolved earlier cases remain in the active six-case panel.
| Case | Generating phase inventory | Intended difficulty |
| c01 | CaWO | Single-phase control with an available parent reference. |
| c02 | ZnWO, ZnO, WO | Low counts; a 7.3 wt% residual precursor. |
| c03 | SrWO, SrCO, WO | All SrWO entries withheld; carbonate-derived mixture. |
| c04 | MnWO, MnO, WO | All MnWO entries withheld; weak residual oxide. |
| c05 | CaWO, CaWO, WO | Withheld 10.7 wt% intermediate and overlapping reflections. |
| c06 | PbWO, PbO, PbO, WO | Four phases; weak PbO and specimen broadening. |
Table 17. Initial synthetic curriculum. Generating phase identities and fractions are visible to the evolver and evaluator, but not supplied to the solver. Withholding removes the specified references from the solver’s library; chemically related structures may remain available.
The generator makes symmetry-preserving cell and coordinate perturbations to the registered structures, then calculates their mass-normalized GSAS-II profiles. Their weighted sum is scaled to the selected peak count, a positive background is added, and independent Poisson counts are drawn once from this noise-free mean. Instrument-profile parameters are shared within a measurement session; angular zero, background, specimen effects, and phase-specific effects use separate seeded random streams. Table 18 gives the frozen laboratory settings. These are synthetic-distribution parameters, not calibrated bounds for experimental refinement. Case c01 fixes the angular offset to zero and disables specimen effects, using a 1 m domain size and Å; instrument uncertainty and structural perturbations remain.
| Component | Setting |
| Radiation and geometry | Cu K/K: 1.540598/1.544426 Å, intensity ratio ; Bragg–Brentano reflection, radius 200 mm. |
| Angular grid | 9.90343322°–100.09455322° in , step 0.0108664°. |
| Count level and noise | Curator-selected signal maximum of 5,000, 15,000, or 40,000 counts, before background; independent Poisson noise on signal plus background. |
| Instrument profile | , , ; . , polarization . |
| Angular corrections | Zero offset ; specimen displacement m. |
| Size and thermal motion | Crystallite size m; per-phase Å. |
| Microstrain | Present with probability 0.6, magnitude ppm. Conditional on strain, probability 0.4 of uniaxial anisotropy with axial/equatorial ratio , along [001], [100], or [010]. |
| Preferred orientation | Probability 0.35; March–Dollase ratio , along [001], [100], [010], [110], or [111]. |
| Surface roughness | Probability 0.5; Suortti parameters , , rejecting draws with an intensity deficit above 50% at . |
| Structural perturbations | Maximum principal cell strain ; requested coordinate RMS Å, with oxygen-specific Å. Symmetry and occupied composition are preserved. |
| Background family | Simple polynomial, exponential tail, or power-law tail, with probabilities 0.1, 0.6, and 0.3 when selected automatically. |
| Background amplitudes | Correlated Bernstein floor coefficients in 200–1,200 counts; with probability 0.15, 2,500–3,500 counts. Tail amplitude counts, or for the high-floor branch. |
| Background shape | Simple floor degree 0–2; quadratic floor for either tail family. Tail decay length ; power-law exponent . |
Table 18. Frozen synthetic-generator settings. and denote uniform and log-uniform draws. Probabilities apply per phase for strain and texture, and per scan for roughness and background. Instrument coefficients use GSAS-II’s native parameter convention. Control recipes can disable effects or specify a background family.
The tail functions are and , where , is the decay length, and the power-law exponent. Each frozen case retains the recipe, realized parameters, perturbed CIFs, clean profile, noisy observation, and source hashes. Generation checks record known-phase fraction recovery and the residual from omitting each phase. The diagnostic strict-recovery screen uses a maximum absolute fraction error of 0.03 and an omission residual of at least 0.25 noise units. These checks describe quantifiability under the generating model; they are separate from solver scoring and do not establish a unique structural solution.
Profile score and project verification. The frozen scorer is version 7.0.1. It reopens final.gpx and calculates the profile on the original observation grid, retaining the submitted structures, scales, background, instrument calibration, and specimen parameters. CIFs and reported fractions must agree with the project within the recorded numerical and export-rounding tolerances. Saved fit exclusions do not remove scoring bins, and the evaluator optimizes no parameters. Only after this prediction is fixed does it read the hidden generating mean.
Let be the noise-free generating count, including background, and the replayed prediction at bin , for bins. The profile error and combined training score are
where is the structural score defined below, and . A zero prediction at a bin with positive gives . Near the generating mean, approximates the mean squared residual in Poisson-noise units; earns half of the profile credit. Scoring against rather than the particular noisy observation discourages fitting counting noise. Observed-pattern is recorded separately as a diagnostic.
Structural score. The structural term uses continuous agreement between phase inventories, atomic models, and mass fractions, rather than a binary StructureMatcher match. For each reference/submission pair , the scorer enumerates compatible common cells and searches origin shifts, using a 20% lattice-length tolerance and angle tolerance. Same-element atomic populations are matched subject to their occupancy capacities. Periodic distances are measured in the mean of the two lattice metrics. Define . For transported population of element , its normalized coefficient is
Here is atomic number, the occupied population in the registered common cell, the corresponding periodic distance, and the root-mean-square logarithmic lattice strain. Atomic-number weighting is a proxy for X-ray scattering sensitivity. The registration search maximizes over the tested cell/origin mappings and population transports. It is a finite deterministic search; unfinished registrations retain explicit score bounds.
For reference mass fractions and submitted fractions , let . For a fully quantified submission, the inventory, mass-overlap, and fraction-precision credits are
The final structural score maximizes total credited agreement over one-to-one phase assignments :
The inventory term gives credit to weak phases while penalizing additional unmatched assertions; the mass term emphasizes abundant phases. Site and fraction precision modulate this recovered-inventory credit. Lattice agreement enters , with no additional fine-scale lattice-precision term.
Incomplete submissions and physical checks. Numerically duplicate phase exports are merged. An invalid individual model remains an unmatched assertion, while valid submitted structures retain their credit. An unknown fraction receives inventory credit with , but no mass or fraction-precision credit. Explicitly unresolved material is not normalized away. Without a complete verified project, the profile contribution is zero and independently evaluable structural credit remains. Malformed top-level submissions and hard physical failures score zero; examples of the latter include invalid occupancies and fully occupied interatomic contacts below Å. Close O–O contacts below Å, partially occupied short contacts, and questionable profile parameters trigger review rather than automatic rejection. Reported means retain these provisional scores and scientific failures. Evaluator errors remain distinct from solver failures, and unresolved structural registration is reported as an interval.
Evolution, promotion, and stopping rules. The evolver uses the same GPT-5.6 Sol model alias and maximum reasoning effort as the solvers. It can inspect synthetic ground truth, trajectories, scripts, and saved projects, and run diagnostic calculations or procedure tests on completed attempts. Such oracle-assisted tests inform package design but are recorded separately from fresh solver evaluations. A package contains at most 600 words of guidance plus optional executable helpers; every evaluated revision is frozen before launch.
The original run followed the written procedure below. The controller enforced freezing, access boundaries, and solve-budget accounting; promotion was an explicit evolver decision with a saved rationale, not an automatic statistical test.
Baseline. Curate six cases and collect three independent baseline attempts per case.
Screen and revise. Run one attempt per case with a new package. Revise on unresolved cases, temporarily setting aside successful cases. The operational target was the intended scientific behavior and scores around 80 or higher. Before repeated evaluation, test one unchanged package across all six cases, including the set-aside cases.
Evaluate and promote. Complete the promising package to three attempts per case, including its screening attempt. The intended success condition was most attempts near the noise floor, most case means above 80, and every case mean above baseline. At the 30-development-solve limit, complete the current or most promising package to three attempts per case and retain it if it improves on the baseline or incumbent, carrying unresolved cases forward.
Test transfer. Add new cases to restore a six-case active panel. On each new case, compare the unchanged baseline and incumbent with three attempts each before using those results for further adaptation. Report these new-case comparisons separately from carried-over cases.
The 30-solve limit counts nonbaseline development attempts when queued. Baseline attempts, prospective transfer comparisons, and the second and third attempts used to complete a package’s three-repeat evaluation are excluded. Once the limit is reached, only one package can be completed to three attempts per already-attempted case in that cycle. Completion of the next prospective comparison starts a new cycle. There is no fixed total-run solve budget: the outer loop continues until stopped by the operator or interrupted by service or infrastructure limits. The later framework-access campaign’s numerical promotion gate is a separate protocol, specified in Appendix A.8.
Retained package and recorded outcomes. The archived run contains 135 completed solver attempts on ten synthetic cases and 15 revisions; v6 was the only promoted package. Its 331 words of guidance and prototype-search helper address source selection, missing-reference recovery, reconsideration of weak phases, instrument/specimen modeling, and saved-project checks. On the initial six cases, its three-repeat mean score was 74.07 versus 50.04 for the baseline, with all six case means improved and complete generating-phase recovery in 17 of 18 attempts. Only two case means exceeded 80, so the evolver promoted v6 under the budget-exhaustion rule. The next two two-case prospective comparisons improved from 58.79 to 72.44 and from 43.02 to 68.97. No later revision replaced v6. The package text and helper description appear in Appendix A.8.
Reproduction records. The frozen run is retained under results/experimental-evolution/divalent-tungstates/ in the research repository. evolve/LAB.md and runtime/generator_defaults.json define the generator; batch manifests record selected phases, fractions, seeds, and withholding rules; scorer/config.json and its source files define score version 7.0.1. evolve/PROMPT.md, evolve/settings.json, and control/progression.py record the evolution instructions and budget accounting. Case records, package hashes, solver outputs, scores, promotion rationale, and evolver-written investigation notes preserve the executed run. The measured-pattern evaluation and its limitations are described in Appendices A.7 and A.8.
Experimental profile comparisons and per-attempt results
Profile evaluation. The experimental comparisons use the same measured counts, evaluation window, and residual definition for each solution. Agent profiles are calculated by reopening the submitted GSAS-II project without further optimization. The replay retains the agent’s phase models, fractions, background, instrument parameters, and specimen effects. Exported profiles and reported scalar residuals are checked against this calculation. Deposited PG profiles are taken from the saved calculated curves accompanying each measurement and interpolated onto the observation grid where necessary. They are rescored directly, without a new refinement or conversion to an agent’s model. All residuals use
where and are the measured and calculated counts, respectively, and contains points from to in . The calculated intensity includes the fitted background. This common residual measures agreement with the observation; the fitted model and its flexibility remain specific to each solution.
For aggregate scores, we average the three attempts within each measurement, then calculate the mean or median of those 15 values. All 45 attempts in each of the five Sol-max conditions have verified profiles. The additional Sr follow-ups are reported separately in Table 7.
Cu: a simpler phase inventory with better profile agreement. Figure 13 compares the PG deposit with baseline and v6 solutions for case 006. The deposited model contains (68.1 wt%), (9.9%), (6.3%), (5.9%), (3.6%), and two models (3.2% and 3.1%). Each of the six baseline/v6 attempts instead submits two models, and , with 96.8–97.9 wt% assigned to . The agent interpretations reproduce the measured profile more closely under the shared residual definition. The disagreement in phase inventory and fractions warrants independent experimental review, particularly because the deposited and agent analyses use different fitted models.
Figure 13. Deposited and agent solutions for the Cu measurement (006). The deposited PG calculated profile is compared with the median-residual attempt in each agent condition (r2 for both baseline and v6). The left panels show the full evaluation window and the right panels expand 22°–40°. Observations are black points, calculations are colored lines, and differences are gray with a vertical offset. All rows share the observed maximum as a display normalization and identical vertical limits within each column. Residuals use the original counts over the full evaluation window. PG’s saved calculated curve is rescored; agent curves are independently replayed from their final projects. No additional fitting is performed for this figure.
All initial attempts and model/compute comparisons. Table 19 reports every initial full-panel attempt for the five Sol-max conditions in Table 6, the PG deposits, and two earlier comparisons using the baseline task without learned guidance. The Sol-high comparison uses gpt-5.6-sol at high reasoning effort in Codex 0.154, with three completed attempts per measurement. The Opus comparison uses claude-opus-5 at high effort in Claude Code 2.1.299, with one attempt per measurement. These runs use the same measured-input hashes, filtered COD library, scientific software, two-CPU/five-GiB worker allocation, and no solver time limit. The Sol-high panel includes a documented replacement for a user-cancelled case-015 r3 that produced no complete submission; the replacement is marked in the table. Ten Opus sessions were resumed after a subscription usage limit, preserving their workspaces and conversation state.
| Case | Run | PG | Sol high | Baseline Sol-max | Evolved v6 | Framework access | Framework + v6 | Evolved framework v5 | Opus |
| 001 (BaCO₃) | 1 | 8.00 | 8.73 | 7.22 | 7.15 | 5.85 | 6.87 | 6.67 | 7.30 |
| 001 (BaCO₃) | 2 | 8.73 | 7.22 | 7.23 | 6.08 | 6.65 | 5.83 | ||
| 001 (BaCO₃) | 3 | 8.82 | 7.15 | 7.58 | 5.92 | 6.79 | 5.96 | ||
| 002 (BaO₂) | 1 | 7.20 | 8.87 | 7.59 | 8.73 | 8.51 | 7.06 | 7.12 | 8.30 |
| 002 (BaO₂) | 2 | 7.68 | 12.67 | 7.65 | 7.24 | 5.78 | 7.21 | ||
| 002 (BaO₂) | 3 | 7.80 | 7.12 | 7.10 | 7.21 | 8.29 | 7.60 | ||
| 003 (CaCO₃) | 1 | 18.28 | 9.00 | 7.79 | 8.09 | 7.31 | 7.37 | 7.34 | 7.23 |
| 003 (CaCO₃) | 2 | 9.28 | 8.39 | 7.88 | 7.26 | 7.76 | 7.45 | ||
| 003 (CaCO₃) | 3 | 10.47 | 7.48 | 7.56 | 7.09 | 5.30 | 7.34 | ||
| Case | Run | PG | Sol high | Baseline Sol-max | Evolved v6 | Framework access | Framework + v6 | Evolved framework v5 | Opus |
| 004 (CoO) | 1 | 6.37 | 6.79 | 7.36 | 5.66 | 4.72 | 4.96 | 5.63 | 5.36 |
| 2 | 6.41 | 6.96 | 4.95 | 6.19 | 4.77 | 5.25 | |||
| 3 | 7.07 | 6.12 | 4.90 | 4.82 | 5.21 | 5.08 | |||
| 005 (Co₃O₄) | 1 | 3.78 | 4.20 | 3.37 | 3.37 | 3.42 | 3.34 | 3.06 | 2.79 |
| 2 | 3.61 | 3.56 | 3.36 | 3.13 | 3.07 | 3.19 | |||
| 3 | 3.42 | 3.44 | 3.22 | 3.07 | 3.26 | 3.17 | |||
| 006 (CuO) | 1 | 45.05 | 7.01 | 6.48 | 6.46 | 3.51 | 6.89 | 5.14 | 3.78 |
| 2 | 7.37 | 6.53 | 5.44 | 3.89 | 6.88 | 7.19 | |||
| 3 | 7.17 | 7.39 | 4.30 | 3.88 | 6.81 | 4.17 | |||
| 007 (MgO) | 1 | 13.12 | 10.18 | 9.68 | 9.31 | 5.73 | 5.73 | 8.35 | 8.24 |
| 2 | 13.21 | 8.91 | 6.74 | 8.29 | 5.81 | 7.64 | |||
| 3 | 9.88 | 8.42 | 6.04 | 5.83 | 7.52 | 8.34 | |||
| 008 (MnO) | 1 | 10.22 | 6.60 | 8.45 | 4.49 | 3.98 | 6.16 | 4.03 | 3.98 |
| 2 | 9.12 | 7.07 | 6.36 | 3.92 | 4.47 | 6.37 | |||
| 3 | 7.23 | 6.75 | 6.75 | 3.85 | 3.91 | 4.25 | |||
| 009 (Mn₃O₄) | 1 | 11.38 | 12.26 | 5.23 | 5.30 | 3.74 | 5.30 | 5.02 | 4.95 |
| 2 | 9.63 | 5.09 | 5.42 | 3.81 | 4.10 | 3.95 | |||
| 3 | 7.31 | 5.27 | 5.06 | 4.30 | 5.03 | 4.97 | |||
| 010 (Mn₂O₃) | 1 | 8.27 | 8.49 | 7.46 | 7.41 | 5.89 | 6.46 | 6.80 | 6.67 |
| 2 | 6.71 | 6.86 | 7.14 | 6.09 | 6.85 | 6.25 | |||
| 3 | 7.10 | 6.83 | 6.92 | 6.24 | 6.17 | 6.13 | |||
| 011 (MnO₂) | 1 | 7.43 | 8.37 | 6.12 | 5.84 | 5.49 | 5.79 | 5.57 | 6.84 |
| 2 | 7.81 | 6.21 | 5.73 | 5.56 | 5.42 | 6.39 | |||
| 3 | 7.81 | 5.99 | 5.88 | 5.43 | 5.78 | 5.54 | |||
| 012 (NiO) | 1 | 6.74 | 5.17 | 4.31 | 4.57 | 4.14 | 4.19 | 4.10 | 4.30 |
| 2 | 4.93 | 4.59 | 4.79 | 4.08 | 4.21 | 4.13 | |||
| 3 | 6.40 | 4.36 | 4.48 | 4.19 | 4.29 | 4.04 | |||
| 013 (PbO) | 1 | 11.54 | 16.06 | 14.70 | 11.70 | 11.47 | 10.52 | 9.72 | 10.61 |
| 2 | 18.35 | 12.58 | 13.01 | 8.02 | 10.47 | 8.84 | |||
| 3 | 12.99 | 10.25 | 12.97 | 8.29 | 13.13 | 9.09 | |||
| 014 (SrCO₃) | 1 | 15.81 | 18.13 | 12.89 | 14.66 | 14.65 | 15.85 | 6.64 | 9.74 |
| 2 | 14.70 | 16.48 | 7.12 | 8.98 | 13.04 | 15.23 | |||
| 3 | 15.70 | 16.72 | 7.88 | 11.73 | 7.53 | 15.59 | |||
| 015 (ZnO) | 1 | 15.70 | 6.38 | 5.49 | 4.82 | 6.21 | 5.58 | 6.96 | 4.74 |
| 2 | 11.72 | 11.25 | 5.89 | 4.89 | 5.26 | 5.33 | |||
| 3 | 9.28† | 13.64 | 5.00 | 5.16 | 6.27 | 6.24 | |||
| Mean | 12.59 | 8.98 | 7.90 | 6.71 | 5.98 | 6.49 | 6.44 | ||
| Median | 10.22 | 8.12 | 7.20 | 5.86 | 5.49 | 6.36 | 6.15 |
Table 19. All initial experimental profile scores, (%). Bold marks the lowest residual among three attempts within each condition and case, before rounding. Baseline and the four adapted/context conditions use Sol-max. Single PG and Opus scores appear only on the first row of each case. Aggregates are over measurement-level means. Every reaction also includes WO₃. †Documented replacement for a user-cancelled attempt.
Mean residuals are 8.98% for Sol-high, 7.90% for the main Sol-max baseline, and 6.32% for Opus. Recorded mean runtimes are 13.9, 27.0, and 86.5 minutes per completed attempt, respectively. The higher-effort Sol condition and the Opus model/harness combination therefore provide useful performance comparisons, with different compute use and execution dates; the reasoning-effort labels are not calibrated across providers. All Sol-high profiles pass the project-replay check. Opus uses a declared secondary grid tolerance of degrees, accommodating coordinates exported to six decimal places; all 15 profiles pass, compared with four at the stricter -degree tolerance. Measured counts and sample counts must still agree, and the residual formula and evaluation window are unchanged.
The Opus Sr solution illustrates why a low residual need not imply phase recovery. It reaches while representing 12 unresolved reflections as independent background peaks, without recovering the structure. Its reported two-phase fit without those peaks has a residual of approximately 15.8%. This comparison provides context for another model/harness combination rather than isolating model strength.
Reproduction. The manuscript’s scripts/plot_experimental_transfer.py builds the comparison figures and tables from compact snapshots of the saved evaluation records. These snapshots retain per-attempt residuals, reported phases and fractions, and source-file hashes; the Cu profile arrays are included separately. data/experimental_transfer/model_comparison.json retains the earlier Sol-high and Opus scores, checked against the archived replay outputs and comparison report, including the replacement’s provenance. The script checks plotted residuals against the saved evaluation results when refreshing inputs and launches no solving or refinement jobs. Full trajectories, projects, and evaluator outputs are retained under results/experimental-eval/divalent-tungstates-PG15/ in the research repositories.
Additional multiphase controls and evolution records
Framework-access evolution protocols
Both campaigns started from framework access and retained it as the incumbent. They used different synthetic curricula and acceptance rules. The earlier campaign, divalent-tungstates-fw, used selective single-attempt screening, as in the original baseline-evolution run. After three baseline attempts on each of six cases, the evolver screened v1 once on every case, then revised the package on unresolved cases. A score of approximately 80 or higher, together with the intended scientific behavior, qualified a case to be set aside. Before promotion, a single package was to complete a full-panel screen followed by two additional attempts per case; at the 30-development-solve limit, the most promising package was to receive the repeated evaluation and be promoted if it improved on the baseline. This run stopped before reaching either route to promotion.
The run completed 18 baseline and 17 revision attempts. v1 passed two cases; v2 resolved two more; and v3 resolved a fifth. The remaining case, containing a withheld intermediate, received revisions through v8. The best of these single attempts scored 70.09 under the early run’s v5. v7 failed to produce final deliverables and receives zero; v9 was stopped before scoring. Scores from successful screens of different versions were not pooled to claim a validated package. The early v5 is a different artifact from the later Evolved framework v5 tested on experimental patterns.
The later campaign, divalent-tungstates-access, replaced screening with three attempts per case for every candidate package. Each active panel contained six cases. The evolver could test up to three versions before forming the next panel, carrying forward unresolved cases and adding new problems. Ten distinct synthetic cases were baselined, requiring 30 attempts. Seven revisions required a further attempts. The active panel changed after v3 and after v6.
For each revision, the controller compared its mean score with the framework baseline on the same cases. Promotion required
with three scored attempts per condition on at least six cases. Here is the difference between case means and SE is the controller’s estimate from pooled within-case replicate variance. With six cases and three attempts per condition, , where each is a sample variance. This is an operational promotion criterion, not a multiple-testing-adjusted significance test.
| Version | Mean change | Required mean gain | Worst case change |
| v1 | -11.81 | 10.13 | -39.66 |
| v2 | -3.55 | 13.94 | -32.62 |
| v3 | +4.16 | 11.95 | -21.83 |
| v4 | -6.68 | 8.46 | -38.34 |
| v5 | +2.96 | 10.06 | -37.62 |
| v6 | -6.04 | 8.43 | -35.93 |
| v7 | [-4.45, -4.41] | [10.80, 10.81] | -20.57 |
Table 20. Later framework-access evolution. Each row compares 18 candidate attempts with baseline attempts on the same six cases. Positive changes improve the synthetic score. No version satisfies either the required mean gain or the permitted worst-case regression.
One v7 attempt has a bounded structural-registration score. Its interval is propagated through the mean and estimated threshold; Figure 10 plots their midpoints. The regression limit is violated throughout the interval. The live controller leaves that case unqualified for promotion until registration is resolved. Provisional physical-review scores are retained, and invalid submissions remain in the denominator.
The later v5 contains 205 words of guidance and no added helper. It prioritizes a complete physical checkpoint, distinguishes numerical failure from scientific rejection, permits cross-chemistry transfer for chemically supported unassigned reflections, and stops further library search once the inventory explains the coherent reflection families. Its subsequent refinement instructions cover coordinates, specimen effects, and phase-profile terms. Figure 14 shows why its positive aggregate change was insufficient for promotion.

Figure 14. Case-level gains and regressions for the later framework v5. Bars compare three attempts with three framework-baseline attempts. The mean improvement of 2.96 score points combines a 38.50-point gain on a missing- case with a 37.62-point loss on the harder Zn problem. The dashed line marks the permitted case-regression limit.
Reproduction. scripts/plot_framework_evolution.py rebuilds the two-campaign figure and version table from the manuscript’s compact job-level snapshot, retaining score intervals, invalid submissions, and the unscored attempt. Source hashes identify the run status, evolver instructions, frozen scorer configuration, and later promotion-rule implementation. Full trajectories, package revisions, procedure tests, and evolver-written diagnoses are retained in the corresponding run directories. The two campaigns have different cases and budgets; their version numbers do not identify comparable stages on a shared learning curve.
Contents of the original v6 package
The 331-word package combines the following guidance with one formula-seeded cross-chemistry prototype-search helper. The helper accepts a formula, reference index and CIF root, with optional crystallographic filters. The command invocation and Markdown formatting are abbreviated below; the released package records the exact text and tool implementation.
Build and revisit the inventory. Screen occupied composition, symmetry, source conditions, disorder, and geometry before fitting. A poor fit or zero scale rejects that record, not a chemically plausible phase: compare a physical alternate record or polymorph under the same restrained nuisance model. Keep plausible weak phases supported by coherent unique or partly overlapped families on a watch list.
If no exact entry exists, search cross-chemistry prototypes with the stoichiometric-analogue helper. Optionally add an indexed space group and cell. Inspect several parents, justify any site substitution, physical-screen the model, and refine scale before cell.
After any phase or source change, reconverge a smooth low-order background, all scales, one angular correction, and cells. Once the provisional inventory is coherent, establish a clean common profile and re-challenge the watch list under it. If inventory changes, rebuild that profile before judging phases.
Require a physically adequate baseline. Neutralize unused phase size/strain, roughness, and texture. Fit one zero or specimen-displacement correction and a nonnegative low-dimensional U/W/X profile; release SH/L when asymmetry repeats across peaks. Judge adequacy from observed-versus-calculated positions, FWHM, and asymmetry for isolated families of the same phase at low, middle, and high angle, plus a smooth plausible background—not from , interior values, or covariance alone. If a systematic trend remains, or all phase-width trials collapse or hit bounds, return to the baseline and inventory rather than adding nuisance terms.
Then compare size and strain from neutral values, one phase at a time. Combine only complementary replay-stable terms. If families remain differently broad, challenge symmetry-appropriate anisotropy. Before texture, decide whether any coherent cross-phase intensity trend is background, polarization, or low-angle roughness; reject scale/background surrogates. Compare no texture with symmetry-distinct axial and diagonal low-index axes, requiring support from multiple isolated families.
After each accepted change, save, reopen, and replay the full range. Prune unsupported or bound terms and verify geometry, fractions, and exact project/export agreement.
Guidance-only and helper-only variants have not been evaluated separately. The observed phase-recovery gain therefore concerns the delivered package as a whole.
Experimental scope and submitted-model diagnostics
The 15 measured patterns form a chemically related family selected from the Precursor Genome ledger (Figure 15). They informed earlier human decisions about the framework and synthetic generator, making this a developmental experimental evaluation. The frozen evolver runs did not receive the measured targets or their deposited solutions. Conditions used a mutable model alias on different dates and consumed different amounts of inference-time compute. Additional Sr repeats were selected after inspecting the initial outcomes and concern the same specimen. These design choices support comparisons on this panel; new specimens and matched-compute controls would test broader transfer and isolate the contributions of context and computation.
Figure 15. Experimental input panel. Each scan is independently max-normalized for display; solvers receive the original counts. Titles give the precursor pair and firing temperature. Background, peak widths, relative intensities, and phase overlap vary across the measurements. The arrangement does not define a train/test partition.
The Pb/Sr structural checks compare submitted CIFs with canonical deposited structures using symmetric pymatgen StructureMatcher: ltol=0.2, stol=0.3, and angle_tol=5 degrees, with primitive reduction and supercell matching enabled, lattice rescaling and subset matching disabled, and elemental comparison. Before matching, each site’s occupied species are retained but occupancy magnitudes are replaced by equal fractions summing to one. A match identifies structural recovery within these tolerances; refined occupancies, phase fractions, and the rest of the mixture require separate assessment.
| Condition | Background terms | Nondefault | Phases |
| Baseline | 15 (6–20) | 12/45 | 2.47 |
| Evolved v6 | 8 (6–12) | 17/45 | 2.91 |
| Framework access | 16 (10–100) | 44/45 | 2.64 |
| Framework + v6 | 8 (3–33) | 45/45 | 2.80 |
| Evolved framework v5 | 12 (8–30) | 45/45 | 2.51 |
Table 21. Diagnostics of the 45 submitted projects per condition. Background terms are polynomial or interpolation coefficients, excluding fixed-background curves and auxiliary peaks; entries give the median and range. Nondefault means a submitted value different from 0.002. Phase counts are means over all attempts.
These diagnostics describe final parameter values; parameters may have been fitted earlier and fixed before export. A CIF contact/occupancy screen flags at least one model in 20/45 baseline, 9/45 v6, 9/45 framework-access, 6/45 combined, and 3/45 evolved-framework-v5 attempts. Flags require chemical adjudication: short O–O contacts can be appropriate in a peroxide such as , while other recurrent flags arise from coarse coordinates in a supplied record. They are retained as diagnostics rather than counted automatically as invalid solutions.
Disagreements with deposited models remain useful questions for experimental reanalysis. Agents assign more on case 003, favor a simpler -dominated inventory on case 006, and omit the small deposited contribution in successful evolved Sr fits. Profile agreement, recovered structures, and chemical plausibility supply complementary evidence for assessing these interpretations.
References
- [1]Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumbh Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/0e9e708b6f48e14fd0ac29e167413f76-Abstract-Conference.html.arxiv.org/abs/2507.19457
- [2]Shayan Ali Akbar, Jiaming Qu, Chen Ling, Madhu Gopinathan, and Erwin Cornejo. From frontier to frugal: Evaluating self-evolution frameworks with small language models. In ACL 2026 Workshop on Sustainable and Efficient Language, Vision, and Action Models (SELVA), 2026. URL https://openreview.net/attachment?id=HyraPzySSX&name=pdf.
- [3]Andy S. Anker, John L. A. Gardner, Louise A. M. Rosset, Andrew L. Goodwin, and Volker L. Deringer. Autonomous interpretation of atomistic scattering data, 2025. URL https://arxiv.org/abs/2510.05938.
- [4]Ilyes Batatia, Dávid Péter Kovács, Gregor N. C. Simm, Christoph Ortner, and Gábor Csányi. MACE: Higher order equivariant message passing neural networks for fast and accurate force fields. In Advances in Neural Information Processing Systems, volume 35, pages 11423–11436, 2022.
- [5]Ilyes Batatia, Chen Lin, Joseph Hart, Elliott Kasoar, Alin M. Elena, Sam Walton Norwood, Thomas Wolf, and Gábor Csányi. Cross learning between electronic structure theories for unifying molecular, surface, and inorganic crystal foundation force fields, 2025.arxiv.org/abs/2510.25380
- [6]Simon J. L. Billinge and Igor Levin. The problem with determining atomic structure at the nanoscale. Science, 316(5824):561–565, 2007. doi: 10.1126/science.1135080.DOI
- [7]Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624:570–578, 2023. doi: 10.1038/s41586-023-06792-0.DOI
- [8]Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D. White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6(5):525–535, 2024. doi: 10.1038/s42256-024-00832-8.
- [9]Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024.arxiv.org/abs/2407.21787
- [10]Bin Cao, Huichi Zhou, Runyu Yang, Jingsong Li, Shuchen Sun, Yan Song, Hanyu Gao, Zhongwei Yu, Tong-Yi Zhang, and Jun Wang. A self-learning scientific agent for X-ray diffraction, 2026. URL https://arxiv.org/abs/2610.07862.
- [11]Kamal Choudhary. DiffractGPT: Atomic structure determination from X-ray diffraction patterns using a generative pretrained transformer. The Journal of Physical Chemistry Letters, 16(8): 2110–2119, 2025. doi: 10.1021/acs.jpclett.4c03137.DOI
- [12]Kamal Choudhary, Kevin F. Garrity, Andrew C. E. Reid, Brian DeCost, Adam J. Biacchi, Angela R. Hight Walker, et al. The joint automated repository for various integrated simulations (jarvis) for data-driven materials design. npj Computational Materials, 6:173, 2020. doi: 10.1038/s41524-020-00440-1.
- [13]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichīro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
- [14]Olympia Dartsi, Lauren N. Walters, Amalie E. Trewartha, Steven B. Torrisi, Amanda Volk, Liam Joyce, Gerbrand Ceder, and Anubhav Jain. Automating chemical reasoning in high-throughput phase identification with a probabilistic, LLM-guided framework. Advanced Science, page e76450, 2026. doi: 10.1002/advs.76450. URL https://doi.org/10.1002/advs.76450.
- [15]William I. F. David and Kenneth Shankland. Structure determination from powder diffraction data. Acta Crystallographica Section A, 64(1):52–64, 2008. doi: 10.1107/S0108767307064252.DOI
- [16]Juan J. de Pablo, Nicholas E. Jackson, Michael A. Webb, et al. New frontiers for the materials genome initiative. npj Computational Materials, 5(1):41, 2019. doi: 10.1038/s41524-019-0173-4.
- [17]Bowen Deng, Bohan Li, Matthew Cox, Hoje Chun, Juno Nam, Artur Lyssenko, Sathya Edamadaka, Jurgis Ruza, Xiāochen Du, Nofit Segal, Jesus Diaz Sanchez, Mingrou Xie, Ty Perez, Yu Yao, Miguel Steiner, Sauradeep Majumdar, Charles B. Musgrave, Anirban Chandra, Abhirup Patra, Detlef Hohl, Connor W. Coley, Ju Li, and Rafael Gómez-Bombarelli. Harnessing AtomisticSkills for agentic atomistic research, 2026. URL https://arxiv.org/abs/2605.24002.
- [18]Irina Espejo Morales, Damon Hinz, Marvin Alberts, Geraud Krawezik, Haewon Jeong, and Shirley Ho. NMR elucidation as an agentic search problem, not a modeling problem, 2026. URL https://arxiv.org/abs/2607.19406.
- [19]Yuxing Fei, Matthew J. McDermott, Christopher L. Rom, Shilong Wang, and Gerbrand Ceder. Dara: Automated multiple-hypothesis phase identification and refinement from powder X-ray diffraction. Chemistry of Materials, 38(3):1364–1376, 2026. doi: 10.1021/acs.chemmater.5c02820.
- [20]Scott Fredericks, Kevin Parrish, Dean Sayre, and Qiang Zhu. PyXtal: A python library for crystal structure generation and symmetry analysis. Computer Physics Communications, 261:107810, 2021. doi: 10.1016/j.cpc.2020.107810.
- [21]Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10835–10866. PMLR, 2023. URL https://proceedings.mlr.press/v202/gao23h.html.arxiv.org/abs/2210.10760
- [22]Zhaolin Gao, Yu (Sid) Wang, Bo Liu, Thorsten Joachims, Kianté Brantley, and Wen Sun. p1: Better prompt optimization with fewer prompts. arXiv:2604.08801, 2026. URL https://arxiv.org/abs/2604.08801.
- [23]Andrew L. Goodwin. Opportunities and challenges in understanding complex functional materials. Nature Communications, 10(1):4461, 2019. doi: 10.1038/s41467-019-12422-z.DOI
- [24]Saulius Gražulis, Adriana Daškevič, Andrius Merkys, Daniel Chateigner, Luca Lutterotti, Miguel Quirós, Natalia R. Serebryanaya, Peter Moeck, Robert T. Downs, and Armel Le Bail. Crystallography open database (cod): An open-access collection of crystal structures and platform for worldwide collaboration. Nucleic Acids Research, 40(D1):D420–D427, 2012. doi: 10.1093/nar/gkr900.DOI
- [25]Daya Guo, Dejian Yang, Haowei Zhang, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645:633–638, 2025a. doi: 10.1038/s41586-025-09422-z.DOI
- [26]Gabe Guo, Tristan Luca Saidi, Maxwell W. Terban, Michele Valsecchi, Simon J. L. Billinge, and Hod Lipson. Ab initio structure solutions from nanocrystalline powder diffraction data via diffusion models. Nature Materials, 24(11):1726–1734, 2025b. doi: 10.1038/s41563-025-02220-y.DOI
- [27]Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154–8173. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.507. URL https://aclanthology.org/2023.emnlp-main.507/.DOI
- [28]Matthew Horton, Jimmy-Xuan Shen, Jordan Burns, Orion Cohen, François Chab bey, Alex M. Ganose, Rishabh Guha, Patrick Huck, Hamming Howard Li, Matthew McDermott, et al. Crystal toolkit: A web app framework to improve usability and accessibility of materials science research algorithms. arXiv preprint arXiv:2302.06147, 2023. doi: 10.48550/arXiv.2302.06147.
- [29]Thomas Hubert, Rishi Mehta, Laurent Sartran, et al. Olympiad-level formal mathematical reasoning with reinforcement learning. Nature, 651:607–613, 2026. doi: 10.1038/s41586-025-09833-y.DOI
- [30]Anubhav Jain, Shyue Ping Ong, Geoffrey Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, and Kristin A. Persson. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL Materials, 1(1):011002, 2013. doi: 10.1063/1.4812323.DOI
- [31]Pengcheng Jiang, Jiacheng Lin, Zhiyi Shi, Zifeng Wang, Luxi He, Yichen Wu, Ming Zhong, Peiyang Song, Qizheng Zhang, et al. Adaptation of agentic AI: A survey of post-training, memory, and skills, 2025.arxiv.org/abs/2512.16301
- [32]Frederik Lizak Johansen, Ulrik Friis-Jensen, Erik Bjørnager Dam, Kirsten Marie Ørnsbjerg Jensen, Rocío Mercado, and Raghavendra Selvan. deCIFer: Crystal structure prediction from powder diffraction data using autoregressive language models. Transactions on Machine Learning Research, 2026. URL https://openreview.net/forum?id=LftFQ35147.arxiv.org/abs/2502.02189
- [33]Mykhailo Kozyrev, Andrei Kozyrev, and Anton Podkopaev. Skill issue: Lessons from optimizing repository SKILLs for coding agents. arXiv:2609.12742, 2026. URL https://arxiv.org/abs/2609.12742.
- [34]Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, et al. Tülu 3: Pushing frontiers in open language model post-training. In Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=i1uGbfHHpH.arxiv.org/abs/2411.15124
- [37]Qingmeng Li, Hao Wang, Dongbo Xiong, Jiajun Zhong, Wenhai Ji, Hao Hu, Yiyu Zhang, Bolun Zhang, Hong Wang, Yongfeng Zhu, Rong Du, Zhengde Zhang, Fazhi Qi, and Junrong Zhang. Rongzai agent: A large language model-based autonomous assistant for rietveld refinement of neutron diffraction data, 2026b. URL https://arxiv.org/abs/2605.13911.
- [38]Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with AlphaCode. Science, 378(6624):1092–1097, 2022. doi: 10.1126/science.abq1158.DOI
- [39]Phillip M. Maffettone, Lars Banko, Peng Cui, Yury Lysogorskiy, Marc A. Little, Daniel Olds, Alfred Ludwig, and Andrew I. Cooper. Crystallography companion agent for high-throughput materials discovery. Nature Computational Science, 1(4):290–297, 2021. doi: 10.1038/s43588-021-00059-2.DOI
- [40]Lynne B. McCusker, Robert B. Von Dreele, David E. Cox, Daniel Louër, and Paolo Scardi. Rietveld refinement guidelines. Journal of Applied Crystallography, 32(1):36–50, 1999. doi: 10.1107/S0021889898009856.DOI
- [41]Bryce Meredig and C. Wolverton. A hybrid computational–experimental approach for automated crystal structure solution. Nature Materials, 12(2):123–127, 2013. doi: 10.1038/nmat3490.DOI
- [42]Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanah Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025.arxiv.org/abs/2501.19393
- [43]Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L. Chevrier, Kristin A. Persson, and Gerbrand Ceder. Python materials genomics (pymatgen): A robust, open-source python library for materials analysis. Computational Materials Science, 68:314–319, 2013. doi: 10.1016/j.commatsci.2012.10.028.DOI
- [44]Juan B. Pérez-Sánchez, Yunheng Zou, Jorge A. Campos-Gonzalez-Angulo, Marcel Müller, Ignacio Gustin, Andrew Wang, Han Hao, Tsz Wai Ko, Changhyeok Choi, Eric S. Isbrand t, Mohammad Ghazi Vakili, Hanyong Xu, Chris Crebolder, Varinia Bernales, and Alán Aspuru-Guzik. El agente quntur: A research collaborator agent for quantum chemistry, 2026. URL https://arxiv.org/abs/2602.04850.
- [45]Periodic Labs. Nature is our learning environment. Technical blog post, 2026. URL https://periodic.com/news/nature-is-our-learning-environment. Published September 15, 2026.
- [46]Eric A. Riesel, Tsach Mackey, Hamed Nilforoshan, Minkai Xu, Catherine K. Badding, Alison B. Altman, Jure Leskovec, and Danna E. Freedman. Crystal structure determination from powder diffraction patterns with generative machine learning. Journal of the American Chemical Society, 146(44):30340–30348, 2024. doi: 10.1021/jacs.4c10244.DOI
- [47]H. M. Rietveld. A profile refinement method for nuclear and magnetic structures. Journal of Applied Crystallography, 2(2):65–71, 1969. doi: 10.1107/S0021889896006558.DOI
- [48]Maxime Robeyns, Martin Szummer, and Laurence Aitchison. A self-improving coding agent. In ICLR 2025 Workshop on Scaling Self-Improving Foundation Models, 2025. URL https://openreview.net/forum?id=rShJCyLs0r.arxiv.org/abs/2504.15228
- [49]Nofit Segal, Mingda Li, Benjamin Kurt Miller, and Rafael Gómez-Bombarelli. XRDiff: Crystal structure prediction from powder X-ray diffraction data using diffusion models, 2026. URL https://arxiv.org/abs/2606.14003.
- [50]Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or RL is suboptimal. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 54058–54094. PMLR, 2025. URL https://proceedings.mlr.press/v267/setlur25a.html.arxiv.org/abs/2502.12118
- [51]Woong Shin, Craig A. Bridges, Marshall T. McDonnell, and Rafael Ferreira da Silva. Fantastic scientific agents and how to build them: AgentBuild for Rietveld refinement, 2026. URL https://arxiv.org/abs/2606.12834.
- [52]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023. doi: 10.52202/075280-0377. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html.DOI
- [53]Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=4FWaWZtd2n.arxiv.org/abs/2408.03314
- [54]Kaixiang Su, Osman Goni Ridwan, Hongfei Xue, and Qiang Zhu. Ab-initio crystal structure determination from powder X-ray diffraction, 2026. URL https://arxiv.org/abs/2605.24594.
- [55]Daniel P. Tabor, Loïc M. Roch, Semion K. Saikin, et al. Accelerating the discovery of materials for clean energy in the era of smart automation. Nature Reviews Materials, 3:5–20, 2018. doi: 10.1038/s41578-018-0005-z.DOI
- [56]Brian H. Toby. R factors in Rietveld analysis: How good is good enough? Powder Diffraction, 21(1): 67–70, 2006. doi: 10.1154/1.2179804.DOI
- [57]Brian H. Toby and Robert B. Von Dreele. GSAS-II: The genesis of a modern open-source all-purpose crystallography software package. Journal of Applied Crystallography, 46(2):544–549, 2013. doi: 10.1107/S0021889813003531.DOI
- [58]Lauren N. Walters, Matthew J. McDermott, Bernardus Rendy, Yuxing Fei, Kristin A. Persson, and Gerbrand Ceder. The Precursor Genome: A pairwise reaction dataset for solid-state synthesis, 2026. URL https://arxiv.org/abs/2607.09903.
- [59]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023a.arxiv.org/abs/2305.16291
- [60]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=1PL1NIMMrw.arxiv.org/abs/2203.11171
- [61]Dongyeop Woo, Nofit Segal, Nayoung Kim, Minkyu Kim, Rafael Gómez-Bombarelli, and Sungsoo Ahn. PXD-Bench: How close are we to determining crystal structures from powder diffraction?, 2026. URL https://arxiv.org/abs/2610.16480.
- [62]Yuetong Wu and Maojun Sun. AutoXRD: Autonomous LLM agents and comprehensive evaluation for powder diffraction analysis, 2026. URL https://arxiv.org/abs/2609.00070.
- [63]Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi S. Jaakkola. Crystal diffusion variational autoencoder for periodic material generation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=O3RLpj-tc_.arxiv.org/abs/2110.06197
- [64]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, volume 37, 2024. doi: 10.52202/079017-1601. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/5a7c947568c1b1328ccc5230172e1e7c-Abstract-Conference.html.
- [65]Lance Yao, Suman Samantray, Ayana Ghosh, Kevin Roccapriore, Libor Kovarik, Sarah Allec, and Maxim Ziatdinov. Operationalizing serendipity: Multi-agent AI workflows for enhanced materials characterization with theory-in-the-loop, 2025. URL https://arxiv.org/abs/2508.06569.
- [66]Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, volume 36, 2023a. doi: 10.52202/075280-0517. URL https://proceedings.neurips.cc/paper/2023/hash/271db9922b8d1f4dd7aaef84ed5ac703-Abstract-Conference.html.DOI
- [67]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=WE_vluYUL-X.arxiv.org/abs/2210.03629
- [68]Dongfang Yu, Zhewen Zhu, Fucheng Leng, and Yizhou Zhu. Equivariant diffusion solution for inorganic crystal structure determination from powder X-ray diffraction data. Nature Communications, 17(1):3274, 2026. doi: 10.1038/s41467-026-70035-9.
- [69]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xia ochen Zuo, Yu Yue, Weinan Dai, et al. DAPO: An open-source LLM reinforcement learning system at scale. In Advances in Neural Information Processing Systems, volume 38, 2025. doi: 10.52202/085713-3775. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/a4277440d50f1f15d2cb4c14f7e0c0d2-Abstract-Conference.html.DOI
- [70]Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models, 2025.arxiv.org/abs/2510.04618
- [71]Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: LLM agents are experiential learners. Proceedings of the AAAI Conference on Artificial Intelligence, 38 (17):19632–19642, 2024. doi: 10.1609/aaai.v38i17.29936.DOI
- [72]Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2505.03335.
- [73]Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 62138–62160. PMLR, 2024. URL https://proceedings.mlr.press/v235/zhou24r.html.arxiv.org/abs/2310.04406
- [74]Yunheng Zou, Austin H. Cheng, Abdulrahman Aldossary, Jiaru Bai, Shi Xuan Leong, Jorge Arturo Campos Gonzalez-Angulo, Changhyeok Choi, Cher Tian Ser, Gary Tom, Andrew Wang, Zijian Zhang, Ilya Yakavets, Han Hao, Chris Crebolder, Varinia Bernales, and Alán Aspuru-Guzik. El agente: An autonomous agent for quantum chemistry. Matter, 8(7):102263, 2025. doi: 10.1016/j.matt.2025.102263.DOI










