WorksComputer Science
InformedXRD: Reproducible Benchmarks and Physics-Informed Evaluation for Powder Diffraction Symmetry Classification
Abstract
Machine learning for powder diffraction (PXRD) symmetry classification is being trained at scale, yet the field lacks a common evaluation standard: papers report results on different subsets of experimental archives, with different target taxonomies, preprocessing, and metrics, and published models are rarely released in a form that permits rescoring. We present InformedXRD, an evaluation framework with three parts: (i) a machine-readable mapping from 230 space groups to 99 extinction groups, the finest classes powder diffraction can identify from systematic absences alone; (ii) a benchmark-construction method whose inclusion rules are specified as code and reproduce from a frozen snapshot, instantiated as RRUFF-473, RRUFF-325, and, as a small example, opXRD-98; (iii) a reporting protocol with prior-only frequency baselines, a label-permutation test, a hierarchy-aware error metric on the condensed translationengleiche subgroup graph, and stratified evaluation. We build on the mapping, RRUFF benchmarks, subgroup graph, and four ViT checkpoints released in our earlier work (Baggett et al., 2026). Using these checkpoints, a 1D residual CNN, and six architectures trained under the XRDBench recipe (Cao et al., 2025), we show that evaluation design changes what conclusions are justified: paired tests point in different directions at Top-1 and Top-5; paired Top-k tests do not separate the three leading XRDBench models, while hierarchy-aware analysis describes error distance and direction across all six; an input-blind frequency predictor matches the models on aggregate Top-k, yet a label-permutation test still detects pattern-specific evidence; and stratified analysis exposes a label-entropy confound that aggregate accuracy hides. All mapping tables, benchmark code, scripts, and checkpoints are released as shared infrastructure for PXRD-ML evaluation.
Introduction
Although powder X-ray diffraction (PXRD) symmetry-classification models are being trained at scale, the field of ML for PXRD has no agreed-upon evaluation standard. Different studies use incompatible subsets of available experimental archives (namely RRUFF [11]), different data filters, target taxonomies, and metrics, and rarely release runnable inference artifacts [18, 13, 20, 5] (Simonnet et al. 2025). Reported accuracy conflates model progress with benchmark and target-design choices [20, 5]. We identify three systematic failures in current PXRD-ML evaluation:
Wrong target. Powder diffraction cannot distinguish space groups, the common classification target, within an extinction group, so 230-way classification penalizes a model for conflating patterns it cannot tell apart.
Non-reproducible benchmarks. Hand-filtered RRUFF subsets remove the hardest cases, exclusion criteria vary between papers without algorithmic specification, and duplicates distort accuracy.
Flat metrics and missing baselines. Top- accuracy treats a model predicting a nearby subgroup the same as predicting an unrelated symmetry, and without a prior-only baseline one cannot tell whether accuracy comes from diffraction evidence or from reproducing the label distribution.
These failures compound: target taxonomy, simulator realism, and post-hoc calibration all affect whether real-data behavior reflects diffraction evidence or dataset bias, and without released checkpoints third parties cannot rescore prior work on a common test set. Neither recent studies Schopmans et al. [ [20] nor SimXRD [Cao et al., [5] releases such artifacts.
Scope and contributions. This paper contributes evaluation infrastructure and standards, not a new PXRD architecture, to make future PXRD-ML claims more comparable, physically meaningful, and auditable.
A curated extinction-group mapping table (), first released with Baggett et al. [ [4], provided here as a processed, verified machine-readable artifact. (§ 3)
A benchmark-construction method specified as code, reproducible from a frozen snapshot, that yields reproducible benchmarks RRUFF-473 and RRUFF-325 from Baggett et al. [ [4], and additionally curated over the opXRD dataset as a case example. (§ 4)
Evaluation tools: prior-only frequency baselines with deployable and oracle variants, a label-permutation test, the hierarchy-aware DAG metric from our earlier work Baggett et al., [4] supplemented with exact nulls, and stratified reporting. (§ 5)
A demonstration on four ViT checkpoints [4], a 1D CNN, and six XRDBench architectures [5] that these choices change rankings and reveal confounds. (§6)
Relationship to Baggett et al. This paper builds off our prior work Baggett et al. [ [4], which first introduced and publicly released the mapping, RRUFF-473 and RRUFF-325, the condensed subgroup graph, and the four ViT checkpoints; each is credited where it is introduced. The two papers ask different questions: Baggett et al. ask whether a physics-informed model can read symmetry from a powder pattern, whereas this paper asks what it takes to know whether any model can. To our knowledge those checkpoints remain the only PXRD-ML transformer models with a public inference pipeline that can be rescored under this protocol.
Released artifacts. The mapping table, both benchmarks with curation code and record lists, the evaluation scripts, and the case-study checkpoints are public; components reused from Baggett et al. [ [4] carry their provenance. Appendix K lists the repositories and the full inventory.
🐙 Code 🤗 Benchmarks 🤗 Checkpoints
Background
Crystal structure and powder diffraction. A crystal structure is a periodic arrangement of atoms, classified by its internal symmetry into 230 space groups. Powder X-ray diffraction (PXRD) experiments produce a 1D intensity trace whose peak positions and relative intensities encode symmetry and atomic arrangement. The relevant fact for classification is a built-in resolution limit: symmetry elements such as glide planes and screw axes produce systematic absences—reflections forbidden by symmetry and therefore missing from the pattern—but different space groups can share identical absence sets, making them experimentally indistinguishable from a powder pattern alone. Friedel’s law further removes the distinction between centrosymmetric and noncentrosymmetric structures (with and without inversion symmetry) when anomalous scattering is negligible. A benchmark scoring models beyond these limits asks for information the measurement does not contain.
Extinction groups. The equivalence classes of space groups that share identical systematic absences are called extinction groups, which reduce 230 space groups to 99 extinction groups [3] (Appendix A gives the construction). They are the finest evaluative label space PXRD can justify, so scoring in 230-way space-group space risks penalizing a physically correct prediction because the measurement cannot distinguish two classes; § 3 describes the penalty of using space groups. One hexagonal/trigonal pair of extinction classes is merged only in the condensed subgroup graph used for the hierarchy metric (§ 5.2) scoring, though not in standard label scoring (see Appendix § 10 for more).
Subgroup hierarchy. Extinction groups are not independent classes but are related through a crystallographic subgroup hierarchy: each connects to others through translationengleiche (-)subgroup relations [28], which preserve the translation lattice while removing point-group operations such as rotations, reflections, and inversion. These form a directed acyclic graph with edges from higher to lower symmetry; the condensed -subgroup graph provides a reproducible first-order notion of error severity with clean ancestor/descendant semantics.
ML for PXRD classification. Labeled experimental patterns are scarce, so PXRD models are trained almost entirely on simulated data. Nearly every study computes its training patterns from structures in a crystal-structure database [18, 26, 16, 23, 30, 13, 12, 19, 7, 17, 2, 5, 27], most often the Inorganic Crystal Structure Database (ICSD) [29], the Crystallography Open Database (COD) [8], or the Materials Project [10], and usually augments them with noise, backgrounds, peak broadening, or preferred orientation; a few studies also perturb those structures or generate synthetic crystals of their own [5, 24, 27, 20, 4], and only rarely do measured patterns enter training [16, 19]. Measured patterns are instead reserved for evaluation, where the common source is RRUFF [11], a public collection of experimental powder diffraction patterns with a primary mineral phase named for every entry; but each study evaluates on its own hand-selected subset of it, under its own target definition and preprocessing. Open experimental archives such as PXRD [Hollaker et al., 2025] are beginning to expand the available real data (§ 4.3). Cao et al. [2025] reported strong transfer to RRUFF from their SimXRD-4M simulator and XRD Bench recipe. Our earlier work [4] contributed one of the few released PXRD-ML inference pipelines that can actually be rescored under a common target and benchmark, which is why we use those checkpoints as case studies in §6.
Evaluating on Extinction Groups and the Mapping Table
Scoring in space-group space understates what a model has learned. Besides Baggett et al. [2026], almost all previous studies use space groups as the label space which PXRD cannot resolve (see § 2). To measure the cost, we ran a matched, convergence-controlled comparison with the training pipeline and 2.0M-pattern synthetic dataset of Baggett et al. [2026]. The same 230-way space-group model, on the same predictions, reaches 15.17% Top-1 scored in its own space and 18.46% once its predictions are mapped into the 99-way extinction-group space. Training directly on extinction groups adds less: the direct model reaches 19.32% at full scale, an advantage of +0.86 %. Yet this gap widens to +2.77 % at one-eighth training scale, so direct supervision is primarily a sample-efficiency advantage that the space-group model recovers implicitly as the corpus grows. The full comparison, including Top-5, the intermediate scales, and the RRUFF transfer check, is in Appendix § 11.
Acting on that choice requires a machine-readable, canonicalized label space. The underlying tabulation is public—the extinction-symbol table distributed with EXPO [1], derived from International Tables for Crystallography [3] and first released in machine-usable form with Baggett et al. [2026]—and we parse, canonicalize, and invert it into a 230-row lookup cross-checked against International Tables (Appendix § 9). The contribution is the processed, verified artifact, released as JSON and CSV, under which any 230-way model can be rescored post hoc and misalignment is auditable.
Standardized Benchmark Construction
The construction method is defined over the label space and the measurement grid, not over any one archive. This section states the method (§ 4.1) and applies it to two archives: RRUFF (§ 4.2) yielding
RRUFF-473 and RRUFF-325, on which §6 evaluates models, and opXRD (§ 4.3), a small example yielding opXRD-98, evaluated in Appendix § 14.
Construction Method
A source archive isn’t always a usable benchmark at first: it may contain multiple scans per material, incomplete or multiphase labels, mixed wavelengths, and exact duplicates. The construction method turns it into a fixed evaluation set whose inclusion rules are specified as code and reproducable from a frozen snapshot:
Single assignable label. Retain records with one phase whose space group is stored or derivable, mapped to the label space of § 3; multiphase records without a primary phase are excluded.
Wavelength and grid gate. Select scans at the target wavelength, require complete angular support at native resolution over the evaluation window, and resample onto a standard grid.
Structural grouping. Assign each record an archive-specific structural-family label, used both to group near-identical records and to stratify results.
Family coherence. Keep only families internally consistent under within-family diagnostics, separating distinct variants and admitting singletons only when they match the family core.
Exact-pattern deduplication, conflict-aware: byte-identical patterns reduce to one representative, and groups carrying conflicting labels are excluded.
Per-pattern strata. Attach nuisance-fit severity from classical Rietveld fitting under fixed cutoffs (§ 5.3), and record any measurement stratum that varies within the archive.
Freeze and audit. Release snapshot-dated identifiers, curation code, and a threshold sweep.
Steps 1, 2, and 7 are identical for every archive; steps 3–6 can require a different instantiation. Figure 1 records both instantiations side by side; the full step descriptions are in Appendix § 12, and the thresholds and gate counts in Appendix § 13 (RRUFF) and Appendix § 14 (opXRD). Sets from different archives are reported separately, never pooled.
Figure 1. The construction method of § 4.1 instantiated on two archives. Left: RRUFF, yielding the frozen RRUFF-473 and RRUFF-325 benchmarks with their nuisance-fit strata. Right: opXRD, a small example yielding opXRD-98. Rows are the steps of the method aligned across the two archives, with dashed boxes marking omitted steps and blue boxes marking the released evaluation sets. Per-step detail—thresholds, diagnostics, and the full gate cascade—is in Appendix § 13 (RRUFF) and Appendix § 14 (opXRD).
Instance 1: RRUFF-473 and RRUFF-325
RRUFF [11] is one of few public sources of experimental PXRD with primary-phase labels, so RRUFF-473 and RRUFF-325, introduced in our earlier work [4], are the primary benchmarks here: real, singly labeled, reproducible from a frozen snapshot. Familycoherence checks reduce the 1,981-scan Cu-K pool to RRUFF-473; families that fail these checks are excluded because they would make the benchmark ambiguous before any model is evaluated. Nuisance-fit cutoffs applied to the frozen RRUFF-473 then give RRUFF-325. Third parties should be able to reconstruct what was evaluated, why those scans were retained, and how composition affects the resulting claims; the full pipeline, released defaults, and threshold-sensitivity results are in Appendix § 13.
Reproducibility, coverage, and intended use. Released scan-ID lists, snapshot date, curation code, and thresholds reproduce both sets exactly, and RRUFF-325 is a deterministic subset of RRUFF-473. Both are fixed evaluation sets, not training corpora, covering 46 and 45 of the 99 extinction groups (Table 4), so claims do not extend to unrepresented groups (Appendix § 13 gives the coverage rationale). Report both—RRUFF-473 for real-data robustness, RRUFF-325 for cleaner error structure—to avoid overfitting to one slice.
| Schomans et al. | SimXRD | RRUFF-473 | RRUFF-325 | |
| Patterns Count | 942 | ~3000 | 473 | 325 |
| Classification target | 145 SGs | 230 SGs | 99 EGs | 99 EGs |
| Curation method | Hand-filtered | Partially documented | Algorithmic | Algorithmic |
| Unique minerals | — | — | 227 | 168 |
| Unique extinction groups | — | — | 46 | 45 |
| Difficult cases retained? | No (85 SGs excluded) | Unknown | Yes | No2 |
| Recoverable / Usable / Poor / Catastrophic | — | — | 172 / 153 / 74 / 74 | 172 / 153 / 0 / 0 |
| Reproducible from raw RRUFF? | No | No | Yes | Yes |
Table 4. Benchmark audit against prior PXRD-ML evaluation sets. The main point is not raw score comparison but the mismatch in target taxonomy, filtering, difficult-case retention, and reproducibility.
Instance 2: a Small Example, opXRD-98
opXRD [Hollarek et al. [ [9]], an open archive of 92,552 experimental patterns from multiple laboratories, is the second instance and tests whether the method transfers beyond RRUFF. Its scale benefits representation learning, but the one-pattern, one-label protocol yields only a small example, opXRD-98, of 98 records; the taxonomy and both readouts transfer without modification, but the result is a stress test rather than a benchmark, and we hope it encourages larger labeled archives. The exact record list and processing scripts are released; the gate table, the reasons the benchmark was not built on opXRD or merged with RRUFF, and the evaluation results are in Appendix § 14. One step is instantiated differently: RRUFF’s mineral family does two jobs—grouping structurally near-identical records and stratifying results—and opXRD carries no such label, so both are done from structure alone by projecting its records into the frozen ICSD structural map of Nguyen et al. [ [14] (Appendix § 14).
Evaluation Protocol
We propose three evaluation tools that should accompany Top- accuracy in any PXRD-ML evaluation. The first (§ 5.1) pairs input-blind prior-only baselines with a label-permutation test: the baselines establish how much Top- accuracy follows from label frequency alone, and the permutation test asks whether a model’s rankings depend on the individual diffraction pattern. The second (§ 5.2) replaces a single scalar with a hierarchy-aware characterization of how the model is wrong. The third (§ 5.3) partitions performance along axes that aggregate accuracy collapses.
Prior-Only Frequency Baseline and Label Permutation Test
Crystallographic training distributions are heavily imbalanced (Figure 4, Appendix § 19), so a model can score well on real data by reproducing the label distribution rather than by reading the diffraction pattern. The prior-only baseline detects this by ranking the extinction groups by their training frequency and predicting that same fixed Top- list for every pattern, ignoring the input. We use the aggregate prior of the case-study calibration pipeline, estimated from its synthetic training corpus and released with the paper (Appendix § 19). We also report each benchmark’s own empirical label frequencies as an oracle prior-only control: computed from the test labels and serving as an upper bound on what any frequency-only predictor can achieve on that fixed set without inspecting the pattern. §6.3 applies both controls to the case-study models, and §6.4 uses the oracle control per severity stratum.
Figure 4. Extinction-group imbalance in the ICSD-derived source distribution underlying the aggregate synthetic training prior used for calibration. Horizontal axis: all 99 extinction groups, ordered by training frequency; vertical axis: prior probability; the ten most frequent groups are numbered on the bars and named in the inset. Heavy class imbalance means a naive prior-frequency baseline can attain non-trivial Top- accuracy with no reference to the input.
Label-permutation test. The label-permutation test asks what the prior-only baselines cannot: whether a model’s rankings depend on the individual diffraction pattern. Hold the model’s Top- candidate set fixed for every diffraction pattern and randomly reassign the benchmark labels among patterns. This preserves both the benchmark label frequencies and the model’s tendency to predict common groups, while destroying only the association between a particular diffraction pattern and its correct label. The shuffled-label mean is what the same candidate sets would score on random scans from the same benchmark, and a one-sided permutation -value compares the observed Top- (or mean true-label rank) against it; we use 10,000 permutations. Together the two controls and the test give a three-part reading of any model: whether it clears the deployable prior, where it sits relative to the oracle prior-only control, and whether its pattern-specific rankings retain a detectable association with the correct scans.
Hierarchy-Aware DAG Metric
Top-1 accuracy treats every misclassification as an equal failure, but from a crystallographic view, errors have structure. Misclassifying into a maximal subgroup—one edge away on the symmetry hierarchy—is a nearby mistake within the same family of symmetries; predicting an extinction group from a completely different crystal-system branch is a much more severe error.
We formalize this as graph distance on the condensed translationengleiche (t-subgroup) DAG released with Baggett et al. [2026], which we adopt unchanged. Nodes: the 99 extinction groups, with the hexagonal/trigonal extinction-class pair P--c / P-c- condensed into a single node (see the rationale in Appendix § 10), giving 98 nodes. Edges: t-subgroup relations, directed from higher to lower symmetry. We use unweighted hop distance on the t-subgroup graph alone (§ 2); weighted or mixed extensions are plausible but would add standardization choices, so the present metric is an auditable default rather than the only possible severity geometry.
For each wrong Top-1 prediction we compute three quantities: distance, direction, and locality. Error distance is the shortest-path distance between the predicted and true extinction group on the DAG, reported as the mean and the full distribution. Directionality records whether the prediction is a descendant of the truth (a subgroup: the model fell to lower symmetry), an ancestor (a supergroup: it asserted higher symmetry), or a branch jump (a different crystal-system family). Locality fraction is the share of wrong predictions within two hops.
Each quantity has a physical reading, and together they separate conservative errors from severe ones. Short distances with a descendant bias are the conservative error mode: noise erases evidence of high symmetry first, so falling to a nearby lower-symmetry class is the expected failure. Long distances with an ancestor bias are the opposite: the model asserts symmetry the pattern contradicts, often by leaning on the label prior. A high branch-jump rate is the most severe mode: the model leaves the local symmetry neighborhood, typically when strong preferred orientation suppresses a diagnostic peak family and the surviving pattern resembles a different branch. Distance and direction must be read together: a short mean distance can arise from genuinely local confusions or from a bias toward low-symmetry nodes, which are near-descendants of many groups.
Reference distributions. A hierarchy-aware summary is interpretable only against what an uninformed predictor would score on the same graph, so we report it against four references (values in Table 9, Appendix § 15). The uniform null enumerates, for every benchmark true label, all 98 incorrect extinction-group predictions with equal weight and measures the shortest distance on the same condensed graph; it is an exact expectation rather than a sample. Because models differ in which cases they get wrong, the model-matched null repeats the computation over each model’s own error set. The frequency-matched null draws the wrong label from the benchmark’s own extinction-group marginal and is the harder reference. The fourth reference, always naming the most frequent group, shows why distance must be read with direction (§ 6.1). All four are reported alongside the observed summaries. § 6.1 reads the ViT checkpoints and the CNN control against them, § 6.2 reads six independently trained architectures the same way, and Appendix § 14 does so for opXRD-98.
| Reference or model | Mean DAG distance | Within 2 hops (%) | Descendant / ancestor / branch (%) |
| Uniform null, RRUFF-473 | 2.997 | 29.76 | 8.65 / 35.61 / 55.69 |
| Uniform null, RRUFF-325 | 2.944 | 30.65 | 10.04 / 29.53 / 60.38 |
| Frequency-matched null, RRUFF-473 | 2.692 | 42.15 | — |
| Frequency-matched null, RRUFF-325 | 2.656 | 43.35 | — |
| Most-frequent-group predictor, RRUFF-325 | 2.116 | 63.4 | — |
| Final mixed-data ViT, RRUFF-325 | 2.46 | 51.55 | 56.7 / 10.7 / 32.6 |
| its model-matched null | 2.981 | 28.92 | 10.8 / 27.4 / 61.8 |
| Preferred-orientation-only ViT, RRUFF-325 | 2.51 | 53.0 | — / — / 34.1 |
| its model-matched null | 2.973 | 28.85 | 11.1 / 25.9 / 62.9 |
| 1D residual CNN (epoch 6), RRUFF-325 | 2.921 | 28.05 | 27.4 / 18.8 / 53.8 |
| its model-matched null | 2.937 | 30.45 | 10.5 / 27.5 / 61.9 |
Table 9. Reference distributions for the hierarchy-aware metric and three RRUFF-325 case-study models. Direction is the descendant / ancestor / branch-jump split over wrong predictions.
Relation to prior hierarchical metrics. Distance-based evaluation on a class hierarchy is a longstanding family of hierarchical-classification metrics [21, 6]; we do not claim hop-distance-on-a-DAG as a new construction, only its crystallographic instantiation (Appendix § 16).
Stratified Evaluation
An aggregate accuracy number over a benchmark collapses two distinct axes of variation: the intrinsic difficulty of the classification problem within a subset of the data, and the number of distinct labels to choose between in that subset. Stratified evaluation partitions the benchmark along both axes and reports per-stratum accuracy alongside the aggregate.
Data quality axis. Each RRUFF-473 pattern carries a nuisance-fit severity label—recoverable, usable, poor, or catastrophic—based on how well classical Rietveld profile fitting, the standard full-pattern least-squares refinement workflow in powder crystallography, handles it. Here “severity” means the severity of the residual mismatch after a standard classical fit, not the importance of the sample itself: usable patterns fit relatively cleanly, recoverable patterns remain interpretable but noticeably degraded, poor patterns retain substantial mismatch, and catastrophic patterns fail badly enough to act as stress tests.
Label-space entropy axis. Each stratum also contains a different number of unique extinction groups, and a narrower label distribution is intrinsically easier to classify, independent of data quality (§ 6.4 gives the RRUFF-473 counts). Reporting aggregate accuracy without the entropy decomposition conflates “the model does well on this stratum” with “this stratum has fewer classes to confuse.”
The protocol therefore reports aggregate Top- alongside per-stratum Top-, the number of unique extinction groups per stratum, and the entropy-reweighted per-stratum Top- that adjusts for label-space concentration. When a per-stratum effect persists after reweighting, it is a property of the model’s interaction with the data; when it disappears, it was a property of the label distribution. § 6.4 applies this decomposition to the severity strata of RRUFF-473, where it changes the interpretation of an apparent quality gap.
What Standard Evaluation Hides
We apply the framework to three model families: four publicly released ViT checkpoints from Baggett et al. [ [4] (no-preferred-orientation, preferred-orientation-only, final mixed-data calibrated, and legacy uncalibrated); an in-house 1D residual CNN control (Appendix § 18); and six independently developed architectures—CNN10, CNN11, PatchTST, BiLSTM, BiGRU, and a generic Transformer—trained under the XRDBench recipe of SimXRD [5] with 230-way outputs, collapsed into the 99-way space with the § 3 mapping and scored on identical RRUFF-473 and RRUFF-325 inputs (§ 4.2). Five findings follow, each about how the field reports results: Top-1 and Top-5 rank the same checkpoints in opposite directions, and the hierarchy-aware metric needs its own null (§ 6.1); paired Top- tests do not resolve differences among CNN10, CNN11, and PatchTST, while hierarchy-aware analysis describes error distance and direction across all six architectures (§ 6.2); an input-blind predictor matches the models on aggregate Top-, and only a permutation test recovers the pattern-specific evidence (§ 6.3); and classical fit quality does not predict neural difficulty once label-space concentration is controlled (§ 6.4). The small opXRD-98 example (§ 4.3) is scored under the same protocol in Appendix § 14.
Flat Accuracy and Hierarchy-Aware Severity Disagree on Model Ranking
On RRUFF-325, flat Top-1 and hierarchy-aware severity rank the five models differently (Table 1; Figure 3 in Appendix G shows some reversal): the model one would pick under flat Top-1, the preferred-orientation-only ViT checkpoint, is not the model one would pick under mean DAG distance, the final mixed-data calibrated checkpoint.
| Model | Top-1 (%) | hops (%) | Mean dist. () | Branch (%, ) | Top-1 rank | Dist. rank |
| ViT: no preferred orientation | 9.54 | 38.4 | 2.72 | 25.8 | 3 | 3 |
| ViT: preferred-orientation only | 13.54 | 53.0 | 2.51 | 34.1 | 1 | 2 |
| ViT: final mixed-data calibrated | 10.46 | 51.55 | 2.46 | 32.6 | 2 | 1 |
| 1D residual CNN | 6.77 | 28.05 | 2.92 | 53.8 | 4 | 4 |
| ViT: legacy uncalibrated | 4.92 | 17.8 | 3.50 | 56.4 | 5 | 5 |
Table 1. Five-model comparison on RRUFF-325: Top-1, -hop locality, mean DAG distance, and branch-jump fraction, with the implied ranks under flat Top-1 versus mean DAG distance. The two ranking columns disagree on the preferred-orientation-only and final mixed-data calibrated ViT checkpoints; the 1D CNN baseline is set apart from the ViT family by its 53.8% branch-jump rate. Hierarchy columns are over wrong Top-1 predictions; matched nulls for the preferred-orientation-only and final mixed-data calibrated ViTs and the CNN are given in Table 9. Boldface marks the best value in each column and does not denote statistical significance.

Figure 3. Flat accuracy and hierarchy-aware severity rank the same five models differently. Each line is one model; its left endpoint is its Top-1 rank, and its right endpoint is its mean-DAG-distance rank on the condensed -subgroup graph. Lower mean DAG distance is better. The preferred-orientation-only ViT checkpoint wins under flat Top-1, while the final mixed-data calibrated ViT checkpoint has the best raw mean-DAG-distance rank. The 1D CNN baseline remains worse on both metrics and, more importantly for this paper, demonstrates that the hierarchy-aware metric distinguishes a separate branch-jump-heavy failure regime that flat Top-1 alone would compress into a single score.
Table 1: Five-model comparison on RRUFF-325: Top-1, -hop locality, mean DAG distance, and branch-jump fraction, with the implied ranks under flat Top-1 versus mean DAG distance. The two ranking columns disagree on the preferred-orientation-only and final mixed-data calibrated ViT checkpoints; the 1D CNN baseline is set apart from the ViT family by its 53.8% branch-jump rate. Hierarchy columns are over wrong Top-1 predictions; matched nulls for the preferred-orientation-only and final mixed-data calibrated ViTs and the CNN are given in Table 9. Boldface marks the best value in each column and does not denote statistical significance.
Standard paired tests make the disagreement concrete: on the same 325 examples, Top-1 and Top-5 separate the ViT pair in opposite directions, while the hierarchy-aware summaries do not separate it at all. Exact McNemar tests give the preferred-orientation-only checkpoint the win at Top-1 (13.54% vs. 10.46%, = 0.002) and the final calibrated checkpoint the win at Top-5 (43.69% vs. 40.92%, = 0.012). Paired permutation tests on hierarchy-aware summaries do not separate the pair (mean severity-burden = −0.034, = 0.58); bootstrap CIs for the final calibrated checkpoint are reported in Appendix § 17. Even under standard paired testing, then, the same predictions support different conclusions depending on whether the benchmark asks for exact rank-1 identification or broader candidate recovery—and the hierarchy-aware metrics still expose distinct notions of error structure (the preferred-orientation-only checkpoint retains the highest -hop locality at 53.0%, while the final calibrated checkpoint achieves the tightest mean distance and best Top-5 behavior).
The framework also distinguishes architecture families, not just training variants. The 1D residual CNN control (Appendix § 18), evaluated at its epoch-6 checkpoint on RRUFF-325, reaches 6.77% Top-1—below the deployable prior-only baseline—but the more informative observation is structural: only 28.05% of its wrong predictions fall within hops, and 53.8% of its mistakes are branch jumps, versus 32.6% for the final mixed-data calibrated ViT checkpoint (Table 1)—a qualitatively different failure regime that flat Top-1 alone would suppress. Because an undertrained control could inflate the transformer family’s advantage, we tested convergence directly: resuming the same lineage for 40 further epochs and selecting on held-out synthetic validation loss alone, epoch 46 reaches the lowest validation loss yet is worse on both real benchmarks at every and less local on the DAG (Table 11). The submitted checkpoint was therefore the more favorable one for the CNN, and synthetic validation improving while both real benchmarks degrade indicates simulation-to-experiment domain shift rather than an intrinsic CNN limitation. The six XRDBench architectures of § 6.2 carry the comparative claims; the CNN stands as one fixed reference among several.
| Checkpoint | Val. loss | Val. Top-1/3/5 (%) | RRUFF-473 acc. | RRUFF-473 hops / mean dist. | RRUFF-325 acc. | RRUFF-325 hops / mean dist. |
| Submitted epoch 6 | 1.6631 | 50.25 / 75.47 / 84.85 | 9.09 / 19.66 / 28.54 | 28.84% / 2.956 | 6.77 / 18.77 / 25.23 | 28.05% / 2.921 |
| Validation-selected epoch 46 | 1.3948 | 58.00 / 81.30 / 89.03 | 3.59 / 8.46 / 15.64 | 17.98% / 3.342 | 2.15 / 4.92 / 12.92 | 20.13% / 3.255 |
Table 11. Convergence test for the 1D residual CNN control. Validation loss and Top-1/3/5 are on held-out synthetic data; benchmark accuracy columns are Top-1 / Top-3 / Top-5 (%); hierarchy columns are the within-2-hop fraction and mean DAG distance over wrong predictions. Epoch 46 is the validation-loss minimum after repeated learning-rate reductions; the formal early-stopping rule (patience 8) had not fired.
Reading the hierarchy metric against its nulls. Against the reference distributions of § 5.2 (Table 9), the final mixed-data ViT is genuinely more local than chance while the CNN control is not. On RRUFF-325 the ViT is well inside its model-matched null on both distance and locality, is strongly biased toward lower-symmetry descendants, and also clears the harder frequency-matched null on both measures. The CNN control sits at its matched null on distance, with a much smaller directional shift. One caution follows. Always naming the most frequent group scores 2.116 and 63.4% on RRUFF-325, better than any model on distance alone, because low-symmetry nodes are near-descendants of many groups; a low mean distance with a strong descendant bias is therefore a different finding from a low mean distance with a branch-heavy error set. The preferred-orientation-only checkpoint, which wins flat Top-1, also sits well inside its own matched null (Table 9), so the ViT pair is separated by flat metrics but not by hierarchy-aware ones.
Accuracy and Error Structure Across Independent Architectures
Paired tests do not resolve Top-1, Top-3, or Top-5 differences among CNN10, CNN11, and PatchTST on either RRUFF benchmark. Across the six architectures, hierarchy-aware analysis reveals differences in error distance and direction that aggregate accuracy does not describe (Table 2). We trained the six models under the XRDBench recipe because four checkpoints from one ViT family are not enough to establish that evaluation design changes conclusions across architectures, and scored them on identical RRUFF-473 and RRUFF-325 inputs. Exact two-sided McNemar tests on every pair among CNN10, PatchTST, and CNN11 fail at Top-1, Top-3, and Top-5 on both benchmarks, the smallest raw being 0.124 (Holm-adjusted 0.371, for PatchTST versus CNN11 at RRUFF-325 Top-3), and every 10,000-replicate paired-bootstrap interval includes zero; the largest tested difference is +4.0 points with 95% CI [−0.6, 8.6].
Table 2: Six XRDBench architectures rescored in extinction-group space. Models were trained on the unchanged public SimXRD train_1 contract [5] and evaluated on RRUFF patterns exactly decimated onto that grid (contract specification in Appendix § 19). Hierarchy columns are over wrong Top-1 predictions and give observed / model-matched null (§ 5.2); on RRUFF-325 matched nulls were computed for the two strongest models. No pair among CNN10, PatchTST, and CNN11 is separable by exact McNemar test at Top-1, Top-3, or Top-5 on either benchmark (smallest raw ). Boldface marks the best value per column, not statistical significance.
| Model | RRUFF-473 | RRUFF-325 | ||||
| Top-1 / 5 (%) | ≤2 hops obs./null | Mean dist. obs./null | Top-1 / 5 (%) | ≤2 hops obs./null | Mean dist. obs./null | |
| CNN10 | 12.47 / 45.67 | 37.68 / 31.43 | 2.797 / 2.910 | 9.85 / 39.08 | 35.15 / 31.42 | 2.877 / 2.901 |
| PatchTST | 12.47 / 45.45 | 40.10 / 30.01 | 2.688 / 2.969 | 9.23 / 40.62 | 37.29 / 30.84 | 2.783 / 2.927 |
| BiLSTM | 8.67 / 30.87 | 34.49 / 31.26 | 2.894 / 2.929 | 4.92 / 24.00 | 34.95 / — | 2.903 / — |
| CNN11 | 11.63 / 45.88 | 38.52 / 31.09 | 2.749 / 2.919 | 8.31 / 40.31 | 36.24 / — | 2.842 / — |
| BiGRU | 10.57 / 33.62 | 32.86 / 31.75 | 2.877 / 2.901 | 6.77 / 25.85 | 31.68 / — | 2.921 / — |
| Transformer | 10.15 / 40.38 | 16.71 / 32.54 | 3.386 / 2.864 | 5.23 / 33.54 | 16.88 / — | 3.461 / — |
Table 2.
The hierarchy-aware summaries provide a different view of the errors. PatchTST, CNN11, and CNN10 are clearly more local than their own model-matched nulls; BiLSTM and BiGRU sit close to theirs; and the generic Transformer is markedly worse than its own null (16.71% against 32.54% within two hops on RRUFF-473; 16.88% on RRUFF-325) while directing 95.8% of its errors toward lower-symmetry descendants against 9.6% under its matched null. Because the distance margins narrow against the frequency-matched null—PatchTST’s RRUFF-473 result (2.688, 40.10%) does not clear it (2.692, 42.15%)—we treat the directional contrast as the firmer result. Together with § 6.1, these results show how hierarchy-aware analysis complements accuracy by describing the distance and direction of errors. Neither readout dominates, and what a benchmark can resolve depends on the evaluation design, not only on the models.
Prior-Only Baselines and the Permutation Test: Where the Evidence Is
The final calibrated model clears the deployable prior of § 5.1 but does not exceed the oracle prior-only control, and the two controls must be kept apart. The deployable prior reaches 9.7% Top-1 and 27.2% Top-5 on extinction-group labels derived from real RRUFF patterns. The oracle prior gives the highest accuracy attainable by a fixed input-blind ranking on each benchmark, at 12.9/51.4% on RRUFF-473 and 12.6/43.7% on RRUFF-325 (Table 6). The final mixed-data calibrated ViT checkpoint reaches 9.94/50.74% on RRUFF-473 and 10.46/43.69% on RRUFF-325: above the deployable prior at Top-5 by a wide margin, below the oracle control except for a tie at RRUFF-325 Top-5. Stated plainly, an input-blind predictor that knew the benchmark’s label histogram would match or exceed this model’s aggregate Top-, and the comparison with the deployable prior does not by itself establish that the model uses the diffraction pattern. A model using the diffraction pattern can outperform the oracle prior-only control. On RRUFF-325 the preferred-orientation-only ViT does so at Top-1, reaching 13.54% against the oracle’s 12.6%. On opXRD-98 (Appendix § 14) the same control is even more decisive: an input-blind predictor using opXRD-98’s own label marginal scores 22.45/66.33%, above every one of the six XRDBench models.
| Benchmark / stratum | Top-1 | Top-5 | Top-10 |
| RRUFF-473 prior | 12.9 | 51.4 | 68.7 |
| RRUFF-325 prior | 12.6 | 43.7 | 62.8 |
| Recoverable prior | 12.2 | 41.3 | — |
| Usable-or-better prior | 15.7 | 48.4 | — |
| Poor prior | 16.2 | 66.2 | — |
| Catastrophic prior | 28.4 | 75.7 | — |
Table 6. Benchmark-matched prior-only controls. “RRUFF prior” denotes the empirical extinction-group distribution of the named benchmark itself rather than the aggregate synthetic-training prior.
The label-permutation test of § 5.1, applied to each pattern’s Top-5 candidate set, answers the narrower question, whether the model’s rankings depend on the individual diffraction pattern, in the affirmative but by a small margin. Observed RRUFF-473 Top-5 is 50.74% against a shuffled-label mean of 48.52% (+2.22 points, one-sided ). On RRUFF-325, observed Top-5 is 43.69% against 41.48% after shuffling (+2.21 points, ), and the mean rank of the true label improves from the shuffled 13.71 to 13.43 (); the RRUFF-473 mean-rank test is weaker (). The tie with the oracle at RRUFF-325 Top-5 is instructive: model and fixed prior are each Top-5-correct on 142 of 325 patterns, but the sets differ (132 shared, 10 model-only, 13 prior-only), which is exactly the pattern-specific component the permutation test isolates. The summary is three-part: the model clears the deployable prior at Top-5, matches or falls below the oracle prior-only control in aggregate, and its pattern-specific rankings retain a small but statistically detectable association with the correct scans.
The Catastrophic Paradox as a Decomposition Exercise
Applying stratified evaluation to RRUFF-473 reveals a counterintuitive observation: classical Rietveld fit quality does not cleanly predict neural classification difficulty. On the final mixed-data calibrated ViT checkpoint, raw Top-1 across the recoverable, usable-or-better, poor, and catastrophic strata is 11.05%, 9.80%, 10.81%, and 6.76%, respectively. After reweighting each stratum to a common extinction-group distribution, the corresponding values become 13.18%, 10.61%, 10.71%, and 10.80%. Thus, differences in label-space composition explain about 45% of the recoverable-versus-catastrophic gap, but not all of it.
The aggregate Top-1 on RRUFF-473 is 9.94%; under standard reporting that is all a reader sees, yet three mechanisms contribute to it and the aggregate cannot separate them: label-space concentration (the catastrophic stratum spans only 13 unique extinction groups to recoverable’s 37, so a model can score well with a narrower effective classifier); intrinsic classification difficulty (the residual gap after entropy reweighting reflects whatever structure in the data actually matters to the model); and aggregation bias (the catastrophic stratum is 74 patterns versus recoverable’s 172, so its contribution to the aggregate is weighted accordingly). Only stratified reporting with entropy reweighting separates these: in the final mixed model, reweighting compresses the three harsher strata into a narrow 10.6–10.8% band, while recoverable remains distinctly higher. The benchmark-matched prior-only ceilings vary sharply across the same strata, from 12.2%/41.3% on recoverable to 28.4%/75.7% on catastrophic at Top-1/Top-5 (Table 6), confirming that label-space concentration differs dramatically across the severity partition. The residual therefore says something specific: recoverable patterns are harder for a classical refinement procedure yet easier for the network than usable-or-better patterns once label-space concentration is controlled—a decoupling that is invisible from aggregate Top-1 (Figure 2).
Figure 2. Decomposition of the catastrophic paradox on RRUFF-473. Left: the severity strata differ strongly in label-space structure, with the catastrophic stratum spanning far fewer unique extinction groups than the recoverable stratum. Right: classical profile-fit quality and neural classification difficulty do not align monotonically, and extinction-group reweighting removes about 45% of the recoverable-versus-catastrophic gap.
Conclusion
We have presented an evaluation framework for PXRD machine learning addressing three failures: wrong targets, non-reproducible benchmarks, and uninformative metrics. We show that evaluation choices change the conclusions reached (§6) and additionally apply our benchmark-construction method to opXRD (Appendix § 14).
Limitations. RRUFF-473 and RRUFF-325 cover only 46 and 45 of the 99 extinction groups, and we do not infer beyond them; both are restricted to Cu-Kα laboratory data and RRUFF’s primary-phase labels, and the paired Top- tests do not resolve differences among CNN10, CNN11, and PatchTST (§6.2). The final calibrated model matches or falls below the oracle prior-only control at Top-1 and Top-5, its pattern-specific rankings retaining only a small but detectable association with the correct scans (§6.3). The DAG is unweighted and -only, and because distance margins narrow against the frequency-matched null we treat cross-architecture distance comparisons as suggestive. Stability is checked by threshold sweeps, not alternative refinement workflows, and the RRUFF transfer check of the target-choice effect uses small, domain-shift-dominated sets, with the synthetic scaling study providing that evidence (Appendix § 11). Only the four ViT checkpoints and our own models can be rescored.
More broadly, the field needs larger, more crystallographically diverse benchmarks with reproducible inclusion rules and routine release of runnable checkpoints, so published systems can be rescored rather than compared through incompatible numbers. The artifacts released here (Appendix § 19) are offered toward that end, and toward treating evaluation as a scientific object rather than a single aggregate score.
Acknowledgments and Disclosure of Funding
The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin (http://www.tacc.utexas.edu) for computational resources, made available to NIST under contract 1333ND25PNB180410, that contributed to the results reported here. Additional computational resources were provided through ACCESS allocation PHY250007, “Applications of AI to Diffraction.” Edward G. Friedman and Elizabeth J. Baggett were supported by the Center for High Resolution Neutron Scattering, a partnership between the National Institute of Standards and Technology and the National Science Foundation under Agreement No. DMR-2010792.
Mapping Construction and Verification
Extinction groups in plain terms. A screw axis or glide plane makes every reflection in some family with a particular index parity vanish, and two space groups whose symmetry elements erase exactly the same reflections leave identical fingerprints in a powder pattern; the extinction group is the name for the set of space groups sharing that fingerprint. Space groups (No. 4) and (No. 11) share a class for this reason: the screw axis produces the same absences in both, and the added mirror produces none. Extinction groups therefore define the finest evaluative label space that PXRD can justify, and a benchmark that scores in full 230-way space-group space risks penalizing a physically correct prediction simply because the measurement cannot distinguish two classes.
Construction. The extinction-group mapping released with this paper is built from the extinction-symbol table distributed with EXPO [1], which derives from International Tables for Crystallography [3] and was first released in machine-usable form by Baggett et al. [4]. The construction is deterministic: we fetch and parse that table, expand its range and list notation into explicit space-group numbers, merge symbols carrying identical space-group sets into canonical classes, assign stable integer labels, annotate crystal system and Laue class, invert the result into a complete 230-row lookup, and cross-check the assignments against International Tables, whose treatment of extinction symbols has itself changed across editions.
The released artifact is provided as JSON and CSV. Each extinction group carries an integer label, a canonical standard crystallographic symbol, and the associated space-group numbers; the inverted 230-row table gives, for every space group, its extinction group, Laue group, and crystal system. The integer labels are stable and are the label space used by every benchmark and metric in this paper, so predictions produced under the mapping remain comparable across studies that adopt it.
The Hexagonal/Trigonal Extinction-Class Pair
The merge referred to in § 2 and § 5.2 is a taxonomy choice for the condensed subgroup graph, not an inference from model performance, and the 99-way label space retains both classes. The two classes are (space groups 159, 163, 186, 190, and 194) and (space groups 158, 165, 185, 188, and 193). Indexing powder lines by the hexagonal invariant , the reflection conditions tabulated in International Tables [3] place the absences on the family at odd and the absences on the family at odd . The two absence sets are the same pattern related by , the basal rotation that rescales the in-plane cell edge by , so the difference between the classes is a basal-direction convention rather than a different kind of absence.
The limit should be stated precisely. The distinction is not erased by powder averaging; it rests entirely on one low-angle reflection family (10.1 absent versus 11.1 absent), so it is recoverable in principle yet fragile on scans where that single line is weak, overlapped, or outside the measured window. We merge the pair in the condensed graph because the two classes are additionally related by -subgroup relations in both directions, which makes them incomparable in the hierarchy and requires condensation to obtain a directed acyclic graph.
The footprint on the benchmarks is exact. Extinction group 74 (-c-) never occurs as a true label in RRUFF-473 or RRUFF-325, while extinction group 73 (--c) occurs 21 and 17 times. The merge therefore changes no reported accuracy; its only effect is that a prediction of group 74 against a true group 73, possible in at most 21 of 473 and 17 of 325 cases, is scored at DAG distance 0 rather than as a near miss. The explicit reflection-condition enumeration is included in the released code.
Space-Group versus Extinction-Group Target
Experimental setup. Table 3 gives the matched, convergence-controlled comparison summarized in §3. Both arms were trained with the pipeline and two-million-pattern synthetic corpus of Baggett et al. [2026] and share the same patterns, split, ViT backbone, initialization, optimizer, augmentations, training budget, and validation-selection rule; only the target and output layer differ. The direct extinction-group arm corresponds to the extinction-group model of Baggett et al.; the space-group arm was trained to convergence under the identical recipe so that the comparison is not confounded by training budget. The space-group model is read three ways: in its own 230-way space, after its probabilities are summed into the 99-way extinction-group space, and against the model trained directly on extinction groups.
Scaling study and transfer rows. The matched scaling study is the informative result. The Top-1 advantage of direct extinction-group training over the collapsed space-group model is +0.59, +1.45, and +2.77 points at one-half, one-quarter, and one-eighth training scale, respectively, against +0.86 at full scale, so direct extinction-group supervision is primarily a sample-efficiency advantage, and as the space-group corpus grows the space-group model learns much of the extinction-group equivalence structure implicitly. The transfer rows score both uncalibrated synthetic-only arms on identical RRUFF inputs in the common extinction-group space; they are not comparable with the calibrated checkpoints of Table 1, and every paired-bootstrap interval includes zero, so these small, domain-shift-dominated sets serve as a transfer check rather than as an estimate of the target-design effect.
| Setting | SG model, SG space | Collapsed SG, EG space | Direct EG | Direct-EG advantage |
| Synthetic validation, full scale | 15.17 / 27.81 / 35.25 | 18.46 / 33.27 / 42.35 | 19.32 / 34.28 / 43.38 | +0.86 / +1.01 / +1.03 |
| Synthetic, scale (Top-1) | — | — | — | +0.59 |
| Synthetic, scale (Top-1) | — | — | — | +1.45 |
| Synthetic, scale (Top-1) | — | — | — | +2.77 |
| RRUFF-473 | — | 3.38 / 5.92 / 8.25 | 3.59 / 5.71 / 8.46 | +0.21 [−1.27, 1.69], |
| RRUFF-325 | — | 1.54 / 4.31 / 7.08 | 2.77 / 5.23 / 8.31 | +1.23 [−0.31, 2.77], |
Table 3. Matched comparison of a 230-way space-group (SG) model and a 99-way extinction-group (EG) model. “Collapsed SG” scores the SG model after its probabilities are summed into EG space. All accuracy cells are Top-1 / Top-3 / Top-5 (%). Advantage is direct EG minus collapsed SG in percentage points; RRUFF advantages are Top-1 with paired-bootstrap 95% CI and exact McNemar (Top-3 and Top-5 differences on both benchmarks likewise have and intervals containing zero).
Construction Method: Full Step Descriptions
A source archive is a collection of heterogeneous measurements, not a benchmark: it may contain multiple scans per material, incomplete or multiphase labels, mixed wavelengths, exact duplicates, no canonical split, and no duplicate policy. The construction method turns such an archive into a fixed evaluation set whose inclusion rules are specified as code and reproduce from a frozen snapshot. Its steps are:
Single assignable label. Retain records with one phase whose space group is stored/derivable and map it to the extinction-group label space of §3. Multiphase records without a designated primary
phase are excluded, because phase order cannot be assumed to identify the dominant diffraction signal.
Wavelength and grid gate. Select scans collected at the target wavelength (Cu-Kα, either an explicit Kα1/Kα2 pair or the conventional 2:1 weighted average), require complete angular support at native resolution over the evaluation window, and resample each onto a standard grid.
Structural grouping. Assign each record a structural-family label, used both to group structurally near-identical records so that they are not counted as independent and to stratify results. The label source is archive-specific: a mineralogical family when the archive supplies one, otherwise a structure-derived community assignment.
Family coherence. Keep only families that are internally consistent under within-family diagnostics (full-pattern correlation, low-angle correlation, and a rank-1 reconstruction angle), separate distinct variants by label and lattice-compatible clustering, and admit singletons only when they remain similar to the family core in both lattice parameters and pattern shape.
Exact-pattern deduplication, conflict-aware: byte-identical patterns are reduced to one representative, and groups whose identical patterns carry conflicting labels are excluded.
Per-pattern strata. Attach nuisance-fit severity from classical Rietveld profile fitting under fixed cutoffs, and record any measurement stratum that varies within the archive (for example wavelength mode) so it can be reported separately.
Freeze and audit. Release the exact record identifiers tied to a snapshot date, the curation code with explicit thresholds, and a threshold-sensitivity sweep; if the upstream archive changes, the frozen identifier list is the reference.
Steps 1, 2, and 7 are identical for every archive; steps 3–6 are where a source can require a different instantiation, and Figure 1 records, step by step, how each was instantiated on the two archives.
Additional RRUFF Details and Discussion
This appendix gives the RRUFF instantiation of the method of Appendix D, summarized in §4.2 and Figure 1, the released defaults, the robustness summaries that support the benchmark-design choices, and the coverage rationale.
Source and label assignment. The frozen visible-filtered Cu-Kα source pool contains 1,981 scans. RRUFF names a primary mineral phase for every entry, and its space group is mapped to the 99-way extinction-group label space before the coherence and quality gates are applied. The scans may contain secondary phases or impurities, but RRUFF identifies the primary mineral.
Curation pipeline. The RRUFF instantiation of the method of Appendix D is as follows. The reduction to RRUFF-473 is an explicit family-coherence curation step, not a metadata screen: retained families must remain internally consistent in lattice-scale metadata and in full-pattern / low-angle correlation and reconstruction diagnostics.
(a) Filter and resample. Select RRUFF scans collected at Cu-like wavelengths () and resample each onto a standard 8501-point grid (–, step).
(b) Group by mineral family and use repeated families to decide which minerals support a stable real-data benchmark at all.
(c) Keep only stable families using within-family consistency diagnostics: full-pattern Pearson correlation, low-angle correlation, and a rank-1 reconstruction angle derived from the leading singular vector of the within-family pattern matrix.
(d) Separate distinct variants within retained families by mineral + space-group assignment and by tight lattice-compatible clusters.
(e) Keep the coherent core algorithmically: admit singleton clusters only when they remain similar to the main family cluster in both lattice parameters and pattern shape. This yields RRUFF-473.
(f) Assign nuisance-fit strata via classical Rietveld profile fitting, i.e., full-pattern least-squares refinement of a crystallographic model against the observed diffraction trace: recoverable (172), meaning the pattern is difficult but still reasonably explained by a standard refinement; usable (153), meaning the fit is cleaner and more deployment-like; poor (74), meaning substantial mismatch remains after fitting; and catastrophic (74), meaning the classical fit fails badly enough that the pattern should be treated as a stress case rather than a routine one. (g) Apply fixed cutoffs ( for the higher-quality usable-or-better subset; for the recoverable stratum) to retain usable and recoverable patterns, producing RRUFF-325.
Released benchmark defaults. The curation defaults are explicit rather than hidden in code: the family-coherence stage uses fixed thresholds on lattice distance, full-pattern and low-angle correlation, anchor-cell agreement, and partial-cluster tie-break margin, and the nuisance-fit labels are deterministic cutoffs (usable-or-better , recoverable , poor , catastrophic above). Exact threshold values, per-scan QC fields, and the fitted lattice/profile/background parameters are listed in the released manifest. We treat those released refinement outputs as fixed benchmark metadata; the robustness analyses below therefore address threshold sensitivity of the benchmark definitions, not invariance across alternative refinement workflows. For reference, the 74 catastrophic patterns of RRUFF-473 span 13 unique extinction groups and 38 minerals, and the 172 recoverable patterns span 37 extinction groups and 111 minerals.
Benchmark stability. A deterministic benchmark must be stable under reasonable variation of its thresholds. Sweeping the RRUFF-325 nuisance-fit cutoffs, the retained subset size ranges from 271 to 359 patterns while the calibrated reference model remains stable at 9.9–10.5% Top-1 and 41.0–45.4% Top-5—conclusions drawn on RRUFF-325 are therefore not brittle to cutoff choice. A local one-at-a-time sweep around the released family-coherence defaults left the benchmark core largely unchanged; only the loosest tested settings introduced a handful of singleton admissions or one-to-two marginal swaps (Table 5). The per-pattern severity labels are themselves a released artifact, so a third party can apply stratified analysis to their own model’s predictions on RRUFF-473 without redoing any Rietveld fits.
| Variant | Selected | Intersection | Selected-only | Reference-only |
| Baseline | 471 | 471 | 0 | 0 |
| Anchor full threshold 0.86 | 473 | 471 | 2 | 0 |
| Anchor cell threshold 0.0020 | 472 | 471 | 1 | 0 |
| Partial tie-break margin 0.04 | 471 | 469 | 2 | 2 |
| Lattice threshold 0.0006 | 471 | 470 | 1 | 1 |
Table 5. Local one-at-a-time sensitivity of the family-coherence curation defaults. Reported variants are the only tested perturbations that changed benchmark composition beyond the baseline 471/473 core match.
Coverage and intended use. Both benchmarks are fixed evaluation sets for extinction-group classification on Cu-K laboratory PXRD, not general-purpose training corpora, and contain only experimental RRUFF measurements; the synthetic corpus used to train the case-study models is training data, not part of the release. To our knowledge, RRUFF remains the largest open experimental powder-diffraction source with broadly usable primary-phase labels, but the archive is strongly imbalanced and lacks complete extinction-group coverage, so a uniform experimental split would leave many classes with too few examples. These limitations are what motivate a fixed benchmark. Existing studies use different RRUFF subsets and preprocessing, preventing direct comparison; InformedXRD freezes the record identifiers and deterministic inclusion rules so that models can be tested on the same experimental cases. Until a larger labeled collection exists, a clearly bounded common benchmark is preferable to investigator-selected subsets, and balanced simulation can cover all 99 labels but cannot replace missing experimental data.
Additional opXRD Details and Discussion
This appendix gives the gate table behind Figure 1 (right), the reasons the benchmark was not built on opXRD and the two sets are not merged, and the evaluation of the six XRDBench architectures on opXRD-98, the small example constructed in §4.3. The exact record manifest and processing scripts implement the freeze-and-audit step.
Complementarity and processing. opXRD [9] is an open archive of 92,552 experimental powder patterns from multiple laboratories, still growing, and parsing cleanly enough that we built pymatgen structures [15] directly from the shipped cells and sites. opXRD and InformedXRD are complementary rather than competing—one is an experimental data archive, the other an evaluation protocol—so the comparison we can offer is not feature-by-feature but what happens when the protocol is applied to the archive. The exact list of records used and the processing scripts are released with the paper.
One protocol step instantiated differently. Steps 1, 2, and 7 transfer unchanged; steps 3–6 are re-instantiated, as §4.1 and Appendix D anticipate, and this is one concrete test of the design’s generality. RRUFF supplies a mineralogical family, which the protocol uses both to group structurally near-identical records so that they are not counted as independent and to stratify results. opXRD carries no such label, so we projected its structures into the frozen ICSD structural map of Nguyen et al. [ [14], whose author list overlaps ours. That map embeds published ICSD structures in a similarity space and partitions them into communities of structurally related materials; a record lying inside its community’s typical spread is in-basin, one at the edge is frontier, which separates well-precedented structure types from unusual ones. The two jobs the mineral family does on RRUFF are therefore done from structure alone: 95 of the 98 records carry an assignment (67 in-basin, 28 frontier), family coherence becomes distance to a community’s own 95th-percentile spread, which stratifies rather than excludes, and structures outside the published projection scope are reported as explicit errors rather than silently labeled outliers. Deduplication runs as an explicit byte-identical pass rather than being absorbed into coherence, and with no Rietveld fits the wavelength mode replaces severity as the reported stratum. The role of each step in the protocol is unchanged; only the source of the label differs. A community assignment does not by itself confirm that a measured pattern matches its deposited structure, which remains a separate profile-consistency step. Because opXRD has no Rietveld-fit severity labels, this instantiation does not produce a second quality-filtered set; opXRD-98 remains a stress test rather than a benchmark.
Why the benchmark was not built on it. Our task is one-pattern, one-label supervised classification, and that is where the archive is currently thin once filtered (Table 7). We chose 10–80° to match the SimXRD input range, not to maximize the count; at 5–90, 5–80, 10–90, and 15–80° the counts are 13, 39, 50, and 139. RRUFF also names a primary mineral phase for every entry; opXRD’s 320 multiphase symmetry-complete records do not list phase order, and order cannot be assumed to identify the dominant phase, so they cannot enter a one-label set. Those records could support a set-valued evaluation, which we think interesting, but they are not one-label examples.
| Gate | Records remaining |
| Initial archive | 92,552 |
| Declares a phase | 2,179 |
| Space group assignable to each phase (316 records with all phase labels stored explicitly; remainder derived from structure with spglib [25]) | 1,228 |
| Single phase with lattice and derivable space group (891 from one source) | 908 |
| Cu-K gate: explicit K/K pair or the conventional 2:1 weighted average | 229 |
| RRUFF angular support at native resolution, 10–80° | 98 (28 space groups, 23 extinction groups) |
Table 7. Filtering gates applied to opXRD to obtain opXRD-98, a one-pattern, one-label Cu-K set comparable with the RRUFF benchmarks.
Why the two are not merged. The direct structural overlap is 6 opXRD records across 23 RRUFF entries, so an intersection is empty for practical purposes. A union would be worse: the label distributions differ measurably (space-group total-variation distance 0.391, Jensen–Shannon divergence 0.216 bits), and the two sets differ in phase semantics, angular support, instrument distribution, and wavelength mode, so pooling would confound source with label distribution—the failure mode this paper argues makes cross-study numbers uninterpretable. Reporting them as separate stratified sets is what our own protocol requires.
What opXRD-98 shows. After exact 0.01-to-0.02° decimation and matched normalization, the six SimXRD-trained XRDBench architectures reach extinction-group Top-1 of 4.08–10.20% and Top-5 of 22.45–38.78% on opXRD-98 (Table 8). We present this as a stress test, not a benchmark, and report its limits: an input-blind predictor using opXRD-98’s own label marginal scores 22.45/53.06/66.33% Top-1/3/5, above every model, and no model improves on its matched DAG null (2.820–2.840; the exact uniform null is 2.876 with 32.62% within two hops).
| Model | EG Top-1 / 3 / 5 (%) | ≤2 hops (%) | Mean DAG dist. |
| PatchTST | 10.20 / 20.41 / 34.69 | 32.95 | 3.034 |
| CNN11 | 6.12 / 20.41 / 26.53 | 31.52 | 3.174 |
| CNN10 | 5.10 / 19.39 / 38.78 | 29.03 | 3.258 |
| BiLSTM | 5.10 / 15.31 / 22.45 | 23.66 | 3.441 |
| BiGRU | 4.08 / 17.35 / 24.49 | 20.21 | 3.394 |
| Transformer | 4.08 / 22.45 / 31.63 | 17.02 | 3.745 |
| Input-blind (opXRD-98 marginal) | 22.45 / 53.06 / 66.33 | — | — |
Table 8. Six XRDBench architectures on opXRD-98 (98 Cu-K records), rescored in extinction-group space. Hierarchy columns are over wrong Top-1 predictions.
Top-5 is also far higher on the 33 weighted-average records (39.39–51.52%) than on the 65 explicit-pair records (13.85–33.85%); wavelength mode is confounded with source and label composition here, so this is descriptive rather than causal, but it shows that wavelength must remain a reported stratum. Finally, one finding we offer back to the archive: among the 908 single-phase records, 13 groups of byte-identical raw patterns (27 records) carry 6 conflicting space-group/extinction-group labels, leaving a conflict-aware exact-pattern set of 888 records.
Reference Distributions for the Hierarchy-Aware Metric
Table 9 lists the four reference distributions defined in § 5.2. The uniform null is model-independent and exact. It uses the 98 wrong labels rather than all 99 so that it is comparable with the observed statistic, which is averaged over the model’s wrong predictions only; letting the null draw the correct label would hand it a free distance of zero. The model-matched null is recomputed over each model’s own error set and is the reference against which the observed summaries in Tables 1 and 2 should be read. The frequency-matched null draws the wrong label from the benchmark’s own extinction-group marginal. The most-frequent-group predictor is included to show why a low mean distance is not by itself evidence of crystallographically informed errors.
Model-matched nulls are reported for the checkpoints whose per-case predictions are in the released failure-mode artifacts; the uniform null is exact over and wrong-label pairs.
Relation to Prior Hierarchical-Classification Metrics
Hierarchical classification metrics fall into four broad families: set-overlap, distance-based, path/level-aware, and cost-weighted [21]. We opted for a distance-based metric to answer the crystallographic question of how far the predicted symmetry is from the true one, but ancestor/overlap, level, and cost-weighted metrics are fair choices. The contribution here is the crystallographic instantiation: the condensed -subgroup graph over extinction groups (a -only restriction avoids conflating point-group symmetry loss with lattice-volume change, which a mixed graph would homogenize), the unweighted-edge choice (auditable, integer-valued, free of subjective severity weights), and the descendant/ancestor/branch-jump disaggregation that reads directly against the diffraction error model. Many scientific classification problems share this structure (biological taxonomy, medical ontologies, material hierarchies), and we hope this work serves as an example for other domains where flat accuracy remains the default.
Bootstrap Confidence Intervals
Table 10 gives nonparametric bootstrap 95% intervals (2,000 resamples over benchmark cases, seed 124) for the final mixed-data calibrated ViT checkpoint. The intervals are wide relative to the between-checkpoint differences in Table 1, which is why the paper relies on paired tests on the same cases rather than on overlapping marginal intervals.
| Benchmark | Top-1 (%) | Top-5 (%) |
| RRUFF-473 () | 9.94 [7.61, 12.68] | 50.74 [46.29, 55.18] |
| RRUFF-325 () | 10.46 [7.38, 13.85] | 43.69 [38.46, 48.92] |
Table 10. Bootstrap 95% confidence intervals for the final mixed-data calibrated ViT checkpoint (point estimate [CI]).
The 1D Residual CNN Control
The control is a 1D residual CNN with 646,596 parameters: a single-channel stem convolution (kernel 13, stride 1) with LayerNorm, GELU, and dropout 0.05; max-pooling with kernel and stride 7; four residual stages of two blocks each with channels , kernels , and strides , each block using two convolutions with LayerNorm and dropout, a projected residual where the shape changes, and GELU; adaptive max-pooling; and a linear 99-class extinction-group head. It was trained with AdamW (learning rate , weight decay ), batch size 128, cross-entropy loss, and reduce-on-plateau learning-rate decay (factor 0.3, patience 2, floor ), seed 1337, on a 2.35M-pattern RRUFF-conditioned synthetic corpus (1,878,370 training and 237,056 validation patterns) on the same 8501-point – grid as the benchmarks. Its exact epoch-6 checkpoint, with weights and configuration metadata, is included in the released model artifact, so the evaluated model is recoverable without the ViT code path. Table 11 reports the convergence test of § 6.1: the submitted lineage was resumed for 40 further epochs with checkpoint selection on held-out synthetic validation loss alone, and the validation-selected epoch 46 is better on synthetic validation and worse on both real benchmarks.
Released Artifacts
Relationship with Prior Release. Baggett et al. [Baggett et al., [4] first introduced and publicly released the extinction-group mapping, RRUFF-473 and RRUFF-325 with their nuisance-fit labels and curation logic, the condensed subgroup graph, and the four ViT checkpoints. This paper reuses those released components and credits them where each is introduced (§3, §4, and §5.2). The new contribution here is the standardized evaluation package—the frozen artifacts, the deployable and oracle prior controls, the permutation test, the null reference distributions for the hierarchy metric, the stratified protocol, and the cross-architecture and cross-archive case studies—and the checkpoints are evidence for how that package changes what claims are justified from the same predictions. The nuisance/entropy observation and the subgroup graph appeared first in Baggett et al.; here they are load-bearing parts of a method rather than findings. opXRD-98, its record list, and its processing scripts exist only in this paper.
XRDBench training contract. The six XRDBench architectures of Table 2 were trained on the unchanged public SimXRD train_1 contract [5]: 597,845 patterns, 230 space-group targets, – at , and 3,501 points per pattern.
The aggregate prior. We use a compact aggregate prior estimated from the synthetic training corpus used by the case-study calibration pipeline. The pattern-level corpus is not redistributed, but the released prior is the complete aggregate quantity the evaluation consumes—a 99-class prior with probabilities, log-priors, seen-class indicators, and the smoothing parameter—so every prior-only and prior-corrected number reported here is reproducible without it. Figure 4 shows the extinction-group imbalance of the source distribution behind it.
All artifacts (mapping tables, DAG adjacency lists, stratification labels, evaluation scripts, benchmark payloads, case-study checkpoints, and the processing scripts and exact record list for the opXRD case study of Appendix F) are public: the GitHub reproducibility repository https://github.com/scattering/InformedXRD carries the curation and evaluation code, and the companion Hugging Face dataset https://huggingface.co/datasets/informedxrd/informedxrd-benchmarks
and model https://huggingface.co/informedxrd/informedxrd-checkpoints repositories carry the external benchmark payloads and checkpoints. The dataset repository is released under CC-BY-4.0 and the checkpoint and code repositories under the MIT license; the underlying RRUFF and opXRD data remain under their original terms of use. The repository is deliberately self-contained: the model implementation, extinction-group lookups, benchmark builders, subgroup-graph code, ViT evaluation code, and the four ViT checkpoints are reused or adapted from the public Baggett et al. release, and each such file is marked with its provenance. The files specific to this paper are the evaluation documentation and manifests, dataset and model metadata, benchmark packaging, the CNN control with its standalone implementation and loader, the prior-only, permutation-test, and null-distribution scripts, the XRDBench rescoring scripts, the opXRD processing pipeline, and a worked example that takes a model’s predicted probabilities, rescores a 230-way output into the 99-way extinction-group space, and reproduces a reported row end to end.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: The abstract and the contribution list in Section 1 name the three parts of the framework (the extinction-group mapping; the benchmark-construction method with its RRUFF-473, RRUFF-325, and opXRD-98 instances; and the evaluation protocol of prior-only baselines, a label-permutation test, the hierarchy-aware metric with null references, and stratified reporting) and the demonstration on four ViT checkpoints, a CNN control, and six XRDBench architectures (Sections 3–6). Both state which components originate in Baggett et al. and which are new here; the “Relationship to Baggett et al.” paragraph and Appendix K identify reused versus new components file by file. Section 1 states the scope as evaluation infrastructure rather than a new architecture.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: A dedicated “Limitations” paragraph in Section 7 collects them: coverage of only 46 and 45 of the 99 extinction groups and restriction to Cu-Kα laboratory data and RRUFF’s primary-phase labels; the inability of paired Top- tests to resolve the leading XRDBench architectures at these benchmark sizes; the final calibrated model matching or falling below the oracle prior-only control in aggregate; the unweighted, -only DAG and the suggestive status of cross-architecture distance comparisons; benchmark stability checked by threshold sweeps rather than alternative refinement workflows; the transfer check of the target-choice effect resting on small, domain-shift-dominated sets; and the fact that only the four ViT checkpoints and our own models can be rescored. Section 4.3 and Appendix F state that opXRD-98 is a stress test rather than a benchmark.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [N/A]
Justification: The paper’s contribution is empirical and methodological (evaluation infrastructure, benchmark construction, hierarchy-aware error analysis). It does not state or prove theorems.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper releases the exact RRUFF scan-ID lists for both benchmarks tied to a frozen upstream snapshot, the curation code with its explicit thresholds and cutoffs, the extinction-group mapping table as a JSON/CSV artifact, the condensed -subgroup DAG adjacency lists, the stratification labels, the aggregate prior, the opXRD-98 record manifest and processing scripts, and the case-study checkpoints. Section 4.2 (“Reproducibility, coverage, and intended use”) states that the released scan-ID lists, snapshot date, curation code, and thresholds reproduce both benchmarks exactly; Appendix D gives the full step descriptions, Appendix E the released defaults and threshold sweeps, Appendix F the opXRD-98 gate table, and Appendices G, J, and K the null distributions, the CNN configuration, and the XRDBench training contract.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All artifacts are public. The GitHub repository (https://github.com/scattering/InformedXRD) provides the curation and evaluation code, environment files, manifests, notebooks, and a worked example that rescores a 230-way output into extinction-group space and reproduces a reported row end to end. The Hugging Face dataset repository (https://huggingface.co/datasets/informedxrd/informedxrd-benchmarks, CC-BY-4.0, with a Croissant file) and model repository (https://huggingface.co/informedxrd/informedxrd-checkpoints, MIT) carry the benchmark payloads, stratification labels, the aggregate prior, and the case-study checkpoints. Appendix K lists the inventory and marks which files are reused from Baggett et al.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies the evaluation pipeline in detail: resampling onto an 8501-point grid (5.00°–90.00°, 0.01° step), the Cu-K wavelength filter, family-coherence thresholds and tie-break margins, the four nuisance-fit strata defined by deterministic cutoffs, the prior-only baseline construction, the label-permutation test, and the condensed -subgroup DAG with its four reference distributions. The case-study models are the released ViT checkpoints of Baggett et al., an in-house 1D residual CNN whose architecture, optimizer, schedule, seed, and training corpus are given in Appendix J together with its 40-epoch convergence continuation, six architectures trained under the published XRDBench recipe on the unchanged SimXRD train_1 contract (Appendix K), and a matched space-group/extinction-group ViT pair with a scaling study (Appendix C). Upstream training procedures are documented in the cited references and the released code.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Model comparisons are reported with exact McNemar tests on Top-1 and Top-5 with explicit -values and paired permutation tests on the hierarchy-aware summaries (Section 6.1); exact McNemar tests with Holm adjustment and 10,000-replicate paired-bootstrap intervals for the XRDBench architectures (Section 6.2); a 10,000-permutation label-permutation test with one-sided -values (Section 6.3); bootstrap confidence intervals for the final calibrated checkpoint (Appendix I); and exact null reference distributions for the hierarchy-aware metric (Appendix G). Threshold-sensitivity ranges are reported as min–max bands over the swept variants, and aggregates are reported alongside their stratified decomposition.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [No]
Justification: Compute is reported only in part. Rescoring a released checkpoint on a benchmark takes under a minute per model on one GPU (the released XRDBench evaluation job ran on a TACC Vista GPU node), and the CNN control of Appendix J trains at about 15 minutes per epoch on one GPU according to the released convergence log, so the submitted epoch-6 checkpoint and the 40-epoch continuation took about 1.5 and 10 GPU-hours. The six XRDBench trainings and the matched space-group/extinction-group ViT pair with its scaling study ran on the TACC and ACCESS allocations named in the acknowledgments, but their GPU-hours were not logged and are not itemized in the paper. The upstream synthetic corpora and the Baggett et al. ViT checkpoints are described in the cited prior work rather than reproduced here.
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification: The authors have read the NeurIPS Code of Ethics and the work conforms in every respect. The paper involves no human subjects, no scraped or personally identifying data, and no dual-use models. The released artifacts are derived from publicly available scientific archives (RRUFF, opXRD) and from previously released model checkpoints credited to their authors.
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The conclusion discusses the positive impact: more reliable, comparable, and physically meaningful evaluation in PXRD-ML, and the field-wide benefit of routine release of runnable checkpoints so that published systems can be rescored; Appendix H notes that hierarchy-aware evaluation transfers to other structured-label scientific domains. The paper does not identify a negative societal impact: the artifacts are evaluation infrastructure for crystallographic classification rather than generative or decision-making systems, and the foreseeable misuse, overfitting to the released subsets, is addressed by the recommendation to report both benchmarks with stratified analysis (§ 4.2).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A]
Justification: The released artifacts are crystallographic evaluation infrastructure: powder-diffraction patterns of natural minerals, mapping tables, evaluation scripts, and small classifier checkpoints whose only outputs are extinction-group labels. They pose none of the misuse risks the question targets (no generative content, no scraped personal data, no dual-use capability).
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites the creators of every external asset it builds on: the RRUFF mineral database, the EXPO extinction-symbol table and the International Tables for Crystallography volumes underlying the mapping and the subgroup graph, the opXRD archive, pymatgen and spglib, the SimXRD-4M simulator and XRDBench recipe, the ICSD structural map used for opXRD-98, and the Baggett et al. checkpoints, mapping, benchmarks, and subgroup graph. Appendix K states the licenses of the released repositories (CC-BY-4.0 for the dataset, MIT for the checkpoints and the code) and that the underlying RRUFF and opXRD data remain under their original terms of use; the aggregate synthetic-training prior is released only as class probabilities, with no upstream structures redistributed.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The released benchmarks, mapping tables, DAG adjacency lists, stratification labels, aggregate prior, evaluation scripts, opXRD-98 record manifest, and case-study checkpoints are documented through (i) the dataset card in the Hugging Face dataset repository (intended use, licensing, contents) and its Croissant file with core and Responsible AI fields; (ii) the model card in the Hugging Face model repository (intended use, provenance, licensing); and (iii) the code repository’s README, manifests, notebooks, per-file provenance markers distinguishing files reused from Baggett et al. from files new to this paper, a standalone CNN implementation and loader, and a worked example that reproduces a reported row. The documentation covers the released benchmark and evaluation assets, not every upstream training corpus used by the case-study models.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: The paper does not involve crowdsourcing or research with human subjects.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: The paper does not involve crowdsourcing or research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: The core methodology (extinction-group mapping, benchmark construction, prior-only baselines, label-permutation test, hierarchy-aware DAG metric, stratified evaluation, and classifier evaluation) does not involve LLMs as an important, original, or non-standard component.
References
- [1]Angela Altomare, Corrado Cuocci, Carmelo Giacovazzo, Anna Moliterni, Rosanna Rizzi, Nicola Corriero, and Aurelia Falcicchio. EXPO2013: a kit of tools for phasing crystal structures from powder data. Journal of Applied Crystallography, 46(4):1231–1235, 2013. doi: 10.1107/S0021889813013113.DOI
- [2]Nina Andrejevic, Ming Du, Hemant Sharma, James P. Horwath, Aileen Luo, Xiangyu Yin, Michael Prince, Brian H. Toby, and Mathew J. Cherukara. AlphaDiffract: Automated crystallographic analysis of powder X-ray diffraction data, 2026.arxiv.org/abs/2603.23367
- [3]Mois I. Aroyo, editor. International Tables for Crystallography, Volume A: Space-Group Symmetry. Wiley / International Union of Crystallography, Chichester, 6th edition, 2016. doi: 10.1107/97809553602060000114.DOI
- [4]Elizabeth J. Baggett, Edward G. Friedman, Abhishek Shetty, Derrick Chan-Sew, Vanellsa Acha, Harshitha Dwaracherla, Paul Kienzle, and William Ratcliff. Attention is not all you need for diffraction. PRX Intelligence, Sep 2026. doi: 10.1103/88gm-4gjx. URL https://link.aps.org/doi/10.1103/88gm-4gjx.arxiv.org/abs/2604.23811
- [5]Bin Cao, Yang Liu, Zinan Zheng, Ruifeng Tan, Jia Li, and Tong-yi Zhang. Simxrd-4m: Big simulated x-ray diffraction data and crystal symmetry classification benchmark. In International Conference on Learning Representations, 2025.
- [6]Yuzhou Cao, Lei Feng, and Bo An. Consistent hierarchical classification with a generalized metric. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 238 of Proceedings of Machine Learning Research, pages 4825–4833. PMLR, 2024. URL https://proceedings.mlr.press/v238/cao24a.html.
- [7]Nicola Corriero, Rosanna Rizzi, Gaetano Settrembre, Nicoletta Del Buono, and Domenico Diacono. CrystalMLA: a new crystallographic machine learning platform for crystal system determination. Journal of Applied Crystallography, 56(2):409–419, 2023. doi: 10.1107/S1600576723000596.DOI
- [8]Saulius Gražulis, Daniel Chateigner, Robert T. Downs, A. F. T. Yokochi, Miguel Quirós, Luca Lutterotti, Elena Manakova, Justas Butkus, Peter Moeck, and Armel Le Bail. Crystallography Open Database – an open-access collection of crystal structures. Journal of Applied Crystallography, 42(4):726–729, 2009. doi: 10.1107/S0021889809016690.DOI
- [9]Daniel Hollarek, Henrik Schopmans, Jona Östreicher, Jonas Teufel, Bin Cao, Adie Alwen, Simon Schweidler, Mriganka Singh, Tim Kodalle, Hanlin Hu, Ashley Ferrigno, Carolin M. Sutter-Fella, Pascal Friederich, et al. opXRD: Open experimental powder x-ray diffraction database, 2025.
- [10]Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, and Kristin A. Persson. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL Materials, 1(1):011002, 2013. doi: 10.1063/1.4812323.DOI
- [11]B. Lafuente, R. T. Downs, H. Yang, and N. Stone. The RRUFF project: an integrated study of the chemistry, crystallography, Raman and infrared spectroscopy of minerals. Highlights in Mineralogical Crystallography, pages 1–30, 2015. Database available at https://rruff.info.
- [12]Byung Do Lee, Jin-Woong Lee, Woon Bae Park, Joonseo Park, Min-Young Cho, Satendra Pal Singh, Myoungho Pyo, and Kee-Sun Sohn. Powder X-ray diffraction pattern is all you need for machine-learning-based symmetry identification and property prediction. Advanced Intelligent Systems, 4:2200042, 2022. doi: 10.1002/aisy.202200042.
- [13]Satvik Lolla, Haotong Liang, A. Gilad Kusne, Ichiro Takeuchi, and William Ratcliff. A Semi-Supervised Deep-Learning Approach for Automatic Crystal Structure Classification. Journal of Applied Crystallography, 55(4):882–889, 2022. doi: 10.1107/S1600576722006069.DOI
- [14]Dan Nguyen, Karen Cao, Brian Chu, Nick Lemoff, Paul Kienzle, and William Ratcliff II. Computed materials proposals depart from the structural memory of experimental discovery, 2026.
- [15]Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffrey Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L. Chevrier, Kristin A. Persson, and Gerbrand Ceder. Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis. Computational Materials Science, 68:314–319, 2013. doi: 10.1016/j.commatsci.2012.10.028.DOI
- [16]Felipe Oviedo, Zekun Ren, Shijing Sun, Charles Settens, Zhe Liu, Noor Titan Putri Hartono, Savitha Ramasamy, Brian L. DeCost, Siyu I. P. Tian, Giuseppe Romano, Aaron Gilad Kusne, and Tonio Buonassisi. Fast and interpretable classification of small X-ray diffraction datasets using data augmentation and deep neural networks. npj Computational Materials, 5:60, 2019. doi: 10.1038/s41524-019-0196-x.DOI
- [17]Hiroyuki Ozaki, Naoya Ishida, and Tetsu Kiyobayashi. Prediction of the space group and cell volume by training a convolutional neural network with primitive ‘ideal’ diffraction profiles and its application to ‘real’ experimental data. Journal of Applied Crystallography, 58(3):718–730, 2025. doi: 10.1107/S1600576725002419.
- [18]Won Bae Park, Jiyong Chung, Jaeyoung Jung, Keemin Sohn, Satendra Pal Singh, Myoungho Pyo, Namsoo Shin, and Kee-Sun Sohn. Classification of crystal structure using a convolutional neural network. IUCrJ, 4(4):486–494, 2017. doi: 10.1107/S205225251700714X.
- [19]Jerardo E. Salgado, Samuel Lerman, Zhaotong Du, Chenliang Xu, and Niaz Abdolrahim. Automated classification of big X-ray diffraction data using deep learning models. npj Computational Materials, 9:214, 2023. doi: 10.1038/s41524-023-01164-8.
- [20]Henrik Schopmans, Patrick Reiser, and Pascal Friederich. Neural networks trained on synthetically generated crystals can extract structural information from ICSD powder X-ray diffractograms. Digital Discovery, 2:1414–1424, 2023. ISSN 2635-098X. doi: 10.1039/D3DD00071K. URL https://xlink.rsc.org/?DOI=D3DD00071K.
- [21]Carlos N. Silla and Alex A. Freitas. A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 22(1–2):31–72, 2011. doi: 10.1007/s10618-010-0175-9.DOI
- [23]Yuta Suzuki, Hideitsu Hino, Takafumi Hawai, Kotaro Saito, Masato Kotsugi, and Kanta Ono. Symmetry prediction and knowledge discovery from X-ray diffraction patterns using an interpretable machine learning approach. Scientific Reports, 10:21790, 2020. doi: 10.1038/s41598-020-77474-4.
- [24]Nathan J. Szymanski, Christopher J. Bartel, Yan Zeng, Qingsong Tu, and Gerbrand Ceder. Probabilistic deep learning approach to automate the interpretation of multi-phase diffraction spectra. Chemistry of Materials, 33(11):4204–4215, 2021. doi: 10.1021/acs.chemmater.1c01071.
- [25]Atsushi Togo, Kohei Shinohara, and Isao Tanaka. Spglib: a software library for crystal symmetry search. Science and Technology of Advanced Materials: Methods, 4(1):2384822, 2024. doi: 10.1080/27660400.2024.2384822.
- [26]Pascal Marc Vecsei, Kenny Choo, Johan Chang, and Titus Neupert. Neural network based classification of crystal symmetries from X-ray diffraction patterns. Physical Review B, 99(24):245120, 2019. doi: 10.1103/PhysRevB.99.245120.DOI
- [27]Hao Wang, Jiajun Zhong, Yikun Li, Junrong Zhang, and Rong Du. Designing a dataset for convolutional neural networks to predict space groups consistent with extinction laws. Journal of Applied Crystallography, 58(3):711–717, 2025. doi: 10.1107/S1600576725002316.
- [28]Hans Wondratschek and Ulrich Müller, editors. International Tables for Crystallography, Volume A1: Symmetry Relations between Space Groups. Wiley / International Union of Crystallography, Chichester, 2nd edition, 2011. doi: 10.1107/97809553602060000110.
- [29]D. Zagorac, H. Müller, S. Ruehl, J. Zagorac, and S. Rehme. Recent developments in the Inorganic Crystal Structure Database: theoretical crystal structure data and related features. Journal of Applied Crystallography, 52(5):918–925, 2019. doi: 10.1107/S160057671900099X.DOI
- [30]Alexander N. Zaloga, Vladimir V. Stanovov, Oksana E. Bezrukova, Petr S. Dubinin, and Igor S. Yakimov. Crystal symmetry classification from powder X-ray diffraction patterns using a convolutional neural network. Materials Today Communications, 25:101662, 2020. doi: 10.1016/j.mtcomm.2020.101662.DOI


