betaFeedback

WorksComputer Science

Quantitative Assessment of Crystal Structure Prediction

Abstract

Crystal Structure Prediction (CSP) aims to predict a material's 3D structure from its chemical composition, with applications from energy storage to biomedical devices. Despite its importance, existing evaluation frameworks suffer from key limitations: they rely on threshold-sensitive or incomparable metrics, focus predominantly on geometric performance while overlooking physical plausibility, and fail to account for polymorphism, where a chemical composition can crystallize into multiple stable structures. In parallel, recent Symmetry-Informed CSP (SICSP) methods often report results alongside standard CSP baselines while receiving crystallographic templates at inference, which constrain the search space and make SICSP a fundamentally different task. To address these limitations, we introduce an Orientation-invariant, Polymorph-Aware, Large Benchmark (OPAL-Bench) that evaluates CSP and SICSP as separate tasks. We evaluate 7 CSP and 2 SICSP methods under OPAL-Bench across 4 datasets using threshold-free geometric and physical distance metrics. Our results show that no method dominates across all metrics, and that geometric performance is not strongly correlated with physical plausibility. We also release OPAL-CSP, an SE(3)-equivariant flow-matching baseline for the CSP track that performs competitively on geometric metrics and ranks among the best on physical ones. By unifying geometric and physical metrics under a polymorph-aware protocol, OPAL-Bench enables a more comprehensive assessment of generative quality in CSP.

Introduction

Crystalline materials play a central role in modern technology and scientific discovery, with applications spanning energy storage [1], microelectronics [2], catalysis [3], and biomedical devices [4]. The periodic organization of atoms in a crystal governs a broad spectrum of mechanical, optical, magnetic, and electrical properties, making the three-dimensional arrangement of crystalline structures the primary target of computational study for materials discovery [5, 6]. Accelerating this process increasingly depends on crystal structure prediction (CSP), the task of predicting how atoms arrange within a crystal structure directly from its composition [7]. Deep generative models have become the dominant paradigm for CSP, displacing traditional search-based approaches that explore vast configurational spaces through costly physical simulations [8]. These methods leverage SE(3)SE(3)- or periodic-E(3)E(3)-equivariant graph neural networks and have rapidly advanced the state of the art through a growing number of diffusion-based [9, 10, 11] and flow matching architectures [12, 13, 14].

As generative methods proliferate, the question of how to evaluate them rigorously has become as consequential as the modeling advances themselves. Current evaluation protocols, however, are insufficient for a complete assessment of generative CSP methods and overlook important dimensions of model behavior. In particular, the field suffers from four main limitations that together obscure the true relative performance of CSP methods.

First, standard metrics depend on arbitrary tolerance thresholds and report errors over structurally distinct subsets of predictions across methods, making reported performance both sensitive to evaluation choices and not directly comparable between methods [9, 10, 13]. Second, most evaluation protocols [9, 10, 11, 12, 13, 14] compare each prediction against a single fixed reference structure, even though a single chemical composition can exhibit polymorphism, adopting multiple distinct yet energetically stable arrangements that differ in geometry and symmetry [15, 16]. Third, existing CSP evaluation frameworks from the machine learning community [9, 10, 13] measure only structural similarity and overlook physical plausibility, even though physically meaningful structures are the ultimate goal of CSP. In contrast, the crystallographic community routinely incorporates physical metrics when evaluating traditional methods [17, 18]. Fourth, methods that receive crystallographic symmetry templates as input at inference [19, 20], a setting we refer to as Symmetry-Informed CSP (SICSP), routinely report results alongside methods that see only the chemical formula, despite the substantial reduction in search space that symmetry information provides.

We address these limitations with an Orientation-invariant, Polymorph-Aware, Large Benchmark (OPAL-Bench) for CSP and SICSP. Figure 1 summarizes the framework. Rather than comparing predictions against a single fixed reference, OPAL-Bench evaluates each prediction against all known polymorphs of the same composition and selects the closest as the reference. Before computing any geometric distance, it applies an explicit SE(3)SE(3) alignment that enforces atom-identity constraints, ensuring that the distances we compute reflect true structural divergence rather than arbitrary orientation differences. OPAL-Bench further extends evaluation with physical plausibility metrics, namely formation energy, relaxed formation energy, smooth overlap of atomic positions (SOAP), X-ray diffraction pattern, and density, providing a more complete picture of generative quality. Finally, OPAL-Bench evaluates CSP and SICSP methods as separate tracks, enabling fair comparisons within each setting.

OPAL-Bench evaluation pipeline

Figure 1. OPAL-Bench evaluation pipeline. OPAL-Bench evaluates seven CSP and two SICSP methods across four datasets, with four geometric and five physical metrics, treating CSP and SICSP as fundamentally distinct tasks. For each composition, a model predicts one structure, and we retrieve all known polymorphs, the distinct stable structures of the same composition, as candidate references. We then apply species-aware SE(3)SE(3) alignment for geometric metrics, while physical metrics are SE(3)SE(3)-invariant and require no alignment, and score each metric against its closest polymorph.

Alongside the benchmark, we release OPAL-CSP, an SE(3)SE(3)-equivariant graph transformer that we train with flow matching to serve as a reference baseline for the CSP track. We use OPAL-Bench to evaluate seven CSP and two SICSP methods, finding that no current method dominates across all metrics and datasets, and that a structured gap separates geometric and physical performance. Our main contributions are as follows.

  1. We introduce OPAL-Bench, a threshold-free, polymorph-aware benchmark that applies explicit SE(3)SE(3) alignment with atom-identity constraints, unifies geometric and physical distance metrics, and separates CSP and SICSP into distinct evaluation tracks.

  1. Through a systematic re-evaluation of seven CSP and two SICSP methods, we empirically demonstrate that geometric performance does not reliably imply physical plausibility, a dimension of generative quality that existing machine learning evaluation frameworks overlook.

  1. We release OPAL-CSP, an SE(3)SE(3)-equivariant flow-matching reference implementation for the CSP track. Tuned with OPAL-Bench in view, it performs competitively on both geometric and physical metrics, showing that the two can coexist in a single model and offering a strong baseline for future work.

OPAL-Bench, along with the OPAL-CSP training code and model checkpoints, is available at https://github.com/BCV-Uniandes/OPAL.

CSP Evaluation

A CSP model takes a chemical formula as input and predicts one plausible crystal structure, comprising a periodic lattice and fractional atomic coordinates [7, 21]. To assess prediction quality, most prior work in the machine learning community [9, 10, 12, 13, 14] reports two output-based metrics, Match Rate and RMSE. Match Rate measures the fraction of structures the model produces that match a ground truth within predefined tolerances, and RMSE quantifies atomic positional errors normalized by the cell volume, computing them only for predictions that successfully match a reference.

These two metrics suffer from fundamental limitations. Match Rate is highly sensitive to tolerance choice, and the relative ranking of methods changes across tolerance settings, so conclusions about which model performs best depend on the threshold. RMSE faces an analogous problem. As each method computes it over its own subset of predictions that match a reference, the reported values reflect different populations, making cross-method comparisons uninformative. We empirically validate both effects in Figure 2 and Figure 3 of Appendix § 8.1. A further bias, which Martirossyan et al. [16] also identify, arises from the protocol itself. Evaluation proceeds per crystal ID rather than per unique composition, so distinct polymorphs of the same formula serve as independent targets, penalizing methods that recover a different known polymorph of the same composition. To mitigate it, they propose METRe and cRMSE, which measure how well a set of generated structures covers the known references, but both still depend on predefined matching tolerances and remain purely geometric.

Figure 2. Distribution of Match Rate (left) and RMSE (right) for each method across 571 tolerance configurations on MPTS-52. We sample 571 tolerance configurations uniformly from stol∈[0.1,0.95]s_{\mathrm{tol}} \in[0.1, 0.95], θtol∈[5∘,85∘]\theta_{\mathrm{tol}} \in[5^\circ, 85^\circ], and ltol∈[0.1,0.95]l_{\mathrm{tol}} \in[0.1, 0.95], and recompute both metrics under each configuration. Each violin shows the full distribution across configurations. The wide spread within each method indicates that reported metric values depend strongly on the choice of tolerance parameters.

Figure 3. Pairwise comparison of Match Rate across 571 tolerance configurations on MPTS-52. We sample 571 tolerance configurations uniformly from stol∈[0.1,0.95]s_{\mathrm{tol}} \in[0.1, 0.95], θtol∈[5∘,85∘]\theta_{\mathrm{tol}} \in[5^\circ, 85^\circ], and ltol∈[0.1,0.95]l_{\mathrm{tol}} \in[0.1, 0.95]. Each point corresponds to one configuration, colored by stols_{\mathrm{tol}}. Points above the dashed diagonal indicate that the yy-axis method achieves higher Match Rate than the xx-axis method under that configuration; points below favor the xx-axis method. Annotations report the number of configurations favoring each method out of 571. No method dominates among the pairs shown, and the standard tolerance setting (stol=0.5s_{\mathrm{tol}} = 0.5, θtol=10∘\theta_{\mathrm{tol}} = 10^\circ, ltol=0.3l_{\mathrm{tol}} = 0.3) sits in regions where small perturbations flip rankings. Table 3 reports the only two pairs in which one method dominates across all configurations.

In contrast, the crystallographic community has evaluated traditional CSP algorithms with geometric and physical metrics. Wei et al. [18] and CSPBench [17] represent the first large-scale attempts to assess CSP algorithms using 3D point set distances, treating crystals as point clouds and computing Chamfer, Hausdorff, Sinkhorn, and RMSE across sets of 16 and 180 test structures, respectively. CSPBench additionally covers physical metrics, including XRD spectrum, fingerprint, and formation energy distances, and compares predictions against available polymorphs using energy distance as the selection criterion.

Despite these contributions, both benchmarks share important geometric limitations. First, none of the geometric metrics enforce atomic identity constraints, meaning distances can reflect cross-species comparisons that distort the true structural divergence. Second, only RMSE applies any form of rotational alignment before distance computation, and even this alignment ignores atom types during superposition, while Chamfer, Hausdorff, and Sinkhorn receive no rotational alignment at all. Although CSPBench takes a step toward physical evaluation, both benchmarks evaluate only traditional methods, omitting the deep generative models that now define the state of the art and the four standard datasets the community has adopted for comparison.

OPAL-Bench addresses these gaps directly by performing explicit SE(3)SE(3) alignment that enforces one-to-one correspondence between atoms of the same type before computing any distance metric, evaluating each prediction against all known polymorphs of the same composition without matching tolerances, applying this protocol to the four standard generative CSP datasets (Perov-5 [22], Carbon-24 [23], MP-20 [24], and MPTS-52 [24]), and extending evaluation with physical metrics.

CSP Methods

Standard CSP. Deep generative approaches to CSP fall into two main paradigms, and in both the atom types are given as input and remain fixed throughout generation. Diffusion-based methods inject noise into lattice parameters and atomic coordinates, then recover the structure using score functions they learn during training. CDVAE [9] follows a diffusion variational autoencoder (VAE) formulation that first predicts the lattice and then refines coordinates through annealed Langevin dynamics [25]. MatterGen [11] employs an SE(3)-equivariant GemNet-dT [26] score network in a diffusion denoising process over lattices and positions, while DiffCSP [10] uses a periodic-E(3) denoiser over fractional coordinates to jointly generate lattices and atom positions.

Flow matching methods instead learn velocity fields that transport a simple prior to the data distribution. CrystalFlow [12] uses Conditional Flow Matching to jointly generate lattice parameters and fractional coordinates. FlowMM [13] generalizes Riemannian Flow Matching [27] to accommodate the translation, rotation, permutation, and periodic boundary conditions that characterize crystalline structures, and FlowLLM [14] combines a large language model with Riemannian Flow Matching to predict metastable structures from text-based representations. As a reference baseline for the CSP track, we release OPAL-CSP, an SE(3)-equivariant graph transformer that we train with flow matching to provide a consistent point of comparison across the four standard datasets.

Symmetry-Informed CSP. SICSP methods constrain the search space with crystallographic symmetry information, namely the space group and, optionally, the Wyckoff positions, which specify the symmetry-equivalent sites that atoms occupy. DiffCSP++ [19] takes the space group as input and enforces symmetry constraints on the lattice and fractional coordinates during diffusion. SGEquiDiff [20] combines space group equivariant diffusion of atomic coordinates with a discrete lattice sampler. Both methods can operate with ground truth symmetry templates or with predicted ones. CSPML [28], for example, predicts them by retrieving structurally similar crystals from a training database via composition-based similarity.

Orientation-invariant, Polymorph-Aware, Large Benchmark

OPAL-Bench evaluates any CSP model through the following protocol. Given a test set, the model generates one prediction per unique chemical composition, and OPAL-Bench scores that prediction by computing metric-specific minimum distances to the known polymorphs of the same composition. For geometric metrics, it first applies the species-aware SE(3) alignment of § 3.4 before computing distances; for physical metrics, it applies them directly to the predicted and reference structures. § 3.1-§ 3.5 describe each component in detail.

Tasks Formulation

Standard CSP. We formally define a crystal structure as the periodic arrangement of NN atoms within a unit cell, with a vector of atom species A=[a1,a2,…,aN]∈ANA = [a_1, a_2, \ldots, a_N] \in\mathcal{A}^N, a lattice matrix L=[l1,l2,l3]∈R3×3L = [l_1, l_2, l_3] \in\mathbb{R}^{3 \times3} with ∣det⁡L∣>0\lvert\det L \rvert> 0 defining the periodic unit cell, and fractional coordinates F=[f1,f2,…,fN]∈[0,1)3×NF = [f_1, f_2, \ldots, f_N] \in[0,1)^{3 \times N}, which relate to Cartesian coordinates X=[x1,x2,…,xN]∈R3×NX = [x_1, x_2, \ldots, x_N] \in\mathbb{R}^{3 \times N} via X=LFX = LF. The crystal structure is invariant under global translations of XX, as well as permutations of atoms sharing the same species aia_i.

CSP consists of learning the conditional distribution p(F,L∣A)p(F,L \mid A) over the space of valid crystal structures, where the search space includes all pairs (F,L)∈[0,1)3×N×{L∈R3×3:∣det⁡L∣>0}(F,L) \in[0,1)^{3 \times N} \times\{L \in\mathbb{R}^{3 \times3} : \lvert\det L \rvert> 0\} consistent with the input composition AA. A CSP model samples from this distribution to produce a plausible crystal structure M=(A,F^,L^)\mathcal{M} = (A,\hat{F},\hat{L}). A key property of this distribution is polymorphism, whereby a single composition AA can correspond to multiple distinct yet energetically stable structures. We denote the set of known polymorphs as Π(A)={M(1),M(2),…,M(K)}\Pi(A) = \{\mathcal{M}^{(1)}, \mathcal{M}^{(2)}, \ldots, \mathcal{M}^{(K)}\}, where each M(k)=(A,F(k),L(k))\mathcal{M}^{(k)} = (A,F^{(k)},L^{(k)}) shares the same composition but differs in geometry.

Symmetry-Informed CSP SICSP extends this formulation by incorporating crystallographic symmetry as additional input. Formally, a SICSP model receives the chemical composition AA, a space group label S∈{1,…,230}S \in\{1,\ldots,230\}, and optionally a set of Wyckoff positions WW, which specify the symmetry-equivalent atomic sites within the unit cell. Together, SS and WW constrain both the admissible lattice geometries and the positions atoms may occupy. With the full template, a SICSP model learns p(F,L∣A,S,W)p(F,L \mid A,S,W), whose support is restricted to the pairs (F,L)(F,L) consistent with SS and WW, a subset of the CSP search space. When the input does not include WW, some methods first predict it from AA and SS with a learned model and then sample from the same distribution. Because the input assumptions of CSP and SICSP differ substantially, the two tasks are not fairly comparable, and we evaluate them under separate tracks in OPAL-Bench.

Metrics

Our benchmark evaluates CSP models along two complementary axes, geometric and physical. All metrics quantify the discrepancy between a predicted structure M^\hat{M} and a reference M(k)∈Π(A)M^{(k)} \in\Pi(A), each capturing a notion of distance appropriate to the property it measures. We provide further details on each metric in Appendix § 8.2.

Geometric metrics treat crystals as labeled point clouds, where atoms act as points in R3\mathbb{R}^{3} and chemical elements serve as labels, and measure structural similarity in terms of point cloud distances. We consider four such metrics: Chamfer distance, Hausdorff distance, Sinkhorn distance, and RMSE. By default, these metrics do not enforce atom identity during comparison and are not invariant to rotations, translations, and same-species permutations. We therefore apply an explicit alignment procedure and a per-species aggregation scheme before computing them, both of which we describe in § 3.4.

Physical metrics quantify the distance between M^\hat{M} and M(k)M^{(k)} in terms of experimentally measurable or computationally derived properties, namely formation energy and relaxed formation energy, which we compute with Machine Learning Interatomic Potentials (MLIPs) following the setup of LeMat-GenBench [29], smooth overlap of atomic positions (SOAP) [30], X-ray diffraction patterns, and density. As these metrics depend only on the physical properties of the structure rather than its coordinate representation, they are invariant to rotations, translations, and permutations by definition, and we apply them directly without any preprocessing. Physical metrics also complement the geometric ones beyond the recovery of known structures. A prediction that differs geometrically from every known polymorph but relaxes to a structure of comparable formation energy receives a large geometric distance and a small relaxed formation energy distance, a distinction that becomes more relevant as CSP methods move toward the discovery of new structures, as in de novo generation. Appendix § 8.§ 2.2 provides full MLIP details. To facilitate comparison with prior work, we additionally report Match Rate and RMSE in Appendix § 8.1.2.

Polymorph Handling

We handle polymorphism by evaluating each prediction against all known polymorphs of the same composition. Formally, let d(⋅,⋅)d(\cdot,\cdot) be any structural distance metric defined between two crystal structures of the same composition. Given a test set where multiple structures may share the same chemical formula, we generate a single prediction M^\hat{M} per unique composition AA, rather than one per crystal ID. We then evaluate M^\hat{M} against the set of known polymorphs Π(A)={M(1),M(2),…,M(K)}\Pi(A)=\{\mathcal{M}^{(1)},\mathcal{M}^{(2)},\ldots,\mathcal{M}^{(K)}\} and select the closest reference as M∗=arg min⁡M(k)∈Π(A)d(M^,M(k))\mathcal{M}^{*}=\operatorname*{arg\,min}_{\mathcal{M}^{(k)}\in\Pi(A)}d(\hat{M},\mathcal{M}^{(k)}), reporting d(M^,M∗)d(\hat{M},\mathcal{M}^{*}) as the final score for that prediction. This procedure computes a metric-specific distance to the set of known valid polymorphs, reflecting that different metrics capture different notions of structural or physical proximity. Consequently, the closest polymorph may differ across metrics for the same prediction, an effect we analyze in depth in Appendices § 11.1 and § 11.4.

Species-Aware Alignment and Aggregation

Computing meaningful geometric distances between two crystal structures requires resolving three sources of ambiguity, namely global rotation, global translation, and atom permutation. We address these jointly through an iterative alignment procedure that finds the rigid transform (R,t)∈SE(3)(R,t) \in SE(3) and atom permutation σ\sigma that minimize the RMSE between a predicted structure M^\hat{\mathcal{M}} and a reference M∗\mathcal{M}^{*}.

Formally, let X^∈R3×N\hat{X} \in\mathbb{R}^{3 \times N} and X∗∈R3×NX^{*} \in\mathbb{R}^{3 \times N} denote the Cartesian coordinates of the predicted and reference structures, respectively. We seek the rigid transform (R,t)(R,t) and permutation σ\sigma that solve min⁡R∈SO(3), t∈R3, σ∈Sym⁡(N)1N∑i=1N∥Rx^i+t−xσ(i)∗∥2\min_{R\in SO(3),\,t\in\mathbb{R}^{3},\,\sigma\in\operatorname{Sym}(N)} \frac{1}{N}\sum_{i=1}^{N}\lVert R\hat{x}_{i}+t-x^{*}_{\sigma(i)}\rVert^{2}, subject to ai=aσ(i)∗a_i=a^{*}_{\sigma(i)} for all ii, with Sym⁡(N)\operatorname{Sym}(N) the set of permutations of NN atoms. We solve this by alternating between two steps. For the current transform (R,t)(R,t), we compute the optimal atom assignment σ∗\sigma^{*} using the Hungarian algorithm with +∞+\infty cost for cross-species matches. For σ∗\sigma^{*}, we update (R,t)(R,t) by minimizing the RMSE via gradient-based optimization. We provide further implementation details in Appendix § 9.2.

Once we compute the alignment, we evaluate each geometric metric in a species-aware manner. Let A={1,…,g}\mathcal{A}=\{1,\ldots,g\} denote the set of distinct atom species in the crystal. For each species a∈Aa\in\mathcal{A}, we extract the sub-clouds X^a\hat{X}_a and Xa∗X^{*}_a containing only the aligned predicted and reference atoms of type aa, with NaN_a atoms each. We then compute each geometric metric dd independently on each pair (X^a,Xa∗)(\hat{X}_a,X^{*}_a) and aggregate the per-species scores as D(X^,X∗)=∑a∈A(Na/N)⋅d(X^a,Xa∗)D(\hat{X},X^{*})=\sum_{a\in\mathcal{A}}(N_a/N)\cdot d(\hat{X}_a,X^{*}_a), where N=∑a∈ANaN=\sum_{a\in\mathcal{A}}N_a is the total number of atoms. This aggregation preserves label consistency, avoids cross-species matches, and reduces to the standard unlabeled metric when all atoms share the same species.

Evaluation Considerations

We report the median score across test compositions for each metric, as it is less sensitive to outliers across diverse chemical systems. SICP predictions require additional handling because some structures have degenerate lattices or atom counts inconsistent with the input composition. For atom-count mismatches, we construct compatible supercells by repeating the predicted and reference structures according to the least common multiple of their atom counts; we consistently exclude cases producing excessively large supercells across SICP methods. Unlike Match Rate and conventional RMSE, which hide such failures by recording non-matches or omitting RMSE values, OPAL-Bench makes them explicit through per-method validity rates, which we report in Appendix § 10.1. Since these validity rates differ across methods, we compute every metric on the intersection of compositions for which all methods produce a valid prediction, ensuring a common evaluation set across methods. The differences in valid prediction sets and supercell construction between standard CSP and SICP further motivate evaluating the two tracks separately rather than comparing them directly.

OPAL-CSP

OPAL-CSP generates crystal structures via flow matching over the joint lattice-coordinate space, and two considerations motivate its design. First, flow matching has produced the strongest geometric results on CSP [12, 13], indicating its continuous-time formulation suits joint lattice-coordinate prediction. Second, SE(3)-equivariant graph transformers respect by construction the symmetries CSP requires [31], so we build on EquiformerV2 [32], which extends this family to higher-degree representations. OPAL-CSP uses separate velocity-field heads for L^\hat{L} and F^\hat{F} and follows the formulation of CrystalFlow [12], with linear interpolation on the lattice and a wrapped operation on fractional coordinates that respects periodic boundary conditions. Full architecture, training, and hyperparameter details are in Appendix § 9.3.

Experiments

Experimental Setup

Datasets We evaluate all models on four CSP datasets: Perov-5 [22], Carbon-24 [23], MP-20 [24], and MPTS-52 [24]. Perov-5 focuses on perovskite materials with 5 atoms per cell, containing 18,928 structures, which span only 5 space groups, making it the most structurally homogeneous dataset. Carbon-24 includes 10,153 carbon-only structures with 6 to 24 atoms per cell, and exhibits the most pronounced polymorphism of the four datasets, as these structures share only 10 distinct formulas, yielding many alternative structures per composition. It spans 56 space groups, although most of its structures adopt low-symmetry triclinic or monoclinic arrangements. MP-20 and MPTS-52 contain selected inorganic crystals from the Materials Project, comprising 45,229 structures with up to 20 atoms and 40,476 structures with up to 52 atoms, respectively. They are also the most structurally diverse, with long-tailed distributions over 179 and 208 space groups. Following prior CSP work, we use the standard splits: 60/20/20 for Perov-5, Carbon-24, and MP-20, and 67.6/12.4/20 for MPTS-52. Appendices § 9.1.1 and § 9.1.2 provide a more detailed statistical analysis of the datasets, including polymorphism rates and space-group distributions.

Baselines We evaluate seven CSP models under OPAL-Bench: six previously published methods, CDVAE [9], DiffCSP [10], MatterGen [11], CrystalFlow [12], FlowMM [13], and FlowLLM [14], together with our reference baseline OPAL-CSP. Although several recent models perform de novo crystal generation [33, 34, 35], our benchmark targets CSP, where the model receives the composition as input. For the SICSP track, we evaluate DiffCSP++ and SGEquiDiff using both ground truth templates (space group and Wyckoff positions) and templates that CSPML predicts. We retrain each model from scratch following its original training procedure and sample structures as the corresponding paper describes; Appendices § 9.4 and § 9.5 provide implementation details and computational resources for all models.

OPAL-Bench Results

Tables 1 and 2 report the median score of each metric across test compositions for the CSP and SICSP tasks, respectively. We provide additional results in Appendix § 10, including 95% bootstrap confidence intervals in Appendix § 10.2 Figure 7 and Figure 8, and qualitative predictions in Appendix § 10.3 Figure 9–Figure 16. We consider a difference between methods statistically significant when their 95% confidence intervals do not overlap. We evaluate each task on its own group-specific intersection of valid predictions (the compositions on which every method in the track has a valid value; Appendix § 10.1).

Full CSP results across all four datasets and nine metrics

Figure 7. Full CSP results across all four datasets and nine metrics. Bars report the median over test compositions and error bars are 95% bootstrap confidence intervals from 10,000 resampling iterations. Lower is better for all metrics. Physical metrics reveal rankings that differ substantially from geometric metrics, confirming that geometric performance does not reliably imply physical plausibility.

Figure 8. Full SICSP results across all four datasets and nine metrics, for both ground truth (GT) and CSPML-predicted symmetry templates. Bars report the median over test compositions on the shared intersection of valid predictions and error bars are 95% bootstrap confidence intervals. GT templates yield substantially better geometric performance on Perov-5, where symmetry constraints are highly informative, while Carbon-24 is a notable exception where CSPML templates outperform GT due to the high polymorphic diversity of carbon structures.

Qualitative comparison on representative samples from the MP-20 dataset in the CSP task

Figure 9. Qualitative comparison on representative samples from the MP-20 dataset in the CSP task. Predictions from the seven CSP methods are shown alongside the corresponding ground truth structures, rendered as ball-and-stick models within their unit cells. Each row corresponds to one test composition, and atom species are color-coded consistently across methods and ground truth.

Qualitative comparison of representative Carbon-24 structures

Figure 16. Qualitative comparison on representative samples from the Carbon-24 dataset in the SICSP task. We show predictions from SGEquiDiff and DiffCSP++ in their CSPML and GT variants, alongside their corresponding selected reference polymorphs. Given the high number of polymorphs in Carbon-24, each prediction is visualized with its closest selected reference polymorph rather than a single shared ground truth structure across all columns. Structures are rendered as ball-and-stick models within their unit cells. Each prediction–reference pair corresponds to one test composition.

CSP Results Table 1 and Appendix Figure 7 show that no single method achieves uniformly best results across all metrics and datasets, confirming that CSP remains an open problem where different methods excel at different aspects of structural quality. OPAL-CSP, FlowMM, and DiffCSP emerge as the most consistently competitive methods overall, though their relative rankings shift considerably across metrics and datasets. We analyze the agreement between the rankings that different metrics produce in Appendix § 11.6.

ModelChamf. ↓\downarrowHaus. ↓\downarrowSink. ↓\downarrowRMSE ↓\downarrowρ\rho (g/cm3^3) ↓\downarrowSOAP ↓\downarrowXRD ↓\downarrowFE (eV/at.) ↓\downarrowR. FE (eV/at.) ↓\downarrow
Geometric Distances (Å)Physical Distances
PEROV-5
FlowLLM [14]1.2180.6230.9700.6100.4348.052.1640.1751.430⋅10−11.430 \cdot10^{-1}
CDVAE [9]2.2951.2721.6551.1720.41310.813.1803.0081.784⋅10−11.784 \cdot10^{-1}
CrystalFlow [12]1.3280.6981.0580.6701.14112.113.1160.4221.542⋅10−11.542 \cdot10^{-1}
MatterGen [11]1.2310.6310.9550.6190.2516.481.6780.1011.330⋅10−11.330 \cdot10^{-1}
DiffCSP [10]1.2290.6250.9210.6160.2366.531.6720.1161.256⋅10−11.256 \cdot10^{-1}
FlowMM [13]1.2260.6310.9560.6160.3457.061.8700.1091.429⋅10−11.429 \cdot10^{-1}
OPAL-CSP1.2110.6170.9090.6070.2856.251.7150.0721.348 · 10−110^{-1}
CARBON-24
FlowLLM1.5531.31712.1200.9310.00217.383.2303.7911.331⋅10−11.331 \cdot10^{-1}
CDVAE1.5721.32413.4450.9470.00410.031.6080.2433.262⋅10−33.262 \cdot10^{-3}
CrystalFlow1.6501.43012.9920.9840.12842.882.3460.4485.228⋅10−35.228 \cdot10^{-3}
MatterGen1.7881.46313.5651.0290.00313.452.0350.1823.198⋅10−33.198 \cdot10^{-3}
DiffCSP1.5871.45013.2551.0040.0098.241.1910.0062.250⋅10−42.250 \cdot10^{-4}
FlowMM1.4881.40110.4010.8330.0088.761.7590.0694.947⋅10−34.947 \cdot10^{-3}
OPAL-CSP1.5611.28112.5630.8890.0015.671.6440.0025.700⋅10−55.700 \cdot10^{-5}
MP-20
FlowLLM4.0392.9808.1542.3291.14211.163.0902.4811.515⋅10−11.515 \cdot10^{-1}
CDVAE3.9872.9588.1292.2890.1553.801.6050.2165.070⋅10−25.070 \cdot10^{-2}
CrystalFlow3.9112.8887.8192.2361.1878.222.4020.2988.100⋅10−58.100 \cdot10^{-5}
MatterGen4.1203.0218.3212.3660.2213.581.6210.0664.113⋅10−24.113 \cdot10^{-2}
DiffCSP3.9192.8847.8692.2310.1282.180.9410.0198.094⋅10−38.094 \cdot10^{-3}
FlowMM3.8662.8537.7512.1940.2032.421.0880.0321.250⋅10−41.250 \cdot10^{-4}
OPAL-CSP3.7002.6686.9522.0840.2142.571.1380.0281.490⋅10−41.490 \cdot10^{-4}
MPTS-52
FlowLLM4.3523.85121.9462.7792.61629.674.1573264.3764.117⋅1004.117 \cdot10^{0}
CDVAE4.4373.83221.6652.7410.4335.722.0840.7851.978⋅10−11.978 \cdot10^{-1}
CrystalFlow4.2753.65320.6492.6000.2844.171.4200.2749.169⋅10−29.169 \cdot10^{-2}
MatterGen4.5233.92722.2912.8030.2644.111.4940.1501.035⋅10−11.035 \cdot10^{-1}
DiffCSP4.3233.70920.8252.6410.2023.961.4640.2341.047⋅10−11.047 \cdot10^{-1}
FlowMM4.2933.68320.6922.6300.2904.861.6490.6191.342⋅10−11.342 \cdot10^{-1}
OPAL-CSP4.3253.69820.9082.6420.1833.471.3380.1137.748⋅10−27.748 \cdot10^{-2}

Table 1. Table 1: CSP Results. For each geometric and physical metric, we report the minimum distance from each prediction to the known polymorphs of the same composition and take the median across test compositions on the four standard datasets. Lower is better for all metrics. Bold marks the best method per dataset and metric; _underline_ marks the second best. Carbon-24 results should be read with caution, as its large number of polymorphs per composition makes comparisons less conclusive.

Regarding geometric metrics, flow-based methods tend to lead. OPAL-CSP and FlowMM perform strongly on Perov-5 and MP-20, whereas CrystalFlow exhibits pronounced dataset dependence, ranking near the bottom on smaller datasets but leading on MPTS-52. Diffusion-based methods generally lag behind on geometric metrics on Perov-5, MP-20, and MPTS-52. On Carbon-24, no method achieves a statistically significant advantage, owing to the high structural variance of carbon polymorphs.

On physical metrics, the flow advantage diminishes. OPAL-CSP and DiffCSP show the strongest performance, with DiffCSP particularly competitive on density, a proxy for unit cell quality independent of atomic positions. This gap between the geometric and physical performance of flow- and diffusion-based methods suggests that the two families of metrics can rank methods differently. FlowLLM illustrates this most clearly. It achieves competitive geometric scores yet produces a median formation energy error of 3264 eV/atom on MPTS-52 (Table 1), orders of magnitude above the eV/atom-scale errors of every other method, as its predictions contain atoms at near-coincident positions (Appendix § 10.3). This dissociation is not specific to FlowLLM. Across all methods, we find that geometric and physical distances are only weakly correlated on Perov-5, MP-20, and MPTS-52 (Appendices § 11.1 and § 11.3 detail the full analysis, including single-polymorph subsets and alternative SOAP and XRD settings). Thus, even under a polymorph-aware protocol, geometric performance does not reliably imply physical plausibility.

OPAL-CSP achieves the best geometric scores on Perov-5 and MP-20, the best formation energy on three of the four datasets, and the best score on every physical metric on MPTS-52, showing that strong geometric and physical performance can coexist in a single model. Consistent with the broader pattern that no method dominates across all metrics, DiffCSP outperforms OPAL-CSP on several physical metrics on Perov-5 and MP-20, and CrystalFlow and FlowMM outperform it on geometric metrics on MPTS-52. OPAL-CSP uses per-dataset hyperparameters, while the baselines run at their released configurations (Appendix § 9.4).

Dataset characteristics further shape relative standings. Perov-5’s structural homogeneity yields clear separations between methods, MPTS-52 produces the most consistent rankings within each metric family despite its structural complexity, MP-20 provides the broadest comparison as the largest dataset, and Carbon-24 remains the most difficult to evaluate conclusively, given its high number of polymorphs per composition.

SICSP Results Table 2 and Appendix Figure 8 present the performance results, and Appendix § 10.1 reports validity rates and evaluation-set intersection sizes. As expected, ground-truth templates yield better performance in most datasets, with Carbon-24 as a notable exception, where CPSML templates outperform ground truth for both methods. This likely arises from the high structural diversity of carbon polymorphs, which may constrain ground truth templates to a space group misaligned with their learned representations. Across datasets, DiffCSP++ tends to outperform SEquiDiff regardless of template source, with two recurring exceptions. On Perov-5, SEquiDiff shows a statistically significant advantage in geometric metrics and formation energy, though differences in the remaining physical metrics fall within confidence intervals. On MPTS-52, SEquiDiff achieves a significant advantage only on density and SOAP, while most other metrics remain competitive. The recurrence of these patterns across both template sources suggests they reflect architectural differences rather than sensitivity to template quality.

Table 2: SICSP results. For each geometric and physical metric, we report the minimum distance from each prediction to the known polymorphs of the same composition and take the median across test compositions, restricted to the intersection of valid predictions across all four method-template combinations. Lower is better for all metrics. Bold marks the best method per dataset and metric; underline marks the second best. Carbon-24 results should be read with caution, as its large number of polymorphs per composition makes comparisons less conclusive.

ModelChamf. ↓Haus. ↓Sink. ↓RMSE ↓ρ (g/cm³) ↓SOAP ↓XRD ↓FE (eV/at.) ↓R. FE (eV/at.) ↓
SGEquiDiff [20] (CSPML)1.1710.5940.8950.5880.3157.211.8720.1522.782 · 10⁻¹
DiffCSP++ [19] (CSPML)1.2000.6090.9010.6020.3377.391.9360.1641.080 · 10⁻¹
SGEquiDiff (GT)0.0760.0400.0580.038<u>0.175</u>1.960.8090.013<u>5.959 · 10⁻²</u>
DiffCSP++ (GT)<u>0.323</u><u>0.165</u><u>0.245</u><u>0.162</u>0.173<u>2.09</u><u>0.856</u><u>0.016</u>2.033 · 10⁻²
PEROV-5
SGEquiDiff (CSPML)1.7521.563<u>13.211</u>1.0480.48983.145.4101.4881.609 · 10⁻¹
DiffCSP++ (CSPML)<u>1.521</u><u>1.352</u>12.8900.8990.0068.681.5220.0038.680 · 10⁻⁴
SGEquiDiff (GT)1.7971.59515.8191.1011.225243.994.5353.9171.602 · 10⁻¹
DiffCSP++ (GT)<u>1.669</u>1.35113.716<u>1.025</u><u>0.007</u><u>9.56</u><u>1.769</u><u>0.078</u><u>1.432 · 10⁻³</u>
CARBON-24
SGEquiDiff (CSPML)4.1583.50915.7682.5190.2232.301.0750.0297.890 · 10⁻⁴
DiffCSP++ (CSPML)4.1043.43915.6552.4750.2121.960.954<u>0.022</u>4.860 · 10⁻⁴
SGEquiDiff (GT)<u>3.988</u><u>3.354</u><u>15.175</u><u>2.392</u><u>0.216</u><u>1.88</u><u>0.944</u>0.023<u>3.680 · 10⁻⁴</u>
DiffCSP++ (GT)3.9493.31315.0852.3690.2061.860.9310.0213.120 · 10⁻⁴
MP-20
SGEquiDiff (CSPML)4.3893.87828.5752.722<u>0.302</u>4.641.5860.280<u>8.320 · 10⁻²</u>
DiffCSP++ (CSPML)4.3423.84027.8602.6960.4305.111.6060.2226.440 · 10⁻²
SGEquiDiff (GT)<u>4.161</u><u>3.593</u><u>26.652</u><u>2.548</u>0.2363.681.332<u>0.202</u><u>5.551 · 10⁻²</u>
DiffCSP++ (GT)4.1143.53325.9032.5050.323<u>3.99</u><u>1.386</u>0.1925.258 · 10⁻²
MPTS-52

Table 2. SICSP results. For each geometric and physical metric, we report the minimum distance from each prediction to the known polymorphs of the same composition and take the median across test compositions, restricted to the intersection of valid predictions across all four method-template combinations. Lower is better for all metrics. Bold marks the best method per dataset and metric; underline marks the second best. Carbon-24 results should be read with caution, as its large number of polymorphs per composition makes comparisons less conclusive.

These results depend on what each template source assumes. Ground truth templates presuppose knowledge of the target symmetry, which is one of the key unknowns that CSP seeks to determine, making this setting an idealized upper bound. CSPML templates avoid this assumption but introduce an additional source of error. The composition that CSPML retrieves may exhibit multiple valid symmetries due to polymorphism, and the correct space group may depend on several factors that the template does not encode. By constraining generation to a single fixed symmetry, both ground truth and CSPML templates implicitly assume that one space group is sufficient to describe the target. This assumption breaks down for compositions with high polymorphic diversity, where multiple distinct symmetries are equally valid and the correct one may not match the one that the template provides.

Ablation Study

We ablate the two main components of the OPAL-Bench protocol: geometric alignment and polymorph handling. We compare four variants: the full protocol, alignment only, polymorph handling only, and neither component. Because alignment affects only geometric metrics, we restrict this analysis to Chamfer, Hausdorff, Sinkhorn, and RMSE; Appendix § 11.2 reports the full results. Geometric alignment consistently reduces distances across datasets and models (roughly 25 to 60% on Carbon-24, MP-20, and MPTS-52 and fourfold on Perov-5) by removing unresolved rotations, translations, and same-species permutations, making geometric errors reflect structural differences rather than coordinate-frame artifacts. Polymorph handling has a smaller effect on Perov-5, MP-20, and MPTS-52 (below 10%), where each composition has relatively few known polymorphs, but a much larger effect on Carbon-24 (30 to 40%), where many polymorphs share the same formula and the reference choice strongly affects the score. As each metric takes the minimum over the known polymorphs, compositions with more reference structures tend to obtain lower distances (we further analyze this effect in Appendix § 11.4). Together, these results confirm that both components are complementary and independently necessary for a fair and informative evaluation of CSP models.

Limitations and Broader Impact

OPAL-Bench operates within the following constraints. First, it evaluates predictions against the polymorphs available in the reference datasets, introducing a bias tied to the polymorph diversity those datasets capture; we expect this bias to diminish naturally as crystal databases grow. The minimum over known polymorphs also favors compositions with more reference structures, an effect that is strongest on Carbon-24 and that we quantify in Appendix § 11.4. We discuss evaluation beyond known polymorphs in Appendix § 11.§ 14.1.

Second, the benchmark currently covers a representative set of physical metrics, and future work may extend it with additional properties and relaxation pipelines as the field advances. Formation energy metrics are necessarily tied to the MLIP and relaxation pipeline that we use for evaluation. We follow the LeMat-GenBench [29] setup and use its ensemble of three MLIPs, so that an external configuration fixes the scoring function in advance rather than us choosing it. Although averaging may not remove limitations shared by the three potentials, we score every method under the same ensemble, and rankings are largely preserved under each individual potential (Appendix § 11.5).

Third, OPAL-Bench evaluates a single prediction per composition. Prior CSP work often draws several samples per composition and reports the best match [9, 10], but under OPAL-Bench this selection would require choosing which metric defines the best sample, favoring that metric over the others. Multi-sample evaluation also raises a distinct question, the generative diversity of a method, for instance how many of the known polymorphs of a composition its samples cover, which a best-of-kk protocol does not capture. Evaluating diversity under a multi-sample protocol therefore remains a natural direction for future investigation.

Fourth, OPAL-Bench is agnostic to how a model is trained, so in principle it can also evaluate pretrained models and compare them with the methods reported here. Doing so, however, requires checking for data leakage between their training data and the test splits, since overlap would inflate their scores. For example, MP-20 and MPTS-52 derive from the Materials Project, which also underlies common pretraining corpora.

This work advances the rigorous evaluation of CSP, with direct positive implications for materials discovery in domains such as energy storage, catalysis, and microelectronics. However, more capable generative models could accelerate the design of materials with hazardous properties, such as toxic or energetic compounds, though we assess this risk as limited given that any computationally predicted structure must still undergo rigorous experimental synthesis and characterization before physical realization.

Conclusion

Crystal structure prediction is central to materials discovery, and the precision of its evaluation bounds progress in the field. We introduce OPAL-Bench, an orientation-invariant polymorph-aware benchmark for crystal structure prediction that addresses four fundamental limitations of existing evaluation frameworks: the sensitivity of Match Rate to arbitrary tolerance thresholds and the comparison of RMSE over method-specific subsets of predictions, the systematic bias of single-reference evaluation, the absence of physical plausibility metrics in machine learning evaluation protocols, and the direct comparison between symmetry-informed and standard CSP methods. By unifying geometric and physical metrics under a principled polymorph-aware protocol with explicit SE(3) alignment, OPAL-Bench enables comprehensive and fair comparisons across generative CSP models. Our experiments reveal that geometric performance does not reliably imply physical plausibility, demonstrating that structural metrics alone are insufficient to assess generative quality and that physical metrics constitute a necessary component of rigorous CSP evaluation. We also release OPAL-CSP, a flow-matching baseline that performs competitively on geometric metrics and ranks among the best on physical ones, showing that both can coexist in a single model. OPAL-Bench provides the CSP community with a standard evaluation framework, enabling reliable measurement of progress in crystal structure prediction for materials discovery.

Acknowledgments and Disclosure of Funding

This work was supported by Azure sponsorship credits granted by Microsoft’s AI for Good Research Lab. The authors declare no competing interests.

Critique and Justification of Metrics

Empirical critique of Match Rate and RMSE

The CSP literature standardizes evaluation around two metrics: Match Rate, which measures the fraction of predictions matching a reference within fixed tolerances, and RMSE, which quantifies positional error among matched predictions. Both metrics depend on a tolerance triplet (stols_{\mathrm{tol}}, θtol\theta_{\mathrm{tol}}, ltoll_{\mathrm{tol}}) that controls the structural similarity threshold for matching. We show that this dependence introduces two failure modes that compromise cross-method comparisons: (i) reported values shift substantially with the tolerance choice, and (ii) the standard tolerance setting yields rankings that contradict the rankings at neighboring configurations. Subsection A.1.1 demonstrates both failures empirically. Subsection A.1.2 reports Match Rate and RMSE for all benchmarked methods at the standard tolerance setting for compatibility with prior work, and we explicitly carry forward the limitations from A.1.1.

Sensitivity to tolerance thresholds

We sample 571 tolerance configurations uniformly from stol∈[0.1,0.95]s_{\mathrm{tol}} \in[0.1, 0.95], θtol∈[5∘,85∘]\theta_{\mathrm{tol}} \in[5^\circ, 85^\circ], and ltol∈[0.1,0.95]l_{\mathrm{tol}} \in[0.1, 0.95], and recompute Match Rate and RMSE on MPTS-52 under every configuration. Recomputing both metrics requires re-running the structural matching algorithm for every prediction under every triplet, which restricts this analysis to four representative methods; since the instability that we document below reflects properties of the evaluation protocol rather than of any particular method, the pattern extends to the remaining methods in the benchmark.

Figure 2 shows the distribution of Match Rate and RMSE across the 571 configurations. Both metrics span nearly their entire admissible range within every method, with Match Rate covering roughly [0,0.85][0, 0.85] and RMSE covering roughly [0,0.65][0, 0.65] Å. Reporting either metric at any single tolerance setting therefore yields one realization of a random variable whose support covers nearly the full range of plausible values.

Table 3 and Figure 3 examine whether relative rankings remain stable across configurations even when individual values do not. Two observations follow. First, most pairs exhibit substantial ranking instability, with reversal rates reaching up to 22.1%, indicating that the relative ranking between two methods is in general a parameter choice rather than an empirical finding; only CDVAE–CrystalFlow and DiffCSP–MatterGen show full dominance among the methods that we analyze, and constitute the only conclusions robust to the tolerance choice. Second, stols_{\mathrm{tol}} accounts for most of the ranking instability, with θtol\theta_{\mathrm{tol}} and ltoll_{\mathrm{tol}} contributing secondarily.

Method AMethod BA winsB winsReversal rate
CrystalFlowDiffCSP44512622.1%
CDVAEMatterGen11545520.1%
CrystalFlowMatterGen4957613.3%
CDVAEDiffCSP395326.8%
CDVAECrystalFlow05710.0%
DiffCSPMatterGen57100.0%

Table 3. Pairwise Match Rate comparison across 571 tolerance configurations on MPTS-52. Each row compares two methods and reports how many tolerance configurations favor each one, where a method is favored when it achieves the higher Match Rate. The reversal rate is the fraction of configurations in which the less frequently favored method wins, indicating how sensitive the pairwise ranking is to the tolerance choice.

Full Match Rate and RMSE results

Tables 4 and 5 report Match Rate and RMSE at the standard tolerance setting (stol=0.5s_{\mathrm{tol}} = 0.5, θtol=10∘\theta_{\mathrm{tol}} = 10^\circ, ltol=0.3l_{\mathrm{tol}} = 0.3) for all CSP and SICSP methods, respectively, on the four standard datasets. We include these results for compatibility with prior work, since most published CSP papers report values under this convention. We retrained every baseline from its released code and configuration. DiffCSP on MP-20 reaches a Match Rate of 0.490 and an RMSE of 0.065, against the published 0.515 and 0.063, and a small number of other baselines, such as CDVAE, show similar isolated gaps. We could not identify a cause beyond stochastic variability, since we followed each method’s released repository and procedure without modification.

Two limitations affect the interpretation of these tables. First, both metrics depend on the chosen tolerance triplet, and the relative ordering of methods is not invariant to that choice; the values we report here correspond to the convention that prior work adopts. Second, each method computes RMSE only over its own matched subset, so each method averages the reported values over different populations of structures: a method matching 30% of test compositions reports RMSE over a different and typically easier subset than a method matching 80%, since the matched subset of a weaker method leans toward compositions where matching succeeded.

The tolerance-free metrics we introduce in Section 3.2 address both limitations directly: they compute distances over the full set of valid predictions without depending on a matching threshold,3 and they evaluate every method over the same set of compositions. Appendix § 11.3 shows that our conclusions are robust to the SOAP and XRD descriptor settings.

ModelPerov-5 MR ↑Perov-5 RMSE ↓Carbon-24 MR ↑Carbon-24 RMSE ↓MP-20 MR ↑MP-20 RMSE ↓MPTS-52 MR ↑MPTS-52 RMSE ↓
CDVAE0.4470.3300.1940.3550.3330.1230.0420.280
DiffCSP0.5160.0810.155<u>0.299</u>0.4900.0650.1270.149
MatterGen0.507<u>0.073</u>0.1670.3370.2930.0630.100<u>0.124</u>
FlowLLM<u>0.520</u>0.097<u>0.194</u>0.4190.2230.0800.0060.395
CrystalFlow0.5230.098<u>0.176</u>0.3160.6180.0510.2230.138
FlowMM0.5110.0800.1570.301<u>0.598</u>0.0630.1250.212
OPAL-CSP0.5110.0660.1470.2770.5550.049<u>0.171</u>0.111

Table 4. Conventional Match Rate and RMSE results for CSP methods. We report the standard threshold-based metrics used by prior CSP methods on Perov-5, Carbon-24, MP-20, and MPTS-52, using (stol=0.5s_{\mathrm{tol}} = 0.5, θtol=10\theta_{\mathrm{tol}} = 10, ltol=0.3l_{\mathrm{tol}} = 0.3). Match Rate (MR; higher is better) measures the fraction of predictions that match a reference structure under these tolerances, while RMSE (lower is better) is computed only over matched predictions. Bold marks the best value and <u>underline</u> marks the second-best value for each dataset and metric.

ModelPerov-5 MR ↑Perov-5 RMSE ↓Carbon-24 MR ↑Carbon-24 RMSE ↓MP-20 MR ↑MP-20 RMSE ↓MPTS-52 MR ↑MPTS-52 RMSE ↓
SGEquiDiff (CSPML)0.5300.0980.1030.2310.6350.0410.2770.078
DiffCSP++ (CSPML)0.5110.0930.2900.1710.7050.0300.361<u>0.093</u>
SGEquiDiff (GT)<u>0.981</u>0.025<u>0.288</u><u>0.147</u><u>0.756</u>0.039<u>0.417</u>0.093
DiffCSP++ (GT)0.984<u>0.051</u>0.2880.1120.787<u>0.034</u>0.4320.096

Table 5. Conventional Match Rate and RMSE results for SICSP methods. We report the standard threshold-based metrics used by prior CSP and SICSP methods on Perov-5, Carbon-24, MP-20, and MPTS-52, using (stol=0.5s_{\mathrm{tol}} = 0.5, θtol=10\theta_{\mathrm{tol}} = 10, ltol=0.3l_{\mathrm{tol}} = 0.3). Results are shown for SGEquiDiff and DiffCSP++ under both CSPML-predicted symmetry templates and ground truth (GT) symmetry templates. Match Rate (MR; higher is better) measures the fraction of predictions that match a reference structure under these tolerances, while RMSE (lower is better) is computed only over matched predictions. Bold marks the best value and <u>underline</u> marks the second-best value for each dataset and metric.

Detailed description of proposed metrics

Geometric Distance Metrics

Geometric metrics quantify structural similarity by treating each crystal as a point cloud in R3\mathbb{R}^{3} and comparing the Cartesian positions of predicted and reference atoms. Unlike physical metrics, they depend on the coordinate representation and are not invariant to rotations, translations, or same-species permutations, so we compute them after the species-aware SE(3) alignment of Appendix § 9.2 and aggregate them per species (§ 3.4). Throughout, P={pi}i=1mP = \{p_i\}_{i=1}^{m} and Q={qj}j=1nQ = \{q_j\}_{j=1}^{n} denote the predicted and reference point sets of a given species.

Chamfer. To quantify the average nearest-neighbor discrepancy between two point sets, we adopt the Chamfer distance,

dChamfer(P,Q)=1m∑p∈Pmin⁡q∈Q∥p−q∥2+1n∑q∈Qmin⁡p∈P∥q−p∥2.(1)d_{\mathrm{Chamfer}}(P,Q) = \frac{1}{m}\sum_{p\in P}\min_{q\in Q}\lVert p-q\rVert_2 + \frac{1}{n}\sum_{q\in Q}\min_{p\in P}\lVert q-p\rVert_2. \tag*{(1)}

The bidirectional formulation penalises both atoms the prediction fails to place and atoms the prediction adds beyond the reference, making it sensitive to the global nearest-neighbor structure of each point cloud.

Hausdorff. To capture the worst-case nearest-neighbor discrepancy between two point sets, we adopt the Hausdorff distance,

dHausdorff(P,Q)=max⁡(max⁡p∈Pmin⁡q∈Q∥p−q∥2,max⁡q∈Qmin⁡p∈P∥q−p∥2).(2)d_{\mathrm{Hausdorff}}(P,Q)=\max\left(\max_{p\in P}\min_{q\in Q}\lVert p-q\rVert_2,\max_{q\in Q}\min_{p\in P}\lVert q-p\rVert_2\right). \tag*{(2)}

Unlike Chamfer, this metric is sensitive to outliers and therefore exposes large local deviations in atomic positions that average-based scores can obscure.

Sinkhorn. As an entropic-regularised approximation to the Earth Mover’s Distance [18], Sinkhorn distance is obtained from the pairwise cost matrix Cij=∥pi−qj∥2C_{ij}=\lVert p_i-q_j\rVert_2 and uniform marginals a=1m1m\mathbf{a}=\frac{1}{m}\mathbf{1}_m, b=1n1n\mathbf{b}=\frac{1}{n}\mathbf{1}_n by solving

dSinkhorn(P,Q)=⟨T∗,C⟩,(3)d_{\mathrm{Sinkhorn}}(P,Q)=\langle T^{*},C\rangle, \tag*{(3)}

where T∗T^{*} minimises ⟨T,C⟩−εH(T)\langle T,C\rangle-\varepsilon H(T) over transport plans with marginals a\mathbf{a} and b\mathbf{b}, with H(T)=−∑ijTijlog⁡TijH(T)=-\sum_{ij}T_{ij}\log T_{ij} and ε=10−3\varepsilon=10^{-3}. We run at most 100 iterations with a convergence threshold of 10−310^{-3}. By balancing sensitivity to global mass transport with robustness to local outliers, this metric complements Chamfer and Hausdorff.

RMSE. Given two point sets of equal size NN, RMSE quantifies the root mean squared deviation after optimal alignment,

dRMSE(P,Q)=1N∑i=1N∥pi−qi∥22,(4)d_{\mathrm{RMSE}}(P,Q)=\sqrt{\frac{1}{N}\sum_{i=1}^{N}\lVert p_i-q_i\rVert_2^2}, \tag*{(4)}

where points pair according to the optimal assignment σ∗\sigma^{*} from the alignment procedure. In contrast to the legacy RMSE reported in prior CSP work, we evaluate it across all test compositions rather than restricting to matched predictions only.

Physical Distance Metrics

Physical metrics quantify structural quality in terms of experimentally observable or computationally derived properties. Each metric depends only on the physical content of the structure rather than its coordinate representation, making it invariant to rigid-body transformations and same-species permutations by construction. We apply all physical metrics directly to predicted and reference structures without any preprocessing.

Density. The density of a crystal structure is ρ=m/V\rho=m/V, where mm denotes the total atomic mass in the unit cell and V=∣det⁡L∣V=\lvert\det L\rvert is the cell volume. We report the absolute difference ∣ρ^−ρ∗∣\lvert\hat{\rho}-\rho^{*}\rvert in g/cm3^3. Density serves as a proxy for unit cell quality independent of atomic positions, capturing lattice errors that geometric point-cloud metrics may miss when atomic coordinates are locally accurate.

SOAP. The Smooth Overlap of Atomic Positions (SOAP) descriptor [30] encodes the local chemical environment of each atom via a rotationally invariant expansion of the atomic density in a basis of spherical harmonics and radial functions. We compute SOAP fingerprints using DScribe [36] with cutoff radius rcut=4.0r_{\mathrm{cut}}=4.0 Å, nmax⁡=12n_{\max}=12 radial basis functions, lmax⁡=9l_{\max}=9 angular momentum channels, Gaussian broadening σ=0.5\sigma=0.5 Å, periodic boundary conditions, and outer product averaging across all atoms in the unit cell. The distance between predicted and reference fingerprints is the ℓ2\ell_2 norm ∥f^−f∗∥2\lVert\hat{f}-f^{*}\rVert_2 between the two averaged descriptor vectors.

X-Ray Diffraction. We simulate powder XRD patterns using the Scatter module of Dans_Diffraction4 over the range 2θ∈[5∘,90∘]2\theta\in[5^\circ,90^\circ] and normalise each pattern by its total integrated area, yielding a probability distribution over 2θ2\theta. We measure the distance between predicted and reference patterns via the Wasserstein distance,

dXRD(M^,M∗)=W1(p^2θ,p2θ∗),(5)d_{\mathrm{XRD}}(\hat{M},M^{*})=W_1(\hat{p}_{2\theta},p^{*}_{2\theta}), \tag*{(5)}

where p^2θ\hat{p}_{2\theta} and p2θ∗p^{*}_{2\theta} are the area-normalised intensity distributions of the predicted and reference structures. The Wasserstein distance is appropriate for XRD patterns because it accounts for peak position shifts rather than treating each 2θ2\theta bin independently, as ℓ2\ell_2-based distances would.

Formation Energy. The formation energy of a crystal structure MM with composition A=a1,…,aNA = a_1, \ldots, a_N is

Ef(M)=E(M)−∑iniμai,E_f(M) = E(M) - \sum_i n_i \mu_{a_i},

where E(M)E(M) is the total energy that the MLIP predicts, nin_i counts atoms of species aia_i, and μai\mu_{a_i} is the elemental reference energy of species aia_i that we take from DFT calculations (PBE) in the LeMat-Bulk dataset [37]. We evaluate E(M)E(M) using the three-model MLIP ensemble from LeMat-GenBench [29] (Orb [38], MACE-MP [39], and UMA [40]) and take the mean of the three predicted energies. We evaluate both the predicted structure M^\hat{M} and the ground truth structure M∗M^* with this ensemble without prior relaxation. We report the absolute difference ∣Ef(M^)−Ef(M∗)∣|E_f(\hat{M}) - E_f(M^*)| in eV/atom.

Relaxed Formation Energy. We relax both the predicted structure M^\hat{M} and the ground truth structure M∗M^* independently using each MLIP in the ensemble, via BFGSLineSearch with a FrechetCellFilter (joint atomic and cell relaxation, fmax⁡=0.02f_{\max} = 0.02 eV/Å, max 50 steps), and then evaluate EfE_f on the respective relaxed geometries. This symmetric evaluation removes the dependence on the original atomic positions and measures how close the two structures are in the energy landscape that the MLIP explores. We report ∣Efrelax(M^)−Efrelax(M∗)∣|E_f^{\mathrm{relax}}(\hat{M}) - E_f^{\mathrm{relax}}(M^*)| in eV/atom.

Metrics not included in OPAL-Bench

We also considered several additional metrics, including those of the CSPBench metric suite [17], and excluded three of them from OPAL-Bench. Graph Edit Distance, the CrystalNN fingerprint distance, and the minimal Wyckoff position distance either returned undefined values or failed to compute for a substantial fraction of the predicted structures. Since OPAL-Bench evaluates all methods on the intersection of valid predictions, these failures reduced the intersections to sets that were not representative of the test splits, which led us to exclude these metrics. For Graph Edit Distance and the CrystalNN fingerprint distance, we hypothesize that the failures arise because both require deciding which atoms are bonded before a distance can be computed, a decision that is unreliable for the degenerate geometries that generative models occasionally produce, such as overlapping atoms or unusually short interatomic distances. For the minimal Wyckoff position distance, we attribute the failures to its definition, which requires the predicted and reference structures to share a space group and Wyckoff configuration, a condition that unconstrained CSP predictions often do not satisfy.

We also excluded the energy above the convex hull, which is redundant in our setting. The energy above the hull of a structure MM with composition AA is ΔEhull(M)=Ef(M)−Ehull(A)\Delta E_{\mathrm{hull}}(M) = E_f(M) - E_{\mathrm{hull}}(A), where Ehull(A)E_{\mathrm{hull}}(A) is the formation energy of the convex hull at composition AA. Since a prediction and its reference share the same composition, Ehull(A)E_{\mathrm{hull}}(A) cancels in the difference,

∣ΔEhull(M^)−ΔEhull(M∗)∣=∣Ef(M^)−Ef(M∗)∣.(6)|\Delta E_{\mathrm{hull}}(\hat{M}) - \Delta E_{\mathrm{hull}}(M^*)| = |E_f(\hat{M}) - E_f(M^*)|. \tag*{(6)}

and the same holds after relaxation. This metric therefore coincides with the formation energy metrics defined above, while requiring a reference phase diagram that does not change the result.

Benchmark Setup and Implementation Details

Statistical characterization of datasets

We characterize the four datasets that we use in our benchmark along two structural axes that determine the difficulty of the CSP and SICSP tasks: polymorphism, which controls the number of valid reference structures per composition, and space group distribution, which determines the symmetry diversity that methods must learn.

Polymorphism distribution

We define a polymorph group as a set of structures that share both chemical formula and number of atoms in the unit cell. This definition is operationally motivated by the CSP task: a generative model receives a chemical formula as input and produces a structure with a fixed number of atoms that the formula determines, so two structures sharing the formula but differing in unit-cell atom count are not interchangeable references for a single prediction. The definition has practical consequences for Carbon-24, where every structure is crystallographically a polymorph of carbon, but our framework distinguishes 10 polymorph groups corresponding to unit cells with 6 to 24 atoms.

Table 6 reports polymorphism statistics independently per split, since the benchmark evaluates each method on the test split and only test-split polymorphs serve as references at evaluation time.

DatasetSplitStructuresPoly. GroupsPolymorphism RateMax Polymorphs
Perov-5train11,3563,33058.6%2
Perov-5val3,78736219.1%2
Perov-5test3,78537619.9%2
MPTS-52train27,3801,26110.4%8
MPTS-52val5,0001164.8%5
MPTS-52test8,0961905.0%6
MP-20train27,1361,49014.0%16
MP-20val9,0472646.7%7
MP-20test9,0462476.3%8
Carbon-24train6,09110100.0%1,980
Carbon-24val2,03210100.0%671
Carbon-24test2,03010100.0%705

Table 6.

Table 6: Polymorphism statistics across the four benchmark datasets, computed independently within each split. Structures are grouped by shared chemical formula and unit-cell atom count. Poly. Groups counts groups containing more than one structure, Polymorphism Rate is the fraction of structures in such groups, and Max Polymorphs is the largest number of structures observed in a single group.

The four datasets span a wide range of polymorphic complexity, with Carbon-24 at one extreme and MP-20, MPTS-52, and Perov-5 sharing a median of 2 polymorphs per group at rates of 6.3%, 5.0%, and 19.9%. This contrast explains the asymmetric magnitude of the polymorph-handling effect: reference choice rarely matters in the first three datasets but is consequential for nearly every Carbon-24 composition. MP-20 and MPTS-52 polymorphism also exhibits structural heterogeneity relevant to the SICSP discussion in § 5.2: in the MP-20 test split, 35 polymorphic groups span monoclinic and orthorhombic systems, 24 span triclinic and monoclinic, and 20 span tetragonal and cubic, with the same pattern at smaller scale in MPTS-52. A SICSP method that conditions on a single space group template therefore restricts generation to a symmetry that may not correspond to the closest polymorph, reinforcing that template-based methods cannot accommodate polymorphic diversity across symmetries.

Space group distribution

The four datasets differ substantially in the diversity of crystallographic symmetries they contain. Table 7 reports the number of distinct space groups that each dataset contains and the share of structures that the most frequent space groups capture, while Figure 4 visualizes the cumulative coverage of structures as a function of the number of space groups, after ranking them by frequency.

Figure 4. Cumulative space group coverage across the four benchmark datasets. Each curve shows the fraction of structures captured by the top-kk most frequent space groups, with kk on a logarithmic scale. Horizontal dashed lines mark the 50% and 90% coverage thresholds.

Table 7: Space group diversity across the four benchmark datasets. Unique SGs denotes the number of distinct space groups observed in each dataset. The last three columns report the cumulative fraction of structures assigned to the top 1, top 5, and top 20 most frequent space groups within each dataset.

DatasetUnique SGsTop 1 SGTop 5 SGsTop 20 SGs
Perov-5556%100%100%
Carbon-245633%73%96%
MP-2017915%31%65%
MPTS-522089%26%56%

Table 7.

Perov-5 contains only 5 distinct space groups, with a single space group (#25, Pmc21Pmc2_1) accounting for 56% of the structures. This reflects the dataset’s design as a collection of synthetic ABO3_3-type perovskites that the dataset curates and constrains to specific symmetries, rather than a representative sample of materials space.

Carbon-24 spans 56 space groups but remains highly concentrated: the two most frequent space groups (#1 and #2, both triclinic) cover half of the dataset, and 84.2% of structures belong to triclinic or monoclinic systems. This concentration is consistent with the dataset’s origin as carbon polymorphs that ab initio random structure searching generated at 10 GPa [23], a regime that favors low-symmetry configurations.

MP-20 and MPTS-52 contain the most diverse symmetry distributions, with 179 and 208 unique space groups respectively, and long-tailed distributions in which the top 20 space groups cover only 65% and 56% of structures. The crystal system composition is also more balanced than in the other datasets, with all seven crystal systems appearing and no single system exceeding 26% of the structures. The most frequent space group in MP-20 is #225 (Fm3ˉm\mathrm{Fm\bar{3}m}, cubic) at 14.8%, while space group #62 (Pnma\mathrm{Pnma}, orthorhombic) dominates MPTS-52 at 9.0%; the difference reflects the broader compositional scope of MPTS-52, which includes structures with up to 52 atoms per cell and consequently lower-symmetry arrangements.

The diversity gap between Perov-5 and the materials-project-derived datasets has direct implications for SICSP. CSPML retrieves space group templates by composition-based similarity to training structures: in Perov-5, where only 5 space groups exist and the top 5 cover the entire dataset, retrieval is essentially constrained to a small set of plausible templates. In MP-20 and MPTS-52, retrieval must select from 179 to 208 distinct space groups, with no top-20 list exceeding 65% coverage; the probability that the retrieved template matches the correct symmetry therefore decreases substantially. This asymmetry partially explains the dataset-dependent gap between CSPML and ground truth template performance that § 5.2 of the main paper reports.

Species-aware alignment

The alignment procedure that § 3.4 describes parametrizes the rotation as a unit quaternion q∈R4q \in\mathbb{R}^{4} with ∥q∥2=1\lVert q\rVert_{2}=1 and the translation as t∈R3t \in\mathbb{R}^{3}. The cost matrix assigns

Cij=∥R(q)xi+t−xj∥2C_{ij}=\lVert R(q)x_i+t-x_j\rVert_{2}

for same-species pairs and +∞+\infty otherwise. The Hungarian algorithm [41] then solves the resulting assignment problem. We update (q,t)(q,t) with Adam (learning rate 0.1) on the RMSE under the current assignment, and recompute the assignment after every gradient step. We run 400 steps from each of 4 random initializations of qq, with tt initialized at zero, and keep the transform with the lowest RMSE across all steps and initializations.

We chose this budget as a trade-off between computational cost and alignment quality. To characterize its convergence, we sample 300 pairs of a prediction and one of its reference polymorphs per dataset, drawn across the seven CSP methods, and rerun the procedure with 8 random initializations of 800 steps each. For each pair, we take the lowest RMSE found with this extended budget as the reference, and the RMSE under the identity rotation and zero translation as the unaligned value. We then measure, for each smaller budget, the fraction of the gap between the unaligned value and the reference that the alignment closes. Figure 5 shows that most of the reduction occurs within the first 200 steps and that additional initializations improve the alignment more than additional steps. Our setting of 4 initializations and 400 steps recovers 98.3%, 97.7%, 97.7%, and 95.9% of the reduction on Perov-5, Carbon-24, MP-20, and MPTS-52, respectively, with MPTS-52, whose unit cells are the largest, still improving slowly beyond 400 steps. Since a suboptimal alignment can only increase a distance, the geometric metrics that we report are upper bounds on their values under the optimal alignment, and the mean RMSE under our setting lies 2 to 4% above the lowest RMSE found with the extended budget. As convergence slows with increasing unit cell size, which the MPTS-52 curves illustrate, datasets with larger or more complex structures may require a larger alignment budget, and we recommend repeating this analysis before applying OPAL-Bench to them.

Figure 5. Convergence of the species-aware alignment. Fraction of the reduction from the unaligned RMSE to the lowest RMSE found with 8 initializations of 800 steps that each budget recovers, averaged over 300 pairs per dataset. Each line corresponds to a number of random initializations, and the black point marks the setting that OPAL-Bench uses (4 initializations of 400 steps).

OPAL-CSP implementation details

Architecture

OPAL-CSP builds on EquiformerV2 [32] and models crystal structures as labeled point clouds evolving under a flow matching objective. We represent the chemical composition as a 3D graph G=(V,E)\mathcal{G} = (\mathcal{V}, \mathcal{E}) and the lattice as a latent vector zlat∈Rdcellz_{\mathrm{lat}} \in\mathbb{R}^{d_{\mathrm{cell}}}.

Each vertex i∈Vi \in\mathcal{V} carries a block-structured feature

hi={Hi,ℓ∈R(2ℓ+1)×C}ℓ=0ℓmax⁡,(7)h_i = \left\{ H_{i,\ell} \in\mathbb{R}^{(2\ell+1)\times C} \right\}_{\ell=0}^{\ell_{\max}}, \tag*{(7)}

where the (2ℓ+1)(2\ell+1) rows index spherical-harmonic components that transform equivariantly under rotations and the CC columns store learnable channels that we initialize from the atomic number aia_i. We follow the default values of ℓmax⁡\ell_{\max} and CC from Liao et al. [32], and set dcell=128d_{\mathrm{cell}} = 128. We encode the invariant polar lattice representation κt∈R6\kappa_t \in\mathbb{R}^{6} into zlat∈Rdcellz_{\mathrm{lat}} \in\mathbb{R}^{d_{\mathrm{cell}}} using a two-layer MLP and concatenate it to every node feature before processing.

We construct edges between each atom and up to KK nearest neighbours within cutoff radius rcr_c, using periodic boundary conditions. For each edge (i,j)(i,j), we compute the relative geometry vij=xj−xi\mathbf{v}_{ij} = \mathbf{x}_j - \mathbf{x}_i, expand the distance dij=∥vij∥2d_{ij} = \lVert\mathbf{v}_{ij} \rVert_2 with a Gaussian radial basis ϕ(dij)∈RB\phi(d_{ij}) \in\mathbb{R}^{B}, and project the direction u^ij=vij/dij\hat{\mathbf{u}}_{ij} = \mathbf{v}_{ij}/d_{ij} through spherical harmonics Y(ℓ)(u^ij)∈R2ℓ+1Y^{(\ell)}(\hat{\mathbf{u}}_{ij}) \in\mathbb{R}^{2\ell+1} to produce SE(3)-equivariant messages.

We process the graph through NN equivariant Transformer layers. Each layer applies an attention block that performs irreaware message passing using eSCN convolutions with attention renormalization, and then applies a feed-forward block with separable S2S^2 activation and separable layer normalization. We embed the time step tt via sinusoidal features and use it to condition each normalization through adaptive separable layer normalization, which predicts scale, shift, and gating parameters.

Two output heads produce the velocity fields vt;θ(κ)v_{t;\theta}^{(\kappa)} and vt;θ(F)v_{t;\theta}^{(F)}. The lattice head averages node features and maps them to the 6-dimensional invariant polar space, while the coordinate head maps node features to the 3×N3 \times N fractional coordinate space. Table 8 reports dataset-specific architectural hyperparameters, Table 9 reports shared hyperparameters, and Figure 6 illustrates the architecture.

Figure 6. OPAL-CSP architecture. (A) Overview of the full model. The chemical composition is encoded into atomic and lattice embeddings which, together with the distance matrix, initialize a 3D graph with SE(3)-equivariant irreducible representations. The graph is processed through a stack of equivariant Transformer layers followed by separable layer normalization. Two output heads predict the flow matching velocity fields for the lattice vt;θ(κ)v_{t;\theta}^{(\kappa)} and fractional coordinates vt;θ(F)v_{t;\theta}^{(F)}. (B) Equivariant Transformer block. Each block applies SE(3)-equivariant graph attention via eSCN convolutions, followed by a feed-forward network, with residual connections and adaptive separable layer normalization conditioned on the time step.

Perov-5Carbon-24MP-20MPTS-52
Cutoff rcr_c (Å)1051212
Layers NN3666

Table 8. Dataset-specific OPAL-CSP hyperparameters. We report the graph-construction and architecture hyperparameters that vary across datasets: radial cutoff rcr_c and number of layers NN.

HyperparameterValue
Max degree ℓmax⁡\ell_{\max}6
Sphere channels CC128
Max neighbours KK20
Attention heads8
Attention hidden channels64
FFN hidden channels128
Gaussian RBF basis BB600
Time embedding dim128
Lattice embedding dim dcelld_{\mathrm{cell}}128

Table 9. Shared OPAL-CSP hyperparameters.

Training and inference

We train OPAL-CSP with a flow matching regression objective that jointly supervises lattice and fractional coordinate velocity fields,

LFM=E[λβ∥vt;θ(κ)−ut(κ)∥22+λα∥vt;θ(F)−ut(F)∥22],(8)\mathcal{L}_{\mathrm{FM}} = \mathbb{E}\left[\lambda_{\beta}\left\|v_{t;\theta}^{(\kappa)}-u_t^{(\kappa)}\right\|_2^2+\lambda_{\alpha}\left\|v_{t;\theta}^{(F)}-u_t^{(F)}\right\|_2^2\right], \tag*{(8)}

with λβ=1\lambda_{\beta}=1 (lattice) and λα=10\lambda_{\alpha}=10 (fractional coordinates). The lattice follows a linear interpolation path κt=(1−t)κ0+tκ1\kappa_t=(1-t)\kappa_0+t\kappa_1, and the fractional coordinates follow a wrapped path

Ft=F0+t⋅wrap⁡(F1−F0).F_t=F_0+t\cdot\operatorname{wrap}(F_1-F_0).

where wrap⁡(x)=(x+12)mod⁡1−12\operatorname{wrap}(x)=(x+\frac{1}{2})\operatorname{mod}1-\frac{1}{2}, following CrystalFlow [12].

Prior to training, we estimate the training-set mean κˉ5\bar{\kappa}_5 of the fifth polar component and shift the lattice source distribution by this offset (κ0(5)←κ0(5)+κˉ5)(\kappa_0^{(5)}\leftarrow\kappa_0^{(5)}+\bar{\kappa}_5), so that the expected velocity for each polar component is centred near zero and all six components contribute equally to the lattice loss gradient.

We train with the Adam optimizer using a learning rate of 10−310^{-3} and a reduce-on-plateau scheduler with factor 0.6 and patience 30, monitoring a scale-normalized validation loss Lcoords/Lcoords0+Llat/Llat0\mathcal{L}_{\mathrm{coords}}/\mathcal{L}_{\mathrm{coords}}^0+\mathcal{L}_{\mathrm{lat}}/\mathcal{L}_{\mathrm{lat}}^0, where we estimate the zero-predictor baselines L0\mathcal{L}^0 from the training set before training so that both tasks contribute equally to the scheduler signal regardless of their natural scale difference. All datasets use a total batch size of 128 structures across GPUs, and we train all models for 1000 epochs. All experiments use random seed 42. We selected dataset specific hyperparameters (Table 8) through a limited number of exploratory runs with OPAL-Bench in view, rather than a systematic search or formal ablation.

At inference, we integrate the learned ODE from t=0t=0 to t=1t=1 using an Euler solver with 1000 steps. At each step, we scale the predicted coordinate velocity by an annealing factor γ(t)=1+5ReLU⁡(t)\gamma(t)=1+5\operatorname{ReLU}(t) before integration, progressively amplifying coordinate updates as t→1t\to1. We adopt this annealing schedule from CrystalFlow [12] without modification. We did not tune the sampling procedure against OPAL-Bench for OPAL-CSP or for any baseline, each of which uses the inference configuration released by its authors. Alternative sampling or integration schemes could improve the results of OPAL-CSP and of the baselines alike, and we leave their exploration to future work.

Baseline implementation details

We use the official implementations and configurations that the authors released for the baseline methods whenever available. We preserve the original training, sampling, and post-processing procedures for each method to avoid introducing implementation-dependent advantages or disadvantages. We restrict any deviations from the original implementations to dataset formatting, checkpoint loading, and evaluation-interface standardization that we need to run all methods under the same OPAL-Bench protocol.

Computational resources

We train all models on NVIDIA Quadro RTX 8000 GPUs (48 GB memory), using between 1 and 7 GPUs depending on the model and dataset. Metric computation runs on 2× Intel Xeon Gold 6230 CPUs (40 physical cores, 2.10 GHz). Training OPAL-CSP takes 15h 5m on Perov-5 and 2h 1m on Carbon-24 using 2 GPUs, and 2d 5h 5m on MP-20 and 4d 3h 22m on MPTS-52 using 4 GPUs.

Complete Results

Validity rates and intersection sizes

For CSP, every metric has a well-defined value for all seven methods on 100% of test compositions on Perov-5 and Carbon-24, and on at least 99.8% on MP-20 and MPTS-52 (at least 8,705 of 8,719 and 7,867 of 7,878 compositions, respectively), so we do not report per-method CSP validity separately. The few excluded compositions correspond to predictions for which a metric cannot be computed, mostly XRD patterns of predictions with degenerate unit cells, and we evaluate each metric on the intersection of compositions where all seven methods have a valid value. For SICSP, validity depends jointly on the method, the metric, and the dataset, since some metrics require structural or symmetry properties that not every prediction satisfies. Table 10 reports validity rates for each method, metric, and dataset across the four SICSP method–template combinations, together with the intersection for each metric and dataset, the fraction of compositions for which all four combinations yielded a valid prediction for that metric. We compute all SICSP results in this paper on these intersections, so for any given metric we score all four methods on identical compositions. Validity is essentially complete on Perov-5 and Carbon-24. On MP-20 and MPTS-52, SGEquiDiff produces a small fraction of predictions with degenerate lattice geometries that we exclude from the affected metrics; on MPTS-52, formation-energy metrics additionally exclude a few DiffCSP++ (CSPML) predictions whose relaxations failed to converge.

ModelChamf.Haus.Sink.RMSEρ\rhoSOAPXRDFER. FE
DiffCSP++ (CSPML)100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%
SGEquiDiff (CSPML)100.0%100.0%100.0%100.0%99.9%99.9%99.9%100.0%99.9%
DiffCSP++ (GT)100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%
SGEquiDiff (GT)99.9%99.9%99.9%99.9%99.9%99.9%99.9%99.9%99.9%
Intersection99.8%99.8%99.8%99.8%99.8%99.8%99.8%99.8%99.8%
DiffCSP++ (CSPML)100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%
SGEquiDiff (CSPML)100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%
DiffCSP++ (GT)100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%
SGEquiDiff (GT)100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%
Intersection100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%
DiffCSP++ (CSPML)100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%
SGEquiDiff (CSPML)98.1%98.1%98.1%98.1%99.6%98.4%98.1%98.1%98.1%
DiffCSP++ (GT)100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%
SGEquiDiff (GT)99.1%99.1%99.1%99.1%99.1%99.1%99.1%99.1%99.1%
Intersection97.4%97.4%97.4%97.4%98.7%97.7%97.4%97.4%97.4%
DiffCSP++ (CSPML)100.0%100.0%100.0%100.0%100.0%100.0%100.0%99.6%99.5%
SGEquiDiff (CSPML)95.1%95.1%95.1%95.1%95.1%95.0%95.1%95.0%95.0%
DiffCSP++ (GT)100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%100.0%
SGEquiDiff (GT)95.4%95.4%95.4%95.4%95.4%95.4%95.4%95.2%95.2%
Intersection91.3%91.3%91.3%91.3%91.3%91.1%91.3%90.6%90.6%

Table 10. Validity rates and evaluation-set intersections for SICSP methods. Each row reports the percentage of test formulas with valid predictions available for each metric. Results are shown for DiffCSP++ and SGEquiDiff under both CSPML-predicted and ground truth (GT) symmetry templates. The Intersection row in each dataset block reports the fraction of formulas for which all listed method–template combinations produced valid predictions for that metric, defining the common evaluation set used for comparison.

95% confidence intervals

Figures 7 and 8 extend Tables 1 and 2 with bar charts covering all metric-dataset combinations. Error bars report 95% bootstrap confidence intervals that we compute via 10,000 resampling iterations over test compositions. Non-overlapping intervals indicate statistically significant differences between methods.

On Perov-5, the structural homogeneity of perovskites yields tight confidence intervals and clear separations between methods on most metrics. Carbon-24 produces wide intervals throughout, consistent with the high structural variance of carbon polymorphs, which makes conclusive rankings difficult for this dataset. MP-20, as the largest dataset, serves as the primary reference for cross-method comparison, though rankings here vary more across metrics than on any other dataset. MPTS-52, despite its structural complexity, yields the tightest confidence intervals and the most consistent rankings within each metric family, making it the most reliable setting for definitive conclusions.

Qualitative Results

Figure 9 to Figure 12 show representative crystal structure predictions from all seven CSP methods, rendered alongside the corresponding ground truth structures as ball-and-stick models within their unit cells.

Qualitative comparison on representative samples from the Carbon-24 dataset in the CSP task

Figure 12. Qualitative comparison on representative samples from the Carbon-24 dataset in the CSP task. Predictions from the seven CSP methods are shown alongside their corresponding selected reference polymorphs. Given the high number of polymorphs in Carbon-24, each prediction is visualized with its closest selected reference polymorph rather than a single shared ground truth structure across all columns. Structures are rendered as ball-and-stick models within their unit cells. Each prediction–reference pair corresponds to one test composition.

Figures 13 to 16 present the analogous qualitative comparisons for the SICSP track, covering SGEquiDiff and DiffCSP++ under both CSPML-predicted and ground truth symmetry templates. Each row corresponds to one test composition, and atom species are color-coded consistently across method predictions and ground truth. Because symmetrization can alter the atom count, SICSP predictions may contain a different number of atoms than the ground truth; we show the raw outputs. Figure 17 illustrates the source of FlowLLM’s anomalously high formation energy distances; predicted structures contain atomic species at near-coincident positions, producing the unphysically large formation energies reflected in Table 1.

Representative structures illustrating failure modes of FlowLLM predictions

Figure 17. Failure modes of FlowLLM predictions. Representative structures generated by FlowLLM exhibit superposed atomic species at near-coincident positions, producing unphysical short-range overlaps. These geometric artifacts explain the anomalously large Formation Energy distances reported in Table 1 (e.g., 3264.4 eV/atom on MPTS-52, 3.79 eV/atom on Carbon-24), since pairwise repulsion diverges as interatomic distances approach zero.

Qualitative comparison on representative samples from the MPTS-52 dataset in the CSP task

Figure 10. Qualitative comparison on representative samples from the MPTS-52 dataset in the CSP task. Predictions from the seven CSP methods are shown alongside the corresponding ground truth structures, rendered as ball-and-stick models within their unit cells. Each row corresponds to one test composition, and atom species are color-coded consistently across methods and ground truth.

Qualitative comparison on representative samples from the Perov-5 dataset in the CSP task

Figure 11. Qualitative comparison on representative samples from the Perov-5 dataset in the CSP task. Predictions from the seven CSP methods are shown alongside the corresponding ground truth structures, rendered as ball-and-stick models within their unit cells. Each row corresponds to one test composition, and atom species are color-coded consistently across methods and ground truth.

Qualitative comparison on representative samples from the MP-20 dataset in the SICSP task

Figure 13. Qualitative comparison on representative samples from the MP-20 dataset in the SICSP task. Predictions from the two SICSP methods are shown under both CSPML-predicted and ground truth symmetry templates alongside the corresponding ground truth structures, rendered as ball-and-stick models within their unit cells. Each row corresponds to one test composition, and atom species are color-coded consistently across methods, template settings, and ground truth.

Qualitative comparison on representative samples from the MPTS-52 dataset in the SICSP task

Figure 14. Qualitative comparison on representative samples from the MPTS-52 dataset in the SICSP task. Predictions from the two SICSP methods are shown under both CSPML-predicted and ground truth symmetry templates alongside the corresponding ground truth structures, rendered as ball-and-stick models within their unit cells. Each row corresponds to one test composition, and atom species are color-coded consistently across methods, template settings, and ground truth.

Qualitative comparison of representative Perov-5 structures

Figure 15. Qualitative comparison on representative samples from the Perov-5 dataset in the SICSP task. Predictions from the two SICSP methods are shown under both CSPML-predicted and ground-truth symmetry templates alongside the corresponding ground truth structures, rendered as ball-and-stick models within their unit cells. Each row corresponds to one test composition, and atom species are color-coded consistently across methods, template settings, and ground truth.

Extended Analysis

Geometric–physical correlation analysis

Since each metric selects its own closest polymorph, the geometric and physical metrics of a single prediction may be computed against different reference structures, which could in principle weaken their correlation. To control for this effect, Figure 18, Figure 19, and Figure 20 plot each physical metric against each geometric metric for all CSP predictions in Perov-5, MP-20, and MPTS-52 for the subset of structures with no polymorphs in the test split (3,033, 8,472, and 7,688 compositions, respectively), so that all metrics share the same reference structure. The Spearman correlation coefficients ρ\rho in each panel confirm the weak relationship between geometric and physical performance.

Figure 18. Spearman correlation between geometric metrics (columns) and physical metrics (rows) across all CSP predictions in Perov-5. Each point is a single prediction, colored by method, restricted to the 3,033 test structures with no polymorph in the test split (i.e., a unique reference per prediction). The low correlation coefficients ρ\rho confirm that geometric performance is weakly correlated to physical plausibility, motivating the inclusion of physical metrics in OPAL-Bench.

Figure 19. Spearman correlation between geometric metrics (columns) and physical metrics (rows) across all CSP predictions in MP-20. Each point is a single prediction, colored by method, restricted to the 8,472 test structures with no polymorph in the test split (i.e., a unique reference per prediction). The low correlation coefficients ρ\rho confirm that geometric performance is weakly correlated to physical plausibility, motivating the inclusion of physical metrics in OPAL-Bench.

Figure 20. Spearman correlation between geometric metrics (columns) and physical metrics (rows) across all CSP predictions in MPTS-52. Each point is a single prediction, colored by method, restricted to the 7,688 test structures with no polymorph in the test split (i.e., a unique reference per prediction). The low correlation coefficients ρ\rho confirm that geometric performance is weakly correlated to physical plausibility, motivating the inclusion of physical metrics in OPAL-Bench.

Formation energy and relaxed formation energy show positive but mostly weak correlations with all four geometric metrics (ρ∈[0.07,0.44]\rho\in[0.07, 0.44] on Perov-5, [0.06,0.42][0.06, 0.42] on MP-20, and [0.08,0.25][0.08, 0.25] on MPTS-52), indicating that lower geometric error is weakly predictive of better energetic quality. Among the geometric metrics, Sinkhorn distance shows the highest correlation with the energy metrics on all three datasets, reaching 0.44 with formation energy on Perov-5, 0.42 with relaxed formation energy on MP-20, and 0.25 with relaxed formation energy on MPTS-52, although its margin over the other geometric metrics is small on Perov-5 and MPTS-52. We hypothesize that this reflects its transport formulation, which weighs the displacement of all atoms jointly rather than relying on nearest-neighbor distances, as Chamfer does, or on the largest deviation, as Hausdorff does.

On MP-20 and MPTS-52, density and XRD show correlations near zero (ρ∈[−0.02,0.12]\rho\in[-0.02, 0.12] and [−0.08,0.05][-0.08, 0.05], respectively), meaning geometric accuracy provides essentially no information about unit cell volume quality or diffraction pattern similarity. Correlations are consistently highest on Perov-5, where all twenty pairs are positive (ρ=0.21\rho= 0.21 for density and 0.33 for XRD), and weaken as structural complexity grows from Perov-5 to MP-20 and MPTS-52. We hypothesize that in simple structures an accurate prediction tends to be both geometrically and physically close to the reference, whereas in more complex structures this coupling is diluted. SOAP fingerprint distance shows weak negative correlations with Chamfer and Hausdorff distances on MPTS-52 (ρ=−0.25\rho= -0.25 and −0.12-0.12, respectively), but this sign is not universal, as ρ\rho(SOAP, Chamfer) is −0.11-0.11 on MP-20 and +0.25+0.25 on Perov-5. We hypothesize that this sign tracks how widely density varies within each dataset, since Chamfer distance shrinks as atoms pack more closely, whereas denser packing places more neighbors within the fixed SOAP cutoff radius and inflates the descriptor distance. These results substantiate the central finding that geometric and physical quality are weakly correlated.

Reference divergence, where the geometric and physical metrics of a prediction select different polymorphs, can only occur for compositions with more than one known polymorph. Within that subset it is frequent, affecting 30 to 44% of predictions on Perov-5, 52 to 55% on MP-20, and 48 to 54% on MPTS-52, but since few compositions have more than one polymorph in these datasets (Table 6), it affects only 4.9%, 1.5%, and 1.3% of all predictions, respectively. Including or excluding these multi-polymorph predictions barely changes the correlations, with shifts of at most 0.006 on MP-20 and 0.005 on MPTS-52 across all twenty pairs. On Perov-5, including them raises the correlations slightly (by 0.036 on average and at most 0.051), so reference divergence, where it occurs, does not inflate the apparent decorrelation. Carbon-24 is the exception, since every composition in its test split has multiple known polymorphs, which leaves no single-polymorph subset for this analysis and makes reference divergence nearly universal (91 to 100% of predictions). Its values in Tables 1 and 2 should therefore be read with caution, and we analyze alternative reference-selection rules in Appendix D.4.

OPAL-Bench ablations

Tables 11, 12, and 13 report scores under three ablated protocol variants. Since physical metrics are SE(3)\mathrm{SE}(3)-invariant, they are unaffected by alignment and reported only in Table 11.

Removing polymorph handling (Table 11) has a modest effect on Perov-5, MP-20, and MPTS-52, consistent with the low polymorph counts per composition in those datasets. The effect is most pronounced on Carbon-24, where the large number of known polymorphs per formula makes the choice of reference consequential. Methods that recover a valid but non-canonical carbon polymorph face inflated distances under the single-reference protocol even when their prediction is physically correct.

Removing SE(3)\mathrm{SE}(3) alignment (Table 12) inflates all geometric distances substantially across datasets and methods. On Perov-5, Chamfer distances increase by roughly 4×4\times relative to the full protocol and method rankings become less discriminative as rotational and permutation ambiguities dominate the signal. This indicates that alignment is necessary to recover true structural divergence.

Removing both components simultaneously (Table 13) combines these effects, producing the most inflated and least discriminative scores. Together, these results indicate that alignment and polymorph handling are complementary components of a fair evaluation, with alignment necessary on every dataset and polymorph handling becoming critical as the number of known polymorphs per composition grows, as on Carbon-24.

ModelChamf.↓\downarrowHaus.↓\downarrowSink.↓\downarrowRMSE↓\downarrowDataset
FlowLLM<u>5.040</u><u>2.786</u>3.8022.683PEROV-5
CDVAE5.0802.8313.7962.667PEROV-5
MatterGen5.1112.8373.8532.716PEROV-5
CrystalFlow5.0702.8123.8192.696PEROV-5
DiffCSP5.0652.7993.7932.698PEROV-5
FlowMM5.0432.8033.7822.692PEROV-5
OPAL-CSP4.9832.782<u>3.784</u><u>2.676</u>PEROV-5
FlowLLM2.8952.95026.0172.222CARBON-24
CDVAE<u>2.825</u>2.91627.7382.596CARBON-24
MatterGen<u>2.860</u>2.952<u>24.379</u><u>2.243</u>CARBON-24
CrystalFlow2.8893.18923.0692.399CARBON-24
DiffCSP3.4743.66529.5022.819CARBON-24
FlowMM3.0112.90725.5132.293CARBON-24
OPAL-CSP3.1353.25425.0082.644CARBON-24
FlowLLM6.2174.73213.6583.936MP-20
CDVAE5.8524.31512.4383.608MP-20
MatterGen6.3654.78814.0904.001MP-20
CrystalFlow5.8384.25012.3743.576MP-20
DiffCSP5.8494.27812.4383.585MP-20
FlowMM<u>5.751</u><u>4.184</u><u>12.143</u><u>3.503</u>MP-20
OPAL-CSP5.3424.03811.2413.375MP-20
FlowLLM6.6426.30333.3065.005MPTS-52
CDVAE5.9135.43830.0844.248MPTS-52
MatterGen6.2825.92633.2004.635MPTS-52
CrystalFlow5.7175.18228.7374.037MPTS-52
DiffCSP<u>5.776</u><u>5.303</u><u>29.258</u>4.141MPTS-52
FlowMM5.7855.255<u>29.287</u><u>4.084</u>MPTS-52
OPAL-CSP5.8745.40029.9114.221MPTS-52

Table 13. Ablation without polymorph handling or SE(3)SE(3) alignment. Geometric distances are reported for the protocol variant that removes both polymorph-aware reference selection and SE(3)SE(3) alignment. Each prediction is evaluated against a single unaligned reference structure. Values are median scores over test compositions; lower is better for all metrics. Bold marks the best value and <u>underline</u> marks the second-best value for each dataset and metric.

PEROV-5
ModelChamf.↓\downarrowHaus.↓\downarrowSink.↓\downarrowRMSE↓\downarrowρ\rho (g/cm3^{3})↓\downarrowSOAP↓\downarrowXRD↓\downarrow
FlowLLM<u>1.252</u><u>0.641</u>1.041<u>0.629</u>0.4718.6102.291
CDVAE2.3251.2881.6811.1860.44711.0653.260
MatterGen1.2730.6511.0290.638<u>0.298</u><u>7.406</u>1.873
CrystalFlow1.3930.7411.1190.7011.17812.4403.178
DiffCSP1.2620.650<u>0.997</u>0.6350.2807.4451.927
FlowMM1.2650.6511.0240.6350.3817.7102.037
OPAL-CSP1.2450.6360.9870.6250.3227.386<u>1.922</u>
CARBON-24
ModelChamf.↓\downarrowHaus.↓\downarrowSink.↓\downarrowRMSE↓\downarrowρ\rho (g/cm3^{3})↓\downarrowSOAP↓\downarrowXRD↓\downarrow
FlowLLM<u>2.035</u>1.84317.3221.2410.35996.8515.303
CDVAE2.1311.88315.9101.2680.18050.4545.241
MatterGen2.1422.00417.4981.2750.27679.9574.362
CrystalFlow2.1071.94017.565<u>1.223</u>0.730181.2515.923
DiffCSP2.0022.08518.6531.2910.12328.349<u>3.857</u>
FlowMM2.0551.657<u>16.044</u>1.209<u>0.134</u><u>40.201</u>3.241
OPAL-CSP2.188<u>1.785</u>17.1061.2680.604156.0247.113
MP-20
ModelChamf.↓\downarrowHaus.↓\downarrowSink.↓\downarrowRMSE↓\downarrowρ\rho (g/cm3^{3})↓\downarrowSOAP↓\downarrowXRD↓\downarrow
FlowLLM4.0512.9908.1962.3381.14411.1693.098
CDVAE3.9982.9708.1802.297<u>0.156</u>3.8181.613
MatterGen4.1303.0358.3582.372<u>0.223</u><u>3.604</u>1.628
CrystalFlow3.9292.8967.8662.2451.1908.2452.408
DiffCSP3.9342.8947.9112.2420.1292.2190.949
FlowMM<u>3.878</u><u>2.866</u><u>7.801</u><u>2.200</u>0.205<u>2.434</u><u>1.093</u>
OPAL-CSP3.7202.6877.0042.0970.2152.5911.144
MPTS-52
ModelChamf.↓\downarrowHaus.↓\downarrowSink.↓\downarrowRMSE↓\downarrowρ\rho (g/cm3^{3})↓\downarrowSOAP↓\downarrowXRD↓\downarrow
FlowLLM4.3573.85721.9642.7822.61929.7024.160
CDVAE4.4453.83821.6862.7450.4355.7282.089
MatterGen4.5253.93722.3472.8080.2644.1221.496
CrystalFlow4.2833.66220.6912.6070.2864.1731.423
DiffCSP4.3263.71420.8492.645<u>0.202</u><u>3.964</u><u>1.466</u>
FlowMM<u>4.299</u><u>3.691</u><u>20.721</u><u>2.635</u>0.2924.8811.652
OPAL-CSP4.3303.70620.9752.6470.1843.4861.341

Table 11. Ablation without polymorph handling. We report geometric and physical distances for the protocol variant in which each prediction is evaluated against a single reference structure rather than the closest known polymorph. The SE(3) alignment step is retained for geometric metrics, while physical metrics are computed directly from the predicted and reference structures. Values are median scores over test compositions; lower is better for all metrics. Bold marks the best value and <u>underline</u> marks the second-best value for each dataset and metric.

ModelChamf.↓\downarrowHaus.↓\downarrowSink.↓\downarrowRMSE↓\downarrowDataset
FlowLLM<u>5.004</u><u>2.768</u>3.7852.665PEROV-5
CDVAE5.0452.811<u>3.774</u><u>2.650</u>PEROV-5
MatterGen5.0662.8143.8252.697PEROV-5
CrystalFlow5.0372.7883.7982.677PEROV-5
DiffCSP5.0282.781<u>3.773</u>2.676PEROV-5
FlowMM5.0132.782<u>3.772</u>2.673PEROV-5
OPAL-CSP4.9442.7583.7522.648PEROV-5
FlowLLM<u>1.940</u><u>1.684</u>16.2981.336CARBON-24
CDVAE2.0901.846<u>15.893</u>1.389CARBON-24
MatterGen2.0632.02616.3211.412CARBON-24
CrystalFlow2.0101.89716.5371.440CARBON-24
DiffCSP2.0171.90417.7901.340CARBON-24
FlowMM2.0091.69416.797<u>1.273</u>CARBON-24
OPAL-CSP1.8861.58914.6931.245CARBON-24
FlowLLM6.1964.70813.5903.920MP-20
CDVAE5.8294.29512.3843.594MP-20
MatterGen6.3384.76614.0093.982MP-20
CrystalFlow5.8194.23312.3113.562MP-20
DiffCSP5.8274.25612.3463.564MP-20
FlowMM<u>5.730</u><u>4.157</u><u>12.059</u><u>3.486</u>MP-20
OPAL-CSP5.3134.00611.1253.356MP-20
FlowLLM6.6306.29033.1634.992MPTS-52
CDVAE5.9015.41930.0024.233MPTS-52
MatterGen6.2655.90533.1224.612MPTS-52
CrystalFlow5.7005.15528.6524.023MPTS-52
DiffCSP<u>5.759</u><u>5.277</u><u>29.126</u>4.125MPTS-52
FlowMM5.7745.24529.173<u>4.073</u>MPTS-52
OPAL-CSP5.8575.38129.7764.205MPTS-52

Table 12. Ablation without SE(3)SE(3) alignment. Geometric distances are reported for the protocol variant that removes the SE(3)SE(3) alignment step while retaining polymorph handling. For each prediction, the closest reference polymorph is still selected, but distances are computed without aligning the predicted and reference structures. Values are median scores over test compositions; lower is better for all metrics. Bold marks the best value and <u>underline</u> marks the second-best value for each dataset and metric.

Sensitivity of the physical metrics to descriptor settings

The SOAP and XRD metrics rely on fixed descriptor settings (Appendix § 8.§ 2.2). These are not matching tolerances, but they are design choices, so we test whether our conclusions depend on them. We use the MPTS-52 subset of compositions with a single known polymorph (7,688 compositions, 53,816 predictions across the seven CSP methods), so that all metrics share the same reference structure. We sweep SOAP over 47 configurations and XRD over 38, varying the parameters over the ranges in Table 14, and recompute each metric on all predictions under every configuration.

MetricParameterRange sweptDefaultWorst-case ρ\rho
SOAPrcutr_{\mathrm{cut}} (Å)3.5–8.04.00.96
SOAPnmax⁡n_{\max}4–13120.95
SOAPlmax⁡l_{\max}3–1291.00
SOAPσ\sigma (Å)0.2–1.00.50.89
XRD2θmin⁡2\theta_{\min} (°)3–1050.94
XRD2θmax⁡2\theta_{\max} (°)60–120900.96

Table 14. Sensitivity of per-structure rankings to descriptor settings on the single-polymorph subset of MPTS-52. Each row sweeps one parameter. The last column reports the worst-case Spearman correlation between the per-structure values under the swept setting and under the default setting.

For each configuration, we compute the Spearman correlation between its per-structure values and those of the default setting, and Table 14 reports the worst case over each sweep. Per-structure rankings stay close to the default for every parameter, with worst cases of 0.89 or higher for SOAP and 0.94 or higher for XRD. The largest change comes from the SOAP Gaussian width σ\sigma (0.89).

The weak relationship between geometric and physical performance persists across these settings. Across all 85 configurations, the mean absolute Spearman correlation between each physical metric and each geometric metric is 0.14 for SOAP (range 0.08 to 0.28) and 0.05 for XRD (range 0.00 to 0.12). The most extreme value is 0.28, between Chamfer and SOAP.

Method rankings are also largely preserved. The ordering of the seven methods under SOAP is unchanged in 41 of the 47 configurations, and under XRD in 34 of the 38. The XRD deviations are a single recurring transposition of DiffCSP and MatterGen, whose median scores differ by 0.03 with overlapping confidence intervals, so all 38 XRD configurations yield one of only two orderings. Of the 21 method pairs, 19 show no reversal in any configuration.

Absolute values do shift, with the median relative change in the metric reaching roughly 3,860% for σ\sigma and 334% for rcutr_{\mathrm{cut}}. Since all methods are scored under identical settings, our conclusions require rank stability rather than magnitude stability, and absolute SOAP and XRD values should only be compared within one configuration.

Effect of the number of known polymorphs

Each metric reports the minimum distance from a prediction to the known polymorphs of its composition, so a larger reference set raises the chance that some known structure lies close to the prediction, and the reported distance decreases as the number of polymorphs grows. We quantify this effect for the RMSE of OPAL-CSP on Carbon-24, the dataset with the most polymorphs per composition (Table 6). We hold each prediction fixed, recompute its minimum distance against random subsets of size kk drawn from its KK known polymorphs, and take the median across compositions at each kk. The median minimum RMSE falls from 1.28 Å at k=1k = 1 to 0.89 Å at k=Kk = K (Figure 21), so the reference count alone drives a substantial decrease.

Median RMSE (Å) vs. polymorph count - Carbon-24 - OPAL-CSP

Figure 21. Median minimum RMSE of OPAL-CSP on Carbon-24 as a function of the fraction of known polymorphs used as references. For each composition, we recompute the minimum RMSE against random subsets of kk of its KK known polymorphs. The line shows the median across compositions and the shaded band the 25th to 75th percentiles.

To assess how this effect changes the comparison between methods, we recompute all metrics under two alternative rules, which use as the single reference for every metric the polymorph closest to the prediction in formation energy or in RMSE, and compare the resulting rankings against those of the original protocol (Table 15 and Table 16). On Perov-5, MP-20, and MPTS-52 the rankings are largely preserved, with occasional swaps between methods with close scores that do not affect our conclusions. Carbon-24 behaves differently, since every metric changes its ranking under at least one alternative rule. For XRD, MatterGen rises from fifth under the original protocol to first under the energy rule, and FlowMM rises from fourth to first under the RMSE rule, while DiffCSP falls from first to second and third, respectively. This is consistent with the overlapping bootstrap confidence intervals on Carbon-24 (Appendix § 10.2), which show no statistically significant advantage between methods.

ModelChamf. ↓Haus. ↓Sink. ↓RMSE ↓ρ\rho (g/cm3^3) ↓SOAP ↓XRD ↓FE (eV/at.) ↓R. FE (eV/at.) ↓
PEROV-5
FlowLLM1.2340.6331.0100.6210.4488.072.1870.1751.532 · 10−1^{-1}
CDVAE2.3271.2911.6761.1880.43910.953.2213.0081.971 · 10−1^{-1}
CrystalFlow1.3800.7331.1070.6961.16112.283.1470.4221.670 · 10−1^{-1}
MatterGen1.2550.6431.0040.6290.2636.561.6880.1011.426 · 10−1^{-1}
DiffCSP1.2440.6380.9630.6250.2426.631.6970.1161.347 · 10−1^{-1}
FlowMM1.2490.6411.0000.6260.3597.111.9010.1091.518 · 10−1^{-1}
OPAL-CSP1.2230.6250.9390.6130.2906.291.7350.0721.455 · 10−1^{-1}
CARBON-24
FlowLLM1.9311.62316.2831.0940.33892.304.8003.7911.331 · 10−1^{-1}
CDVAE2.1021.83017.5071.2420.17152.554.6710.2439.816 · 10−2^{-2}
CrystalFlow2.1161.87615.2741.2410.515140.317.5780.4482.085 · 10−1^{-1}
MatterGen2.0171.84315.8191.1770.24264.003.9700.1821.581 · 10−1^{-1}
DiffCSP2.0392.20017.5641.2710.20655.774.2030.0068.335 · 10−2^{-2}
FlowMM2.0281.85416.4911.2110.26668.424.3350.0691.647 · 10−1^{-1}
OPAL-CSP2.0261.78916.4451.2880.19060.564.7510.0023.141 · 10−3^{-3}
MP-20
FlowLLM4.0522.9908.1962.3371.14611.173.0992.4811.515 · 10−1^{-1}
CDVAE4.0002.9738.1862.2970.1563.821.6120.2165.073 · 10−2^{-2}
CrystalFlow3.9312.9007.8832.2471.1898.252.4100.2988.760 · 10−5^{-5}
MatterGen4.1323.0338.3392.3730.2233.591.6260.0664.120 · 10−2^{-2}
DiffCSP3.9332.8947.9102.2410.1292.200.9490.0198.197 · 10−3^{-3}
FlowMM3.8752.8657.7932.1990.2052.431.0920.0321.551 · 10−4^{-4}
OPAL-CSP3.7152.6867.0042.0960.2152.581.1420.0281.816 · 10−4^{-4}
MPTS-52
FlowLLM4.3583.85821.9792.7832.61929.734.1623264.3764.115 · 100^{0}
CDVAE4.4453.83921.7022.7450.4355.732.0900.7851.979 · 10−1^{-1}
CrystalFlow4.2823.66120.6702.6050.2854.171.4220.2749.189 · 10−2^{-2}
MatterGen4.5253.93822.3422.8090.2654.121.4950.1501.036 · 10−1^{-1}
DiffCSP4.3293.71520.8602.6460.2023.961.4660.2341.049 · 10−1^{-1}
FlowMM4.2973.69120.7232.6340.2924.871.6510.6191.343 · 10−1^{-1}
OPAL-CSP4.3303.70620.9712.6460.1843.481.3400.1137.749 · 10−2^{-2}

Table 15. CSP results under the formation-energy reference rule. Each prediction is evaluated against a single reference, the known polymorph closest to it in formation energy, which is used for every metric. Formation energy distances therefore coincide with those of the original protocol (Table 1). Values are median scores over test compositions; lower is better for all metrics. Bold marks the best method per dataset and metric; _underline_ marks the second best.

ModelChamf. ↓Haus. ↓Sink. ↓RMSE ↓ρ\rho (g/cm³) ↓SOAP ↓XRD ↓FE (eV/at.) ↓R. FE (eV/at.) ↓
PEROV-5
FlowLLM1.2180.6230.9710.6100.4598.302.2250.1971.527 · 10−110^{-1}
CDVAE2.2951.2721.6551.1720.43810.953.2423.0381.931 · 10−110^{-1}
CrystalFlow1.3280.6981.0580.6701.17812.423.1740.4501.659 · 10−110^{-1}
MatterGen1.2310.6310.9550.6190.2767.011.7730.1281.442 · 10−110^{-1}
DiffCSP1.2290.6250.9220.6160.2576.981.8020.1411.368 · 10−110^{-1}
FlowMM1.2260.6310.9560.6160.3717.441.9700.1441.523 · 10−110^{-1}
OPAL-CSP1.2110.6170.9090.6070.3036.72<u>1.795</u>0.0901.462 · 10−110^{-1}
CARBON-24
FlowLLM1.6991.37313.7680.9310.370101.355.6883.8983.232 · 10−110^{-1}
CDVAE<u>1.615</u>1.47913.8630.9470.37593.424.3210.388<u>1.151 · $10^{-1}</u>
CrystalFlow1.6641.44512.9920.9840.569147.645.3100.5072.014 · 10−110^{-1}
MatterGen1.8761.46313.7621.0290.31686.455.3090.4022.686 · 10−110^{-1}
DiffCSP1.7311.64713.7641.0040.19954.774.5080.1117.047 · 10−210^{-2}
FlowMM1.499<u>1.401</u>10.4010.833<u>0.224</u><u>55.27</u>4.1340.2251.999 · 10−110^{-1}
OPAL-CSP<u>1.615</u>1.481<u>12.563</u><u>0.889</u>0.324103.805.769<u>0.194</u>2.186 · 10−110^{-1}
MP-20
FlowLLM4.0392.9818.1572.3291.14611.163.0992.4811.516 · 10−110^{-1}
CDVAE3.9882.9608.1292.289<u>0.156</u>3.821.6120.2165.090 · 10−210^{-2}
CrystalFlow3.9112.8897.8212.2361.1908.252.4080.2988.665 · 10−510^{-5}
MatterGen4.1203.0238.3212.3660.2233.601.6280.0664.135 · 10−210^{-2}
DiffCSP3.9202.8867.8692.2310.1292.210.9490.0198.327 · 10−310^{-3}
FlowMM<u>3.868</u><u>2.854</u><u>7.760</u><u>2.194</u>0.205<u>2.43</u><u>1.094</u>0.032<u>1.515 · $10^{-4}</u>
OPAL-CSP3.7002.6686.9552.0840.2152.591.143<u>0.028</u>1.850 · 10−410^{-4}
MPTS-52
FlowLLM4.3543.85321.9492.7792.61929.724.1623297.9534.056 · 10010^{0}
CDVAE4.4393.83321.6712.7410.4365.722.0890.7861.980 · 10−110^{-1}
CrystalFlow4.2763.65520.6562.6000.2854.171.4220.274<u>9.206 · $10^{-2}</u>
MatterGen4.5233.92822.3112.8030.2654.121.496<u>0.151</u>1.038 · 10−110^{-1}
DiffCSP4.3233.71020.8252.641<u>0.202</u><u>3.96</u>1.4660.2341.049 · 10−110^{-1}
FlowMM<u>4.294</u><u>3.684</u><u>20.699</u><u>2.630</u>0.2924.871.6510.6201.343 · 10−110^{-1}
OPAL-CSP4.3253.70020.9432.6420.1843.481.3410.1137.764 · 10−210^{-2}

Table 16. CSP results under the RMSE reference rule. Each prediction is evaluated against a single reference, the known polymorph closest to it in RMSE, which is used for every metric. RMSE values therefore coincide with those of the original protocol (Table 1). Values are median scores over test compositions; lower is better for all metrics. Bold marks the best method per dataset and metric; <u>underline</u> marks the second best.

These results do not change our conclusions on Perov-5, MP-20, and MPTS-52, where few compositions have more than one known polymorph, but they show that Carbon-24 scores depend on the size of its reference sets and on the rule used to select among them. Carbon-24 values in Table 1 and Table 2 should therefore be read with caution.

Sensitivity of the energy metrics to the choice of MLIP

We recompute formation energy and relaxed formation energy using each of the three potentials individually for all seven CSP methods on Perov-5 and Carbon-24. Since every method is scored under the same ensemble, a shift shared by all methods cancels in the ranking. For unrelaxed formation energy on Perov-5, the variability between potentials is 2.5 to 6.3%, and the ranking under every potential is identical to the ensemble ranking. On Carbon-24 the magnitudes vary more, but only MACE-MP reorders the methods, transposing the top two, whose scores differ by about 1%.

Relaxed formation energy varies more across potentials, with relative differences of 68.9 to 91.3% on Perov-5. This variation comes from the definition of the metric rather than from unstable rankings. Since the predicted and reference structures are relaxed independently, both move toward low-energy configurations and the resulting difference is small for every method, between 0.126 and 0.178 eV/atom on Perov-5 (Table 1), so even modest absolute changes translate into large relative differences. These changes also affect all methods in a similar way. For every method, Orb gives the lowest values and UMA the highest, and moving from Orb to MACE-MP or UMA raises the scores of all methods by a similar amount, 0.086 and 0.119 eV/atom on average. As a result, UMA reproduces the ensemble ranking exactly, and Orb and MACE-MP only swap methods whose scores differ by at most 0.010 eV/atom. On Carbon-24, six of the seven methods reach values between 10−410^{-4} and 10−210^{-2} eV/atom. At this scale the differences between methods are mostly numerical noise, so the ranking changes across potentials, consistent with the overlapping confidence intervals in Appendix § 10.2.

We also compare the ensemble with DFT on LiNbO3_3, a representative Perov-5 composition. We compute the energy of the ground truth and of each prediction with Quantum ESPRESSO5 under PBE, using cutoff and kk-point settings converged for this structure, and obtain the formation energy distance as in Appendix § 8.§ 2.2. DFT and the ensemble order the seven methods identically, with a mean absolute error of 0.075 eV/atom between them. This error is smaller than the differences between most methods but larger than the gap between the closest ones, so the ensemble follows DFT at a coarse level, although this comparison does not validate the fine-grained ordering.

For relaxed formation energy, we relax the ground truth and the seven predictions with DFT. Two relaxations of the ground truth from different perturbed starting points reach the same minimum, which coincides with the one reached by the prediction of MatterGen (relaxed distance below 0.001 eV/atom). CrystalFlow and DiffCSP relax to higher energies (0.024 and 0.049 eV/atom), while the predictions of OPAL-CSP, CDVAE, FlowLLM, and FlowMM reach a common minimum 0.036 to 0.041 eV/atom below the ground truth. The ensemble agrees with DFT in placing MatterGen first and CrystalFlow among the closest methods, but it places OPAL-CSP second, whereas DFT relaxes it to the lower minimum, and the Spearman correlation between the two rankings is 0.21. This suggests that relaxed formation energy can be more sensitive than its unrelaxed counterpart to the relaxation protocol, since MLIP and DFT relaxations may end in different minima. Since the metric is an absolute difference, it also assigns a nonzero distance to predictions that relax to a lower energy than the reference. This is a single example, however, and assessing the actual extent of these effects would require repeating the analysis on more structures.

A related consideration is that each potential relaxes the structures independently, so the ensemble value for relaxed formation energy averages energies evaluated at up to three different relaxed geometries. We use the mean over potentials, although the median could be a more robust alternative when one potential deviates from the other two.

Agreement between metric rankings

Removing matching tolerances eliminates one source of ranking instability, but a protocol with several metrics can still rank methods differently depending on the metric. To assess how structured this variation is, we compute for each dataset the Spearman correlation between the rankings of the seven CSP methods under every pair of metrics, and average it within three groups, namely the six pairs of geometric metrics, the ten pairs of physical metrics, and the twenty pairs that combine one geometric and one physical metric. As a reference for the previous convention, we also compute the correlation between the rankings under Match Rate and RMSE (Table 4).

DatasetGeometricPhysicalGeometric vs. physicalMR vs. RMSE
Perov-50.8460.8320.608−0.180
Carbon-240.7860.5430.059−0.955
MP-200.9760.6040.5210.649
MPTS-520.9640.9140.2430.714

Table 17. Agreement between the method rankings of different metrics. Mean Spearman correlation between the rankings of the seven CSP methods under pairs of metrics, averaged within each group of pairs. The last column reports the correlation between the rankings under the conventional Match Rate and RMSE.

Rankings agree strongly within the geometric family (0.786 to 0.976) and within the physical family (0.543 to 0.914). In contrast, Match Rate and RMSE range from reasonable agreement on MP-20 and MPTS-52 (0.649 and 0.714) to low or negative agreement on Perov-5 and Carbon-24 (−0.180 and −0.955), even though both aim to capture geometric similarity. Agreement drops mainly when one geometric and one physical metric are compared (0.059 to 0.608), which reflects the separation between geometric and physical performance that the benchmark is designed to expose. The variation that remains is therefore organized by metric family rather than arbitrary. Carbon-24 shows the lowest agreement within each family, consistent with its structural diversity and the closely spaced scores of its methods (Appendix § 10.2).

Evaluation beyond known polymorphs

OPAL-Bench evaluates each prediction against the polymorphs available in the reference datasets, so it measures how well a method recovers known structures. A prediction that corresponds to a valid polymorph absent from these datasets therefore receives a large distance, even though it may be physically correct, and the benchmark cannot distinguish it from an incorrect prediction. As crystal databases incorporate newly reported polymorphs, future versions of OPAL-Bench can extend the reference sets accordingly, which reduces this effect over time.

OPAL-Bench also generates a single prediction per composition, which does not capture the diversity of the structures that a method can produce. With several predictions per composition, future versions could measure diversity as the number of distinct structures after relaxation and deduplication, and coverage as the fraction of known polymorphs that these predictions recover, stratified by the number of polymorphs per composition. Finally, for predictions that do not match any known polymorph, the MLIP ensemble that we use for the energy metrics can still compute their formation energy without a known reference structure, which provides a first estimate of their physical plausibility.

NeurIPS Paper Checklist

Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction state three contributions, all substantiated in the paper: (1) OPAL-Bench, a threshold-free polymorph-aware benchmark with SE(3) alignment unifying geometric and physical metrics (Section 3); (2) a systematic re-evaluation of 7 CSP and 2 SICSP methods showing geometric performance does not reliably imply physical plausibility (Section 5.2, Tables 1–2, Appendix D.1); (3) OPAL-CSP, a flow-matching reference baseline (Section 4, Appendix B.3). Limitations and scope are discussed in Section 5.4.

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper includes a dedicated limitations section discussing four key aspects of OPAL-Bench: (1) the polymorph-aware protocol penalizes predictions matching undocumented polymorphs; (2) systematic biases in the fixed relaxation and property-evaluation pipeline propagate into physical metric scores; (3) the single prediction per-composition design does not capture generative diversity; (4) evaluating pretrained models requires checking for data leakage between their training data and the test splits.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [N/A]

Justification: The paper does not include theoretical results; it focuses on the design and empirical evaluation of a benchmark for crystal structure prediction.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides all information necessary to reproduce the main experimental results. The benchmark methodology is fully described, including the formal problem formulation, the complete set of geometric and physical metrics, the polymorph-aware evaluation protocol, and the species-aware SE(3)SE(3) alignment procedure with its mathematical formulation (§ 3.1–§ 3.5). Dataset splits are reported in § 5.1, and model training details (optimizer, learning rate, scheduler, batch size, number of epochs and GPUs) and inference settings for OPAL-CSP in Appendix § 9.§ 3.2, with architecture details in Appendix § 9.§ 3.1 and baseline details in Appendix § 9.4. The re-evaluation of the seven CSP and two SICSP methods follows the original training procedures described in their respective papers, and any protocol-specific preprocessing steps, such as supercell construction for SICSP geometric evaluation, are explicitly described. Together, these disclosures are sufficient to reproduce the benchmark results and the central finding that geometric performance does not reliably imply physical plausibility.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed

instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We release the full OPAL-Bench evaluation framework, including the polymorph-aware protocol, species-aware SE(3)SE(3) alignment, and all geometric and physical metric implementations, at https://github.com/BCV-Uniandes/OPAL. Training and data preprocessing scripts for OPAL-CSP are available at the same link.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: Training and architectural details of the baseline model, including hyperparameters and optimizer settings, are fully specified in the appendix. Data splits follow the official partitions of the established benchmark datasets used, as described in the main paper, and all metric definitions and computation details are presented in both the main paper and the appendix.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports median metric scores across all test compositions for each method, where the median is computed over the distribution of per-prediction metric values. Bootstrap confidence intervals are derived by resampling this distribution and computing the median over each resample, yielding a robust estimate of uncertainty around the reported scores. These confidence intervals are reported alongside all main benchmark results, providing appropriate statistical grounding for the comparisons made across methods and datasets.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper reports compute resources in Section B.5: all training was performed on NVIDIA Quadro RTX 8000 GPUs (48 GB memory), using between 1 and 7 GPUs per run depending on the model and dataset, and metric computation ran on 2× Intel Xeon Gold 6230 CPUs (40 physical cores, 2.10 GHz). Section B.3.2 specifies the OPAL-CSP training schedule (1000 epochs, total batch size 128 across GPUs, Adam optimizer with reduce-on-plateau scheduling), and Appendix § 9.4 documents that baseline methods were re-trained following their original procedures. Appendix § 9.5 reports the training time of OPAL-CSP on each dataset; we do not report the compute used during preliminary development.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: The research conforms with the NeurIPS Code of Ethics in all applicable respects. The work is purely computational and involves no human subjects, personally identifiable data, or sensitive information. The datasets used are publicly available and are used in accordance with their respective licenses. The research poses no foreseeable risks related to safety, security, discrimination, or surveillance, as it focuses on the development of an evaluation benchmark for crystal structure prediction. Computational resources are transparently reported, and all research artifacts are released publicly with sufficient documentation to enable reproducibility and external scrutiny.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses societal impacts in § 5.4. On the positive side, OPAL-Bench supports more rigorous evaluation of crystal structure prediction methods, which may benefit materials discovery in areas such as energy storage, catalysis, and microelectronics. On the negative side, the paper acknowledges that more capable generative models could potentially accelerate the design of undesirable materials. We assess this risk as limited because computationally predicted structures still require experimental synthesis and characterization before physical realization.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: The models and datasets released in this paper pose no significant risk of misuse. OPAL-CSP is a domain-specific crystal structure prediction model whose outputs require rigorous experimental validation before physical realization, and the benchmark datasets are established publicly available resources that contain no sensitive or personally identifiable information.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets and methods used in this paper are properly credited and their licenses are respected. The four benchmark datasets are publicly available and appropriately cited: Carbon-24 is released under a CC BY 4.0 license via Materials Cloud, and MP-20 and MPTS-52 are sourced from the Materials Project, which distributes its data under a CC BY 4.0 license; Perov-5 is a publicly available dataset cited from its original publication. The code of the six previously published CSP methods and the two SICSP methods re-evaluated under OPAL-Bench is publicly available through open-source repositories accompanying their respective published papers, and we use them in accordance with their stated licenses. We also use the MLIP ensemble of LeMat-GenBench, the LeMat-Bulk dataset, CSPML, Dans_Diffraction, and Quantum ESPRESSO, all cited or linked in the paper and used under their respective licenses.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper introduces two new assets: OPAL-Bench, a benchmark for crystal structure prediction evaluation, and OPAL-CSP, a reference baseline model. We release OPAL-Bench at https://github.com/BCV-Uniandes/OPAL. The repository includes the full implementation of the benchmark protocol, covering the polymorph-aware evaluation procedure, the species-aware SE(3)SE(3) alignment, and the computation of all geometric and physical metrics. The implementation, training scripts, and inference code for OPAL-CSP are available at the same link. Documentation is provided in the repository itself, with additional details on the benchmark methodology in the main paper (Sections 3.1–3.5) and further architecture, training, and implementation details in the appendix, ensuring that both assets are sufficiently documented to be understood, reproduced, and extended by the research community.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: The paper does not involve crowdsourcing or research with human subjects, as it focuses entirely on computational methods for crystal structure prediction evaluation.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A]

Justification: The paper does not involve human subjects. All experiments are purely computational and operate exclusively on crystal structure datasets.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [N/A]

Justification: LLMs are not used as any important, original, or non-standard component of the core methodology. The paper focuses on crystal structure prediction evaluation and does not incorporate LLMs in any aspect of its methods or experiments.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [1]Myunghyun Paik Suh, Hye Jeong Park, Thazhe Kootteri Prasad, and Dae-Woon Lim. Hydrogen storage in metal–organic frameworks. Chemical reviews, 112(2):782–835, 2012.DOI
  2. [2]Arun Mannodi-Kanakkithodi, Anand Chandrasekaran, Chiho Kim, Tran Doan Huan, Ghanshyam Pilania, Venkatesh Botu, and Ramprasad. Scoping the polymer genome: A roadmap for rational polymer dielectrics design and beyond. Materials Today, 21(7):785–796, 2018.
  3. [3]Qiang Shi, Bing Liu, Jing Li, Xuping Wang, and Leyong Wang. Catalysis in single crystalline materials: From discrete molecules to metal-organic frameworks. Chemistry–An Asian Journal, 16(22):3544–3557, 2021.DOI
  4. [4]Qifa Zhou, Kwok Ho Lam, Hairong Zheng, Weibao Qiu, and Kirk Shung. Piezoelectric single crystal ultrasonic transducers for biomedical applications. Progress in materials science, 66:87–111, 2014.pubmed.ncbi.nlm.nih.gov/25386032
  5. [5]Dongdong Xiao and Lin Gu. Origin of functionality for functional materials at atomic scale. Nano Select, 1(2):183–199, 2020.DOI
  6. [6]Stefano Curtarolo, Gus LW Hart, Marco Buongiorno Nardelli, Natalio Mingo, Stefano Sanvito, and Ohad Levy. The high-throughput highway to computational materials design. Nature materials, 12(3):191–201, 2013.DOI
  7. [7]Artem R Oganov, Chris J Pickard, Qiang Zhu, and Richard J Needs. Structure prediction drives materials discovery. Nature Reviews Materials, 4(5):331–348, 2019.DOI
  8. [8]Artem R Oganov and Colin W Glass. Crystal structure prediction using ab initio evolutionary techniques: Principles and applications. The Journal of chemical physics, 124(24), 2006.pubmed.ncbi.nlm.nih.gov/16821993
  9. [9]Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Crystal diffusion variational autoencoder for periodic material generation. In International Conference on Learning Representations, 2022.arxiv.org/abs/2110.06197
  10. [10]Rui Jiao, Wenbing Huang, Peijia Lin, Jiaqi Han, Pin Chen, Yutong Lu, and Yang Liu. Crystal structure prediction by joint equivariant diffusion. Advances in Neural Information Processing Systems, 36:17464–17497, 2023.arxiv.org/abs/2309.04475
  11. [11]Claudio Zeni, Robert P Pinsler, Daniel Zügner, Andrew Fowler, Matthew Horton, Xiang Fu, Zilong Wang, Aliaksandra Shysheva, Jonathan Crabbé, Shoko Ueda, et al. A generative model for inorganic materials design. Nature, 639(8055):624–632, 2025.DOI
  12. [12]Xiaoshan Luo, Zhenyu Wang, Qingchang Wang, Xuechen Shao, Jian Lv, Lei Wang, Yanchao Wang, and Yanming Ma. Crystalflow: A flow-based generative model for crystalline materials. Nature Communications, 16(1):9267, 2025.DOI
  13. [13]Benjamin Kurt Miller, Ricky T. Q. Chen, Anuroop Sriram, and Brandon M Wood. Flowmm: Generating materials with riemannian flow matching. In International Conference on Machine Learning, 2024.arxiv.org/abs/2406.04713
  14. [14]Anuroop Sriram, Benjamin K Miller, Ricky T Chen, and Brandon M Wood. Flowllm: Flow matching for material generation with large language models as base distributions. Advances in Neural Information Processing Systems, 37:46025–46046, 2024.arxiv.org/abs/2410.23405
  15. [15]Aurora J Cruz-Cabeza and Joel Bernstein. Conformational polymorphism. Chemical reviews, 114(4):2170–2191, 2014.DOI
  16. [16]Maya Martirossyan, Thomas Egg, Philipp Höllmer, George Karypis, Mark Transtrum, Adrian Roitberg, Mingjie Liu, Richard Hennig, Ellad Tadmor, and Stefano Martiniani. All that structure matches does not glitter. Advances in Neural Information Processing Systems, 38, 2025.
  17. [17]Lai Wei, Sadman Sadeed Omee, Rongzhi Dong, Nihang Fu, Yuqi Song, Edirisuriya Siriwardane, Meiling Xu, Chris Wolverton, and Jianjun Hu. Cspbench: a benchmark and critical evaluation of crystal structure prediction. arXiv preprint arXiv:2407.00733, 2024.arxiv.org/abs/2407.00733
  18. [18]Lai Wei, Qin Li, Sadman Sadeed Omee, and Jianjun Hu. Towards quantitative evaluation of crystal structure prediction performance. Computational Materials Science, 235:112802, 2024.arxiv.org/abs/2307.05886
  19. [19]Rui Jiao, Wenbing Huang, Yu Liu, Deli Zhao, and Yang Liu. Space group constrained crystal generation. arXiv preprint arXiv:2402.03992, 2024.arxiv.org/abs/2402.03992
  20. [20]Rees Chang, Angela Pak, Alex Guerra, Ni Zhan, Nick Richardson, Elif Ertekin, and Ryan P Adams. Space group equivariant crystal diffusion. arXiv preprint arXiv:2505.10994, 2025.arxiv.org/abs/2505.10994
  21. [21]Scott M Woodley and Richard Catlow. Crystal structure prediction from first principles. Nature materials, 7(12):937–946, 2008.DOI
  22. [22]Ivano E Castelli, David D Landis, Kristian S Thygesen, Søren Dahl, Ib Chorkendorff, Thomas F Jaramillo, and Karsten W Jacobsen. New cubic perovskites for one-and two-photon water splitting using the computational materials repository. Energy & Environmental Science, 5(10):9034–9043, 2012.DOI
  23. [23]Chris J. Pickard. Airss data for carbon at 10 gpa and the c+n+h+o system at 1 gpa, 2020.
  24. [24]Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, and Kristin A. Persson. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL Materials, 1(1):011002, 07 2013. ISSN 2166-532X. doi: 10.1063/1.4812323. URL https://doi.org/10.1063/1.4812323.
  25. [25]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1907.05600
  26. [26]Johannes Gasteiger, Florian Becker, and Stephan Günnemann. GemNet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34:6790–6802, 2021.arxiv.org/abs/2106.08903
  27. [27]Ricky TQ Chen and Yaron Lipman. Flow matching on general geometries. arXiv preprint arXiv:2302.03660, 2023.arxiv.org/abs/2302.03660
  28. [28]Minoru Kusaba, Chang Liu, and Ryo Yoshida. Crystal structure prediction with machine learning-based element substitution. Computational Materials Science, 211:111496, 2022. doi: 10.1016/j.commatsci.2022.111496.DOI
  29. [29]Siddharth Betala, Samuel P Gleason, Ali Ramlauoi, Andy Xu, Georgia Channing, Daniel Levy, Clémentine Fourrier, Nikita Kazeev, Chaitanya K Joshi, Sékou-Oumar Kaba, et al. LeMat-genbench: A unified evaluation framework for crystal generative models. arXiv preprint arXiv:2512.04562, 2025.arxiv.org/abs/2512.04562
  30. [30]Felix Musil, Andrea Grisafi, Albert P Bartók, Christoph Ortner, Gábor Csányi, and Michele Ceriotti. Physics-inspired structural representations for molecules and materials. Chemical reviews, 121(16):9759–9815, 2021.DOI
  31. [31]Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in neural information processing systems, 33:1970–1981, 2020.arxiv.org/abs/2006.10503
  32. [32]Yi-Lun Liao, Brandon Wood, Abhishek Das*, and Tess Smidt*. EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=mCBKZmrzD.arxiv.org/abs/2306.12059
  33. [33]Zhendong Cao, Xiaoshan Luo, Jian Lv, and Lei Wang. Space group informed transformer for crystalline materials generation. Science Bulletin, 2025.DOI
  34. [34]Sherry Yang, KwangHwan Cho, Amil Merchant, Pieter Abbeel, Dale Schuurmans, Igor Mordatch, and Ekin Dogus Cubuk. Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235, 2023.arxiv.org/abs/2311.09235
  35. [35]Nate Gruver, Anuroop Sriram, Andrea Madotto, Andrew Gordon Wilson, C Lawrence Zitnick, and Zachary Ulissi. Fine-tuned language models generate stable inorganic materials as text. arXiv preprint arXiv:2402.04379, 2024.arxiv.org/abs/2402.04379
  36. [36]Jarno Laakso, Lauri Himanen, Henrietta Homm, Eiaki V Morooka, Marc Ojl Jäger, Milica Todorović, and Patrick Rinke. Updates to the describe library: New descriptors and derivatives. The Journal of Chemical Physics, 158(23), 2023.
  37. [37]Martin Siron, Inel Djafar, Ali Ramlauoui, Etienne du Fayette, Amandine Rossello, Edvin Fako, Matthew McDermott, Felix Therrien, Luis Barroso-Luque, Flaviu Cipcigan, et al. Lemat-bulk: aggregating, and de-duplicating quantum chemistry materials databases. arXiv preprint arXiv:2511.05178, 2025.arxiv.org/abs/2511.05178
  38. [38]Benjamin Rhodes, Sander Vandenhaute, Vaidotas Šimkus, James Gin, Jonathan Godwin, Tim Duignan, and Mark Neumann. Orb-v3: atomistic simulation at scale. arXiv preprint arXiv:2504.06231, 2025.arxiv.org/abs/2504.06231
  39. [39]Ilyes Batatia, Philipp Benner, Yuan Chiang, Alin M Elena, Dávid P Kovács, Janosh Riebesell, Xavier R Advincula, Mark Asta, Matthew Avaylon, William J Baldwin, et al. A foundation model for atomistic materials chemistry. The Journal of chemical physics, 163(18), 2025.
  40. [40]Brandon M Wood, Misko Dzamba, Xiang Fu, Meng Gao, Muhammed Shuaibi, Luis Barroso-Luque, Kareem Abdelmaqsoud, Vahe Gharakhanyan, John R Kitchin, Daniel S Levine, et al. Uma: A family of universal models for atoms. arXiv preprint arXiv:2506.23971, 2025.arxiv.org/abs/2506.23971
  41. [41]Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2 (1-2):83–97, 1955.

Paper details

vlnc:W_wmc4gnjv
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents