How good are universal MLIPs for alloys? A benchmark is only as trustworthy as its controls

Introduction

Universal machine-learned interatomic potentials (uMLIPs) are atomistic models trained to reproduce density-functional theory (DFT) across much of the periodic table. They predict the energies, forces, and relaxed geometries of crystals at a tiny fraction of the cost of DFT. Their speed is reshaping how candidate materials are screened and discovered. Public leaderboards most often rank them by the mean absolute error (MAE) of predicted formation energies against a reference database such as the Materials Project (MP) [1], and by convex-hull stability classification, the central metric of Matbench Discovery [2].

Rankings are good for spurring progress, but two assumptions deserve a harder look. The first is that a smaller energy MAE means a model that describes the potential-energy surface (PES) more accurately. However, a model can have a lower formation-energy error simply because its training data used the same per-element energy conventions as the reference database. That shared convention is a bookkeeping advantage, not a better PES. A low energy MAE is also an unreliable predictor of which phases a model will actually place on the convex hull [2, 3].

uMLIPs are fitted, black-box models, and we make no claim about what they represent internally. What we can test is the part of a model’s behavior that survives a change of elemental reference. At a fixed composition, any per-element shift cancels exactly, so the energy ordering a model predicts among competing structures is a property of its PES and of nothing else.

The second assumption, that the reference database is ground truth, matters more, and it does not hold. Even with a shared exchange-correlation functional, independent high-throughput DFT databases do not agree exactly with one another about formation energies [4], because they differ in pseudopotentials, settings, corrections, and elemental references. Benchmarks almost never compare a model’s error with the disagreement between two careful DFT calculations of the same quantity.

We address both benchmarking problems by changing the DFT reference, the chemistry, and the metrics. For an independent reference, we use AFLOW data [5], a high-throughput DFT database that is independent of the data the models were trained on (MPtrj [6], OMat24 [7], and Alexandria [8] [9], none of which includes AFLOW). We focus on intermetallics, a chemically rich and under-benchmarked region of materials space where DFT results are relatively accurate to the underlying physics. AFLOW offers systematically enumerated ordered-prototype libraries and precomputed elastic constants (benchmark section).

Two design choices keep the test clean. First, we compare formation energies rather than total energies and lead with properties that depend only weakly on the elemental reference: the ordering of phases on the convex hull, the curvature of the elastic tensor, the geometric drift of a structure as a model relaxes it, and the energy ordering of polymorphs at fixed composition. Polymorph ordering is exactly reference-free; hull metrics are immune to a self-consistent change of convention but not to a wrong elemental reference (reference-free section). Each metric is still scored against AFLOW, so nothing here is true by construction.

Second, we compare every energy and stability error with the DFT-vs-DFT noise floor. The question is how a model’s error compares with the disagreement between the DFT calculations themselves. On our set of intermetallics, three independent DFT databases (AFLOW [5], Alexandria [8], and the Open Quantum Materials Database (OQMD) [10, 11]) agree with one another to 6–8 meV/atom on typical structures. The mean disagreement increases to 16–30 (noise-floor section) because of heavy tails of large disagreements. These floors, not zero, are the realistic target for a model.

We test each suspicious result with a control, a second version of the measurement that changes one suspected

Results

cause (the geometry, the elemental reference, or the Hamiltonian) and holds everything else fixed. Three apparent model failures shrank or vanished under these controls: shear moduli that tracked AFLOW poorly (reference-free section), some fifteen thousand apparent failures, most of which a third database traces to AFLOW where it can settle them (bad-references section), and the leading model’s 7 meV/atom lead, which shrinks to 0.4 at a common geometry (leaderboard section).

We report the results that survive these controls, and the controls themselves. Leaderboards for these models are now numerous, and other benchmarks can reuse the controls more readily than any ranking.

The closest existing benchmark is Matbench Discovery [2], which scores the same kind of models on convex-hull stability, a metric we adopt in the reference-free section. It differs in the reference database. Matbench Discovery generates and relaxes its test structures in the Materials Project’s own DFT convention and scores them against the MP hull. It therefore measures how faithfully a model reproduces its training database.

Matbench Discovery uses AFLOW only for its prototype labels and never as an energy reference [2]. The two benchmarks are complementary. Matbench Discovery is a broad-chemistry stability test within the training convention. Our benchmark tests transfer to an independent reference on metals, compares every energy and stability error with the DFT-vs-DFT floor, and adds elastic, geometric, and reference-convention diagnostics beyond the convex hull.

Other general-purpose platforms such as MLIP Arena [12], LAMBench [13], and CHIPS-FF [14] score many potentials across many tasks, as do broad multi-model assessments [15, 16], while targeted studies probe elasticity [17], phonons [18], mechanical softening [19], and distribution shift [20]. None of them scores against a reference the models never trained on or compares errors with a measured DFT-vs-DFT floor.

We estimate how much of each model’s error a change of elemental reference could explain by realigning the model to AFLOW’s elemental references, a correction that changes no model parameter and that we use as a diagnostic. Once broken elemental references are screened, the stability ranking agrees with the energy ranking almost everywhere, while forces and symmetry retention order the models differently (reference-free section). Because each metric answers a different question, an objective benchmark reports them side by side.

Results

The benchmark: reference, test set, potentials

The reference is AFLOW, a high-throughput DFT database of several million compounds built largely by combinatorially “decorating” a fixed library of crystallographic prototypes with elements from across the periodic table [5] [21]. Three features matter here. First, none of the potentials we test was trained on it. Second, its binary (LIB2) and ternary (LIB3) prototype libraries systematically enumerate ordered intermetallic structures. Formation-energy leaderboards, weighted toward oxides and the Materials Project’s Inorganic Crystal Structure Database (ICSD) entries, undersample these chemistries. (AFLOW’s own ICSD oxides would add little, since 98% of its 18,943 ICSD entries containing oxygen have a composition already in MP.)

Third, for a small subset, AFLOW reports elastic constants through AEL [22]. Because elastic moduli measure the curvature of the PES rather than its absolute level, they let us compare a model to DFT without choosing any energy reference at all—a probe of $\sim3{,}500$ compounds that a formation-energy benchmark cannot provide.

The test set is $\sim1.45$ million ordered structures. The core is 1.38M binary and ternary intermetallic prototype decorations. The remainder supplies the elemental references, the lanthanide-alloy and extended-chemistry strata, the elastic subset, and a small quaternary set. We exclude the 1,575 elemental entries that AFLOW did not compute with PBE (Methods).

The potentials are fourteen uMLIPs (Table 1), chosen to span different ML architectures and training corpora, since the training data determines which DFT a model has effectively learned. They fall into two generations, an earlier one trained on smaller datasets (MPtrj [6], MatPES [23], or the MatterSim set [24]) and a newer one trained mainly on the much larger OMat24 corpus.¹

We include the earlier MP generation deliberately, alongside the latest OMat models. If an effect moves all fourteen models together, it almost certainly comes from the benchmark. If it moves only the OMat models or only the MP models, it comes from their training data. The main figures show twelve of the fourteen. The two superseded checkpoints (an early MatterSim and SevenNet-0) appear in the closing leaderboard (Discussion) and the elastic comparison (reference-free section).

The protocol starts every structure from AFLOW’s relaxed geometry and relaxes it again with each potential, so that geometry, formation energy, and relaxation drift are all measured on the same footing (ASE FIRE with a FrechetCellFilter that relaxes cell and positions together; Methods) [38, 39]. We use the model’s own elemental energies to compute formation energy, so every value starts in that model’s native convention. Each element’s reference is the lowest energy the model assigns to any structure of that element, after a screen that removes elemental relaxations gone unphysical (Methods; bad-references section returns to them).

The DFT-vs-DFT noise floor {#sec:The DFT-vs-DFT noise floor}

A uMLIP cannot aim for zero error. Two independent high-throughput DFT databases do not return the same formation energy for identical structures. Even if they share the exchange–correlation functional (here, the Perdew–Burke–Ernzerhof functional, PBE [40]), they differ in the rest of the protocol: pseudopotentials, k-point density, smearing, relaxation tolerances, and elemental references. Those choices shift formation energies measurably. A model’s agreement with one such database therefore cannot be ranked more finely than two such databases agree with each other. Before scoring any potential, we measure the DFT-vs-DFT disagreement and treat it as the noise floor.

We first measure the energy difference for structures that appear in both AFLOW and a second, independent DFT database. Two structures are the same if pymatgen’s StructureMatcher matches them and they have the same space group, and we compare every pair of structures with the same reduced formula across the databases to find them (Methods). Table 2 gives the shared counts; 27,427 structures are common to all three databases. Across these shared structures, the median $|\Delta E_\mathrm{f}|$ between AFLOW and Alexandria is 5.8 meV/atom, and between AFLOW and OQMD 8.4 meV/atom (Table 2, Figure 1, Methods).

All three use the PBE functional, so this 6–8 meV/atom agreement is the reproducibility of a protocol, not of the PBE functional itself. Carefully converged codes that run with matched settings agree on elemental equations of state to about 1 meV/atom [41, 42], but a formation energy is far more settings-sensitive. It is a difference of total energies of chemically different systems, so the errors that cancel within a single element’s equation of state do not cancel here.

The three share this floor because they all report raw-PBE energies (raw meaning no post-hoc adjustment). The one exception is small. OQMD fits the elemental references of four of our metals (Na, Ti, Sn and Hg) to experiment, which touches 20% of its pairs with AFLOW and raises its median by about 1 meV/atom (Methods). No AFLOW structure behind these comparisons carries a Hubbard +U (Methods). The one +U-heavy subset, the elastic library, is treated in the reference-free section.

As with the models, the size of the floor depends on which average we quote, the median or the mean. The disagreement distributions are heavy-tailed. Most matched pairs agree to a few meV/atom, but a very small population disagrees catastrophically (the bad-reference class of the bad-references section, present on both sides of every pair). We score every model (leaderboard section) with a trimmed mean, the average after setting aside the few structures that disagree by more than 1 eV/atom. The large disagreements that remain below that cutoff put the floor’s trimmed mean at 16–30 meV/atom, well above the median.

DFT pair	median $\lvert\Delta E_f\rvert$ (meV/atom)	trimmed MAE (meV/atom)	shared metallic structures
AFLOW – Alexandria	5.8	16.4	$\sim$448,500
AFLOW – OQMD	8.4	25.8	$\sim$30,300
OQMD – Alexandria	7.8	29.9	$\sim$56,200
AFLOW – Materials Project	18.0	30.3	$\sim$6,400

The median and the mean answer different questions, and neither is wrong. Comparing model means with the median floor, however, puts every model about 11 meV/atom farther from the floor than it is. The leader would sit 8 meV/atom above it instead of 3 below (Supplementary Table 8 gives every model). Every distance from the floor that we quote below therefore uses the same trimmed mean for the model and the floor, on the structures on which the floor is measured (Methods).

This floor might still be illusory. The 6–8 meV/atom median could be merely a per-element reference convention, a constant offset a model could absorb for free, rather than genuine disagreement. We test this possibility by fitting a per-element, composition-linear offset between each database pair and asking how much of the structure-to-structure spread it removes. On the metallic set, a per-element offset fit on 70% of the chemical systems explains under 7% of the AFLOW–Alexandria disagreement on the other 30%, and subtracting it does not lower the floor. The pairs that include OQMD are higher (13–14%), consistent with OQMD’s fitted elemental references.

The Materials Project, the one database here that adjusts compound energies after the calculation [43, 44], sits farther from AFLOW than the other two. On our intermetallic subset (where none of MP’s anion or +U corrections are applied), MP disagrees with AFLOW by 18.0 meV/atom on a typical structure, compared with 6–8 for the raw-PBE databases. A per-element offset removes little of that gap (in-sample $R^2 = 0.06$; Supplementary Note 5). The gap is not MP’s corrections, which are inactive on metals, and not a re-referenceable convention. It is a genuine protocol difference that we do not isolate further. Hedge et al., in a pairwise comparison of AFLOW, the Materials Project, and OQMD, reached the same conclusion [4]. Across all chemistries, MP is the common term in the two largest gaps, 105 meV/atom for AFLOW–MP and 87 for MP–OQMD, compared with only 19 for AFLOW–OQMD. Hedge et al. trace these gaps to oxide-dominated references and post-hoc corrections, the very corrections that fall away on metals.

The formation-energy leaderboard

We begin with the familiar metric, each potential’s formation-energy error against AFLOW. For each potential we relax every structure, compare its formation energy to AFLOW’s, and report the trimmed MAE of the noise-floor section (structures with $\lvert\text{error}\rvert> 1$ eV/atom are counted separately; Methods). We determine 95% confidence intervals from a bootstrap that resamples whole chemical systems (Methods D). The rankings and confidence intervals are given in Fig. 2 and Table 3.

Model	Raw MAE [95% CI]	Single-point	After Rung-0	SP + Rung-0	Above floor
eqV2	18.3 [17.9–18.7]	14.4	16.6	12.7	−2.6 [−2.8, −2.4]
eSEN	25.5 [25.0–26.0]	14.8	23.8	13.3	+2.3 [+1.9, +2.6]
SevenNet-MF	28.2 [27.8–28.7]	19.4	27.0	17.9	+5.8 [+5.5, +6.1]
UMA	28.5 [28.1–29.0]	17.9	25.1	13.3	+4.6 [+4.2, +5.1]
GRACE-2L	29.8 [29.3–30.3]	21.0	28.1	20.0	+7.2 [+6.9, +7.5]
MACE-OMAT	32.7 [32.2–33.1]	24.2	30.7	22.2	+9.8 [+9.4, +10.2]
MACE-MPA	33.0 [32.5–33.5]	21.6	31.3	21.2	+9.0 [+8.6, +9.4]
DPA-3	33.0 [32.6–33.5]	23.9	30.6	22.3	+10.2 [+9.9, +10.6]
Orb-v3	35.8 [35.2–36.3]	20.2	33.4	17.8	+11.6 [+11.0, +12.3]
MatterSim-5M	52.0 [51.4–52.5]	39.4	40.7	31.6	+21.3 [+20.7, +22.0]
M3GNet	63.3 [62.7–63.9]	50.1	55.8	46.1	+38.8 [+37.8, +39.8]
CHGNet	126.5 [125.4–127.6]	101.7	72.3	60.9	+54.9 [+53.6, +56.2]

The models fall into two tiers, with one model above both. eqV2 leads outright at 18.3 meV/atom, and eSEN is next at 25.5. Behind them the OMat-trained models cluster tightly, 28 to 36. The MP-trained models trail well behind, with MatterSim-5M at 52, M3GNet at 63, and CHGNet at 127. MACE-MPA breaks that split. At 33 it sits in the middle of the OMat-trained models, away from its MP-trained siblings, for a reason we come to below. The separations are far larger than the confidence intervals, none of which is wider than ±2 meV/atom (Table 3).

eqV2’s 7 meV/atom lead over eSEN is statistically significant. Before interpreting this as a ranking of PES accuracy, we ask what a low MAE against AFLOW measures. In part, it measures how closely the DFT conventions each model was trained on agree with AFLOW’s. eqV2 and eSEN were both fine-tuned on subsampled Alexandria, a database that agrees with AFLOW to within 6 meV/atom on the typical structure (noise-floor section). The fine-tune is worth 1.5–3.6 meV/atom of raw error, almost all of it a per-element offset (Supplementary Note 7).

A second question is how eqV2 reached its score. In the relax-again protocol, it barely moved. Its forces come from a separate output head, a final part of the network trained to predict forces directly, rather than from the gradient of its energy. That head treats AFLOW’s minima as already converged (Methods). It was therefore scored at or very near the DFT geometry, while the other thirteen models absorbed relaxation drift, the change in energy as a structure relaxes away from AFLOW’s geometry (Methods). The control that puts every model on the same footing scores each model at AFLOW’s geometry without re-relaxing it (single-point), using the energy recorded before its optimizer’s first step (Methods).

The single-point control changes two things (Table 3, Single-point). Every model improves, by 4 to 25 meV/atom, and the lead all but vanishes. eqV2’s 7 meV/atom margin over eSEN becomes four tenths (14.4 versus 14.8). A paired bootstrap still resolves that order (eqV2 lower by 0.43 meV/atom [0.39–0.48]), but four tenths of a meV is about 3% of the trimmed floor. The ordering is statistically resolved but too small to matter. Orb-v3 moves the most in rank, from ninth to fifth (35.8 to 20.2). Only CHGNet moves more in meV/atom (126.5 to 101.7). Nearly half of Orb-v3’s apparent error (15.6 of 35.8 meV/atom) came from relaxing structures away from AFLOW’s geometry rather than from mis-describing the energy at a fixed one.

Two explanations compete for why the MP-trained models trail so far behind. The first is genuine model error, that their potential-energy surfaces are less accurate. The second is bookkeeping. A model trained on Materials Project data inherits MP’s per-element energy conventions, its anion and DFT+U corrections, its elemental references. Because a formation energy is computed by subtracting elemental reference energies, any per-element gap between MP’s references and AFLOW’s shifts every formation energy by a fixed composition-weighted amount. When the model is scored against AFLOW, that inherited convention appears as error even when the underlying PES is sound.

The two explanations make opposite predictions about what happens when we correct the elemental references, so a simple reference correction, which we call Rung-0, can rule bookkeeping out, though it cannot confirm it. We fit one energy constant per chemical element, choosing the constants that minimize the same mean absolute error we score, and subtract them from the model’s formation energies. The correction involves no neural network, no weight update, and no per-structure parameter, only one number per element that re-references the model’s energies to AFLOW’s. The name Rung-0 suggests its place at the bottom of a hierarchy whose higher rungs (per-structure terms, fine-tuning, retraining) would add real fitting capacity. Rung-0 adds nothing beyond a shift of each element’s reference energy.

Two safeguards make Rung-0 a sharp diagnostic rather than a fit (Methods F). The first, a capacity bound, limits what Rung-0 could remove to one constant per element, which can shift a model’s reference frame but cannot reshape its PES. Rung-0 still removes anything linear in composition: a difference of reference convention, a broken elemental reference, or an element-wise bias in the model. It cannot distinguish them, so its gain is the most that any change of elemental reference could remove, not a measured convention.

The second, the held-out split, shows that what Rung-0 did remove is transferable. The ∼50 constants are fit on 70% of the chemical systems and scored on the other 30% (Methods F; fold robustness in Supplementary Note 7), so they cannot flatter the model by absorbing in-sample noise. The test is therefore one-sided. If Rung-0 leaves a model’s error unchanged, that error is not a per-element offset. If Rung-0 removes part of it, that part could be a convention or an element-wise bias.

Re-referencing removes little where a model’s error has no per-element part. For nine of the twelve models, which are already nearly aligned with AFLOW’s reference frame, it removes only 1.2 to 3.4 meV/atom (Table 3).

Three of these nine are aligned only once the elemental references are screened. UMA, eSEN, and MACE-MPA each give their lowest energy for some element to a structure that is not a sound bulk crystal (bad-references section). That structure becomes the element’s reference (benchmark section), so every compound containing the element is mis-scored by the same per-element amount. Rung-0 detects and removes this signature but cannot tell such a mistake from a genuine inherited convention. Discarding such structures before the elemental reference is set (bad-references section; Methods) removes the error rather than absorbing it. The raw MAEs fall, MACE-MPA’s from 48.2 to 33.0, eSEN’s from 38.6 to 25.5, and UMA’s from 35.0 to 28.5. All three then behave like every other aligned model.

The three models trained without Alexandria or OMat24 carry a large per-element component on top of a large structure-dependent error. CHGNet is the clearest case. Re-referencing lowers its error from 127 to 72 meV/atom. Two fifths of its apparent error (43%) is removable with one constant per element, without changing any weight. Those constants barely track the differences between the Materials Project’s and AFLOW’s elemental references, the convention of CHGNet’s training data (r = 0.19, p = 0.21; Supplementary Note 7). They are therefore more likely an element-wise bias in the model than an inherited convention. The 72 that remains is structure-dependent error that no per-element correction can erase. M3GNet and MatterSim-5M behave the same way, improving to 56 and 41 meV/atom.

The residual after Rung-0 contains no per-element offset, so it is an estimate of each model’s structure-dependent error. Ranked on this residual, the group behind the leader forms a four-way near-tie. eSEN (24), UMA (25), SevenNet-MF (27), and GRACE-2L (28) span 4 meV/atom. Those gaps are smaller than the 6–8 meV/atom uncertainty of the reference itself, so at this scale a ranking measures agreement with AFLOW’s particular protocol, not PES accuracy. An equally legitimate raw-PBE reference could plausibly put these four in a different order.

Even after re-referencing, eqV2’s lead over eSEN, about 7 meV/atom, sits at the edge of that resolution.

On the structures that define the floor, the re-referenced leader sits 3 meV/atom below the AFLOW–Alexandria floor, eSEN 2 above it, and UMA, SevenNet-MF, and GRACE-2L 5 to 7 above it, all inside the 16–30 band at which the independent raw-PBE databases disagree (Table 3; noise-floor section). These distances depend more on how strictly structures are matched when the floor is built than on sampling. Across the matching tolerances we tried they move by up to about 5 meV/atom for the leading models, compared with intervals of a few tenths, and the leader stays below the floor at every tolerance (by 2.3 to 7.3 meV/atom; Supplementary Table 8).

Before re-referencing, the leader sits just below the floor on the structures that define it, with a trimmed MAE of 15.1 meV/atom compared with the floor’s 16.4 and a median of 5.3 compared with 5.8. On the OQMD-matched set its trimmed MAE is 16.9 compared with that pair’s 25.8 (Methods). Re-referencing lowers the leader’s error on the held-out structures further, from 15.4 to 14.1 meV/atom.

A model can sit below the floor because the floor measures how far two independent DFT databases disagree, not how closely any calculation can match AFLOW. Alexandria and AFLOW differ in pseudopotentials, settings, and convergence, and a potential trained on other

Units meV/atom. Raw MAE [95% CI]: trimmed mean over the full metallic set; intervals from a bootstrap over whole chemical systems (Methods). Single-point: the same trimmed MAE at AFLOW’s geometry, without re-relaxation (Methods). After Rung-0: corrected MAE on a held-out 30% of the chemical systems, with one energy constant per element fit on the other 70% by minimizing the absolute error (Methods F). (Uncorrected, the held-out MAE is within 0.7 meV/atom of the full-set value, so the text’s before→after figures use the full-set raw MAE.) Above floor: after-Rung-0 trimmed MAE minus the 16 meV/atom AFLOW–Alexandria floor of the noise-floor section, both on the held-out structures that define the floor (Methods). Intervals are paired 95% bootstrap intervals over chemical systems, with the per-element constants held at their fit. A negative entry means the model agrees with AFLOW more closely than Alexandria does on those structures. SP + Rung-0: both controls at once, an estimate of structure-dependent error at the reference geometry (Rung-0 fit and scored on the single-point errors, on the same held-out split as After Rung-0).

PBE data can land closer to AFLOW than Alexandria does. At that point, AFLOW can no longer resolve which of the two is closer to the PBE answer. The floor can be measured only on structures both databases hold, and these structures are in the training data of the models fine-tuned on Alexandria. The OMat24-only checkpoints of eqV2 and eSEN, which never saw Alexandria’s labels, also sit below the floor after Rung-0 (Supplementary Note 7), and any memorization of Alexandria compounds is bounded at under 1 meV/atom (generalization section).

We close the error decomposition by applying both controls at once. We fit Rung-0 to the single-point errors, so that neither relaxation drift nor any per-element offset remains (Table 3, SP + Rung-0; the filled diamonds of Figure 3). Here eqV2’s energy margin all but disappears. UMA, whose remaining drift masked it in the relaxed ranking, drops from 17.9 to 13.3 meV/atom, eSEN from 14.8 to 13.3, and eqV2 from 14.4 to 12.7. The fully controlled top three cluster within 0.7 meV/atom. That spread is small next to the floor, so their order says little about which model describes the PES better. Behind them come Orb-v3, SevenNet-MF and GRACE-2L, with MACE-MPA beside them (Table 3).

eqV2 still leads with both controls applied, by about 0.6 meV/atom over eSEN and UMA (Supplementary Table 12). Together, the two controls show that its raw lead came mostly from relaxation drift, which eqV2 barely incurs, rather than from a uniquely better PES at the DFT geometry. The separation that survives most clearly is in hull stability (reference-free section). eqV2 ranks first there both after relaxing and, by a small margin, at AFLOW’s geometry.

Paired difference (meV/atom)	Estimate	95% CI
UMA − eqV2	+0.66	[+0.35, +1.00]
eqV2 − eSEN	−0.58	[−0.63, −0.53]
UMA − eSEN	+0.08	[−0.22, +0.41]

Rung-0’s ingredients are not new. Fitted reference energies are standard practice, from fitted elemental-phase reference energies for every element [?] and OQMD’s fitted chemical potentials for 13 elements [?] to the Materials Project’s per-atom corrections for anions and +U cations [?]. Machine-learned potentials use them too. MACE shifts each atom’s energy by the energy of the isolated atom [?], CHGNet normalizes its training energies with elemental reference energies [6], and multi-fidelity and multi-task training reconcile datasets computed with different settings [?, ?, ?, ?]. Composition-only baselines are textbook [?, ?, ?, ?]. A single energy per element is also the lowest-order term of the cluster expansion of alloy configurational energy [?, ?, ?, ?, ?], the framework on which the formation energies of ordered intermetallics are conventionally built. What is new is using Rung-0 as a diagnostic, which tells us what a leaderboard score means without changing the model’s PES.

Reference-free metrics: independence, not reordering

The leaderboard section raises a concern that part of a model’s standing on the formation-energy leaderboard is an accident of a reference convention. Rung-0 can remove a per-element offset, only after the fact, and only if one chooses to apply it, which leaderboards as ordinarily reported do not. A cleaner route is to lead with properties that need no elemental energy reference, so that bookkeeping cannot affect them. Polymorph ordering is the cleanest such property, and the only energy-based metric that is exactly invariant. Competing polymorphs share a composition, so a per-element shift adds exactly the same constant to every member of the set and cannot change the ordering.

The convex hull is nearly as good. Stability depends on a structure’s energy relative to competing phases of the same elements, and a per-element shift cancels in that difference as long as the elemental endpoints shift with the compounds. A shift applied to the compounds alone does not cancel. By construction, a model’s elemental reference is its own lowest-energy elemental structure, which sits at

$E_f = 0$ and anchors that endpoint. If that structure is the right one, the cancellation holds and the metric is blind to the convention. If it is the wrong one, the hull is anchored to a bad structure, and phases move on and off it. Hull metrics are invariant to a self-consistent convention but not immune to a mis-set reference. MACE-MPA’s Fe reference before screening was one (bad-references section).

Stability on the convex hull

Matbench Discovery scores a potential by how well it classifies thermodynamic stability ⁴. We use the same kind of metric, built against AFLOW’s hull rather than the Materials Project’s, and call the resulting score hull-F1.

For every structure we ask whether it is stable. AFLOW’s energies give the true hull, and each model’s own energies give its predicted hull. A classification can err in two ways, and precision and recall measure them. Precision is the fraction of the structures a model calls stable that truly are, and recall is the fraction of the truly stable structures it finds.⁴ The F1 score is their harmonic mean, so it is high only when both are high.

We call a structure stable if it lies within 25 meV/atom of the hull, a few times the 6–8 meV/atom DFT-vs-DFT floor (noise-floor section), so no model is asked to resolve stability more finely than two DFT databases agree. The choice of window does not drive the result, since eqV2 leads at every window from 10 to 50 meV/atom (Supplementary Note 9).

Broken elemental references make the energy and stability rankings look more different than they are. If we

rerank the twelve models by stability instead of by energy, a model moves two places on average with the unscreened references, and less than one with them screened. These averages score each model on all the structures it relaxed, not on the common set of Table 4.

Model	Precision	Recall	Hull-F1 [95% CI]	$E_{\mathrm{hull}}$ MAE	Hull-F1, single-point [95% CI]	$E_{\mathrm{hull}}$ MAE, single-point
eqV2	0.91	0.94	0.921 [0.910, 0.929]	18.7	0.930 [0.920, 0.938]	14.7
eSEN	0.85	0.95	0.893 [0.882, 0.902]	26.5	0.923 [0.913, 0.932]	15.6
Alexandria (second DFT)	0.91	0.89	0.901	—	—	—
UMA	0.84	0.91	0.872 [0.860, 0.883]	27.1	0.897 [0.883, 0.909]	15.7
SevenNet-MF	0.81	0.93	0.868 [0.857, 0.879]	29.0	0.901 [0.890, 0.911]	19.6
GRACE-2L	0.77	0.93	0.843 [0.827, 0.856]	31.4	0.874 [0.860, 0.886]	22.6
MACE-MPA	0.77	0.92	0.839 [0.826, 0.851]	35.5	0.878 [0.867, 0.888]	23.9
Orb-v3	0.78	0.90	0.837 [0.825, 0.849]	35.8	0.875 [0.863, 0.886]	19.8
MACE-OMAT	0.75	0.90	0.821 [0.806, 0.835]	34.0	0.857 [0.843, 0.870]	25.2
DPA-3	0.74	0.90	0.815 [0.799, 0.830]	34.0	0.847 [0.831, 0.861]	24.9
MatterSim-5M	0.62	0.91	0.736 [0.717, 0.751]	53.0	0.777 [0.759, 0.792]	41.0
M3GNet	0.51	0.78	0.616 [0.594, 0.637]	62.7	0.653 [0.631, 0.674]	50.8
CHGNet	0.31	0.84	0.448 [0.426, 0.469]	113.1	0.522 [0.499, 0.543]	91.8

eqV2 is first in both rankings and eSEN second. Both gaps are resolved, 7 meV/atom on energy and 0.028 [0.023, 0.033] on hull-F1 (0.921 versus 0.893). At AFLOW’s geometry, the single-point control of the leaderboard section, the hull-F1 gap narrows to 0.007 [0.002, 0.011] (0.930 versus 0.923) but stays resolved. Only two models move by more than one place. Orb-v3 ranks ninth on energy but seventh on stability, and MACE-OMAT moves the other way, from sixth to eighth.

The stability metric is useful because a self-consistent change of elemental reference can move the energy ranking but not the stability ranking. A broken reference can move the stability ranking as well (above). Once the references are screened, the two rankings agree almost everywhere.

The divide between the OMat and MP models (leaderboard section) survives the change of metric and is wider here than on energy. The OMat-trained models and MACE-MPA score hull-F1 0.82–0.92, and the other MP-trained models 0.45–0.74. CHGNet is the extreme case. Its recall of 0.84 means it finds most stable phases, but its precision is only 0.31, so more than two of every three structures it calls stable are not. Precision matters for screening new materials, where the hull is the filter. A model this imprecise suggests many candidates that are actually unstable.

Residual forces

Residual forces need no elemental reference. At AFLOW’s relaxed geometry the DFT forces are essentially zero, so the force a model reports there measures how well its PES agrees with DFT’s near the minimum. On this metric the panel does not split by training set. Ten of the twelve models report a median maximum residual force of 0.052 eV/Å or less. Eight of them lie between 0.040 and 0.052 (UMA lowest of those), and eSEN is lower still at 0.022.

eqV2 reports a median below 0.001 eV/Å. As the leaderboard section noted, it predicts forces with a separate output head, so a near-zero force at a DFT minimum partly reflects what that head was trained to output. (On the energy ranking, the single-point control of the leaderboard section corrects for the same asymmetry.)

Only M3GNet and CHGNet stand apart, near 0.10 eV/Å, with about half of their structures above 0.1 eV/Å compared with a quarter to a third for the other models. The two models the hull metric ranked last are also the two whose PES is farthest from DFT’s at the minimum.

Space-group retention

After relaxing each structure, we ask whether it keeps its space group. This is a check on geometry, separate from the energy-based hull-F1. At first sight, space-group retention separates the models sharply. Most of that separation, however, comes from the symmetry tolerance (symprec) used to assign the space group. As we loosen symprec (Figure 4), the models converge. Over the whole benchmark, and leaving eqV2 aside, retention at a tight 0.01 Å runs from 77.5% (UMA) to 92.0% (MatterSim-5M). At the conventional 0.1 Å it runs from 88.6% to 92.2% (all fourteen models at every tolerance in Supplementary Note 10). eqV2 retains 99% at every tolerance because it barely relaxes (leaderboard section).

UMA, eSEN, and Orb-v3 lose the most at the tight tolerance (77.5–84.0%) and gain the most as it loosens, the signature of small distortions that a looser tolerance forgives. A small real difference remains at 0.1 Å, where every interval is about ±0.1 point. Orb-v3 retains the fewest of the OMat-trained models (90.3%), and the four oldest MP-trained models the fewest overall (88.6–89.0%). But the large apparent spread came from one analysis choice, so we report retention as a function of tolerance, never as a single number.

Polymorph ordering

Polymorph ordering asks whether a model ranks the competing structures at one composition in the same energy order as DFT. We score it by the Spearman correlation $\rho$ across each composition’s polymorphs and keep the compositions whose AFLOW energies span at least 25 meV/atom. The two-tier divide appears again. The OMat-trained models reproduce AFLOW’s ordering at median $\rho$ 0.92–0.98 (eqV2 highest, then eSEN, UMA, SevenNet-MF), MACE-MPA sits with them at 0.92, and the MP generation trails at 0.79–0.88 (CHGNet lowest). At AFLOW’s geometry every model orders the polymorphs better, and the leaders converge. eqV2 and eSEN then tie (mean $\rho$ 0.962 each; each finds the lowest polymorph 62% of the time), so eqV2’s lead over eSEN in the relaxed protocol is a relaxation effect.

A stricter test lowers every score. Asked to pick the single lowest-energy polymorph, the best model (eqV2) is right only 59% of the time, and CHGNet 28%, counting a tie for lowest as a hit. Both are far better than chance. Random guessing among the polymorphs of each set, whose median size is 25, would be right 8% of the time. The models capture the shape of the energy landscape but often miss its exact minimum.

Alexandria can be scored only on the compositions it shares with AFLOW, which have far fewer polymorphs (10 on average, compared with 21 on the full set). There it finds AFLOW’s minimum 91% of the time. Scored against AFLOW on the same compositions, eqV2 does slightly better (92%, mean $\rho$ 0.963 versus Alexandria’s 0.956), again matching AFLOW’s protocol more closely than a second DFT database does, as in the leaderboard section. eSEN is 2 points below Alexandria and the other leading models 4 to 7 points below (Supplementary Note 11).

The pairs that include OQMD agree less well. OQMD and Alexandria share at least three polymorphs in 2,565 compositions and agree at mean $\rho$ 0.85, finding the same minimum 84% of the time; AFLOW and OQMD, in 2,318 compositions, reach 0.83 and 85%. Chance is 28–29%. AFLOW and Alexandria, reweighted to the same set sizes, give 0.95 and 94%. (We quote means because nearly all of these sets have only three to five members.) On the AFLOW–OQMD compositions, every OMat-trained model and MACE-MPA orders the polymorphs more consistently with AFLOW than OQMD does (mean $\rho$ 0.84–0.93, versus 0.83), and the MP-only models less so.

Where the rankings differ

Figure 5 collects these rankings into one picture and spaces the models by how far apart they are, not by rank. Three things stand out. The first and coarsest is the two-tier split above. On the energy axis the gap is very large, with CHGNet at 127 meV/atom compared with the leader’s 18.3.

Second, there is little reshuffling inside the leading tier. eqV2 and eSEN rank first and second on all four metrics, and UMA and GRACE-2L hold their places to within one. Orb-v3 is ninth on energy MAE, seventh on the hull and

on residual force, and ninth on polymorph ordering. Its relaxation drift (nearly half of its formation-energy error;

Elastic response

leaderboard section) affects the reference-free metrics too. At AFLOW’s geometry it rises to sixth on the hull and fourth on polymorph ordering. Rung-0 reduces its energy error only slightly ($36\rightarrow33$). The order in the middle of the energy ranking also depends on the cutoff of the trimmed mean, while the leader and the tiers do not. Orb-v3 moves three places between cutoffs of 0.5 and 1.0 eV/atom (Supplementary Note 6; Supplementary Fig. 1).

Third, MACE-MPA does not stay with its group. It is the one dotted curve in the leading tier. We class it with the MP generation, but on the reference-free metrics it rises into the OMat-trained tier.

The clearest comparison for MACE-MPA is with MACE-OMAT, which has the same architecture and differs only in training data [7] [35]. The two are close on every metric we have, with no consistent winner: hull-F1 0.839 versus 0.821, residual force 0.047 versus 0.045 eV/Å, polymorph ordering $\rho$ 0.920 versus 0.925, and, once the elemental references are screened, energy MAE 33.0 versus 32.7 meV/atom. Before screening, their energy MAEs differ by 16 meV/atom in MACE-OMAT’s favor, which would suggest that the larger OMat corpus separates them. In fact, that gap comes from MACE-MPA’s broken Fe reference (bad-references section).

MACE-MPA most likely keeps pace with the OMat-trained models because of its training set. It is the one MP-generation model in our panel trained on subsampled Alexandria as well as MPtrj, and Alexandria reports formation energies in the same raw-PBE convention as AFLOW. Capability therefore follows the OMat-versus-MP label only roughly.

Elastic response

The elastic response is another reference-free test. For about 3,500 of our structures, AFLOW’s AEL library stores the full DFT elastic tensor [22, 59], from which we read a bulk modulus $K$ and a shear modulus $G$. We relax each structure with a model and recompute its tensor by the same finite-strain protocol, which yields a model $K$ and $G$ for the same compound. Because moduli are curvatures of the PES rather than absolute energies, the comparison needs no reference at all.

Across these compounds, the bulk modulus tracks AFLOW well for every model (log-space correlation $r$ from 0.91 to 0.99). The shear modulus does not. Its log-space $r$ runs from 0.76 for CHGNet and M3GNet to 0.94 for UMA. Taken at face value, several models predict shear stiffness poorly. Systematic softening has been reported across uMLIPs [19], and elastic response is now a benchmark target in its own right [17].

Before blaming the models, we ask what the poorly predicted structures share. It is the reference Hamiltonian. AFLOW computes its alloy libraries (LIB2 and LIB3, 95% of our benchmark) without $+U$, but elsewhere it applies a Hubbard $+U$ correction [21, 60] to compounds containing any of 28 transition and post-transition metals, intermetallics included (Methods H). $+U$ compounds are therefore rare in the benchmark (1.9%, and none of the structures behind its energy metrics) but make up 78% of the AEL comparison set. The Materials Project applies $+U$ only to oxides and fluorides of eight of those elements, so for most of these compounds the models were trained on a different Hamiltonian than AFLOW used. The shear response, which measures the response to a change of shape, is more sensitive to that choice than the bulk response to uniform compression.

When we restrict the comparison to the AFLOW structures computed without $+U$ (about 795 of them), the shear correlation recovers (Figure 6), to 0.85–0.98 (CHGNet 0.76 → 0.88, MACE-MPA 0.92 → 0.98, UMA 0.94 → 0.98). Splitting the $+U$ structures by chemistry (Supplementary Note 12) shows shear agreement worst for the $+U$ intermetallics, a treatment the Materials Project never uses, and lowest there for five of the eight models. The picture is not purely one of convention, though. The $+U$ oxides and fluorides of elements the Materials Project treats without $+U$ agree well for most models, while the 89 structures where both databases apply $+U$ are poorly predicted by the older models.

Working in log space shows what kind of error is left (Supplementary Note 12). On the subset without $+U$, every model’s log–log slope against AFLOW’s shear moduli lies between 0.91 and 1.02. The older models therefore do not drift further from AFLOW as the modulus grows. They are off by a constant factor. CHGNet predicts shear at 0.48× AFLOW across the board, M3GNet at 0.71×, and SevenNet-0 at 0.73×, while MACE-MPA, GRACE-2L, Orb-v3, and UMA sit at 0.97–1.03×.

A linear correlation would tell a different story for MACE-MPA, $r=0.16$ on all structures. That value is set almost entirely by one structure, NiB$_2$, predicted with a shear modulus of −12,719 GPa; without it the linear $r$ is 0.66. Its median shear ratio to AFLOW is 0.97.

Much of the apparent scatter in shear therefore comes from comparing models trained on one Hamiltonian with a reference computed with another. The older models’ constant-factor softening (0.48–0.73×) survives the control. A difference in Hamiltonian can look like a model error, so the structures should be stratified by how the reference was computed before an error is blamed on a model’s PES. The argument has a limit. Removing the $+U$ structures from the elastic set also removes every compound containing any of the 28 elements AFLOW treats with $+U$, so we cannot separate $+U$ from element identity or from correlated-electron effects.

The elastic set is not metals-only, since AFLOW’s elastic data live mainly on ICSD compounds with nonmetals; a metals-only recut leaves the conclusion intact at lower statistical power (Supplementary Note 12).

“Bad” references: mostly the benchmark’s, sometimes the model’s

Most disagreements between a model and AFLOW are a few tens of meV/atom or less. A small population of structures tures miss by whole electron-volts. There, either the model or the reference is badly wrong, and deciding which needs evidence from outside the pair.

We therefore look for the structures where AFLOW disagrees with the consensus of eight of the models rather than those a single model gets wrong. When the potentials cluster within a few tens of meV and AFLOW sits an electron-volt away, the reference is more likely to be wrong. A structure on which the models agree closely and AFLOW sits far from all of them is a discrepancy (the consensus test; thresholds in Methods I). The consensus test finds 15,278 discrepancies among the 1.42M structures it can score, and the count changes smoothly with the threshold.

Generalization

"Bad" references: mostly the benchmark's, sometimes the model's

Fifteen thousand bad references would be a serious problem for AFLOW, but part of each disagreement is the per-element reference offset of the noise-floor and leaderboard sections [45]. We fit that offset to the gap between the consensus and AFLOW, subtract it, and apply the same cut again. The discrepancies fall from 15,278 to 10,861. Of the 4,419 structures that drop out, 91% contain a magnetic 3d element (V, Cr, Mn, Fe, Co, or Ni), compared with 35% of all structures. These elements carry the largest fitted offsets (Mn −119, Cr −103, and Fe −87 meV/atom). Magnetism itself does not make the models worse. No model has a larger trimmed MAE on magnetic structures than on nonmagnetic ones (Supplementary Note 16).

The offset alone does not explain the 4,419 discrepancies that drop out. After we remove the offset, each of them still sits more than 200 meV/atom from the consensus. Each drops out because the offset carries it just below the 0.3 eV/atom cut. Where Alexandria has the same structure, it sides with the models on 86% of the dropouts, so most of them are AFLOW disagreements a little below the threshold. None of the discrepant structures, before or after re-referencing, is computed with +U (the alloy libraries carry none; Methods), so none of these discrepancies can be the Hamiltonian mismatch of the reference-free section.

The 10,861 discrepancies that remain after re-referencing are real disagreements, and for these we consult a third database. Where a discrepant structure exists in another DFT database, we ask whether that database agrees with the models or with AFLOW. Alexandria is the most useful third database, because it shares AFLOW’s raw-PBE convention (noise-floor section). We count a result only when the other database holds the same crystal as the discrepant AFLOW structure (Methods). Alexandria has the same structure for 3,794 of the 10,861. It sides with the models on 3,151 of them (83%), with a median disagreement of 13 meV/atom with the model consensus compared with 412 meV/atom with AFLOW. When a second independent DFT database agrees with the models against AFLOW, the reference is almost certainly the one at fault.

The result depends little on which models form the consensus. We rebuild the whole test from the six MP-generation models alone, so that no model in it was trained on OMat24 (one of them, MACE-MPA, saw a subset of Alexandria). Alexandria then has the same structure for 3,794 discrepancies and sides with the models on 3,032 of them, 80% (Methods). OQMD has the same structure for 322 of the 10,861 discrepancies that remain after re-referencing, and sides with the models on 301 of them (93%). The Materials Project has the same structure for 97 of the remaining 10,861 and sides with the models on 96, but about half of these are elemental crystals. Where a third database has the same structure, it sides with the models about five times in six. Of the remaining 10,861, 7,067 have no same-crystal Alexandria entry, and for those we cannot say whose error it is.

We also ask whether Alexandria has bad references of its own. We apply the same consensus test to the 448,397 same-crystal AFLOW–Alexandria metallic pairs whose AFLOW structure the models scored, now measuring Alexandria’s distance from the models. The models were run on the AFLOW structure, and every pair is the same crystal, so the models’ energies apply to Alexandria’s entry as well. In 435 pairs the test finds a discrepancy for Alexandria and none for AFLOW, and StructureMatcher confirms the match again on the structure behind Alexandria’s energy for 426 of them. We count these 426 as genuine Alexandria errors (Supplementary Note 13).

These 426 are nearly all of one kind. In all but two of them, Alexandria’s energy is too high (by a median of 426 meV/atom). In 362, AFLOW finds a magnetic ground state where Alexandria’s entry carries no moment. (Supplementary Table 18 gives the chemistry, sources and third-database checks.) On the same pairs and with the same structure test, AFLOW is the outlier in 2,768 pairs, so Alexandria has fewer bad references than AFLOW, and most of Alexandria’s are a missed magnetic state.

	Count
Alexandria above C	424 (median excess 426 meV/atom; the other two entries are 2.76 and 0.58 eV/atom below)
AFLOW within 0.1 eV/atom of C	405
Alexandria cell smaller than AFLOW’s	408
Containing V, Cr, Mn, Fe, Co or Ni	414 (Mn 244, Cr 120, Fe 68)
AFLOW magnetic and Alexandria not	362
Both magnetic	44
Alexandria magnetic and AFLOW not	2
From Alexandria’s high_throughput/heusler set	293
From substitutions/batch-001	57
From high_throughput/tI10	11
OQMD covers / closer to AFLOW than to Alexandria	225 / 216
MP covers / closer to AFLOW than to Alexandria	6 / 6

In 643 of the 3,794 cases Alexandria settles, the two DFT databases agree and the models are wrong. These are genuine model errors. The models share these errors, because a discrepancy requires a tight consensus among them (Methods I) that sits more than 0.3 eV/atom from AFLOW. The errors concentrate on the 4d/5d refractory metals. Of the 643, 437 (68%) contain a 4d/5d refractory element, compared with 874 (28%) of the 3,151 on which the models are right. Platinum appears in 139 of the 643 and in 182 of the 3,151, iridium in 79 and 127, and several other refractory metals are over-represented as well (Supplementary Note 13). Manganese and chromium are comparatively rare among the errors (13% and 11% of the 643, compared with 45% and 29% of the 3,151), so this is not a magnetic-3d effect.

One class of bad reference belongs to the models. The third-database check above cannot see this class, because it asks only whether a model is right about a compound’s energy, not about the elemental reference the model subtracted to get that energy. MACE-MPA places a diamond-Fe cell 0.94 eV/atom below every sound Fe structure (the cell also collapses 30% in volume). DPA-3 places a bcc Gd cell more than an electron-volt below its own hcp Gd (the cell expands 59%), and eSEN places a Cs cell about 3.6 eV/atom below its sound Cs structures (the cell expands 44%). UMA (Pd) and eSEN (Ti, Sn, V, Ga) also score isolated atoms from AFLOW’s unary set below the bulk crystal. Each model puts a minimum where none belongs, and because the elemental reference is the lowest energy a model assigns to that element, one such structure mis-references every compound containing that element.

No third database can settle these cases, because AFLOW is right, and so are the model’s compound energies. Only its elemental reference is wrong. We find these structures with the elemental-reference screen (Methods). The leaderboard section shows what removing them does to the raw error of MACE-MPA, eSEN, and UMA.

Three controls thus take 15,278 apparent model failures to 643 confirmed model errors, and leave 7,067 unsettled. Excluding the 10,861 discrepancies that remain after re-referencing, or all 15,278, lowers every model’s trimmed MAE by 2.5–4.0 meV/atom. On the structures that define the floor, the models and the floor drop by similar amounts (1.8–2.9 and 1.8–2.4 meV/atom), so no ranking and no comparison with the floor changes (Supplementary Note 14).

Generalization

A model trained on a very large dataset might top the leaderboard because it generalizes, or only because it has already seen structures in the test set. How these potentials behave under such distribution shift is under active study [20]. AFLOW’s prototype libraries let us tell the two cases apart. A structure is familiar to a model if it appears in that model’s training corpus, and novel if it does not. Detecting familiarity across a large database is the hard part. We use the same test as for the noise floor: a structure is present if StructureMatcher matches it to a training structure with the same space group. We compare every benchmark structure with every training structure of the same reduced formula, so no match is missed for want of a lookup key (Methods J).

Familiarity is a property of each model, so we tag each structure separately for each model, against public corpora that stand in for that model’s training data (Methods J). The familiar fraction separates cleanly by training corpus. Of our alloys, 48.1% are familiar to the four models trained on the Materials Project altogether (MACE-MPA, eqV2, eSEN, SevenNet-MF), 39.6% to DPA-3, trained on OMat24 and the Materials Project, 38.9% to the four trained on OMat24 alone, and 2.3% to the five trained on the Materials Project only. For OMat24, whose training rows are perturbed configurations, we match against the Alexandria structure each row was generated from. Of the 1,419,259 scored benchmark structures, 552,041 match one, and every one of those also matches an Alexandria structure (Methods). A single pooled figure would be correct for the first four models and wrong for the other ten, by up to 46 percentage points. About half or more of the benchmark is novel for every model (52% for the four trained on the Materials Project and Alexandria, 60% for DPA-3, 61% for the four trained on OMat24 alone), and nearly all of it (98%) for the MP-trained ones.

To measure how much accuracy a model loses on a structure it has never seen, we compare the familiar structures with every novel one, whose structure is not in the model’s training data. Most novel structures have a composition that is in training, with a different structure. Fewer have a new composition in a chemical system that is in training, and for the models trained without OMat24 or Alexandria many stay in chemical systems absent from training altogether (Methods J). For the OMat-trained models, the mean error is 16–29 meV/atom on familiar structures and 5.3 to 11.6 meV/atom higher on novel ones (Fig. 7). eqV2 loses the least (+5.3) and eSEN the most (+11.6), and the MP-trained MACE-MPA loses 15.6. The gap is largest, 12.3 to 22.1 meV/atom, on the novel structures with a new composition in a familiar chemical system, 8–12% of each OMat-trained model’s novel set, because those sit far above the hull (below).

The familiar-structure error of ∼24 meV/atom sits inside the trimmed-mean band at which independent DFT databases agree (16–30, noise-floor section). A model that exactly reproduced the DFT it was trained on would score the same. The familiar structures alone therefore cannot separate memorization from skill, but the novel ones can, because memorization would show up as a large jump in error on novel structures. The rise we see is real but moderate, a quarter to three fifths of the familiar-structure error, and far smaller than that of the models trained without OMat24 or Alexandria.

The MP-trained models lose more, CHGNet most at +78, but they were the least accurate to begin with. Memorization does not explain what separates the OMat-trained models from the MP-trained ones, since the largest-data models have the lowest error of the panel on novel structures.

Changing the reference database also tests generalization. We ran all fourteen models on 34,681 metallic compounds that appear in OQMD but not in AFLOW (Methods), and scored them against OQMD, a second independent PBE database. After a per-element re-alignment fit on the scored compounds themselves (Methods J) and the same trimming as above, the same models lead on the 12,409 compounds whose elements carry unary references. eqV2 (15), eSEN (17), UMA (20), and SevenNet-MF (21 meV/atom) lead, as they do on the re-referenced AFLOW leaderboard, and the first-generation MP-trained models trail far behind (M3GNet 61, CHGNet 63). About two fifths of the scored compounds (4,821 of 12,409) have a structure that appears in none of the models’ training sets. On those compounds every model is less accurate than on the 7,588 seen ones, by 1.3 to 17.7 meV/atom (1.3 to 14.9 for the models trained on OMat24 or Alexandria), and the ranking barely changes (Spearman $\rho$ 0.978 with the full set, with the top five in the same order; eqV2 15.8, eSEN 20.7, UMA 22.0 meV/atom). The test does not rule out chemical familiarity, because the elements and neighboring compositions are in training.

A direct test bounds memorization itself. On the OQMD compounds that only Alexandria contains, we compare how much the four models trained on Alexandria (MACE-MPA, eqV2, eSEN, SevenNet-MF) improve over never-seen compounds with how much the OMat24-only models, which never saw these compounds, improve. None of the three

ways of matching composition detects a memorization advantage (−1.4 [−2.2, −0.6], −0.1 [−0.9, +0.6] and −1.2 [−2.3, −0.2] meV/atom), so the data bound any such advantage at under 1 meV/atom (the largest upper limit is 0.6). The same test does detect a training-set advantage where one exists. On Materials Project compounds, models trained only on the Materials Project improve by 5 to 15 meV/atom more than OMat24-only models (range over the three ways of matching composition; Supplementary Note 15). Having seen a compound in Alexandria therefore does not make a model predict it measurably better.

The alignment itself exposes a convention. Every one of the fourteen models needs the same ∼0.7 eV/atom shift on Tb compounds, an OQMD-vs-AFLOW reference convention that the alignment removes identically for all.

The complementary test scores the models on data they did train on. We ran all fourteen on 50,002 configurations from OMat24’s held-out validation split (Methods) and scored forces only. OMat24’s VASP settings differ from the Materials Project’s by 13.5 meV/atom, more than the entire published energy-MAE range, so its energies cannot be compared without re-referencing. The models separate completely by training corpus. The eight OMat-trained models score 0.055–0.082 eV/Å and the six MP-trained ones 0.135–0.254, and the closest pair across the divide is still a factor of 1.6 apart. This test measures accuracy on the training distribution itself, which the familiar/novel comparison cannot reach. The split holds out individual configurations, not whole materials. Distortions of the same crystal mostly sit in the training set, so each model has seen close relatives of everything it is tested on.

On this test UMA leads at 0.055 and eqV2 follows at 0.061, so the model that tops our independent-reference leaderboard does not top OMat24’s own held-out data. As the data move toward its training distribution, eqV2’s advantage does not grow, as a memorization advantage would. The order does depend on how many strongly distorted frames the set holds (Supplementary Note 15). Forces make this argument in a way energies could not, because no elemental reference convention affects them.

Supplementary Note 15 breaks this test down by reference force and by how the frames were generated.

The error is not spread evenly across the benchmark. Because AFLOW’s libraries are combinatorial enumerations of prototypes, the formation-energy distribution of the structures peaks near zero but skews strongly toward unfavorable energies, with a median of +172 meV/atom and 71% of structures above zero (Figure 8).

We bin every structure by its distance above the hull and track each model’s median error (Figure 9). The relationship is tight, shared, and close to monotonic. Error stays low and flat for the near-hull structures, where the best models reach or fall below the 6–8 meV/atom floor, then rises by a factor of 1.6 to 3.3 toward the most metastable structures. The model ordering is preserved across the range. The same rise appears against raw formation energy, only less cleanly, since $E_{\mathrm{hull}}$ is the sharper axis. The rank correlation runs 0.84 to 0.99 for eleven of the fourteen models, and 0.87 to 0.99 for all fourteen once the sparse bins above +2 eV/atom are excluded (Supplementary Note 15; Supplementary Fig. 2).

Every model’s error rises with distance above the hull, and two explanations compete. Far from equilibrium the PES is more anharmonic and near-degenerate, so these structures may be intrinsically harder. Even a familiar prototype is also rarely seen at such an unfavorable composition, so the training data may be less relevant there. This curve cannot separate the two, and the matched comparison below separates them only in part. The rise also depends on the sign of the energy, not only its size. At the same $|E_f|$, about 1 eV/atom, the models are 1.6 to 4.7 times worse (median 2.8×) for unfavorable compounds ($E_f > 0$) than for stable ones ($E_f < 0$).

We separate familiarity from hull distance by comparing familiar and novel structures at matched $E_{\mathrm{hull}}$ and number of elements (Methods J). For the eight OMat-trained models, the gap falls only to 4.3–11.0 meV/atom (eqV2 from +5.3 to +4.3, eSEN from +11.6 to +10.3, UMA unchanged at +7.9). On the novel structures with a new composition in a familiar chemical system, the same matching takes the 12.3–22.1 meV/atom gap down to 2.9–10.4. That tier’s gap came largely from where those structures sit, far above the hull, where every model is worse.

Over all novel structures, hull distance accounts for little of the OMat-trained models’ gap (at most 20%, and Orb-v3’s gap grows slightly under matching). A residual of 4.3 to 11.0 meV/atom remains for every one of them, each with an interval that excludes zero. Once hull distance and the number of elements are matched, these models are thus 4 to 11 meV/atom less accurate on novel structures than on familiar ones. Matching on formation energy and the number of elements instead leaves 4.4–11.7 meV/atom. Balancing element content as well as hull distance and the number of elements leaves 6.8–10.4 meV/atom for seven of the eight and +2.7 [+2.1, +3.2] for eqV2 (Supplementary Table 20), so no control we tried removes the gap of any model trained on OMat24 or Alexandria. Supplementary Note 15 gives the gap under every matching we tried.

The five models trained without OMat24 or Alexandria need one more step. Their familiar sets are small (about 33,200 structures here), and 60% of those structures are elements. Far above the hull the familiar side is therefore mostly high-energy elemental polymorphs, and matching on hull distance alone compares elements against compounds. For CHGNet that comparison even reverses the sign of the gap. When we match on hull distance and the number of elements together, CHGNet keeps 43 meV/atom, SevenNet-0 29, MatterSim-1M and M3GNet 12 each, and MatterSim-5M 10, every interval excluding zero. Matching removes a third to over two thirds of these gaps (35–71%). A model trained without OMat24 or Alexandria loses 33 to 78 meV/atom on novel structures before matching, far more than an OMat-trained one. These models’ remaining gaps are less certain. M3GNet’s interval includes zero once element content is also balanced, as do SevenNet-0’s and CHGNet’s on the same-system subset (Supplementary Table 20).

As causes of the rise, we can rule out reference noise, and training sparsity only in part. Alexandria, a major training source, samples AFLOW’s unfavorable metallic structures about as heavily as AFLOW does (Figure 8). But coverage by Alexandria and the Materials Project falls with distance above the hull, from 84% of structures within 25 meV/atom of it to 43% at 400–1,600 meV/atom (Supplementary Note 15). Measured the same way, the DFT-vs-DFT floor stays flat in its 5–10 meV/atom band across the entire $E_{\mathrm{hull}}$ range (Figure 1), so the rising error belongs to the models, not to the benchmark, and every model shows it.

Near the hull, where the typical error is smallest, the leading models are as close to AFLOW as a second DFT database is (Figure 10). eSEN (median 2.6 meV/atom), MACE-MPA (3.5), and eqV2 (3.8) are lowest, and UMA, Orb-v3, and MACE-OMAT reach 5.3–6.1. The tails are alike, with each of these models missing by more than 100 meV/atom on 0.1–0.2% of its near-hull structures. As an absolute yardstick, Figure 10 also shows the same distribution for AFLOW against Alexandria (median 5.0, mean 7.2 meV/atom). Every model is scored on the same 29,175 near-hull structures of the common set (Methods G), and the yardstick on the 25,546 of them that Alexandria also holds, so the models and the yardstick are measured on nearly the same population. Even the two DFT databases disagree by more than 100 meV/atom on a few of these structures (36 of 25,546), the same bad-reference class the bad-references section examines.

The degradation on novel structures is real for every model, and for the OMat-trained models it is moderate, 5 to 12 meV/atom before matching and 4 to 11 after. A cohort that had merely memorized its training set would not stay within 5 to 12 meV/atom of its familiar-structure error on a set that is about half or more novel for every model. The loss of accuracy is also smallest where accuracy matters most. Within 100 meV/atom of the hull, among the thermodynamically plausible phases a discovery campaign actually pursues, novel structures are predicted only 0.5 to 3 meV/atom worse than familiar ones. The loss grows to 6 to 20 meV/atom only 400 meV/atom or more above the hull, among strongly metastable structures a screen would discard anyway.

Discussion

The apparent failures of the leaderboard through generalization sections sort into five kinds a benchmark must tell apart: a reference convention, an error in the reference itself, an irreducible disagreement between two DFT codes, an artifact of the benchmark’s own analysis, and a genuine model limitation. The third kind is what the floor measures, where beating it would mean fitting one DFT code’s settings, not a better PES. The magnetic 3d discrepancies are of the first two kinds, a per-element offset or an error in one database’s entry (for Alexandria, usually a missed magnetic state). The +U shear cases are a mismatch between the Hamiltonian of the reference and that of the training data, which no floor pair measures, since none carries $+U$.

Only the fifth kind, a genuine model limitation that neither re-referencing nor a protocol control can remove, can be charged to the model. A disagreement earns that label only after a reference error and a protocol mismatch have both been ruled out. Table 5 gathers the controls that do the ruling out.

Control	What it guards against	Section	What it revealed
Independent reference (AFLOW)	Memorization posing as accuracy	Benchmark	52–98% of the alloys are novel to a model, depending on its training corpus, so the benchmark measures transfer at least as much as recall
DFT-vs-DFT noise floor	Scoring error against zero	Noise floor	Independent raw-PBE databases agree to only 6–8 meV/atom on the typical structure; below that scale a ranking is not transferable
Floor and models under the same trimmed mean, on the same structures	Comparing a median floor with mean model errors	Noise floor	Under the trimmed mean the floor is 16–30 meV/atom, not 6–8; after re-referencing, the leader sits 3 below it, eSEN 2 above, and UMA, SevenNet-MF, and GRACE-2L 5–7 above
Clean metallic cut (no oxides or nitrides)	A chemistry confound, anion and +U corrections posing as a metallic convention	Noise floor	MP sits 18.0 meV/atom from AFLOW on a typical structure (the raw-PBE databases 6–8), and a per-element offset removes little of it (in-sample R² = 0.06)
System-resampled bootstrap	Pseudo-replication, near-duplicate decorations posing as independent evidence	Leaderboard	Confidence intervals no wider than ±2 meV/atom
Single-point scoring at the reference geometry (all fourteen models)	A protocol asymmetry: the direct-force leader barely relaxed, while thirteen models absorbed relaxation drift	Leaderboard	Every model improves by 4–25 meV/atom; the leader’s margin over eSEN shrinks from 7 to 0.4 meV/atom
Rung-0 per-element re-referencing (fit on 70% of the chemical systems, scored on the other 30%)	Error linear in composition (a reference convention, a broken reference, or element-wise bias) counted as structure-dependent error	Leaderboard	1–3 meV/atom for nine of the twelve models; 7–54 meV/atom for the three trained without Alexandria or OMat24 (CHGNet 127→72); the group behind the leader spans 4 meV/atom
Symprec tolerance sweep	An analysis parameter posing as a fidelity difference	Reference-free	The spread in retention across models (eqV2 aside) shrinks from 14.5 points to 3.6
Reference-Hamiltonian stratification (drop +U structures)	A Hamiltonian mismatch posing as a model failure	Reference-free	Shear agreement recovers for every model (log-space r 0.85–0.98 without +U; MACE-MPA 0.92 → 0.98)
Elemental-reference screen (S0 isolated atoms, S1 low outlier, S2 volume change)	A model’s own pathological elemental structure setting the reference for every compound containing that element	Benchmark	Ten references across five models; removing them lowers raw MAE by 15 meV/atom for MACE-MPA, 13 for eSEN, and 7 for UMA
Consensus test (the models agree and AFLOW stands apart)	Blaming the model when the reference is the outlier	Bad references	Finds 15,278 discrepancies (structures where AFLOW sits far from a tight model consensus) among 1.42M structures, 10,861 after the per-element offset; the count changes smoothly with the threshold
Third-database check (Alexandria, OQMD)	An unresolved model-versus-reference disagreement	Bad references	Where Alexandria has the same structure (3,794 discrepancies), it sides with the models in 83% of cases (80% when the whole test is rebuilt on the six MP-generation models); 643 are genuine model errors, concentrated in 4d/5d refractory metals
Cross-database validation (OQMD-only compounds)	A conclusion that is AFLOW-specific	Generalization	The same leading models on the 12,409 OQMD compounds absent from AFLOW whose elements carry unary references (4,821 also absent from every training set), so the lead reflects the models and not AFLOW

Below the leaders, the rankings on forces, elastic moduli, and symmetry retention differ from the ranking on energy, while energy and stability largely agree (reference-free section). No single leaderboard therefore stands in for the rest. Multi-metric reporting is not new (Matbench Discovery already reports several statistics of its stability task). What is new here is that these axes measure different properties and are scored against a reference no model trained on

Discussion

Cost is an axis of its own, and it interacts with the choice of metric (Figure 11). The efficient frontier shifts from panel to panel, so the cost-effective model depends as much on the budget and the target property as on any single leaderboard.

On most of those axes (Fig. 12), the strongest predictor of where a model lands is its training set. The OMattrained models lead the energy ranking and the reference-free metrics, degrade least on unseen structures, and share their errors. Two models trained on the same corpus make more similar errors than two trained on different corpora (mean Spearman $\rho$ 0.61 versus 0.40 over the 45 pairs of ten models; no other way of assigning four of the ten to the MP group gives as large a gap, $p = 1/210 = 0.005$; Methods I). Readers can add their own model to Fig. 12. The scripts in scripts/byom/ run any model with an ASE calculator through the same protocols and redraw the figure with it included, on a subset that runs on a workstation or on the full benchmark (Code availability).

	energy raw ↓	single-point ↓	SP + Rung-0 ↓	hull-F1 ↑	$E_{\mathrm{hull}}$ MAE ↓	polym. $\rho$ ↑	force (eV/Å) ↓	shear log-r ↑	G @0.1Å ↑	SG ret.
eqV2	18.3 ±0.4	14.4 ±0.3	12.7 ±0.6	0.921 ±0.010	18.7 ±0.5	0.983 ±0.000	0.0004 ±0.0000	—	0.991 ±0.000
eSEN	25.5 ±0.5	14.8 ±0.3	13.3 ±0.6	0.893 ±0.010	26.5 ±0.6	0.949 ±0.001	0.022 ±0.0004	—	0.921 ±0.001
SevenNet-MF	28.2 ±0.5	19.4 ±0.3	17.9 ±0.6	0.868 ±0.011	29.0 ±0.5	0.942 ±0.001	0.044 ±0.0010	—	0.922 ±0.001
UMA	28.5 ±0.4	17.9 ±0.3	13.3 ±0.4	0.872 ±0.012	27.1 ±0.5	0.944 ±0.001	0.040 ±0.0007	0.983 ±0.005	0.911 ±0.001
GRACE-2L	29.8 ±0.5	21.0 ±0.3	20.0 ±0.6	0.843 ±0.014	31.4 ±0.5	0.938 ±0.001	0.042 ±0.0009	0.984 ±0.005	0.922 ±0.001
MACE-OMAT	32.7 ±0.5	24.2 ±0.4	22.2 ±0.7	0.821 ±0.015	34.0 ±0.5	0.925 ±0.001	0.045 ±0.0010	—	0.915 ±0.001
MACE-MPA	33.0 ±0.5	21.6 ±0.3	21.2 ±0.7	0.839 ±0.012	35.5 ±0.6	0.920 ±0.001	0.047 ±0.0011	0.977 ±0.007	0.915 ±0.001
DPA-3	33.0 ±0.5	23.9 ±0.3	22.3 ±0.6	0.815 ±0.015	34.0 ±0.5	0.928 ±0.001	0.051 ±0.0010	—	0.921 ±0.001
Orb-v3	35.8 ±0.5	20.2 ±0.3	17.8 ±0.6	0.837 ±0.012	35.8 ±0.6	0.915 ±0.001	0.045 ±0.0009	0.983 ±0.006	0.903 ±0.001
MatterSim-5M	52.0 ±0.5	39.4 ±0.4	31.6 ±0.7	0.736 ±0.016	53.0 ±0.6	0.883 ±0.001	0.052 ±0.0012	—	0.922 ±0.001
MatterSim-1M †	61.9 ±0.6	48.4 ±0.4	44.9 ±0.7	0.634 ±0.022	59.5 ±0.6	0.834 ±0.002	0.089 ±0.0023	0.938 ±0.034	0.887 ±0.001
M3GNet	63.3 ±0.6	50.1 ±0.5	46.1 ±0.8	0.616 ±0.022	62.7 ±0.6	0.827 ±0.002	0.096 ±0.0024	0.848 ±0.034	0.886 ±0.001
SevenNet-0 †	87.1 ±1.0	66.1 ±0.8	54.7 ±1.1	0.597 ±0.022	77.5 ±0.9	0.800 ±0.001	0.107 ±0.0029	0.923 ±0.017	0.889 ±0.001
CHGNet	126.5 ±1.1	101.7 ±0.9	60.9 ±1.0	0.448 ±0.021	113.1 ±1.1	0.786 ±0.003	0.098 ±0.0024	0.878 ±0.030	0.890 ±0.001

The panel’s one same-recipe pair tests this claim. eSEN and eqV2 were trained on the same corpora (OMat24 pre-training, MPtrj + sAlex fine-tune) but differ in architecture, scale, and force objective. With both controls applied, their energy gap closes to 0.6 meV/atom (leaderboard section), so the raw gap came mostly from relaxation drift. eqV2 still leads eSEN on hull-F1, by 0.028 after relaxing (0.921 versus 0.893) and narrowly at AFLOW’s geometry (0.930 versus 0.923). The data do not say whether architecture or closeness to AFLOW’s protocol is the reason. On polymorph ordering the two tie at AFLOW’s geometry, so eqV2’s relaxed lead there is a relaxation effect. The OMat24 test agrees. Within the OMat tier, architecture spreads the force error only 1.5-fold, compared with a gap of at least 1.6-fold between tiers, and UMA, not eqV2, leads there. For model developers, training data has the larger effect, but it is not the only one.

Judging the next generation well takes three inexpensive habits. Score against at least one DFT reference the models were not trained on, so the test measures transfer rather than recall. Report the error before and after re-referencing, since the difference bounds how much of it a convention mismatch could explain. Finally, compare with a measured DFT-vs-DFT floor, quoted with the same trimmed mean as the model error and on the same structures, so a 7 meV/atom gap is not mistaken for skill when two careful codes disagree by that much on typical structures.

Re-referencing also matters to anyone who uses these potentials. A per-element shift cancels exactly in any comparison at fixed composition, such as polymorph ordering, so for those uses the re-referenced error is the one a user meets. It does not cancel when a model’s energies are compared with a database’s. A candidate relaxed with a potential and judged against the Materials Project’s or AFLOW’s convex hull mixes two conventions. The databases themselves re-reference routinely [?, ?], and a user can remove the same mismatch the same way, with one constant per element fit to a set of DFT calculations in the chemical system of interest. For the three models trained without Alexandria or OMat24 (MatterSim-5M, M3GNet, CHGNet) that step removes 7 to 54 meV/atom; for the other nine, 1 to 3. Rung-0 also tells a user which remedy a model needs. An error that is mostly per-element is removed by a few constants. An error that survives re-referencing is structure-dependent, either a PES error or a protocol difference of the kind the floor measures, and only fine-tuning or better training data reduce it.

The same caution applies to “novelty.” On databases this large, a structure a model has never seen is still almost always close in composition and chemistry to ones it has, so a truly out-of-distribution test is nearly impossible to construct. The generalization we can actually measure is transfer to an independent DFT reference, not escape from the training distribution.

Our benchmark and Matbench Discovery agree at tier level (the models atop Matbench Discovery are the models atop this leaderboard) and disagree exactly where our floor puts the limit of a transferable ranking. Matbench Discovery’s leader, eSEN [25], and its same-recipe sibling eqV2 swap places here.

The benchmark has limits. It uses one reference database (AFLOW) and one functional (PBE), so every conclusion is about agreement with raw-PBE DFT, and we drop the elemental entries AFLOW computed otherwise (benchmark section and Methods). It covers only ordered metallic structures (benchmark section), except for the elastic set, which includes nonmetals (reference-free section). The shear test cannot separate $+U$ from element identity or correlated-electron effects (reference-free section). Of the discrepancies in the bad-references section, 7,067 have no same-crystal Alexandria entry, so their cause is unsettled. Rung0 bounds the effect of a convention rather than measuring it, because it cannot tell a convention from an element-wise bias (leaderboard section). Familiarity is tagged against public stand-ins for each model’s training data, some of which hold more than a model saw, and it depends on the tolerances of the structure match (generalization section; Methods J). The elastic task ran on the original eight-model panel (Methods A).

The next generation therefore has a clear, narrow target. Re-referenced, the best model sits 3 meV/atom below the floor, eSEN 2 above it, and UMA, SevenNet-MF, and GRACE-2L 5 to 7 above it. What still separates the models lies far above the hull. Every model’s error rises with distance above the hull (Figure 9), where training coverage thins out, so that is where targeted data on an independent reference could help most. As the models improve past the floor, a gain will be visible only if the comparison follows the three habits above and controls the protocol before crediting the model. We hope the controls assembled here make that easier for the benchmarks that follow.

Data availability. Everything the analysis reads is archived at Zenodo (DOI: 10.5281/zenodo.21301563): the per-structure benchmark records for every structure–potential pair, stored as JSONL with one row per pair keyed by AFLOW AUID and model, carrying energies, relaxation metrics, and space groups; the matching records for the elastic subset, the symmetry-tolerance sweep (which also stores every model’s relaxed structure), and the OQMD cross-database test; the per-structure formation-energy table derived from them; the test-set manifest; the same-crystal match tables (every StructureMatcher match from the exhaustive search, with the space-group test applied) behind the noise-floor and overlap analyses; and the prototype-fingerprint tables they are compared with. AFLOW, Alexandria, OQMD, and Materials Project data were obtained from their public APIs and bulk downloads (AFLOW via the AFLUX REST API), and OMat24 from its release [7]. The exact queries and the versions of the bulk downloads are recorded in the archive.

Code availability. All analysis and figure-generation code, per-model environment lockfiles, and the model-checkpoint provenance are archived in the same Zenodo record (DOI: 10.5281/zenodo.21301563). Every data figure regenerates from the summary tables in the archive with the provided scripts, and every value in Tables 2–4 is read or computed from them. The scripts that compute those summary tables from the per-structure data are included. Several of them need cluster-scale resources. The scripts in scripts/byom/ benchmark any model with an ASE calculator under the same protocols and redraw Fig. 12 with it included, on a stratified subset that runs on a workstation or on the full benchmark.

Funding. The authors received no funding for this work.

Author contributions. G.L.W.H. conceived and designed the benchmark and its controls-first methodology (the independent-DFT reference, the DFT-vs-DFT noise-floor calibration, and the held-out per-element re-referencing, Rung-0), and led the relaxation-fidelity, elastic, and $+U$ analyses, the analysis and figure pipeline, and the original draft. J.N.P. led the convex-hull stability and training-overlap generalization analyses, contributed the leaderboard-table figure, drafted portions of the Methods and Supplementary Information, and provided an extensive section-by-section review of the manuscript. Both authors interpreted the results, reviewed and edited the manuscript, and contributed equally to this work.

Competing interests. The authors declare no competing interests.

Methods

The subsections trace the pipeline in the order it runs: build the test set, relax, reference, score, then apply each control. Every “(Methods)” pointer in the main text lands on the procedure that produced its number.

The benchmark: reference, test set, and potentials

The reference is three parts of AFLOW [5] [21]: the binary (LIB2) and ternary (LIB3) prototype libraries for the ordered intermetallics that are our core test, the unary entries for elemental reference states, and the Automatic Elasticity Library (AEL) for the DFT elastic tensors of Methods H [22]. For each structure we take AFLOW’s DFT-relaxed geometry and its formation enthalpy per atom (enthalpy_formation_atom) as ground truth. AFLOW’s LIB2/LIB3 alloy libraries carry no post-hoc correction (raw PBE). We access AFLOW through its AFLUX REST API [61], keyed by each entry’s AFLOW unique identifier (AUID). On the exhaustive fetch, the inline geometry matched the CONT-CAR to within 0.002%.

The test set has a metals-only core, and to enter it a structure must pass four filters: every element in the 46-element METALS set of scripts/build_testset.py or one of twelve metalloids/nonmetals (B, C, N, O, Si, P, S, Ge, As, Sb, Se, Te); fewer than half its atoms nonmetals; an arity (number of elements) of one to three and a reduced cell of at most 40 atoms (60 where AFLOW elastic data exist); and no Hubbard U. Hubbard U enters only the labeled extended strata and the lanthanide probes: all of E2 (19,882) and its elastic subset (2,749), 4,857 of the 9,153 E3 structures, 105 of the 884 lanthanide unaries, and one lanthanide alloy, 1.9% of the benchmark in all. No energy metric includes a +U structure. Within that scope we take LIB2 and LIB3 essentially in full: ∼1.45M ordered structures (1,454,458)—∼1.38M binary/ternary decorations at the core (202,724 LIB2, 1,178,736 LIB3), plus the elemental unaries that supply references (∼29k), a lanthanide probe (E3, 9,153), lanthanide unaries (ln_unary, 884), lanthanide alloys (ln_alloy, 3,189), the labeled extended-chemistry strata, the AEL elastic subset, and a 32-structure quaternary set (AFLOW’s alloy libraries stop at ternary). The extended strata are never pooled into headline metrics. E1 holds standard-DFT (no-U) nonmetal compounds, E2 holds +U entries admitted for geometry only (AFLOW reports no formation enthalpy for any +U intermetallic), and E3 is the lanthanide probe (lanthanide intermetallics of any U status). The lanthanide unary and alloy strata (ln_unary, ln_alloy) admit only the trivalent lanthanides (La, Pr, Nd, Gd, Tb, Dy, Ho, Er, Lu; mixed-valent Ce, Sm, Eu, Tm, Yb excluded). Actinides are excluded. We also drop the 1,575 elemental entries that AFLOW computed without plain PBE (LDA, SCAN, older GGA pseudopotentials, RPBE, and one spin-orbit Bi), whose energies are not on the PBE scale (results/analysis/unary_nonpbe_exclusions.csv; Supplementary Note 2). Sampling is monotone: each stratum’s entries are taken in ascending md5 of the AUID up to a cap raised over five waves, so each wave is a strict superset of the last (wave caps in scripts/build_testset.py, superset selection in src/umlip_bench/sampling.py). The same ordering partitions the work across cluster jobs, and the runner skips any structure–potential pair already on disk, so the campaign restarts cleanly across shards, waves, and resubmissions.

The potentials of Table 1 joined the panel in three waves: an original eight-model panel ran the full production campaign including the elastic task, four models joined mid-campaign, and eSEN and eqV2 arrived last. All fourteen run the OQMD cross-check of Methods J. Each model ships an ASE calculator in its own pinned virtual environment, which isolates incompatible dependency stacks (TensorFlow for GRACE-2L, PyTorch/fairchem for the rest). Checkpoint identifiers, package versions, and loaders are in docs/model_provenance.md (Supplementary Note 1).

Model	Trained	Architecture	Training data	Ref.
SevenNet-MF	OMat	E(3)-equivariant message-passing (NequIP-class [27]), multi-fidelity	OMat24 + MPtrj + Alexandria (multi-fidelity)	[28]
GRACE-2L	OMat	Graph atomic cluster expansion (2-layer, tensor)	OMat24	[29]
MACE-OMAT	OMat	Equivariant message-passing, ACE-order messages (MACE)	OMat24	[7] [30, 31]
DPA-3	OMat	Attention-based message-passing (DeePMD / DPA)	Multi-source, incl. OMat24 and MPtrj; OMat24 head	[32]
UMA	OMat	Equivariant message-passing (eSEN) + Mixture of Linear Experts	OMat24 (multi-domain, with OMol25, OC20, ODAC25 and OMC25)	[33]
Orb-v3	OMat	Graph network (non-equivariant), energy-conserving	OMat24	[34]
eSEN	OMat	Equivariant Smooth Energy Network, energy-conserving (30M)	OMat24 pretrain; MPtrj + sAlex fine-tune	[25]
eqV2	OMat	EquiformerV2 equivariant transformer, direct-force (86M)	OMat24 pretrain; MPtrj + sAlex fine-tune	[7] [26]
MACE-MPA	MP	Equivariant message-passing, ACE-order messages (MACE)	MPtrj + Alexandria subset	[30, 35]
MatterSim-5M	MP	Graph network (M3GNet-based)	MatterSim set (MP-consistent PBE+U)	[24]
M3GNet	MP	Graph network (3-body interactions)	MatPES-PBE	[23, 36]
CHGNet	MP	Graph network, charge-informed (uses magnetic moments)	MPtrj	[6]
SevenNet-0 †	MP	E(3)-equivariant message-passing (NequIP-class)	MPtrj	[37]
MatterSim-1M †	MP	Graph network (M3GNet-based)	MatterSim set (MP-consistent PBE+U)	[24]

Relaxation and formation energies

Relaxation always starts from AFLOW’s DFT-relaxed geometry (rationale in the benchmark section), with an identical protocol across all fourteen potentials: ASE FIRE with a FrechetCellFilter relaxing cell and positions together, maximum force 0.02 eV/Å, capped at 500 steps [38, 39]. No symmetry is imposed, so space-group retention (Methods G) is a measured outcome. A guard aborts and records a per-structure failure if the cell volume leaves the window 0.3–6.0× its start or any interatomic distance drops below 0.5 Å (this also converts GRACE-2L’s occasional collapsing-cell neighbor-list blow-up into a recorded failure rather than a node crash). Geometry fidelity is scored against AFLOW with pymatgen’s StructureMatcher [62] (ltol 0.3, stol 0.5, angle_tol 10°, no supercell attempts) and with space groups from pymatgen’s SpacegroupAnalyzer (which wraps spglib [63]) at symprec 0.01 Å. A post-relaxation interatomic distance below 0.7 Å marks a structure collapsed, distinct from the 0.5 Å abort guard.

Formation energies use the model’s own elemental references: $E_f = E/\text{atom} - \sum_{\mathrm{el}} x_{\mathrm{el}} E_{\mathrm{ref}}(\mathrm{el})$, with $x_{\mathrm{el}}$ the atomic fraction and $E_{\mathrm{ref}}(\mathrm{el})$ the minimum relaxed energy per atom over the element’s unary stratum, without the non-PBE entries of Methods A. Candidate ground states (fcc, bcc, hcp, and ASE’s default bulk reference) are seeded at the element’s reference atomic volume (the per-atom volume of ASE’s bulk crystal, or a covalent-radius fcc volume where unavailable, e.g. Mn, Ga) and then fully cell-relaxed, so the seed volume is only a starting point (src/umlip_bench/references.py; production references in analyze.py). Each molecular reference (H, N, O, F, Cl) is a diatomic molecule in a box, relevant only at the extended-chemistry edges. An element whose per-potential reference fails to converge is held on a per-model list, and its structures are dropped for that model on both sides of every comparison, which is why more cross-dataset subsets (Methods J) shrink to compounds whose elements all carry a converged reference.

Elemental-reference screening is needed because the reference is a minimum (benchmark section). A structure can be a reference only if it passes three clauses, applied per model and element and identically to all fourteen models (scripts/screen_unary_refs.py, projected onto results/analysis/unary_blocklist.csv by scripts/build_unary_blocklist.py). S0 rejects a one-atom cell without an AFLOW space group (AFLOW’s isolated-atom references, one atom in a 15 Å cube). S1 rejects any low-energy group that sits more than 500 meV/atom below the rest of that element’s structures and holds at most 20% of them. S2 rejects a structure whose relaxed volume differs from AFLOW’s by more than 25% with its space group retained. The 500 meV/atom gap lies between the largest physical polymorph gap (250 meV/atom; W, Mo, Cr, Nb) and the smallest pathological one (942 meV/atom, MACE-MPA Fe). The 25% bound lies between the α-Sn control (−1.8% under MACE-MPA, −0.3% under eSEN) and the collapsed diamond-Fe basin (−29% to −31%), and raising it anywhere below 29% moves no reference by more than 3 meV/atom. The screen rejects 4,500 structures (3,015 by S0 and 1,485 by S2; S1 rejects none beyond these). It moves ten references by more than 10 meV/atom, across five models (results/analysis/unary_ref_changes.csv): DPA-3’s Gd (+1.02 eV/atom, structures whose volume grows by 59–87%), MACE-MPA’s Fe (+0.94, a collapsed diamond cell), and, set by isolated atoms, UMA’s Pd (+0.34) and eSEN’s Ti, Sn, V, and Ga (+0.15 to +0.29). The rest (UMA’s and Orb-v3’s Cs, eSEN’s Be) move by 10–84 meV/atom. The screen replaces a hand-curated list of 189 structures, which had already removed eSEN’s Cs structure (+44% volume, 3.6 eV/atom below its sound Cs structures).

The single-point control

The single-point control scores every model at AFLOW’s geometry, from the energy each relaxation recorded before its optimizer’s first step (e_initial_per_atom), with no re-relaxation and no new model evaluations (scripts/singlepoint_control.py). It costs nothing, since the optimizer must evaluate the starting point anyway. It uses the same elemental references and the same 1 eV/atom trim as the relaxed column, with the trim applied to the relaxed error so both columns share one row set. The difference between the columns is then the geometry effect alone. As a check, eqV2’s median per-structure $|\text{singlepoint} - \text{relaxed}|$ difference is 0.0 meV/atom: its direct-force head reads AFLOW’s minima as already converged, and 65% of its structures take no optimizer step, compared with 3–5% for eSEN and UMA. Its relaxation drift is 3.9 meV/atom, compared with about 10.6 for those two. That 3.9 meV/atom gap between its two Table 3 columns comes from the 35% in which its relaxation did move.

The trimmed mean and its uncertainty

The trimmed mean is our error statistic. The per-structure error is $E_f(\text{model}) - E_f(\text{AFLOW})$ per atom, and we report a trimmed mean absolute error, the mean of $|\text{error}|$ after setting aside the few structures whose $|\text{error}|$ exceeds 1 eV/atom, which we count and report separately rather than discard. We did not tune the 1 eV/atom cutoff. A sensitivity sweep checks the cutoff (Supplementary Note 6). We apply the same trim to the model errors and to the DFT-vs-DFT floor (Methods E), so both sides of every comparison use the same trimmed mean.

Uncertainty comes from resampling whole chemical systems, not individual structures, so near-duplicate decorations do not make the intervals look narrower than they are. Each metric is computed once per system, then bootstrapped by resampling systems with replacement (B = 2000, percentile 2.5/97.5, fixed seed; scripts/bootstrap-ci_all.py). Hull, polymorph, and energy metrics all resample the same unit (the chemical system). Hull and polymorph metrics use the common set C of Methods G. Energy metrics use every row that passed the trim.

Conventions used throughout the paper are these. In Table 3 and wherever the text quotes an energy MAE, every energy metric uses all benchmark structures with an AFLOW formation enthalpy and a model energy, about 1.42M per model, after the 1 eV/atom trim. A model error is quoted before re-referencing unless labeled after Rung-0 or re-referenced. “All fourteen” means every model of Table 1, and the twelve headline models omit the two superseded checkpoints (SevenNet-0 and MatterSim-1M, marked †). The consensus test of the bad-references section uses the original eight-model panel (CHGNet, MACE-MPA, M3GNet, SevenNet-0, Orb-v3, MatterSim-1M, GRACE-2L, UMA), and the six MP-generation models are the six MP rows of Table 1. The five models of the generalization section trained without OMat24 or Alexandria are those six less MACE-MPA, which also trained on an Alexandria subset, and none of the five trained on OMat24 or Alexandria. The four models trained on the Materials Project and Alexandria are MACE-MPA, eqV2, eSEN and SevenNet-MF. DPA-3 trained on OMat24 and MPtrj but, of Alexandria, only on its two-dimensional subset, so it is tagged against OMat24 and the Materials Project alone (Methods J). Its other training sets were not matched against AFLOW, so its familiar fraction is a lower bound. The four OMat24-only models are GRACE-2L, MACE-OMAT, Orb-v3 and UMA. AFLOW entries were fetched through AFLUX in five waves between 11 and 17 June 2026 (cluster logs logs/testset_build*.log, wave4_fetch.log). Alexandria energies come from release 2024.12.15, OQMD’s from the qmpy v1.8 database dump, and the Materials Project’s from a June 2026 query. Structure matching against Alexandria uses the 2024.12.15 structures, the release the energies come from. Only the OMat24 parent structures absent from that release come from release 2025.07.02 (Methods J). The medians of the floor and of the other check use all pairs, before the trim, while the binned model medians of Figure 9 and Figure 10 use the rows of the hull metrics.

The DFT-vs-DFT noise floor

Matching treats two structures from different databases as the same crystal if pymatgen’s StructureMatcher [62] matches them directly at its default tolerances (ltol 0.2, stol 0.3, angle_tol 5°, primitive cells, no supercell), in the sense that get_rms_dist returns a mapping, and their space groups, assigned by pymatgen’s SpacegroupAnalyzer at symprec 0.1 Å, are the same. The search is exhaustive within each composition. We compare every pair of structures with the same reduced formula across AFLOW, Alexandria, the Materials Project, OQMD, and the Alexandria parents of the OMat24 training rows, 16.77 million pairs, and the matcher completes every one (scripts/smfam_build_stores.py, scripts/smfam_match.py, scripts/smfam_smsg.py). AFLOW’s own XtalFinder [64] compares structures against AFLOW’s prototype library [65–67]; we use pymatgen throughout so that one tool compares structures from every database. The same-crystal matches between two databases join their structures into connected components. For the floor, each component gives one pair, its lowest-$E_f$ AFLOW structure and that structure’s lowest-$E_f$ direct partner in the other database (scripts/floor_pairs_smsg.py). The space-group requirement removes 100,067 of the 815,539 AFLOW–Alexandria matches. A prototype fingerprint (the reduced formula, the space-group number, and the occupied Wyckoff positions) is a cheaper lookup but a different test. Of the 448,532 metallic AFLOW–Alexandria pairs it would have found 321,725 (72%), missing mostly the same crystal described with different Wyckoff labels, and it proposes 17,344 pairs (5.1% of its 339,202) that are not the same crystal. For AFLOW–OQMD, OQMD–Alexandria, and AFLOW–MP it would have found 42%, 56%, and 72% of the pairs, and 0.7%, 0.5%, and 2.2% of its own pairs are not the same crystal (scripts/floor_smsg_vs_fingerprint.py). At tolerances (ltol/stol/angle_tol) of 0.1/0.2/3°, 0.2/0.3/5° (the default, used throughout), 0.3/0.5/10°, and 0.4/0.6/15°, the AFLOW–Alexandria trimmed MAE is 15.4, 16.4, 19.2, and 20.9 meV/atom, and the median stays between 5.7 and 6.1 (results/analysis/floor_smsg_summary*.csv).

Settings differ less than one might expect. AFLOW and Alexandria report raw-PBE formation energies on metals. Alexandria applies the Materials Project’s corrections, but they are zero on every metallic entry. OQMD fits the chemical potentials of 13 elements to experiment [11]. Four of them are metals in our set (Na, Ti, Sn and Hg), and 20% of the AFLOW–OQMD floor pairs contain one of the four. On those pairs the fit shifts OQMD’s formation energies by a median of −23 meV/atom. Removing the shift would lower the AFLOW–OQMD floor from 8.4 to 7.4 meV/atom (median) and from 25.8 to 22.6 (trimmed MAE), and the Alexandria–OQMD floor from 7.8 to 7.0 and from 29.9 to 27.0 (scripts/f305_oqmd_offset.py). We report the floor with OQMD’s energies as published. Supplementary Note 4 gives each database’s settings. No floor pair carries +U (scripts/ldau_floor_sm.py, results/analysis/ldau_floor_sm.csv; every population’s +U fraction is in Supplementary Table 6).

Median, trimmed mean, and checks together define the floor. We report it two ways, each computed exactly as the model error is: the median $|\Delta E_f|$ and the trimmed MAE under the same 1 eV/atom trim (Table 2). Every pair is restricted to metallic prototypes, by the same nonmetal-exclusion set used throughout (Supplementary Note 5). Two checks establish that the floor is not an easy artifact. First, we fit a composition-linear per-element offset between each pair (result in the noise-floor section). We did not attempt richer structure-level corrections. Second, to rule out an easier structure set than the floor’s, we recompute the leader’s error on exactly the floor-defining structures, one per same-crystal component (scripts/floor_structure_matched_downstream.py; results in the leaderboard section). Adding a volume criterion ($|\Delta V/V| \le10\%$) lowers the AFLOW–Alexandria floor further, to 5.7 and 13.3. We do not apply it, because a volume difference between two relaxations of the same crystal is part of the protocol disagreement for the floor measures.

Materials Project needs its own cut. An overlap cut that merely looks metallic is still nearly a third oxides and nitrides, and its removable gap is an oxide effect, so the MP row of Table 2 uses a clean metallic cut (results in the noise-floor section).

Rung-0: a held-out per-element re-referencing

Rung-0 fits one energy constant per chemical element [45] and subtracts it, composition-weighted, from the model’s formation energies. The constants minimize the absolute error, the statistic we score (a least-absolute-deviation fit). The fit uses the element-fraction design matrix with a small ridge term ($\lambda= 10^{-3}$, needed because the fractions sum to one and the matrix is collinear), restricted to the metallic alloy strata (LIB2/LIB3, elemental and lanthanide unaries, and the ternary alloys, and the quaternary set) and to rows with $|\mathrm{error}| < 500$ meV/atom. We evaluate the corrected errors with the same 1 eV/atom trim as the raw leaderboard. The fit is solved by iteratively reweighted least squares (weights $1/\max(|r|,\epsilon)$, with $\epsilon$ lowered from 1 to $10^{-4}$ meV/atom). We confirm that it is a minimum: neither moving any single constant by ±0.01 or ±0.1 meV/atom nor substituting the least-squares, Huber, or one-constant fits lowers the absolute error (scripts/rung0_lad.py). Supplementary Note 7 compares it with the least-squares fit of our first version and with a Huber fit.

The constants are fit on 70% of the chemical systems and scored on the other 30%, with no chemical system in both (scripts/rung0_alloy.py, split materialized by scripts/rung0_hash_split.jl); Supplementary Note 7 gives a five-fold check. Rung-0 removes anything linear in composition, so its gain is a bound on what any change of elemental reference could remove (leaderboard section). We correlate CHGNet’s offsets with the per-element MP−AFLOW and Alexandria−AFLOW reference differences over the 47 of its 49 fitted elements with at least 20 supporting structures in both fits (results/analysis/rung0_chgnet_diagnosis.csv; results in Supplementary Note 7).

Two derived columns follow. “SP + Rung-0” fits Rung-0 on the single-point errors, again held-out, so neither drift nor any per-element offset remains. It uses the alloy strata, trimmed on their own held-out single-point errors. “Above floor” is the model’s after-Rung-0 trimmed MAE minus the AFLOW–Alexandria trimmed MAE, with both terms computed on the same structures. These are the held-out structures that define the floor, namely the AFLOW structures of the same-crystal AFLOW–Alexandria pairs of Methods E that fall in the 30% of chemical systems held out of the Rung-0 fit, 125,455–125,947 per model. Rung-0 is fitted exactly as for the “After Rung-0” column. On these structures the floor is 16.7 meV/atom (scripts/floor_sm_rung0.py, results/analysis/floor_sm_rung0.csv). The column’s interval resamples the held-out chemical systems and recomputes the model’s and the floor’s trimmed MAE on the same resampled structures (B = 2000), with the per-element constants held at their fit. Repeating the comparison at the strictest and loosest matching tolerances of Methods E gives the tolerance columns of Supplementary tbl:8 (scripts/floor_sm_rung0_ci.py).

Reference-free metrics

The hull and polymorph metrics are scored against AFLOW on one common set C for every model: the 1,407,872 structures that all fourteen models score, with both the relaxed and the single-point error within 1 eV/atom (scripts/reffree_singlepoint.py, scripts/reffree_common_regen.py). We do not keep, for each model, only the structures it relaxed without a change of space group. That filter would remove exactly the structures a model relaxes away from AFLOW’s and give each model a different set (1.41 million for eqV2, 1.18 million for UMA), so it would select on the outcome. The single-point columns score the same set from the energies at AFLOW’s geometry (Methods C), where no structure can lose its symmetry. The Alexandria ceiling uses every AFLOW structure with an AFLOW energy, so no model enters it.

Convex-hull stability (hull-F1) uses hulls we build ourselves. AFLOW provides its own hull analysis, AFLOW-CHULL [68]; we build every hull with one solver so that AFLOW’s and each model’s hulls are built identically. For each chemical system we build the formation-energy hull twice—from AFLOW’s energies and from the model’s—and label a structure stable if it lies within 25 meV/atom of the hull (hull-F1 as defined in the reference-free section). We also report the MAE of the energy above the hull. We compute the energy above the hull with a direct scipy lower-convex-hull solver [69, 70] over each structure’s whole chemical space, so a ternary is measured against the binary phases of its sub-systems as well as the other ternaries (scripts/hull_proper.py). Each element’s corner is its AFLOW reference energy, $E_\mathrm{f}=0$. An elemental structure that AFLOW places below its reference is scored against the lowest structure of its own element but is not a corner of compound hulls (Supplementary Note 2). The models measure formation energies from their own lowest-energy structure of each element, so holding AFLOW’s corners at the reference gives the AFLOW and model hulls the same corners. A structure is scored only in a system with at least three entries. The solver agrees with pymatgen’s PhaseDiagram [62] to within $10^{-15}$ eV on 80 sampled chemical spaces. The 25 meV/atom window is swept from 10 to 50 meV/atom (Supplementary Note 9). The Table 4 Alexandria row scores Alexandria itself as a “model” against the AFLOW hull, on the same-crystal pairs of Methods E. Alexandria covers only part of the benchmark, so we also score Alexandria and the models on the structures that AFLOW and Alexandria share as the same crystal (scripts/alex_ceiling_shared.py, rerun on these pairs by scripts/dft_ceilings_sm.py and scripts/reffree_common_regen.py).

Residual forces are the maximum absolute force component that a model predicts at AFLOW’s relaxed geometry, where the true forces are essentially zero (fmax_initial in each relaxation record; src/umlip_bench/relax.py).

Space-group retention and the symprec sweep come from re-relaxing every benchmark structure with all fourteen models, with the production protocol on the production shards (scripts/run_shard_symprec.py). We report retention at symprec 0.01, 0.05, and 0.1 Å, and assign the space group of AFLOW’s structure and of the relaxed structure alike with spglib at the swept tolerance (scripts/symprec_reference_sg.py, scripts/symprec_full_retention.py). A structure enters a model’s rate when its relaxation succeeded and both space groups exist; where ties left duplicate rows, the last one counts. Intervals are a percentile bootstrap over chemical systems. A looser tolerance can still give the two structures different higher-symmetry groups, so retention can dip slightly as the tolerance loosens; the largest dip is 0.4 points (MatterSim-1M). Retention is reported as a function of tolerance, never as a single number.

Polymorph energy ordering is scored per composition. For each composition with at least three distinct structures whose AFLOW energies span at least 25 meV/atom, we compute the Spearman $\rho$ between the model’s and AFLOW’s formation energies, check whether the model’s lowest-energy structure has AFLOW’s minimum energy (ties count as hits), and report the median $\rho$ and the ground-state hit rate across compositions (scripts/analyze_polymorph_ranking.py). The OQMD–Alexandria comparison (scripts/polymorph_floor_oqmd_alex.py) pairs the two databases’ structures by the same-crystal rule of Methods E, counts each same-crystal class once, and applies the same 25 meV/atom spread cut to OQMD’s energies, on the 2,565 compositions with at least three shared polymorphs, grouped by reduced composition. Because 1,308 of those have exactly three, we report means, and compare them with AFLOW–Alexandria reweighted to the same set sizes. The AFLOW–OQMD comparison is built the same way, with OQMD’s lowest-energy entry for each structure, on 2,318 compositions (scripts/polymorph_floor_aflow_oqmd.py). OQMD’s fitted elemental references shift every polymorph of a composition by the same constant and so leave its ordering unchanged. To compare models and databases like for like, we also score each model against AFLOW on exactly the structures a DFT pair uses, less any the model did not keep, re-apply the three-structure and 25 meV/atom rules, and score the second database on the same members. Intervals resample chemical systems.

Elastic response

For the AEL subset, AFLOW stores the full DFT elastic tensor. Moduli are curvatures of the energy surface, so the comparison needs no energy reference. We recompute each compound’s tensor with the model by a finite-strain protocol mirroring AEL’s [22]: relax the structure, apply normal strains of $\pm0.5\%$ and $\pm1\%$ and shear strains of $\pm1\%$ and $\pm2\%$, relax the ions at fixed cell for each deformation (FIRE, fmax 0.02 eV/Å, at most 200 steps), and fit the tensor from the model’s analytic stress in the ASE sign convention (no sign flip; a check showed that flipping the sign gives unphysical moduli; pymatgen DeformedStructureSet and ElasticTensor.from_independent_strains; src/umlip_bench/elastic.py).

From the tensor we read the Voigt–Reuss–Hill [71] bulk modulus K and shear modulus G and the Debye temperature [59]. We stratify by U status. Except in the LIB2 and LIB3 alloy libraries, AFLOW applies +U to every compound containing any of 28 elements (Sc–Ga, Nb–Sn and Ta–Au), whatever its anion (Supplementary Table 6 gives each population’s +U fraction). The Materials Project’s +U elements are eight of these, in oxides and fluorides only. Supplementary Note 12 splits the +U structures into oxides and fluorides with and without one of the Materials Project’s eight elements, other anion compounds, and metallic compounds (scripts/elastic_u_split.py).

Population	+U fraction
Full benchmark (1.45M)	1.90%
E2 stratum	100%
E3 stratum	53%
ln_unary	12% (105 of 884)
ln_alloy	1 entry
LIB2/LIB3 alloy libraries, unary, AFLOW∩MP overlap	0%
AFLOW–Alexandria floor pairs (448,532; the AFLOW structure that sets each pair’s value)	0%
AFLOW–OQMD floor pairs (30,336)	0%
AEL elastic set	78% (2,749 of 3,544)
Consensus-test discrepancies (15,278 before re-referencing; 10,861 after the offset)	0%

Correlations are computed in log space (log10 predicted against log10 AFLOW; scripts/elastic_logspace.py), which leaves out the structures a model predicts with a non-positive modulus (32–155 per model on the full set). We also report the log–log slope (ordinary least squares) and the median multiplicative bias (ten to the median of log10 predicted − log10 AFLOW). The AEL ground truth lives mostly on ICSD compounds containing nonmetals, so this comparison is not metals-only. The metals-only recut is in Supplementary Note 12. The finite-strain task was run on the original eight-model panel, which is why it includes the two superseded checkpoints.

Bad references: detection and the third-database check

The consensus test works as follows. For each structure scored by at least six models of the eight-model panel (Methods A), we take every model’s energies after the elemental-reference screen (Methods B) and compute the consensus C (the median model formation energy), the robust inter-model spread S (1.4826× the median absolute deviation), the distance dA = |C − E_f(AF​​LOW)|, and the z-score dA / max(S, 0.02 eV/atom) (scripts/secvi_chain_s_dA_z.py). A model’s formation energy on a structure is discarded as a numerical failure when it differs from AFLOW’s by more than 50 eV/atom, and the 20 structures this leaves with fewer than six models are not scored. A structure on which AFLOW and the model consensus disagree beyond the cut (S < 0.1 eV/atom, dA > 0.3 eV/atom, and z ≥ 5; all strict inequalities on meV/atom values) is a discrepancy. The test scores 1,419,025 structures. Sweeping z from 3 to 20 moves the count smoothly between 16,367 and 3,633, by about 5% per unit around the setting (scripts/flags_vs_threshold_post.py).

Re-referencing fits one offset per element to the signed gap C − E_f(AF​​LOW), as a ridge regression (λ = 10⁻³) on composition over the structures with |C − E_f(AF​​LOW)| < 0.5 eV/atom (scripts/reference_cut_aligned.jl). We subtract the fitted offset and apply the same cut again (S < 0.1 eV/atom, dA > 0.3 eV/atom, z ≥ 5). Two of the structures that remain were not discrepancies before re-referencing. The chain is scripts/secvi_chain_s_dA_z.py and scripts/secvi_chain_structmatch.py.

Third-database checks adjudicate what remains. For each of the 10,861 discrepancies left after re-referencing, we take the lowest formation energy X among the entries of Alexandria, OQMD, and MP that are the same crystal as the discrepant AFLOW structure (Methods E). The database sides with the models if |X − C| < |X − E_f(AF​​LOW)|. As robustness checks we also consult results from a prototype-fingerprint lookup with no structure check, and with a volume check (volumes per atom agreeing within 10%) added to the structure check, and we rebuild the whole chain with a consensus from only the six MP-generation models, none trained on OMat24 (Supplementary Note 13; scripts/secvi_robustness_smsg.py).

The genuine residual is the cases where Alexandria sides with AFLOW are genuine model errors. We count their enrichment in the 4d/5d refractory metals (Ir, Os, Tc, Re, Mo, W, Ru, Rh, Pt, Ta, Nb, Hf) against the cases where the models are right (scripts/secvi_chain_structmatch.py; per-element counts in results/analysis/secvi-post_enrichment.csv).

Corpus correlation is measured on every structure scored by all ten models of the panel that predates eSEN and eqV2 (Table 1 less those two and the two superseded checkpoints), excluding any on which one of them misses by more than 5 eV/atom. On these structures we compare the mean Spearman ρ of the models’ per-structure errors over the 21 same-corpus pairs with that over the 24 crosscorpus pairs (Table 1 grouping), with an exact permutation test over all 210 corpus labelings, whose smallest attainable $p$ is 1/210 (scripts/model_corr.jl, scripts/corpus_contrast_exact.py).

Familiarity, novelty, and generalization

We call a structure familiar if it is present in a model’s training corpus and novel if it is not. A structure is present if it is the same crystal (Methods E) as a training structure, found by the same exhaustive comparison within each reduced formula. We tag every AFLOW structure against Materials Project relaxed structures, standing in for MPtrj, against the Alexandria PBE bulk set, and against the Alexandria parent structure of every OMat24 training row, since the rattled OMat24 frames themselves are triclinic (space group 1) almost without exception (scripts/smfam_match.py, scripts/smfam_smsg.py, scripts/smfam_tags.py; results/analysis/familiarity_smsg.csv, familiarity_fractions.csv). We report familiar fractions over the 1,419,259 AFLOW structures that at least one model scores. Of these, 552,041 match an OMat24 parent, and all of them also match an Alexandria structure. A prototype fingerprint would have tagged 519,479 of the 712,394 structures that match Alexandria and called another 31,851 familiar that are not (results/analysis/familiarity_smsg_summary.json). For the novel-minus-familiar gap of Figure 7, we tag each model against the public corpora that stand in for its training data (scripts/familiarity_per_model.py). The Materials Project stands in for MPtrj and for the proprietary MatterSim training set, and the full Alexandria set for the subsampled Alexandria used in fine-tuning, so a stand-in that is a superset of the training data can over-count familiarity. Pooled tagging would call every model familiar with the same structures and misstate the gaps (on the new-composition tier below, CHGNet’s gap is 52 meV/atom pooled and 73 per model). A novel structure falls in one of three tiers: its composition is in training with a different structure; its composition is new but its chemical system is in training; or its chemical system is absent from training. Figure 7 compares the familiar structures with all three tiers together and marks the second separately. Structures whose tier cannot be determined, and those without an AFLOW $E_{\mathrm{hull}}$, are left out of both groups. The per-model familiar fractions use scripts/dedup_omat24_overlap.py. The sensitivity of the familiar fractions to the matching tolerances is in results/analysis/familiarity_smsg_tolerance.csv.

Matched comparisons contrast familiar and novel structures at matched properties by stratifying them and averaging the within-stratum difference in MAE, weighted by the pooled number of structures in each stratum (scripts/matched_ehull_generalization.py, scripts/r1_2_matched_controls.py). The strata are AFLOW $E_{\mathrm{hull}}$ bins (edges 25, 100, 400, and 1,600 meV/atom), AFLOW $E_{\mathrm{f}}$ bins (edges −400, −200, −100, 0, 100, 200, 400, and 800 meV/atom), and each of these crossed with the number of elements. A stratum enters only when both groups hold at least 200 structures (100 for the five models trained without OMat24 or Alexandria, whose familiar sets hold about 33,200 structures), and we report an estimate only when the retained stratum hold at least half of each group. To match on chemistry over the whole novel set, we reweight the familiar structures by entropy balancing [72] so that the element frequencies, and optionally the $E_{\mathrm{hull}}$-bin and element-count distributions, equal those of the novel set exactly (scripts/r1_2_chemistry_controls.py). We also match exactly on chemical system crossed with $E_{\mathrm{hull}}$ bin (at least 5 structures per group), which 60–65% of the OMat-trained models’ novel structures permit and 2% of the others’. Intervals resample chemical systems (Methods D), with the balancing weights re-solved in each replicate.

Near the hull and across databases, within 10 meV/atom of the hull, we compare each model’s per-structure [error] distribution to the AFLOW–Alexandria disagreement on the same window and bins over the 25,546 of the 29,175 near-hull common-set structures (Methods G) whose Alexandria partner is the same crystal, chosen without reference to any model’s results (scripts/near_hull_yardstick_common.py). The cross-database test runs all fourteen models on 34,681 metallic compounds that were in OQMD but not in AFLOW when the set was built (180 of them match a later AFLOW fingerprint table). It scores the models against OQMD (scripts/analyze_oqmd_aligned_trim.py) on the 12,409 of 34,681 whose elements all carry unary references (Methods B; 12,408 scored for SevenNet-0), with per-element re-alignment fit in-sample on the scored set by least absolute deviation, as in Methods F but without the held-out split, and the same trim. We split the scored compounds by whether they are the same crystal as a structure of any training corpus above (scripts/oqmd_generalization_split.py; results/analysis/oqmd_familiarity_smsg.csv): 7,588 match at least one and 4,821 match none. To bound memorization, a difference-in-differences test compares each model’s improvement on a group of seen compounds, relative to the never-seen ones, between models trained on that group and models that were not (scripts/oqmd_memorization_did.py). For the compounds that only Alexandria contains, the four models trained on Alexandria are compared with the four OMat24-only models; for the Materials Project compounds, the five models trained only on the Materials Project are compared with the same four. The never-seen compounds are reweighted to the seen group’s composition, by exact strata of element count and crystal system or by a logistic propensity, and the bootstrap resamples chemical systems jointly across models.

OMat24 held-out forces are scored for all fourteen models on 50,002 configurations of OMat24’s held-out validation split, a subsample drawn by uniform stride within each sub-dataset so that rattled and molecular-dynamics frames keep their natural proportion (scripts/omat24_subsample.py). Each configuration is a single-point evaluation with no relaxation (scripts/score_omat24.py).

We report the per-component force MAE in $\mathrm{eV}/\text{\AA}$, averaged over configurations after setting aside those whose force MAE exceeds 1 $\mathrm{eV}/\text{\AA}$, overall and binned by reference force and frame type (scripts/omat24_aggregate.py; results/analysis/omat24_force_mae.csv, omat24_force_binned.csv).

Compute cost

Production ran the large models on GPU and the small ones on CPU. For comparability, the cost figure (Figure 11; Supplementary Note 17) uses CPU-measured relaxation time throughout—the median per-structure cost in core-seconds (wall-clock seconds $\times$ allocated cores: 8 for the production array including UMA, 12 for GRACE-2L; scripts/timing_extract.py), with eSEN re-measured on a matched CPU sample (scripts/cpu_timing_sample.py). We omit eqV2: its direct-force head barely relaxes (Methods C), so its relaxation time is not comparable. The full study used about 372,000 CPU core-hours and 830 GPU-hours on Brigham Young University’s research cluster, roughly half for the production relaxations and half for re-relaxing the whole benchmark to measure space-group retention.

Use of AI assistance

The analysis code, figure scripts, and manuscript drafts were developed with the assistance of an AI coding agent (Claude Code, Anthropic; Claude Opus 4.8, Claude Sonnet 4.6, Claude Fable 5, Claude Opus 5, and Claude Opus 5.5, June–October 2026) operating under the authors’ direction. The authors specified every analysis and made every scientific decision, but did not review the generated code line by line. Correctness was established instead by checks that do not depend on reading it. The manuscript’s numbers were given written specifications taken from the text, and separate agents, working from the raw data without seeing the printed values, wrote new code to recompute them. 801 numbers were re-derived this way, and every mismatch was traced through the original scripts and resolved. Afterward, a regenerated result replaced an earlier one only after the new pipeline had reproduced the earlier output exactly, and a separate AI review, instructed to act as an additional referee, compared the manuscript’s claims with the data files. The exhaustive structure matching of Methods E and the analyses regenerated from it came after these audits and passed the same kind of checks: each script reproduced its earlier output before switching to the new pairs, and an independent re-derivation confirmed the new numbers. The authors reviewed every change to the text. Each reported number is produced by a released script and traces to the released data, and environment lockfiles together with a checkpoint-provenance manifest pin the software stack. The authors take full responsibility for the work.

Supplementary Information

Every procedure is in the Methods. The Supplementary Notes below hold the tables and robustness analyses that the Methods subsections and the main-text callouts defer to. The reader’s map below is the index.

Reader’s map—which note answers which callout

Looking for	Where
model provenance: checkpoints, loaders, envs	Note 1
test-set strata, counts, wave manifest	Note 2
relaxation protocol, formation energies, references	Methods B
single-point control, eqV2	Methods C
extended bootstrap intervals	Note 3
database DFT settings, +U exposure	Note 4
floor detail, MP metallic cut, Hegde	Note 5
trim-cutoff sweep	Note 6
Rung-0: leading constants, held-out behavior, CHGNet decomposition	Note 7
the full leaderboard (Fig. 12)	Note 8
hull threshold sweep, ceiling on the shared overlap	Note 9
symprec sweep	Note 10
polymorph detail, Alexandria ceiling	Note 11
elastic no-+U recut, log-space, metals-only	Note 12
discrepancy anatomy, panel robustness, Alexandria’s own bad references	Note 13
“do the discrepancies distort the headline?”	Note 14
$E_f$ binning, matched comparisons, near-hull, OQMD cross-DB, OMat24 forces by force bin and frame type	Note 15
magnetism slice	Note 16
cost / Pareto (Figure 11)	Note 17

Supplementary Note 1. Model provenance (model_provenance.md)

Model selection and the run protocol are Methods A. The table gives each potential’s exact loader, checkpoint, and isolating environment (envs/<name>), rebuilt from the registry src/umlip_bench/potentials.py. Package version pins and checkpoint hashes ship with the released dataset.

Supplementary Table 1. Each potential’s loader, checkpoint, and isolating environment.

Model	Package	Loader / checkpoint	Env (envs/)
SevenNet-MF	sevenn	7net-mf-ompa, modal omat24	sevenn
GRACE-2L	tensorpotential (TF)	grace_fm("GRACE-2L-OMAT", pad_atoms_number=40)	grace
MACE-OMAT	mace-torch	mace_mp(model="medium-omat-0", default_dtype="float64")	core
DPA-3	deepmd	DP(model="dpa3.pth")	deepmd
UMA	fairchem (v2)	uma-s-1p1, task omat	fairchem
Orb-v3	orb-models	orb_v3_conservative_inf_omat (float32-high)	orb
eSEN	fairchem-core 1.10.0	facebook/OMAT24 : esen_30m_oam.pt (OCPCalculator)	fairchem-v1
eqV2	fairchem-core 1.10.0	facebook/OMAT24 : eqV2_86M_omat_mp_salex.pt	fairchem-v1
MACE-MPA	mace-torch	mace_mp(model="medium-mpa-0", default_dtype="float64")	core
MatterSim-5M	mattersim	MatterSim-v1.0.0-5M.pth	mattersim
M3GNet	matgl	M3GNet-PES-MatPES-PBE-2025.2 (stress_unit eV/Å$^{3}$)	matgl
CHGNet	chgnet	CHGNetCalculator (0.3.0)	core
SevenNet-0 †	sevenn	7net-0	sevenn
MatterSim-1M †	mattersim	default MatterSimCalculator	mattersim

† superseded within-family checkpoint. Among the main-text figures it appears only in the closing leaderboard (Fig. 12) and the eight-model elastic comparison (Figure 6); the Supplementary tables list it for completeness. Three reproducibility caveats: the eSEN [1] and eqV2 [2] checkpoints come from the gated facebook/OMAT24 Hugging Face repo [3] and load only through the legacy OCPCalculator in fairchem-core 1.10.0 (a separate environment from UMA’s fairchem v2); MACE runs in float64 because float32 stress noise stalls FIRE near minima; and M3GNet’s stress_unit must be $\mathrm{eV}/\mathrm{\AA}^3$ or ASE cell relaxation breaks silently.

Test-set strata and counts

Test-set construction—scope, filters, strata, and sampling—is Methods A. The strata and counts come from the final manifest (results/analysis/per_stratum.csv).

Stratum	What	Count
LIB2	binary intermetallic prototype decorations	202,724
LIB3	ternary intermetallic prototype decorations	1,178,736
unary	elemental reference states	29,305
E3 (lanthanide)	4f-alloy “edge of validity” probe	9,153
E1 / E1-ael	standard-DFT (no-U) nonmetals	5,818 / 470
E2 / E2-ael	+U entries (geometry only)	19,882 / 2,749
icsd_ael	AEL elastic subset	325 core 2
overlap	ICSD alloys matched to a Materials Project entry	1,191
ln_unary / ln_alloy	trivalent-lanthanide references / alloys	884 / 3,189
quaternary	metal-only 4-component alloys	32
Total		1,454,458

The unary and ln_unary counts exclude the non-PBE elemental entries (see “AFLOW’s elemental references” below).

The elastic comparison of the reference-free section of the main text uses $\sim$3,500 AEL structures across all admitted chemistries.

The manifest accumulated in five monotone waves (Methods A). We selected entries by ascending md5(auid), so each wave is a strict superset of the last.

Wave	Strata introduced	Entries
1	unary, lib2, lib3, icsd_ael, overlap	88,430
2	e1, e2, e3	16,956
3	exhaustive expansion of the prototype libraries	1,343,099
4	quaternary; four- and five-element E1/E2 entries	3,403
5	ln_unary, ln_alloy	4,145
Total		1,456,033

From wave 3 on, geometries are taken inline from AFLUX [4] (positions_fractional plus cell parameters). The earlier waves use fetched POSCARs. After construction we apply the non-PBE exclusion below, which leaves the 1,454,458 of Supplementary Table 2.

Of AFLOW’s ICSD oxides, 95.5% match an MP entry at the same formula and space group (scripts/icsd_oxide_full.py; the scope argument is in the benchmark section). Production ran the 1,456,033-entry manifest against up to fourteen models at a $\sim$0.03% hard-failure rate.

AFLOW’s elemental references. AFLOW measures every formation enthalpy from one fixed reference structure per element (for example $\beta$-Sn, A7 Bi and hcp Cd), and that structure is not always the lowest-energy PBE polymorph. Of the 28,421 PBE elemental structures we score, 1,882 sit below their element’s reference. Most sit there by less than a meV/atom: they are the reference structure recomputed with slightly different settings (bcc Cr at $-3.7$ meV/atom, for example). The 509 that sit more than 1 meV/atom below include genuine lower polymorphs in PBE, as much as 43 meV/atom below the reference: $\alpha$-Sn lies 43 meV/atom below $\beta$-Sn, and $\alpha$-Ga 24 below AFLOW’s Ga reference. AFLOW’s unary library also holds calculations made with other functionals and settings, whose energies are not on the PBE scale: 876 LDA, 212 SCAN, 481 with older GGA pseudopotentials and 5 RPBE entries, some of them as much as 1.09 eV/atom below the reference (LDA Cr). One ICSD Bi entry lies 0.86 eV/atom below the reference in the same A7 structure because it was computed with spin-orbit coupling (LSORBIT in its input) and measured against the non-spin-orbit reference. We remove these 1,575 entries from the benchmark (results/analysis/unary_nonpbe_exclusions.csv). Such structures, $\alpha$-Sn among them, are scored against the lowest structure of their own element but are not hull corners (Methods G). Where AFLOW’s reference is not the lowest PBE polymorph, every compound of that element carries the same offset per atom of it, the per-element convention that Rung-0 removes (leaderboard section).

Extended bootstrap intervals

The system-resampled bootstrap behind every “±” in the paper is Methods D. Further intervals (scripts/bootstrap_ci_phase2.py; same conventions—system-resampled, $B = 2000$, percentiles 2.5/97.5) extend it to the reference-free and elastic metrics.

Metric	Model	Estimate	95% CI
Residual force, median max (eV/Å)	eqV2	0.00038	[0.00037, 0.00039]
Residual force, median max (eV/Å)	eSEN	0.0217	[0.0214, 0.0221]
Residual force, median max (eV/Å)	UMA	0.0401	[0.0394, 0.0409]
Elastic G $r$, all-cut linear	MACE-MPA	0.160	[0.06, 0.89]
Elastic G $r$, no-+U	MACE-MPA	0.965	[0.939, 0.979]
Novel−familiar gap, all novel structures (meV/atom; Supplementary Table 20, Raw)	eqV2	+5.3	[+4.8, +5.9]
Novel−familiar gap, all novel structures (meV/atom; Supplementary Table 20, Raw)	CHGNet	+77.7	[+65.0, +88.7]

The residual-force intervals of the top three models do not overlap. MACE-MPA’s all-cut linear G $r$ is outlier-dominated (the quantified log-space caveat of Supplementary Note 12), while its no-+U recovery is robust. The novel-minus-familiar gaps all exclude zero.

DFT settings of the four databases

The floor comparison rests on the four databases sharing PBE but differing in protocol. We compiled the settings from the primary sources (docs/database_protocol_notes.md).

Setting	AFLOW (LIB2/3)	Alexandria (PBE 3D)	OQMD	Materials Project
XC	PBE	PBE	PBE	PBE(+U)
ENCUT	AFLOW HT-standard	520 eV	520 eV (final static)	520 eV
k-points	AFLOW HT-standard	KPPRA $\approx1{,}000$	KPPRA 6,000 relaxation (after a 4,000 pre-relaxation), 8,000 final static	KPPRA $\approx1{,}000$
+U	none (alloys)	oxides/fluorides (MP-style U)	correlated-TM oxides	oxides/fluorides (MP2020)
Elemental refs	raw DFT bulk elements	raw DFT bulk elements	DFT ground states, except 13 elements fit to experiment (of our metals, Na, Ti, Sn and Hg)	fitted / corrected
Post-hoc corrections	none	MP2020 anion/+U mixing, zero on every metallic entry	+U chemical potentials, for +U compounds only	MP2020 anion/+U mixing
VASP	HT-standard	PAW datasets of VASP 5.2	5.3.2	—

OQMD’s fitted chemical potentials are those of Table 3 of Kirklin et al. [5]; Methods E gives their effect on the floor. AFLOW’s exact plane-wave and k-point settings are its high-throughput standard [6, 7], and we do not re-tabulate them.

Supplementary Table 6 groups every population the paper scores by AFLOW’s ldau_type flag (scripts/ldau_groupby.py). The 1.38M core alloy libraries carry no +U entries.

The no-+U cut of the elastic section (Supplementary Note 12) is the remaining 795 structures.

Supplementary Note 5. DFT-vs-DFT noise floor

The floor’s construction—same-crystal matching, the choice of median and trimmed mean, and the two reality checks—is Methods E. The table below is Table 2 with the columns the body omits.

Pair	median $\lvert\Delta E_f\rvert$ (meV/atom)	trimmed MAE (meV/atom)	removable per-element offset	shared metallic structures
AFLOW–Alexandria	5.8	16.4	none ($R^2$ = 0.01)	$\sim$448,500
AFLOW–OQMD	8.4	25.8	none ($R^2$ = 0.04)	$\sim$30,300
OQMD–Alexandria	7.8	29.9	none ($R^2$ = 0.03)	$\sim$56,200
AFLOW–Materials Project	18.0	30.3	largely not removable ($R^2$ = 0.06)	$\sim$6,400

Every row counts one pair per connected component of same-crystal matches (Methods E; scripts/floor_pairs_smsg.py → floor_sm_table2.csv). A prototype-fingerprint lookup would have found 321,725 of the 448,532 AFLOW–Alexandria pairs, 12,701 of the 30,336 AFLOW–OQMD, 31,604 of the 56,206 OQMD–Alexandria, and 4,658 of the 6,433 AFLOW–MP. Most of the pairs it misses are the same crystal described with different Wyckoff labels (106,033 of the 126,807 AFLOW–Alexandria misses), and they disagree a little more than the rest (AFLOW–Alexandria median 6.3 and trimmed MAE 20.3 meV/atom, compared with 5.6 and 14.8 for the pairs both find). Of the fingerprint’s own pairs, 17,344 of 339,202 AFLOW–Alexandria (5.1%), 92 of 12,794 AFLOW–OQMD, 155 of 31,764 OQMD–Alexandria, and 103 of 4,761 AFLOW–MP are not the same crystal (scripts/floor_smsg_vs_fingerprint.py → floor_smsg_vs_fingerprint.csv). Requiring $\lvert\Delta V/V\rvert\le10\%$ as well gives median and trimmed MAE of 5.7 / 13.3, 8.0 / 22.4, 7.5 / 25.5, and 17.8 / 29.2 meV/atom for the four rows.

$R^2 \le0.04$ in the raw-PBE rows means composition explains almost none of the structure-to-structure disagreement. The table’s $R^2$ values are in sample, for a least-squares fit to all pairs (floor_sm_r2.csv). A second fit leaves out the pairs that disagree by 500 meV/atom or more, is fit on 70% of the chemical systems, and is scored on the rest. On those held-out systems the per-element offset explains 6.7% of the AFLOW–Alexandria disagreement, 12.9% of the AFLOW–OQMD and 13.9% of the Alexandria–OQMD, the last two consistent with OQMD’s fitted elemental references, and 17.9% of the AFLOW–MP (scripts/floor_smsg_r2_heldout.py → floor_smsg_r2_heldout.csv).

Every row is on the metallic cut. MP is the one pair for which that matters: uncut, AFLOW–MP gives a median of 42.1 and a trimmed MAE of 162.0 meV/atom over 10,983 same-crystal pairs, which measures MP’s anion and +U conventions rather than any floor. On the 6,433 metallic pairs the same database pair gives 18.0 and 30.3.

Per-element offset test (Methods E)—element sets. The metallic cut uses the nonmetal-exclusion set H, He, B, C, N, O, F, Ne, Si, P, S, Cl, Ar, Ge, As, Se, Br, Kr, Te, I, Xe, Po, At, Rn. The Figure 1 floor-versus-$E_{\mathrm{hull}}$ binning uses a 22-element variant that counts Ge and Po as metals (both appear in the pipeline). The AFLOW\MP metallic per-element $R^2$ is 0.059 in sample (scripts/floor_structure_matched_downstream.py, floor_sm_r2.csv).

Floor versus distance from the hull (Figure 1). Binned against $E_{\mathrm{hull}}$, all three pairs stay between 5.7 and 9.8 meV/atom in every bin (AFLOW–Alexandria, the largest sample, at 5.7–6.6, and 8.8 in the last bin, n = 384; AFLOW–OQMD 6.4–9.8; OQMD–Alexandria 7.6–9.4). Binned against formation energy, AFLOW–Alexandria stays at 5.2–6.1 meV/atom from $E_f = -1{,}000$ to 1,250 meV/atom (7.9 in the lowest bin, n = 212), and AFLOW–OQMD at 4.6–9.6 up to +750, rising to 10.5–14.9 in the bins above (n = 1,035 in the last) (floor_sm_vs_ef.csv, floor_sm_vs_hull.csv). The reference does not degrade appreciably far from equilibrium, and no single database is the noisy one.

The MP metallic-cut correction (Methods E), in numbers. The “removable convention” that the first, contaminated decomposition found was dominated by O, S, and N offsets specifically (rather than any floor). Supplementary tbl:8 uses the definition of Methods F (results/analysis/floor_sm_rung0.csv). Quoting the median floor (5.8 meV/atom) instead of the trimmed one (16.7) adds the same 10.9 meV/atom to every model, so the factor by which it inflates the distance shrinks as the model gets worse.

Supplementary tbl:8. Distance from the floor, trimmed versus median, with its sensitivity to how the floor’s structures are matched. Intervals are paired 95% bootstrap intervals over held-out chemical systems, with the per-element constants held at their fit (scripts/floor_sm_rung0_ci.py). The strictest and loosest matching use StructureMatcher tolerances (ltol/stol/angle_tol) of 0.1/0.2/$3^\circ$ and 0.4/0.6/$15^\circ$, where the floor over all pairs is 15.4 and 20.9 meV/atom (Methods E); the trimmed floor is recomputed on each set’s own held-out structures (floor_sm_rung0_tol.csv). The tolerance moves the distances by up to about 5 meV/atom for the leading models and up to 7 for the weakest, more than any interval, and the leader stays below the floor at every tolerance.

Model	After Rung-0, floor structures	Above trimmed floor [95% CI]	Above median floor	Above trimmed floor, strictest matching	Above trimmed floor, loosest matching
eqV2	14.1	−2.6 [−2.8, −2.4]	+8.3	−2.3	−7.3
eSEN	18.9	+2.3 [+1.9, +2.6]	+13.2	+2.3	−2.5
UMA	21.3	+4.6 [+4.2, +5.1]	+15.5	+4.8	−0.3
SevenNet-MF	22.4	+5.8 [+5.5, +6.1]	+16.7	+6.0	+1.1
GRACE-2L	23.9	+7.2 [+6.9, +7.5]	+18.1	+7.5	+2.4
MACE-MPA	25.7	+9.0 [+8.6, +9.4]	+19.9	+9.2	+4.3
MACE-OMAT	26.4	+9.8 [+9.4, +10.2]	+20.7	+10.1	+4.9
DPA-3	26.9	+10.2 [+9.9, +10.6]	+21.1	+10.5	+5.4
Orb-v3	28.3	+11.6 [+11.0, +12.3]	+22.5	+11.4	+6.8
MatterSim-5M	38.0	+21.3 [+20.7, +22.0]	+32.2	+21.6	+15.9
MatterSim-1M †	53.0	+36.4 [+35.5, +37.2]	+47.3	+36.7	+30.5
M3GNet	55.4	+38.8 [+37.8, +39.8]	+49.7	+39.0	+32.6
SevenNet-0 †	63.2	+46.5 [+45.4, +47.7]	+57.4	+46.9	+40.0
CHGNet	71.5	+54.9 [+53.6, +56.2]	+65.8	+55.3	+48.7

† superseded checkpoints.

Trim-cutoff sensitivity

The 1 eV/atom trim (Methods D) removes only exceptional events, but where it sits matters in the middle of the ranking, where the error distributions have long tails. Between cutoffs of 0.25 and 1.0 eV/atom, Orb-v3’s trimmed MAE grows 56% and eSEN’s 48%, so the order in the middle of the ranking depends on the cutoff while the leader and the tiers do not. The table gives trimmed MAE (relaxed protocol, meV/atom) at representative cutoffs (columns in eV/atom; model rows from scripts/trim_sensitivity.py → trim_sensitivity.csv, floor rows from floor_sm_trim_sweep.csv).

Supplementary Table 9. Trimmed MAE at representative cutoffs.

Model / DFT pair	0.25	0.5	1.0	2.0	5.0
eqV2	13.6	17.3	18.3	18.7	19.4
eSEN	17.2	23.0	25.5	26.3	27.1
SevenNet-MF	21.5	26.4	28.2	28.9	29.6
UMA	21.6	26.3	28.5	29.3	30.1
GRACE-2L	23.3	28.1	29.8	30.4	31.3
MACE-OMAT	26.3	31.0	32.7	33.2	33.9
MACE-MPA	25.7	31.1	33.0	33.6	34.4
DPA-3	26.6	31.3	33.0	33.6	34.3
Orb-v3	22.9	30.3	35.8	37.6	38.6
MatterSim-5M	44.1	49.8	52.0	52.7	53.4
M3GNet	51.7	59.2	63.3	64.6	65.4
MatterSim-1M †	52.6	59.7	61.9	62.6	63.4
SevenNet-0 †	70.6	83.4	87.1	88.1	88.8
CHGNet	98.8	121.2	126.5	127.7	128.4
AFLOW–Alexandria floor	12.4	15.6	16.4	16.6	16.7
AFLOW–OQMD floor	19.5	24.1	25.8	26.7	27.1
OQMD–Alexandria floor	18.9	24.8	29.9	31.7	32.4

† superseded checkpoints. From 1 to 5 eV/atom the model order holds, except that MACE-MPA and DPA-3, 0.05 meV/atom apart at 1 eV/atom, trade places at 5 eV/atom. Below 1 eV/atom, near-ties reorder. Orb-v3 moves three places between the 0.5 and 1.0 cutoffs, and UMA and SevenNet-MF swap, as do M3GNet and MatterSim-1M. eqV2 leads and eSEN is second at every cutoff. Measured on the structures that define the floor, and before re-referencing, the leader sits 0.9 to 1.2 meV/atom below the AFLOW–Alexandria floor at every cutoff, with the leader and the floor each trimmed at that cutoff (si_s6_trim_recheck_floorset.csv). At the untrimmed limit, two checkpoints’ single corrupted rows dominate (GRACE-2L 6.6×10³, SevenNet-0 4.0×10⁷ meV/atom), which is why raw means are never reported. The full sweep including the untrimmed limit is plotted in Supplementary Figure 1.

Supplementary Figure 1. Trim-cutoff sensitivity: each model’s trimmed MAE against the error cutoff, including the untrimmed limit where single corrupted rows dominate. The ranking is the same for every cutoff at or above 1 eV/atom, except for one near-tie between MACE-MPA and DPA-3 (Methods D). The curves are errors over the whole benchmark. The shaded band and the dashed line show the size of the DFT-vs-DFT floor; distances from the floor measured on the structures that define it are in Supplementary Table 8.

Rung-0

The fit, its capacity bound, and the system-disjoint held-out split are Methods F.

Fitted per-element constants. Fit by least absolute deviation, most models’ constants are small. Fe, for example, runs from +13 to −29 meV/atom for every model except CHGNet (−153) and SevenNet-0 (−124), and eqV2’s largest constants are Sn +38 and Mg +21 meV/atom. A least-squares fit gives the magnetic 3d elements much larger constants (Fe −49 to −178 meV/atom), because least squares is pulled by the minority of badly predicted structures that contain them. The consensus-versus-AFLOW fit of Methods I, also by least squares, gives Mn −119, Cr −103, and Fe −87 meV/atom.

Held-out behavior (Methods F). The MP-trained models lose a large part of their error that is linear in composition, which may be a reference convention, a broken reference or an element-wise bias (leaderboard section; Table 3).

CHGNet, decomposed. CHGNet’s large Rung-0 gain separates cleanly: a single global constant (−101.4 meV/atom, the median of its errors, overbinding against its own unaries) accounts for most of it (126.2 → 81.2 meV/atom), and element-specific offsets carry the rest, largest on the refractory metals.

Element	Centered offset (meV/atom)
Ru	−183
W	−160
Os	−131
Mo	−113
Al	−83

These offsets correlate only weakly with the per-element reference differences over the 47 elements both fits support: $r = 0.19$ ($p = 0.21$) against MP−AFLOW, the convention of CHGNet’s training data, and $r = 0.30$ ($p = 0.04$) against Alexandria−AFLOW. They are therefore mostly CHGNet-specific systematic error, not an inherited Materials Project convention. This is the paradox Rung-0 is built to resolve: model errors can be composition-linear (held-out $R^2$ up to 0.17, for CHGNet) while the AFLOW–Alexandria disagreement barely is ($R^2$ 0.01 in sample on all pairs, 0.07 held out with pairs beyond 500 meV/atom left out; Supplementary Note 5), so the same one-constant-per-element fit strips much from the former and little from the latter (rung0_chgnet_diagnosis.csv, floor_smsg_r2_heldout.csv). The pairs that include OQMD reach a held-out $R^2$ of 0.13–0.14, consistent with OQMD’s fitted elemental references.

Choice of fit. We fit the constants by least absolute deviation, so the fit minimizes the statistic we score. A least-squares fit centers each model’s residuals on their mean rather than their median, and the error distributions have long one-sided tails. For the aligned models, least squares therefore raises the held-out MAE above its raw value, by 0.2 to 4.1 meV/atom (eqV2 20.9, eSEN 29.4, UMA 29.2, SevenNet-MF 30.7, GRACE-2L 31.1, compared with 16.6, 23.8, 25.1, 27.0, and 28.1 by least absolute deviation). A single global constant shows the cause: set to the mean of eqV2’s errors (−6.8 meV/atom) it raises eqV2’s held-out MAE from 18.2 to 21.9, while set to their median (+2.9) it lowers it to 17.7. A Huber fit ($\delta= 1.345\cdot\sigma_{\mathrm{MAD}}$) lands within 0.35 meV/atom of least absolute deviation for every model. For the models trained without Alexandria or OMat24 the choice matters less. Least squares raises their held-out MAE by 0.5 to 2.5 meV/atom (CHGNet 72.9 against 72.3, MatterSim-5M 43.2 against 40.7), small next to the 7 to 54 meV/atom that Rung-0 removes from them, because their per-element component is large (scripts/rung0_robustness.py, scripts/rung0_lad.py; results/analysis/rung0_lad.csv).

The fine-tune and AFLOW’s convention (scripts/protocol_proximity.py → protocol_proximity_test.csv). eqV2 and eSEN were pretrained on OMat24 and fine-tuned on MPtrj and subsampled Alexandria. We scored the public OMat24-only checkpoints of the same architectures (EquiformerV2-86M-OMAT24, eSEN-30M-OMAT24) at AFLOW’s geometry on a random 100,000 of the benchmark structures, with their own elemental references formed as in the main benchmark, and compared them with the fine-tuned checkpoints on the same held-out structures (paired intervals over chemical systems). Raw, the fine-tune lowers the error by 3.6 [3.0, 4.2] meV/atom for eqV2 and 1.5 [1.1, 1.9] for eSEN; without it, eqV2 ties UMA. After Rung-0 the fine-tune gives nothing for eSEN (−0.3 [−0.6, +0.1]) and costs eqV2 1.1 [0.6, 1.7] meV/atom. Its gain is therefore a per-element offset. The OMat24-only checkpoints keep the leaders’ re-referenced edge over Orb-v3, GRACE-2L and MACE-OMAT (4.7–10.7 meV/atom), and all four checkpoints sit below the AFLOW–Alexandria floor on the sampled floor structures after Rung-0. So does UMA, which was also never trained on Alexandria (−3.9 [−4.9, −3.0] meV/atom).

Robustness. A 5-fold system-disjoint cross-validation reproduces the 70/30 corrected MAEs within 0.25 meV/atom (fold standard deviation 0.12–0.49), so the results do not depend on the split. Fitting on the rows with $|\operatorname{error}| < 1$ eV/atom instead of $< 0.5$ eV/atom changes no corrected MAE by more than 0.01 meV/atom.

The full leaderboard (all metrics, fourteen models)

The table gives Figure 12’s values for every model on every axis, with 95% system-resampled bootstrap half-intervals. Columns and dashes are as in Fig. 12’s caption, plus an After-Rung-0 column that Fig. 12 does not show. The elastic panel used the superseded MatterSim-1M checkpoint, so MatterSim-5M carries no shear value.

Model	Raw MAE (meV/atom)	Single-point (meV/atom)	After Rung-0 (meV/atom)	SP + Rung-0 (meV/atom)	Hull-F1	$E_{\mathrm{hull}}$ MAE (meV/atom)	Residual force (eV/Å)	Polymorph $\rho$	No-+U shear r (log-space)	SG retention @ 0.1 Å
eqV2	18.3 ± 0.4	14.4 ± 0.3	16.6	12.7 ± 0.6	0.921 ± 0.010	18.7 ± 0.5	0.0004 ± 0.0000	0.983 ± 0.000	—	0.991 ± 0.000
eSEN	25.5 ± 0.5	14.8 ± 0.3	23.8	13.3 ± 0.6	0.893 ± 0.010	26.5 ± 0.6	0.0217 ± 0.0004	0.949 ± 0.001	—	0.921 ± 0.001
SevenNet-MF	28.2 ± 0.5	19.4 ± 0.3	27.0	17.9 ± 0.6	0.868 ± 0.011	29.0 ± 0.5	0.0435 ± 0.0010	0.942 ± 0.001	—	0.922 ± 0.001
UMA	28.5 ± 0.4	17.9 ± 0.3	25.1	13.3 ± 0.4	0.872 ± 0.012	27.1 ± 0.5	0.0401 ± 0.0007	0.944 ± 0.001	0.983 ± 0.008	0.911 ± 0.001
GRACE-2L	29.8 ± 0.5	21.0 ± 0.3	28.1	20.0 ± 0.6	0.843 ± 0.014	31.4 ± 0.5	0.0420 ± 0.0009	0.938 ± 0.001	0.984 ± 0.005	0.922 ± 0.001
MACE-OMAT	32.7 ± 0.5	24.2 ± 0.3	30.7	22.2 ± 0.6	0.821 ± 0.015	34.0 ± 0.5	0.0447 ± 0.0010	0.925 ± 0.001	—	0.915 ± 0.001
MACE-MPA	33.0 ± 0.5	21.6 ± 0.3	31.3	21.2 ± 0.7	0.839 ± 0.012	35.5 ± 0.6	0.0466 ± 0.0011	0.920 ± 0.001	0.977 ± 0.007	0.915 ± 0.001
DPA-3	33.0 ± 0.5	23.9 ± 0.3	30.6	22.3 ± 0.6	0.815 ± 0.015	34.0 ± 0.5	0.0510 ± 0.0011	0.915 ± 0.001	—	0.921 ± 0.001
Orb-v3	35.8 ± 0.5	30.2 ± 0.3	33.4	17.8 ± 0.6	0.837 ± 0.012	35.8 ± 0.6	0.0449 ± 0.0009	0.915 ± 0.001	0.983 ± 0.006	0.903 ± 0.001
MatterSim-5M	52.0 ± 0.5	39.4 ± 0.4	40.7	31.6 ± 0.7	0.736 ± 0.017	53.0 ± 0.6	0.0520 ± 0.0012	0.883 ± 0.001	—	0.922 ± 0.001
M3GNet	52.3 ± 0.5	50.1 ± 0.5	55.8	46.1 ± 0.8	0.616 ± 0.022	62.7 ± 0.6	0.0388 ± 0.0001	0.928 ± 0.001	0.848 ± 0.034	0.922 ± 0.001
MatterSim-1M †	61.9 ± 0.5	48.4 ± 0.5	54.3	44.9 ± 0.7	0.634 ± 0.022	59.5 ± 0.6	0.0893 ± 0.0023	0.834 ± 0.002	0.938 ± 0.030	0.887 ± 0.001
SevenNet-0 †	87.1 ± 1.0	66.1 ± 0.8	63.6	54.7 ± 1.1	0.597 ± 0.022	77.5 ± 0.9	0.1067 ± 0.0029	0.800 ± 0.001	0.923 ± 0.017	0.889 ± 0.001
CHGNet	126.5 ± 1.1	101.7 ± 0.9	72.3	60.9 ± 1.1	0.448 ± 0.021	113.3 ± 1.1	0.0980 ± 0.0024	0.786 ± 0.003	0.878 ± 0.030	0.890 ± 0.001

± are 95% system-resampled bootstrap half-intervals (B = 2000; bootstrap_ci_all.csv, bootstrap_ci_phase2.csv, bootstrap_ci_sprung0.csv; the retention column, symprec_ci_full.csv). † superseded checkpoints. The After-Rung-0 column carries no interval—no bootstrap covers the relaxed+Rung-0 combination—and the two Rung-0 columns are scored on the 30% held-out systems. The hull and polymorph columns use the common set of Methods G (relaxed energies; the single-point values are in Table 4 and Supplementary Note 11). The held-out uncorrected MAE is within 0.6 meV/atom of the full-set value (Table 3), so the columns are comparable but not computed on identical sets. The shear column is the log-space $r$ of Supplementary Note 12 (elastic_logspace.csv). eqV2’s polymorph and residual-force intervals render as ±0.000 because the bootstrap collapses at a plateau (residual-force CI [0.00037, 0.00039]).

The fully controlled energy order, resolved. Bootstrap CIs on the SP + Rung-0 column (Table 3), paired over the shared held-out systems, have marginal half-widths of ±0.4–0.6 meV/atom for the top three, and two of the three paired differences exclude zero (scripts/bootstrap_ci_sprung0.py). Pairing is the correct test, since the models’ errors on a common structure set are highly correlated.

The fully controlled order is therefore eqV2 lowest, with eSEN and UMA indistinguishable—but these intervals do not propagate the Rung-0 fit’s own noise (the five-fold spread of 0.12–0.49 meV/atom in Supplementary Note 7, the same scale as the differences).

Hull-F1 robustness

Hull construction, the stability window, and the validation against pymatgen are Methods G. Threshold sweep (on the common set of Methods G; scripts/refree_common_regen.py → results/analysis/hull_f1_threshold_sweep.csv, single-point in hull_f1_threshold_sweep_singlepoint.csv). eqV2 leads and eSEN is second at every window, relaxed and at AFLOW’s geometry. Behind them only adjacent near-ties swap, each by at most 0.010 in F1. SevenNet-MF passes UMA at 20 and 50 meV/atom (at 50 by less than 0.001), Orb-v3 passes MACE-MPA at every window except 25 and GRACE-2L at 15 and 50, and DPA-3 passes MACE-OMAT at 10. The OMat-trained models behind eqV2 score F1 0.815–0.893 at the 25 meV/atom operating point.

Supplementary tbl:13. Hull-F1 at seven stability windows, relaxed energies on the common set. † superseded checkpoints.

Model	10 meV/atom	15	20	25	30	40	50
eqV2	0.909	0.915	0.913	0.921	0.924	0.925	0.935
eSEN	0.864	0.874	0.882	0.893	0.895	0.902	0.907
UMA	0.830	0.844	0.858	0.872	0.876	0.882	0.889
SevenNet-MF	0.829	0.842	0.861	0.868	0.873	0.881	0.890
GRACE-2L	0.805	0.816	0.830	0.843	0.850	0.858	0.868
MACE-MPA	0.796	0.808	0.826	0.839	0.848	0.856	0.865
Orb-v3	0.798	0.818	0.830	0.837	0.850	0.858	0.870
MACE-OMAT	0.766	0.789	0.809	0.821	0.827	0.839	0.850
DPA-3	0.767	0.787	0.808	0.815	0.827	0.836	0.846
MatterSim-5M	0.688	0.701	0.721	0.736	0.745	0.763	0.777
MatterSim-1M †	0.567	0.585	0.612	0.634	0.647	0.666	0.689
M3GNet	0.551	0.577	0.598	0.616	0.632	0.659	0.684
SevenNet-0 †	0.527	0.551	0.575	0.597	0.604	0.624	0.643
CHGNet	0.424	0.425	0.438	0.448	0.462	0.484	0.506

Ceiling on the shared overlap. Table 4’s Alexandria row (hull-F1 0.901, on same-crystal pairs) is computed on a different structure set than the model rows. Recomputed on the shared same-crystal AFLOW∩Alexandria overlap, restricted to the common set (scripts/reffree_common_regen.py → alex_ceiling_shared_sm.csv; 674,683 structures, 13,686 systems), the ceiling is 0.913. eqV2 (0.936) and eSEN (0.923) clear it, and SevenNet-MF (0.905) and UMA (0.898) sit below. At AFLOW’s geometry eqV2 (0.942), eSEN (0.936) and SevenNet-MF (0.920) clear it, and UMA (0.906) sits just below it (alex_ceiling_shared_sm_singlepoint.csv).

Space-group retention versus symprec

The tolerance problem and the whole-benchmark rerun, every structure relaxed by all fourteen models, are Methods G. The table gives retention at each tolerance, with the reference and the relaxed structure both assigned at the swept tolerance, and 95% intervals from resampling chemical systems (scripts/symprec_full_retention.py → results/analysis/symprec_full_retention.csv).

Model	symprec 0.01 Å	0.05 Å	0.10 Å
eqV2	0.989 [0.989, 0.989]	0.992 [0.992, 0.993]	0.991 [0.991, 0.992]
eSEN	0.811 [0.809, 0.813]	0.903 [0.902, 0.904]	0.921 [0.920, 0.921]
SevenNet-MF	0.910 [0.909, 0.911]	0.913 [0.912, 0.914]	0.922 [0.921, 0.923]
UMA	0.775 [0.773, 0.777]	0.885 [0.884, 0.886]	0.911 [0.910, 0.912]
GRACE-2L	0.911 [0.910, 0.912]	0.913 [0.912, 0.914]	0.922 [0.921, 0.923]
MACE-OMAT	0.904 [0.903, 0.905]	0.905 [0.904, 0.906]	0.915 [0.914, 0.915]
MACE-MPA	0.902 [0.901, 0.903]	0.905 [0.904, 0.906]	0.915 [0.914, 0.916]
DPA-3	0.906 [0.905, 0.907]	0.911 [0.910, 0.912]	0.921 [0.920, 0.922]
Orb-v3	0.840 [0.838, 0.841]	0.890 [0.889, 0.891]	0.903 [0.902, 0.904]
MatterSim-5M	0.920 [0.919, 0.921]	0.917 [0.917, 0.918]	0.922 [0.921, 0.923]
M3GNet	0.864 [0.863, 0.865]	0.876 [0.875, 0.877]	0.886 [0.885, 0.888]
MatterSim-1M †	0.882 [0.880, 0.883]	0.878 [0.877, 0.879]	0.887 [0.885, 0.888]
SevenNet-0 †	0.886 [0.885, 0.887]	0.886 [0.885, 0.887]	0.889 [0.888, 0.890]
CHGNet	0.888 [0.887, 0.889]	0.888 [0.887, 0.889]	0.890 [0.889, 0.891]

† superseded checkpoints.

Every model retains at least as much at 0.1 Å as at 0.01 Å, and the models that lose most at the tight tolerance (UMA, eSEN, Orb-v3) gain most as it loosens. Leaving aside eqV2, which barely relaxes, the spread is 77.5–92.0% at 0.01 Å and 88.6–92.2% at 0.1 Å (symprec_full_retention_spread.csv), and the gap between GRACE-2L and UMA falls from 13.6 points to 1.1. The large gap at the tight tolerance is a symprec artifact.

Polymorph ordering in detail

The metric—Spearman ρ and ground-state hit rate per qualifying composition—is Methods G. The table gives per-model medians at the three AFLOW-spread thresholds (scripts/analyze_polymorph_ranking.py).

Supplementary tbl:15. Polymorph ordering: per-model medians at the three AFLOW-spread thresholds, and each model scored on the compositions of the two DFT pairs that include AFLOW, beside the second database on the same members (Methods G; scripts/polymorph_floor_aflow_oqmd.py).

Model	median $\rho$ (spread $\ge25$ / 50 / 100 meV/atom)	ground-state hit rate (25; ties count as hits)	AFLOW–Alexandria sets: mean $\rho$ / top-1 (Alexandria)	AFLOW–OQMD sets: mean $\rho$ / top-1 (OQMD)
eqV2	0.98 / 0.98 / 0.98	0.59	0.963 / 0.92 (0.956 / 0.91)	0.93 / 0.92 (0.83 / 0.85)
eSEN	0.95 / 0.95 / 0.95	0.51	0.941 / 0.89 (0.956 / 0.91)	0.91 / 0.91 (0.83 / 0.85)
UMA	0.94 / 0.94 / 0.95	0.50	0.933 / 0.87 (0.956 / 0.91)	0.89 / 0.88 (0.83 / 0.85)
SevenNet-MF	0.94 / 0.94 / 0.94	0.48	0.931 / 0.85 (0.956 / 0.91)	0.88 / 0.88 (0.83 / 0.85)
GRACE-2L	0.94 / 0.94 / 0.94	0.47	0.926 / 0.84 (0.956 / 0.91)	0.87 / 0.87 (0.83 / 0.85)
DPA-3	0.93 / 0.93 / 0.93	0.45	0.912 / 0.81 (0.956 / 0.91)	0.84 / 0.83 (0.83 / 0.85)
MACE-OMAT	0.93 / 0.93 / 0.93	0.45	0.912 / 0.81 (0.956 / 0.91)	0.84 / 0.83 (0.83 / 0.85)
MACE-MPA	0.92 / 0.92 / 0.92	0.44	0.912 / 0.81 (0.956 / 0.91)	0.85 / 0.83 (0.83 / 0.85)
Orb-v3	0.92 / 0.92 / 0.92	0.46	0.903 / 0.83 (0.956 / 0.91)	0.86 / 0.85 (0.83 / 0.85)
MatterSim-5M	0.88 / 0.88 / 0.88	0.40	0.874 / 0.74 (0.956 / 0.91)	0.77 / 0.72 (0.83 / 0.85)
MatterSim-1M †	0.83 / 0.83 / 0.84	0.33	0.802 / 0.64 (0.956 / 0.91)	0.66 / 0.64 (0.83 / 0.85)
M3GNet	0.83 / 0.83 / 0.83	0.33	0.797 / 0.63 (0.956 / 0.91)	0.64 / 0.61 (0.83 / 0.85)
SevenNet-0 †	0.80 / 0.80 / 0.80	0.30	0.770 / 0.58 (0.956 / 0.91)	0.64 / 0.65 (0.83 / 0.85)
CHGNet	0.79 / 0.79 / 0.79	0.28	0.752 / 0.57 (0.956 / 0.91)	0.57 / 0.59 (0.83 / 0.85)

The two-tier divide of the reference-free section holds at every threshold. The Alexandria ceiling is top-1 0.908 [0.904, 0.911] at mean $\rho$ 0.955 [0.954, 0.957] over 37,742 compositions (median 1.00). The two pairs that include OQMD agree less well, OQMD–Alexandria at 0.841 [0.826, 0.856] and 0.855 [0.840, 0.867] over 2,565 compositions, and AFLOW–OQMD at 0.849 [0.835, 0.864] and 0.833 [0.818, 0.847] over 2,318, with chance at 0.29 and 0.28. The two OQMD pairs have small sets (mean 3.7 and 3.8 members, compared with 9.9 for AFLOW–Alexandria) (polymorph_floor_aflow_oqmd_table.csv). Scored on the same members (Supplementary Table 15), eqV2 exceeds Alexandria on the AFLOW–Alexandria sets, and every OMat-trained model and MACE-MPA exceeds OQMD’s mean $\rho$ on the AFLOW–OQMD sets; the MP-only models fall below both databases. At AFLOW’s geometry (polymorph_ranking_singlepoint_common.csv) every model orders the polymorphs better, and eqV2 and eSEN tie (mean $\rho$ 0.962 each).

Elastic response

The finite-strain protocol and the U-status stratification are Methods H. Supplementary Table 16 gives the shear correlation before and after removing +U structures, and within the +U structures by chemistry (Figure 6; scripts/elastic_logspace.py, scripts/elastic_u_split.py).

Supplementary tbl:16. Log-space shear correlation (95% interval, resampling chemical systems) on all AEL structures, without +U, and in four +U groups: oxides and fluorides containing one of the Materials Project’s eight +U elements (89 structures), other oxides and fluorides (204), other anion compounds (1,311) and metallic compounds (1,117), counts for UMA (scripts/elastic_u_split.py). Structures a model predicts with a non-positive shear modulus are left out. Slope and multiplicative bias are on the no–+U set.

Model	all AEL	no +U	+U oxide/fluoride, MP +U element	+U oxide/fluoride, other	+U, other anion	+U, metallic	log–log slope (no +U)	mult. bias (no +U)
MACE-MPA	0.92 [0.90, 0.94]	0.98 [0.97, 0.98]	0.85 [0.75, 0.95]	0.85 [0.91, 0.97]	0.86 [0.81, 0.91]	0.86 [0.81, 0.91]	0.94	0.97
CHGNet	0.76 [0.73, 0.78]	0.88 [0.84, 0.90]	0.56 [0.23, 0.86]	0.81 [0.64, 0.92]	0.74 [0.70, 0.77]	0.64 [0.59, 0.68]	1.02	0.48
M3GNet	0.76 [0.73, 0.78]	0.85 [0.81, 0.88]	0.59 [0.45, 0.70]	0.84 [0.21, 0.46]	0.76 [0.73, 0.79]	0.72 [0.66, 0.78]	0.91	0.71
SevenNet-0 †	0.79 [0.76, 0.81]	0.52 [0.90, 0.94]	0.73 [0.59, 0.87]	0.90 [0.86, 0.94]	0.77 [0.72, 0.81]	0.62 [0.56, 0.69]	0.95	0.73
UMA	0.94 [0.92, 0.95]	0.98 [0.97, 0.99]	0.90 [0.83, 0.97]	0.97 [0.96, 0.98]	0.92 [0.89, 0.95]	0.89 [0.86, 0.91]	0.99	1.03
Orb-v3	0.93 [0.91, 0.94]	0.98 [0.98, 0.99]	0.94 [0.92, 0.98]	0.94 [0.91, 0.97]	0.93 [0.91, 0.95]	0.83 [0.79, 0.88]	0.97	0.98
GRACE-2L	0.93 [0.91, 0.95]	0.98 [0.98, 0.99]	0.94 [0.89, 0.97]	0.97 [0.95, 0.98]	0.92 [0.90, 0.95]	0.84 [0.78, 0.89]	0.97	0.98
MatterSim-1M †	0.91 [0.90, 0.93]	0.94 [0.91, 0.96]	0.93 [0.88, 0.97]	0.96 [0.94, 0.97]	0.91 [0.88, 0.94]	0.84 [0.81, 0.87]	0.97	0.91

The reference-free section interprets the table. Bulk modulus tracks AFLOW for every model in every group (log-space r 0.85–1.00). Removing PtTi (2,389 versus 20 GPa) as well as NiB₁₂ takes MACE-MPA’s linear all-AEL r to 0.86.

Metals-only recut. The AEL ground truth covers mostly ICSD nonmetal compounds. Only 1,291 of the 3,544 AEL structures (and 158 of the 795 no–+U ones) are purely metallic (scripts/elastic_metals_recut.py). Restricting to metals leaves the conclusion intact—+U-stratified shear correlations run 0.84–0.99 and the MP generation’s +U sensitivity if anything sharpens—but at far lower statistical power, so the main text reports the full AEL set, and this recut is the metals-only check.

Bad references—anatomy of the discrepancies

The consensus test, its thresholds, the per-element re-referencing and the third-database check are in Methods I, and their results in the bad-references section. This section holds what the discrepancies are and how robust the third-database check is.

Robustness of the third-database check (scripts/secvi_chain_structmatch.py, scripts/secvi_robustness_smsg.py → secvi_robustness_smsg.csv). Alexandria holds the same crystal for 3,794 of the 10,861 discrepancies that remain after re-referencing and sides with the models on 3,151 of them (83%). A prototype-fingerprint lookup with no structure check finds an Alexandria entry for 3,541 and sides with the models on 3,139 of them (88.6%). Adding a volume check (volumes per atom agreeing within 10%) to the structure check leaves 1,798 settled cases, 1,167 of them (65%) for the models. The volume check removes 1,984 of the 3,151 cases settled for the models and 12 of the 643 settled for AFLOW, because it discards the AFLOW cells whose volume is wrong; in 1,764 of the 1,984, Alexandria’s cell is more than 10% larger. The 643 genuine model errors of the structure check become 631 with the volume check as well, and the fingerprint lookup alone gives 402. With a consensus from only the six MP-generation models, Alexandria sides with it on 3,152 of the same 3,794, and rebuilding the whole chain on those six models (17,339 discrepancies before re-referencing, 10,845 after the offset) gives 3,032 of 3,794 (80%). OQMD holds the same crystal for 322 of the 10,861 and sides with the models on 301 (93%). MP holds it for 97 and sides with the models on 96; 50 of the 97 are elemental crystals.

Discrepancy geometry. The failure is one-sided, and many of the discrepant structures also have a wrong AFLOW cell (scripts/s13_alex_volume_post.py, scripts/s13_anatomy_post.py). Of the 15,278 discrepancies, 10,861 remain after the per-element offset (Methods I). AFLOW sits above the consensus in 99.7% of the 15,278 and 99.6% of the remaining 10,861 (median AFLOW − consensus +402 meV/atom before re-referencing, +450 after). Where Alexandria shares a fingerprint with a discrepancy (4,982 of the 15,278), its volume per atom differs from AFLOW’s by a median 11% (compared with 0.3% for structures that are not discrepancies), and AFLOW’s cell is the smaller one in 84% of them. The models’ relaxations change the volume of the discrepant structures by a median 9–12%, compared with at most 1.4% for the other structures. Where Alexandria’s volume differs from AFLOW’s by more than 10% (2,779 structures), 87% of the models’ relaxations end within 10% of Alexandria’s volume. The wrong cell does not account for the whole gap, though. On AFLOW’s own structure, before relaxation, thirteen of the fourteen models already sit a median 247–327 meV/atom below AFLOW on the 15,278 discrepancies, compared with 359–456 meV/atom after relaxation. Only UMA is close to AFLOW there (48 meV/atom, compared with 367 after relaxation).

Two systematic families stand out among the 10,861 discrepancies that remain after re-referencing. Three Fmmm decorations (space group 69, Wyckoff patterns 1b,2i and 1a,2i, both AB$_2$, and 1a,2f,2i, mostly ABC$_2$) hold 463 of the remaining 10,861. That is a 30.3% discrepancy rate on a 1,526-member family, 40× the 0.77% rate over all scorable structures. The other 1,197 SG-69 structures carry 25 (2.1%), twelve of them in a fourth AB$_2$ pattern, 4b,8i (12 of its 86 structures). The family comes from a chemistry-dependent relaxation collapse that leaves over-compressed published cells, which the models re-expand by a median 72%. It is enriched on Sn (4×), Ba (4×), Li (3×), and Be (3×). Duplicated broken unaries are another 253 of the remaining 10,861 (2.3%; identical stuck copies, $E_f$ +450 to +1,204 meV/atom, volumes 11–21% above the ground state).

The reverse population, where a third database sides with AFLOW and the models are wrong together, is 553 (11%) of the 4,982 discrepancies (before re-referencing) that Alexandria shares a fingerprint with, and 877 (16%) of the 5,466 where Alexandria holds the same crystal. It is concentrated in the 4d/5d refractories (Ir, Os, Tc, Re) and has unremarkable geometry. Relaxation changes the volume of these structures by a median +0.4% where Alexandria holds the same crystal, compared with +11% where Alexandria sides with the models. It is an energy-only collective miss on the same refractory elements that dominate CHGNet’s fitted offsets (Supplementary Note 7).

Panel-expansion robustness. Rebuilding the consensus test on all fourteen models (scripts/consensus_panel14_post.py) gives 15,417 discrepancies, on which Alexandria sides with the models in 90.2% (4,500 of 4,987), compared with 88.9% for the eight-model test’s 15,278 (4,429 of 4,982; 84.0% where Alexandria holds the same crystal, 4,589 of 5,466). The fourteen-model test keeps 91.7% of the eight-model discrepancies, and the two tests agree on 84% of the structures either one finds (14,015 of 16,680). The paper’s quoted counts stay with the eight-model test.

The genuine model errors by element. Of the 643 genuine model errors of the bad-references section and the 3,151 discrepancies Alexandria settles against AFLOW, these refractory metals appear in: Pt 139 and 182, Ir 79 and 127, Os 64 and 72, Tc 52 and 55, Re 49 and 44, Ru 47 and 90, W 39 and 41, Mo 31 and 47, Ta 31 and 38 (results/analysis/secvi_post_enrichment.csv, same-crystal third-database check).

Bad references in Alexandria. We mirror the consensus test of the bad-references section onto Alexandria (scripts/alex_badref_mirror.py, scripts/alex_badref_mirror_geom.py, scripts/alex_mirror_post_genuine.py → results/analysis/alex_mirror_post_*.csv). The pairs are the 448,397 same-crystal AFLOW–Alexandria pairs of the metallic set (Methods E) whose AFLOW structure at least six models of the post-screen panel scored. On each pair, C and S are the panel median and spread on that AFLOW structure. The test finds a discrepancy for Alexandria when $S < 0.1$ eV/atom, $|C - E_f(\mathrm{Alexandria})| > 0.3$ eV/atom, and $|C - E_f(\mathrm{Alexandria})|/\max(S, 0.02\ \mathrm{eV/atom}) \ge5$. We repeat the cut after subtracting a per-element offset. The offset is fitted to $C - E_f(\mathrm{Alexandria})$ exactly as for AFLOW (a composition-linear ridge fit on the pairs with $|C - E_f| < 0.5$ eV/atom). Alexandria’s offsets are small. The largest are Dy −96, Bi −76 and Sn −64 meV/atom, and Mn is −32, so the offset removes few discrepancies. A discrepancy for Alexandria counts as a genuine Alexandria error when AFLOW passes the same cut and StructureMatcher, run again on the Alexandria structure behind the energy, confirms the same crystal; it does for 426 of the 435 raw Alexandria-only discrepancies. We apply no volume criterion, because a lost magnetic moment also shrinks the cell, and Alexandria’s cell is smaller in 408 of the 426 raw genuine errors. Supplementary tbl:17 gives the counts, and Supplementary Table 18 describes the 426. On the same pairs and with the same structure test, AFLOW is the outlier in 2,768 pairs.

Supplementary tbl:17. Consensus-test discrepancies on the 448,397 same-crystal AFLOW–Alexandria metallic pairs, post-screen eight-model panel. “Raw” uses formation energies as reported. “Offset removed” subtracts each database’s per-element offset from the gap to the consensus.

	Raw	Offset removed
Discrepancy for Alexandria	1,050	892
Discrepancy for AFLOW	3,392	2,521
Alexandria only (AFLOW passes the cut)	435	428
… of which AFLOW lies within 0.1 eV/atom of C	412	363
Discrepancy for both, databases within 0.1 eV/atom of each other (models wrong)	594	447
Discrepancy for both, databases disagree	21	17
AFLOW only	2,777	2,057
Alexandria only, volume within 10%	193	207
Genuine Alexandria errors (Alexandria only, same crystal confirmed)	426	420
Genuine, with volume within 5% as well	45	82
Genuine, with volume within 10% as well	185	200
Genuine, with volume within 15% as well	330	328

Notes on the tables. - Background. Chemistry is compared with the 441,643 same-crystal pairs on which no version of the Alexandria test finds a discrepancy. Of these 441,643, 152,177 (34.5%) contain V, Cr, Mn, Fe, Co or Ni (alex_mirror_post_summary.csv). Magnetism is compared with 2,000 random pairs without a discrepancy, in which AFLOW is magnetic and Alexandria is not in 31. - Magnetic. AFLOW $|\mathrm{spin\_atom}| > 0.1\ \mu_{\mathrm{B}}/\mathrm{atom}$. For Alexandria, $|\mathrm{total\_mag}| / \mathrm{sites} > 0.1\ \mu_{\mathrm{B}}/\mathrm{atom}$. - Releases. Every Alexandria energy, structure and volume in this note is from release 2024.12.15. - Third opinion. Among all pairs with a discrepancy for Alexandria but not AFLOW, OQMD sides with AFLOW and the models in 215 of 225 (raw) and 213 of 222 (offset removed), and MP in 6 of 6. For the pairs with a discrepancy for AFLOW but not Alexandria, OQMD sides with Alexandria in 70 of 131 and with AFLOW in 7, and the rest lie in between.

Supplementary Note 14. Do the discrepancies distort the headline numbers?

Slightly. On most discrepancies AFLOW and the models differ by 0.3–1 eV/atom, so the 1 eV/atom trim removes only 4.7–6.3% of each model’s predictions on the 15,278 discrepancies and 6.7–8.7% on the 10,861 that remain after re-referencing. On the whole benchmark, excluding the remaining 10,861 lowers every model’s trimmed MAE by 2.5–3.1 meV/atom, and excluding all 15,278 lowers it by 3.3–4.0 (scripts/secvi_chain_s_dA_z.py, results/analysis/secvi_post_mae_impact.csv). The same exclusions lower the AFLOW–Alexandria floor by 1.8 (16.4 to 14.6) and 2.4 (to 14.0), on the all-chemistry cut alike (scripts/floor_sm_badref_post.py, results/analysis/floor_sm_badref_post.csv). Most of the difference between the two shifts comes from comparing different structures. On the structures that define the floor, with both terms computed on the same structures, the exclusions lower the models’ raw trimmed MAE by 1.8–2.2 and 2.3–2.9 meV/atom and the floor by 1.8 and 2.4. On the held-out split after Rung-0 they lower the models by 1.9–2.3 and 2.4–3.0 and the floor by 1.9 and 2.5 (scripts/floor_sm_badref_samestruct.py, results/analysis/floor_sm_badref_samestruct.csv). On the same structures the two sides therefore shift together, and no ranking changes in any case. The discrepant structures sit on both sides of the model-versus-floor comparison, so removing them can neither manufacture nor erase a model’s standing.

Hull-F1 is barely affected. Of the 10,861 discrepancies that remain after re-referencing, 32 have an AFLOW energy that is too low and sits on AFLOW’s own hull, where it can distort the hull (31 of all 15,278). None is a pure element. Six of the 32 are in the common set every model is scored on (Methods G). Removing them raises hull-F1 by 0.0001–0.0002 for eqV2, eSEN, SevenNet-MF and UMA, and changes no ranking (scripts/badref_hull_poison_post.py, results/analysis/badref_hull_sensitivity_post.csv).

The two targeted exclusion rules are materialized in badref_exclusions.csv—the three-pattern SG-69 family (1,612 structures across all strata, found by rule and then inspected, a superset of the 1,526 counted in Supplementary Note 13) and a keep-one-representative dedup of broken unary copies (11,477 entries)—but headline metrics in this paper are computed on the full test set. The list ships with the reproduction repository as a robustness artifact whose adoption as a default filter is left to future revisions.

Familiarity, novelty, and generalization

The familiarity test, the matched comparisons, the near-hull comparison, and the OQMD campaign design are Methods J.

Formation-energy binning (the SI form of Figs. 8–9; scripts/analyze_err_vs_ef.py → results/analysis/error_vs_ef.csv; Supplementary Fig. 2). Binned in 50-meV/atom slices of AFLOW $E_{\mathrm{f}}$ (73 bins per model with ≥100 structures each, −1.1 to +2.7 eV/atom), every model’s median $|\mathrm{error}|$ climbs with formation energy. The table gives the Spearman correlation between per-bin median error and bin center.

Model	$\rho$	Model	$\rho$
SevenNet-0 †	0.99	MatterSim-1M †	0.91
M3GNet	0.97	GRACE-2L	0.91
MACE-MPA	0.97	eqV2	0.89
CHGNet	0.96	MatterSim-5M	0.84
eSEN	0.95	Orb-v3	0.80
MACE-OMAT	0.93	SevenNet-MF	0.78
DPA-3	0.93	UMA	0.73

† superseded checkpoints.

Eleven of the fourteen sit at 0.84–0.99. The three low outliers (Orb-v3 0.80, SevenNet-MF 0.78, UMA 0.73) go noisy only in the sparse bins above +2 eV/atom—restricted to $E_f \le2$ eV/atom, all fourteen sit at 0.87–0.99, the numbers the main text quotes.

The rise tracks the sign of the energy, not its magnitude: at matched $|E_f| \approx1$ eV/atom (bins centered −975 vs +975 meV/atom), median error is 1.6–4.7× larger on the unfavorable side (panel median 2.8×; MACE-MPA 6.7 → 26.1 meV/atom, CHGNet 36.1 → 146.0, eqV2 3.4 → 8.2; smallest DPA-3 16.2 → 26.3). $E_{\mathrm{hull}}$ is the sharper axis (scripts/analyze_err_vs_hull.py → err_vs_hull.csv, quartiles_ef.csv, quartiles_ehull.csv): across the ten $E_{\mathrm{hull}}$ bins (edges 0–6,400 meV/atom, against the corrected hulls of Methods G) twelve of the fourteen models rise monotonically and DPA-3 and MACE-MPA reach Spearman 0.988. The lowest-to-highest $E_{\mathrm{hull}}$ quartile gives the main text’s Figure 9 numbers exactly—SevenNet-MF 7.3 → 17.2 meV/atom, MACE-MPA 7.1 → 23.1, CHGNet 52.1 → 145.8; across the panel the rise runs from 1.6× (eqV2) to 3.3× (MACE-MPA)—while the same models binned on $E_f$ quartiles start higher and rise less cleanly (9.1 → 16.3, 9.0 → 20.7, 81.3 → 127.4, the first two $E_f$ quartiles non-monotonic for SevenNet-MF and CHGNet).

Formation-energy distribution (Figure 8; ef_dist_compare.csv, ef_overlap_compare.csv, oqmd_metals_ef_dist.csv; full-OQMD pipeline in scripts/oqmd_dump/). On the 50-meV/atom histograms, AFLOW’s deduplicated metals (1,042,241 entries) put 71.2% of structures above zero, with the median in the +150–200 meV/atom bin (the caption’s unbinned +172). The 448,532 AFLOW structures that Alexandria holds as the same crystal (one per metallic same-crystal pair) track that shape at 72.1% above zero (also 72.1% on Alexandria’s own energies, median +176 versus AFLOW’s +178). OQMD’s complete metallic set (111,901 entries from the qmpy dump) sits at 66.6% above zero with its median in the same bin (the caption’s +159)—three independently constructed databases agree that the combinatorial libraries skew unfavorable. (Provenance: the exact medians, the 448,532 Alexandria-overlap and the 111,901 OQMD counts come from scripts/ef_distribution_medians.py → results/analysis/ef_distribution_medians.csv, which also writes the histograms of ef_overlap_compare.csv.) The scored benchmark itself (ef_dist.csv, 1,420,601 structures) is marginally less skewed: 68.8% above zero, median bin +100–150. The floor along the same axis is in Supplementary Note 5.

Matched comparisons (Methods J; scripts/r1_2_matched_controls.py → results/analysis/r1_2_matched_controls.csv, per-cell counts in r1_2_matched_cells.csv; scripts/r1_2_chemistry_controls.py → r1_2_chemistry_controls.csv). Supplementary Table 20 gives the novel-minus-familiar gap of Figure 7 under each pair of control, with 95% intervals from 2,000 resamples of chemical systems. Every control uses the structures that carry an AFLOW $E_{\mathrm{hull}}$, whose unmatched gap is the Raw column, the gap Figure 7 shows. Matching on hull distance or formation energy alone is not meaningful for the five models trained without OMat24 or Alexandria. Three fifths of their familiar structures are elements, and they dominate the familiar side far above the hull.

Matching on hull distance removes little of the OMat-trained models’ gap (the main text’s generalization section), and the result depends little on which property of the structures we match. Matching on formation energy instead, or on either one together with the number of elements, leaves every OMat-trained residual between 4.3 and 11.7 meV/atom. Element content alone explains none of the gap. When we reweight the familiar structures so that each element occurs in them as often as in the novel ones ⁵, the gap stays about the same or grows (eqV2 5.3 to 5.5, eSEN 11.6 to 13.1, Orb-v3 10.6 to 14.2). Balancing element content on top of hull distance and on top of the number of elements leaves 6.8 to 10.4 meV/atom for seven OMat-trained models and +2.7 [+2.1, +3.2] for eqV2, each interval excluding zero (Supplementary Table 20).

The strictest comparison pairs familiar and novel structures from the same chemical system and the same hull-distance bin, which 60–65% of the OMat-trained models’ novel structures allow and 2% of the others’. On that subset the residual is 4.4 meV/atom for eqV2 and 8.9 to 14.7 for the other OMat-trained models (Supplementary Table 22). Hull matching alone gives residuals of similar size on the same subset (larger for eqV2, DPA-3 and MACE-OMAT, equal for GRACE-2L, smaller for the other four), so matching chemistry exactly neither removes the gap nor explains it.

Supplementary Table 20. Novel-minus-familiar gap (meV/atom, [95% interval]), familiar = the same crystal is in the model’s training data, novel = every other structure (the three novel tiers of Methods J), with per-model familiarity tags. Arity is the number of elements. “Raw” is the unmatched gap on the structures that carry an AFLOW $E_{\mathrm{hull}}$, the population every control uses, and is the gap Figure 7 shows. “System × hull” is computed on the subset of the novel structures that have familiar partners in the same chemical system and hull bin. Supplementary Table 21 gives its coverage.

Model	Raw (Fig. 7)	Hull	Hull × arity	$E_f$	$E_f$ × arity	Elements	Elements + hull + arity	System × hull (subset)
eqV2	+5.3 [+4.8, +5.9]	+4.3 [+3.8, +4.8]	+4.3 [+3.8, +4.8]	+4.6 [+4.1, +5.1]	+4.4 [+3.9, +4.9]	+5.5 [+5.1, +5.9]	+2.7 [+2.1, +3.2]	+4.4 [+4.0, +4.8]
eSEN	+11.6 [+11.0, +12.2]	+9.6 [+9.1, +10.2]	+10.3 [+9.5, +11.3]	+10.5 [+9.9, +11.1]	+10.3 [+9.8, +11.0]	+13.1 [+12.6, +13.5]	+9.5 [+8.9, +10.1]	+14.7 [+13.9, +15.6]
SevenNet-MF	+10.1 [+9.5, +10.7]	+8.0 [+7.5, +8.6]	+8.9 [+8.1, +10.0]	+9.1 [+8.5, +9.6]	+8.9 [+8.3, +9.6]	+12.2 [+11.7, +12.6]	+8.5 [+7.9, +9.1]	+12.4 [+11.6, +13.2]
DPA-3	+8.6 [+8.1, +9.2]	+7.7 [+7.2, +8.2]	+7.8 [+7.3, +8.3]	+8.7 [+8.2, +9.2]	+8.3 [+7.8, +8.9]	+10.5 [+10.0, +10.9]	+7.1 [+6.5, +7.6]	+8.9 [+8.5, +9.3]
UMA	+7.9 [+7.4, +8.4]	+7.7 [+7.2, +8.2]	+7.9 [+7.5, +8.5]	+8.4 [+7.9, +8.9]	+8.4 [+7.9, +9.0]	+9.3 [+8.9, +9.7]	+6.8 [+6.3, +7.3]	+10.1 [+9.6, +10.5]
GRACE-2L	+7.8 [+7.2, +8.3]	+7.4 [+6.9, +8.0]	+7.7 [+7.2, +8.3]	+8.2 [+7.6, +8.8]	+8.2 [+7.6, +8.8]	+10.2 [+9.7, +10.6]	+7.3 [+6.7, +7.8]	+9.3 [+8.9, +9.8]
MACE-OMAT	+7.9 [+7.3, +8.5]	+7.5 [+6.9, +8.0]	+7.8 [+7.3, +8.4]	+8.3 [+7.7, +8.8]	+8.4 [+7.7, +9.0]	+10.4 [+10.0, +10.8]	+7.4 [+6.8, +7.9]	+8.9 [+8.5, +9.4]
Orb-v3	+10.6 [+9.9, +11.3]	+10.3 [+9.7, +11.0]	+11.0 [+10.4, +11.6]	+11.4 [+10.8, +12.1]	+11.7 [+11.1, +12.4]	+14.2 [+13.6, +14.7]	+10.4 [+9.8, +11.1]	+14.4 [+13.9, +15.0]
MACE-MPA	+15.6 [+15.0, +16.2]	+12.9 [+12.3, +13.5]	+13.8 [+13.0, +15.0]	+14.3 [+13.7, +15.0]	+14.0 [+13.4, +14.6]	+16.6 [+16.1, +17.0]	+11.4 [+10.7, +11.9]	+14.1 [+13.3, +15.1]
MatterSim-5M	+35.1 [+32.1, +37.5]	—	+10.0 [+2.9, +16.4]	—	+11.3 [+8.0, +14.2]‡	+32.6 [+30.6, +34.4]	+10.9 [+2.8, +17.4]	+6.5 [+3.7, +9.0]
MatterSim-1M †	+38.9 [+34.9, +42.4]	—	+12.1 [+5.0, +18.3]	—	+17.6 [+13.0, +20.6]‡	+36.2 [+34.1, +38.0]	+13.2 [+5.3, +19.8]	+7.9 [+3.9, +12.1]
M3GNet	+32.6 [+26.9, +37.8]	—	+11.8 [+4.5, +18.1]	—	+17.1 [+13.7, +19.8]‡	+32.8 [+30.7, +34.5]	+8.6 [−0.4, +15.0]	+8.7 [+3.0, +15.0]
SevenNet-0 †	+44.1 [+31.7, +55.4]	—	+28.7 [+21.3, +35.5]	—	+29.5 [+26.0, +33.4]‡	+55.0 [+52.7, +57.0]	+26.8 [+18.1, +33.2]	+5.4 [−0.9, +11.2]
CHGNet	+77.7 [+65.0, +88.7]	—	+42.8 [+34.8, +50.0]	—	+48.5 [+45.0, +52.9]‡	+82.6 [+79.6, +84.9]	+38.2 [+27.7, +46.2]	+5.0 [−0.7, +10.3]

—Not reported for the five models trained without OMat24 or Alexandria. Without the number of elements, the familiar side of the high-energy bins is dominated by elemental polymorphs, which have no novel counterpart. For those five models a stratum enters the hull × arity and $E_f$ × arity controls at 100 structures per group rather than 200, because their familiar sets are small (Methods J). ‡ An estimate that the 50% coverage rule withholds ($E_f$ × arity for those five models: the retained strata hold 41% of the novel structures).

† superseded checkpoints.

Supplementary Table 21. Structures behind each estimate (familiar / novel). Per-cell counts, systems and MAEs for the hull, hull × arity, $E_f$ and $E_f$ × arity controls are in r1_2_matched_cells.csv.

Model	Hull	Hull × arity (share of novel)	$E_f$	$E_f$ × arity (share of novel)	Elements + hull + arity (familiar effective sample size)	System × hull (systems)
eqV2	681,675 / 735,399	681,675 / 735,399	681,070 / 735,399 (100%)	681,675 / 735,399	677,534 / 735,257 (100%)	681,577 / 735,399 (91,737)	553,769 / 476,929 (11,566)
eSEN	682,046 / 735,021	682,046 / 735,399	681,439 / 735,020 (100%)	682,046 / 735,399	677,846 / 734,890 (100%)	681,948 / 735,021 (91,692)	553,867 / 476,479 (11,566)
SevenNet-MF	682,070 / 735,153	682,070 / 735,153	681,465 / 735,145 (100%)	682,070 / 735,153	677,927 / 735,019 (100%)	682,070 / 735,153 (91,703)	553,863 / 476,801 (11,566)
DPA-3	559,047 / 852,726	559,047 / 852,726	558,494 / 852,672 (100%)	559,047 / 852,726	557,720 / 852,710 (100%)	559,047 / 852,726 (95,585)	473,100 / 524,164 (10,828)
UMA	551,455 / 865,603	551,455 / 865,603	550,856 / 865,694 (100%)	551,455 / 865,603	551,455 / 865,603 (100%)	551,440 / 865,603 (98,416)	473,668 / 516,975 (10,834)
GRACE-2L	551,537 / 865,767	551,537 / 865,767	550,901 / 865,604 (100%)	551,537 / 865,767	551,537 / 865,767 (100%)	551,537 / 865,767 (98,430)	473,709 / 517,064 (10,834)
MACE-OMAT	551,470 / 865,875	551,470 / 865,875	550,911 / 865,821 (100%)	551,470 / 865,875	551,470 / 865,875 (100%)	551,470 / 865,875 (98,438)	473,749 / 517,155 (10,834)
Orb-v3	551,037 / 864,772	551,037 / 864,772	550,518 / 864,658 (100%)	551,037 / 864,772	551,037 / 864,772 (100%)	551,037 / 864,772 (98,305)	473,458 / 516,706 (10,834)
MACE-MPA	682,143 / 735,219	682,143 / 735,219	681,413 / 735,152 (100%)	682,143 / 735,219	677,843 / 735,024 (100%)	682,143 / 735,219 (91,732)	553,886 / 476,804 (11,566)
MatterSim-5M	33,128 / 1,383,954	33,081 / 1,361,991	33,081 / 1,361,991 (98%)	33,128 / 1,383,954	30,741 / 560,758 (41%)	33,128 / 1,383,954 (568)	15,135 / 384,379 (532)
MatterSim-1M †	33,179 / 1,384,070	33,112 / 1,361,927	33,179 / 1,384,070 (98%)	33,179 / 1,384,070	30,770 / 560,684 (41%)	33,179 / 1,384,070 (568)	15,366 / 34,561 (532)
M3GNet	33,187 / 1,383,288	33,112 / 1,361,402	33,187 / 1,383,288 (98%)	33,187 / 1,383,288	30,778 / 560,678 (41%)	33,187 / 1,383,288 (568)	15,371 / 34,521 (532)
SevenNet-0 †	33,189 / 1,383,524	33,113 / 1,361,499	33,189 / 1,383,524 (98%)	33,189 / 1,383,524	30,779 / 560,699 (41%)	33,189 / 1,383,524 (568)	15,371 / 34,582 (532)
CHGNet	33,179 / 1,383,337	33,103 / 1,361,332	33,179 / 1,383,337 (98%)	33,179 / 1,383,337	30,770 / 560,608 (41%)	33,179 / 1,383,337 (568)	15,358 / 34,453 (532)

† superseded checkpoints.

Supplementary Table 22. The shared-system subset (chemical system × hull bin with ≥ 5 familiar and ≥ 5 novel structures), under four comparisons (meV/atom, [95% interval]). For the OMat-trained models, hull matching alone and matching on chemical system and hull bin together give residuals of similar size on this subset.

Model	Raw	Hull only	System	System × hull
eqV2	+3.4 [+2.9, +4.0]	+5.7 [+5.2, +6.3]	+5.5 [+5.1, +5.9]	+4.4 [+4.0, +4.8]
eSEN	+9.3 [+8.6, +10.0]	+12.7 [+12.0, +13.4]	+16.2 [+15.4, +17.2]	+14.7 [+13.9, +15.6]
SevenNet-MF	+7.7 [+7.1, +8.4]	+10.7 [+10.0, +11.3]	+13.9 [+13.1, +14.8]	+12.4 [+11.6, +13.2]
DPA-3	+5.9 [+5.4, +6.5]	+9.5 [+8.9, +10.0]	+9.9 [+9.5, +10.4]	+8.9 [+8.5, +9.3]
UMA	+6.1 [+5.6, +6.7]	+9.8 [+9.2, +10.3]	+10.7 [+10.2, +11.2]	+10.1 [+9.6, +10.5]
GRACE-2L	+5.7 [+5.1, +6.3]	+9.3 [+8.8, +9.9]	+10.2 [+9.7, +10.7]	+9.3 [+8.9, +9.8]
MACE-OMAT	+5.5 [+4.9, +6.0]	+9.1 [+8.5, +9.7]	+9.8 [+9.4, +10.3]	+8.9 [+8.5, +9.4]
Orb-v3	+7.9 [+7.3, +8.6]	+13.3 [+12.7, +14.0]	+15.7 [+15.1, +16.4]	+14.4 [+13.9, +15.0]
MACE-MPA	+11.2 [+10.5, +11.9]	+14.4 [+13.8, +15.0]	+15.9 [+15.0, +16.9]	+14.1 [+13.3, +15.1]
MatterSim-5M	+10.1 [+5.3, +14.7]	+6.8 [+1.7, +10.8]	+9.4 [+6.5, +12.4]	+6.5 [+3.7, +9.0]
MatterSim-1M †	+9.9 [+3.5, +16.2]	+4.4 [−2.9, +11.5]	+12.0 [+7.5, +17.5]	+7.9 [+3.9, +12.1]
M3GNet	+4.2 [−6.1, +13.5]	−3.2 [−15.7, +7.5]	+13.6 [+6.1, +22.1]	+8.7 [+3.0, +15.0]
SevenNet-0 †	−11.7 [−30.5, +6.3]	−27.3 [−40.7, −10.9]	+12.9 [+6.1, +19.5]	+5.4 [−0.9, +11.2]
CHGNet	−1.5 [−20.4, +15.9]	−17.8 [−30.3, −3.4]	+15.4 [−8.5, +22.7]	+5.0 [−0.7, +10.3]

† superseded checkpoints.

Coverage against distance above the hull (results/analysis/coverage_vs_ehull.csv). The table gives the fraction of benchmark structures that Alexandria or the Materials Project holds as the same crystal, by AFLOW $E_{\mathrm{hull}}$ bin (identical across models to within 1 point; eqV2’s set shown).

Supplementary Table 23. Coverage against distance above the hull.

$E_{\mathrm{hull}}$ bin (meV/atom)	structures	familiar	fraction
0–25	48,659	40,734	0.837
25–100	136,992	84,767	0.619
100–400	636,792	295,523	0.464
400–1,600	572,394	247,033	0.432
> 1,600	22,237	13,618	0.612

Every OMat24 parent that matches a benchmark structure is also an Alexandria structure (Methods J), so OMat24’s coverage in each bin is no higher than these fractions. Counted on its own, it is 56% within 25 meV/atom of the hull and 37% at 400–1,600 meV/atom (coverage_vs_ehull_by_corpus.csv).

Near-hull disagreement (Figure 10; yardstick from scripts/near_hull_yardstick_common.py → near_hull_dft_floor_sm_hist_summary_common.csv, models from scripts/near_hull_err_hist_common.py → near_hull_err_summary.csv; design in Methods J). Results are in the generalization section. Per model, between 0.10% (UMA) and 0.16% (MACE-OMAT) of near-hull errors exceed 100 meV/atom.

OQMD cross-database campaign (scripts/analyze_oqmd_aligned_trim.py, analyze_oqmd_metals.py; design in Methods J). Results are in the generalization section.

Memorization bound (Methods J; scripts/oqmd_memorization_did.py → oqmd_memorization_did.csv). For each group of seen OQMD compounds we take each model’s improvement over the never-seen compounds and compare it between models trained on that group and models that were not, with the never-seen compounds reweighted to the group’s composition three ways (unweighted, exact strata of element count × crystal system, logistic propensity). On compounds that only Alexandria contains, the four Alexandria-trained models (MACE-MPA, eqV2, eSEN, SevenNet-MF) improve by −1.4 [−2.2, −0.6], −0.1 [−0.9, +0.6] and −1.2 [−2.3, −0.2] meV/atom more than the four OMat24-only models, which never saw them. No scheme detects memorization. Two of the estimates are small and negative, with intervals that exclude zero, so the models that saw these compounds gain no more on them than the models that did not. The largest upper limit, 0.6 meV/atom, bounds any memorization advantage. The positive control behaves as it must. On Materials Project compounds, models trained only on the Materials Project improve by 11.9 [9.7, 14.1], 15.4 [8.9, 25.4] and 5.3 [2.1, 8.9] meV/atom more than OMat24-only models. Any memorization of Alexandria compounds is therefore under 1 meV/atom, compared with 5 to 15 for the training-set advantage the same test detects.

OMat24 held-out forces, by force bin and frame type (scripts/omat24_subsample.py, scripts/score_omat24.py, scripts/omat24_aggregate.py → results/analysis/omat24_force_mae.csv, omat24_force_binned.csv, omat24_by_source.csv). The aggregate ordering of UMA and eqV2 on OMat24’s held-out validation split (the main text’s generalization section) is fragile. If we split the set by reference force, eqV2 is the better model over most of it, beating UMA in seven of the eight bins below 4 eV/Å. UMA’s win comes entirely from strongly distorted configurations: it leads above 4 eV/Å and decisively above 8, where its trimmed force MAE is 0.097 eV/Å compared with eqV2’s 0.142 eV/Å on 7,274 structures. Which of the two leads therefore depends on how much of the evaluation set sits far from equilibrium.

A second feature of this test is the gap between rattled and molecular-dynamics frames, and it needs a control that the raw split does not supply. On the molecular-dynamics frames, UMA, eqV2, and eSEN converge to 0.041 eV/Å, while on the rattled ones they spread to 0.070, 0.085, and 0.085 eV/Å—but the rattled frames also sit six times farther from equilibrium, at a median reference force of 5.7–7.3 eV/Å compared with 1.0–1.1 for the 1,000 K trajectories. Most of that spread therefore reflects distance from equilibrium rather than the manner of the displacement. The same holds for the tail of large errors: SevenNet-0 exceeds the 1 eV/Å trim 1,632 times on rattled structures compared with 53 on molecular-dynamics frames, but the rattled configurations also have far larger reference forces. When we bin by reference force, the gap largely closes. Between 2 and 4 eV/Å, the two kinds of frame agree to within 0.003 eV/Å for all three models.

A small effect survives the control, and one of OMat24’s generation recipes isolates it. The rattled-relax configurations are statically displaced yet carry the smallest reference forces in the set, a median of 0.36 eV/Å compared with 1.05 for the 1,000 K trajectories. Every model is still worse on them: eqV2 0.043 versus 0.037, eSEN 0.044 versus 0.037, UMA 0.050 versus 0.037. Static displacement is therefore genuinely harder than thermal displacement at matched magnitude, but the effect is a few thousandths of an eV/Å compared with the twofold gap the raw split suggests. A benchmark built at relaxed geometries samples neither regime.

Magnetism

Magnetic systems are a labeled slice, never pooled into the nonmagnetic headline. With magnetic defined here as an AFLOW moment above 0.05 μB/atom (Supplementary Note 13, which compares AFLOW’s moments with Alexandria’s, uses 0.1 μB/atom), no model has a larger trimmed MAE on magnetic structures than on nonmagnetic ones (MACE-MPA 29.0 versus 34.0 meV/atom; scripts/slices_post.py → results/analysis/magnetic_slice.csv; the slice pools all rows and omits eqV2 and eSEN, so its MAEs are not Table 3’s). By median error, DPA-3, MACE-OMAT and SevenNet-MF are 0.4 to 0.9 meV/atom worse on the magnetic structures. MACE-MPA’s earlier gap, 90.4 versus 37.4, came from the broken Fe reference that the screen removes (benchmark section), not from magnetism.

Compute cost and the Pareto frontier

Cost measurement—core-seconds, the matched CPU re-measure for the GPU-run models, and eqV2’s exclusion—is Methods K. The efficient set depends on the metric (Figure 11). Frontier membership is in its caption. eSEN costs ~73 core-seconds per structure, the most of any model, and the cheap M3GNet and MatterSim-5M anchor every frontier.