betaFeedback

WorksPhysics and Astronomy

MLIP Arena: Advancing Fairness and Transparency in Machine Learning Interatomic Potentials via an Open, Accessible Benchmark Platform

Abstract

Machine learning interatomic potentials (MLIPs) have revolutionized molecular and materials modeling, but existing benchmarks suffer from data leakage, limited transferability, and an over-reliance on error-based metrics tied to specific density functional theory (DFT) references. We introduce MLIP Arena, a benchmark platform that evaluates force field performance based on physics awareness, chemical reactivity, stability under extreme conditions, and predictive capabilities for thermodynamic properties and physical phenomena. By moving beyond static DFT references and revealing the important failure modes of current foundation MLIPs in real-world settings, MLIP Arena provides a reproducible framework to guide the next-generation MLIP development toward improved predictive accuracy and runtime efficiency while maintaining physical consistency. The Python package and online leaderboard are available at https://github.com/atomind-ai/mlip-arena.

Introduction

The accurate prediction of molecular and material properties has driven innovation for decades and remains crucial for addressing challenges in energy technology, climate change, and drug discovery. While first-principles electronic structure methods have long served as the primary workhorse for property prediction, their computational cost remains prohibitive for scaling atomistic modeling beyond hundreds of atoms. Machine learning interatomic potentials (MLIPs), trained on extensive databases comprising millions of density functional theory (DFT) calculations, have emerged as an efficient and accurate alternative. These models have demonstrated remarkably accurate approximations of the DFT potential energy surface (PES)—the high-dimensional landscape that maps atomic configurations to their corresponding energies and forces—across a wide range of chemical compositions at a fraction of the computational cost of direct DFT evaluations.

Despite excelling in error-based metrics for bulk systems [1], MLIPs trained on the DFT total energy and interatomic forces do not necessarily capture the correct dynamic interactions of atomistic systems [2]. Analogously, classical force fields [3] fit to describe near-equilibrium radial distribution functions cannot capture the energetics of bond-breaking. These limitations may also extend to MLIPs predominantly trained on near- or on-equilibrium configurations. In particular, energy and force regression metrics based on near-equilibrium structures may not reflect performance in downstream scientific tasks. We highlight some specific limitations below.

First, energy and force regression metrics are vulnerable to data leakage, failing to accurately assess a model’s extrapolation and generalization capabilities. This issue is evident in Matbench Discovery [1], where non-compliant models rank highly for crystal stability metrics due to energy overfitting at the expense of forces and finite-temperature capabilities. This may result in poor generalization to structures more diverse in chemistry and away from the energy convex hull. Additionally, high-ranking models often rely on large datasets, risking test set contamination without proper safeguards.

Second, benchmarks tied to specific datasets or DFT functionals lack flexibility in a rapidly evolving field, where larger, more chemically diverse, or higher-accuracy datasets frequently emerge [4, 5, 6, 7]. Static dataset benchmarks quickly become outdated and misleading as newer models trained on larger or proprietary datasets are introduced.

Third, conventional error-based regression metrics often fail to reflect the practical utility and generalizability of MLIPs in real-world applications. Póta et al. [8] recently demonstrated that while some MLIPs exhibit zero-shot capabilities for lattice thermal conductivity prediction, many top-ranked Matbench Discovery models perform worse due to broken crystal symmetry and rough PES derivatives. This underscores that relying solely on regression metrics while ignoring physical priors can widen the gap between model predictions and experimental observables.

To address these challenges, we introduce MLIP Arena, a fair and transparent benchmarking platform for foundation MLIPs. This platform evaluates both the quality of the learned PES and the extent to which models respect the physical laws and symmetries essential to atomistic modeling. Unlike previous error-based DFT reference benchmarks [1, 9, 10, 11, 12], MLIP Arena focuses on assessing physical soundness to better evaluate the utility of MLIPs for downstream applications. By moving beyond error-centric evaluations, it provides more actionable insights for model development and training. Specifically, we assess how well foundation MLIPs capture physics-informed phenomena, their reliability for accurate atomistic simulations, and their readiness for practical scientific research and discovery. The MLIPs evaluated in this work are listed in Table S4.

ModelPrediction1NVTNPTTraining Set2CodeReferenceLicenseCheckpointFirst Release
MACE-MP(M)EFS✓✓MPTrjGitHubBatatia et al. [27]MIT2023-12-03-mace-128-L1_epoch-199.model2023-12-29
CHGNetEFSM✓✓MPTrjGitHubDeng et al. [26]BSD-3-Clausev0.3.02023-02-28
M3GNetEFS✓✓MPFGitHubChen and Ong [28]BSD-3-ClauseM3GNet-MP-2021.2.8-PES2022-02-05
MatterSimEFS✓✓MPTrj, Alex, ProprietaryGitHubYang et al. [29]MITMatterSim-v1.0.0-5M.pth2024-05-10
ORBEFS✓✓MPTrj, AlexGitHubN/AApache-2.0orbff-v1-20240827.ckpt2024-09-03
ORBv2EFS✓✓MPTrj, AlexGitHubNeumann et al. [23]Apache-2.0orb-v2-20241011.ckpt2024-10-15
SevenNetEFS✓✓MPTrjGitHubPark et al. [30]GPL-3.07net-02024-07-11
eqV2(OMat)EFS✓✗OMat, MPTrj, AlexGitHubBarroso-Luque et al. [5]Apache-2.0*eqV2_86M_omat_mp_salex.pt2024-10-18
eSENEFS✓✓OMat, MPTrj, AlexGitHubFu et al. [51]Apache-2.0*esen_30m_oam.pt2025-04-14
EquiformerV2(OC22)EF✓✗OC22GitHubLiao et al. [22]Apache-2.0EquiformerV2-1E4-1F100-S2EFS-OC222023-06-21
EquiformerV2(OC20)EF✓✗OC20GitHubLiao et al. [22]Apache-2.0EquiformerV2-31M-S2EF-OC20-All+MD2023-06-21
eSCN(OC20)EF✓✗OC20GitHubPassaro and Zitnick [91]Apache-2.0eSCN-L6-M3-Lay20-S2EF-OC20-All+MD2023-02-07
DeepMDEFS✓✓MPTrjGitHubZhang et al. [92]GNU LGPLv3.0dp0808c_v024mixu.pth2024-10-09
ALIGNNEFS✓✓MP22GitHubChoudhary and DeCost [93]NIST2024.5.272021-11-15

Table S4. List of first-class supported open-source, open-weight models in MLIP Arena. Custom models could be incorporated through convenient class inherited from ASE Calculator.

Figure 1. Overview of MLIP Arena. Four benchmark categories beyond error-based regression metrics provide actionable insights agnostic to underlying model architecture and DFT reference. Tasks are defined as Prefect (https://www.prefect.io/) workflows to enable advanced task caching, chaining, and parallel/concurrent execution on HPC. Atomic simulation environment (ASE) [13] calculator and database are used. Codebase (https://github.com/atomind-ai/mlip-arena) and online leaderboard on Hugging Face Space (https://huggingface.co/spaces/atomind/mlip-arena) are available.

MLIP Arena benchmarks

MLIP Arena assesses the limitations of MLIPs through four primary perspectives. In § 2.1, we focus on the asymptotic behaviors of MLIP predictions and propose metrics that enable robust and well-balanced ranking from multi-rank aggregation, reducing susceptibility to overfitting on any single metric. § 2.2 tests MLIP robustness and reactivity under extreme conditions using molecular dynamics (MD) simulations, exposing their instabilities and unphysical behaviors. § 2.3 investigate the robustness of MLIP to scenarios with quantified distribution shifts to higher uncertainty. Section 2.4 assesses the predictive capabilities of MLIPs in determining thermodynamic properties and physical phenomena, which requires multiple model passes, higher-order gradients, and more advanced workflows.

Asymptotic analyses on off-equilibrium conditions

Asymptotic Behaviors: The benchmarks evaluate the asymptotic behavior of MLIP predictions on the equation of state (EOS) of stable crystals derived from WBM structures [14] and on the potential energy curves (PECs) of homonuclear diatomics across the periodic table. The quality of prediction is quantified using physical and geometric measures of PECs (including derivative flips, tortuosity, and Spearman’s coefficient), assessed in terms of deviations from physically correct values agnostic to DFT references.

Robust MLIPs should predict reasonable asymptotic behaviors of an atomic system under extreme conditions and symmetry transformations. We specifically focus on the metrics agnostic to underlying DFT functional the model has been trained on and propose physical and geometric measures to assess the general performance of MLIPs. A new suite of metrics to reflect the important aspects of MLIPs for atomistic modeling beyond regression errors is proposed as follows.

Metrics

Smoothness. The major utility of modern MLIPs is the accurate approximation of DFT PES. High-quality PES should be smooth since DFT, as the ground-state electronic structure theory, predicts smooth PES under the assumption of adiabatic approximation on the lowest-energy Born-Oppenheimer surface. Common training objectives on the energy and force of bulk crystals near equilibrium are subject to many-body error cancellation and do not guarantee smoothness. To quantify this effect, we propose tortuosity ( eq. (2)), energy jumps ( eq. (3)), and force/gradient flips to measure the quality of PES.

Tortuosity measures the arc-chord ratio of potential energy curves (PECs, one dimensional slice of PES) projected in the energy dimension. Smooth PECs with a single equilibrium point, like the Lennard-Jones pairwise PEC, have a tortuosity strictly equal to 1. Energy jump detects the change in the sign of energy gradients and sums up the discontinuity with neighboring points. The number of force/gradient flips count the times force/gradient changes sign along the slice.

Short-range repulsion. Atoms at close distances should experience strong repulsion. We use Spearman’s coefficients to measure the monotonicity of PECs at short interatomic distances or under high compression. Robust MLIPs should have Spearman’s coefficients of energy and force close to −1-1 when approaching the repulsive regime. This metric detects the short-range PES holes. The absence of these PES holes and the reasonable repulsion are important for the correct samplings of thermodynamic ensembles essential for correct long-time dynamics and physical property calculations.

Conservative field. Conservative forces are important for energy conserving molecular simulations, and non-conservative forces are known to degrade the stability of thermostats [15]. We calculate the conservation deviation as the MAE between force and the central difference approximation of the derivative of the energy along the PECs ( eq. (1)). We note that energy conservation is a constraint that can be agnostic of the architecture itself, as the standard way it is enforced is by taking gradients of the predicted potential energy in the loss function.

Results

The Birch–Murnaghan equation of state (EOS) ( eq. (4)) [16, 17] describes the relationship between the energy and volume of crystalline solids under external pressure and has been computed at scale for materials in the Materials Project [18]. The detailed EOS curves for a set of representative models—each consisting of 21 sampled points evaluated after ionic relaxation at fixed volume (21,000 ionic relaxation trajectories per model)—are visualized in Figure 2. In Table 1, we present the corresponding metrics and their aggregated rankings to assess the quality of the predicted EOS across diverse crystal structures. Both Figure 2 and Table 1 show that most models exhibit the expected concave-up behavior for the majority of structures, although some models display characteristic failure modes, including short-range holes, shifted energy minima, and spurious spikes.

Figure 2. EOS benchmark on 1,000 WBM structures [14]. The reduced relative energy, ΔEBV0\frac{\Delta E}{BV_{0}}, is normalized by the bulk modulus BB and equilibrium volume V0V_{0} through a rearrangement of the Birch–Murnaghan EOS (eq. (5)). Color indicates the EOS curve of each crystal structure. The number of valid predictions for each model is shown after the model name.

ModelDerivative flips ↓Tortuosity ↓Spearman’s coefficient dEdV\frac{dE}{dV}: compression ↑EE: compression ↓EE: tension ↑Missing ↓
MACE-MPA1.037 (0.283)1.005 (0.054)-0.999368 (0.012)0.996332 (0.039)0.993186 (0.077)2
eSEN1.042 (0.314)1.008 (0.090)-0.999330 (0.012)0.996857 (0.037)0.992097 (0.073)5
MACE-MP(M)1.042 (0.345)1.009 (0.129)-0.999330 (0.011)0.994116 (0.059)0.991586 (0.088)5
MatterSim1.045 (0.376)1.006 (0.055)-0.997350 (0.039)0.992790 (0.078)0.988098 (0.115)3
CHGNet1.105 (0.540)1.015 (0.123)-0.996499 (0.051)0.992997 (0.052)0.986642 (0.117)3
SevenNet1.109 (0.555)1.019 (0.275)-0.998128 (0.026)0.988912 (0.077)0.985958 (0.117)3
M3GNet1.175 (0.676)1.018 (0.149)-0.996321 (0.052)0.989743 (0.065)0.980169 (0.133)5
ORBv21.316 (0.870)1.037 (0.215)-0.991846 (0.082)0.970143 (0.132)0.963746 (0.198)7

Table 1. Equation of state (EOS) benchmark on 1,000 WBM structures [14]. Boldface and underline indicate the best and worst metrics across all MLIPs, respectively. Standard deviations are given in parentheses. Derivative flips are ranked by their absolute deviation from 1. For up-to-date models and aggregated rankings, see the online leaderboard.

We further perform energy–volume scans under more extreme volumetric strains, ranging from −49%-49\% to 75%75\%, but without initial structure or ionic relaxation; that is, the fractional coordinates of ions remain fixed after deformation. See section A.4 for further analysis. To evaluate whether the models truly capture the underlying interactions, we also analyze the potential energy curves (PECs) of homonuclear diatomics, with interatomic distances spanning 0.9 times the covalent radius rcovr_{\mathrm{cov}} to 3.1 times the van der Waals radius rvdwr_{\mathrm{vdw}} across the entire periodic table. This range approximately covers the equilibrium covalent bond length (2rcov2r_{\mathrm{cov}}) and the decay of dispersion interactions (see section A.2 for details). Interestingly, many top-ranked models on bulk crystal EOS and Matbench Discovery perform poorly in pairwise interactions, suggesting that apparent benchmark success may result from plausible many-body error cancellation in bulk systems. See Table S1 for rankings and Figure S1 for select PECs in section A.2. We encourage reader to visit our interactive leaderboard for the complete set of all elements.

ModelConservation deviation [eV/Å]Spearman’s coefficient E: repulsionSpearman’s coefficient F: descendingEnergy jump [eV]Force flipsTortuosity
MACE-MPA0.077-0.997-0.9750.0101.3711.006
MACE-MP(M)0.070-0.997-0.9800.0381.4491.161
MatterSim0.013-0.980-0.9720.0082.7661.021
M3GNet0.026-0.991-0.9470.0293.5281.016
ORBv29.751-0.883-0.9880.9910.9911.287
eSCN(OC20)2.045-0.939-0.9840.8060.6405.335
CHGNet1.066-0.992-0.9250.2912.2552.279
ORB10.220-0.881-0.9541.0191.0261.798
SevenNet34.005-0.986-0.9280.3922.1121.292
eqV2(OMat)15.477-0.880-0.9764.1183.1262.515
eSEN1.170-0.692-0.9195.5624.0001.838
ALIGNN5.164-0.913-0.3109.87630.6691.818
EquiformerV2(OC20)21.385-0.680-0.89138.28222.7758.669
EquiformerV2(OC22)27.687-0.415-0.85564.83721.67415.880

Table S1. PEC quality of homonuclear diatomics based on physical and geometric measures. Boldface and underline represent the best and the worst metrics across all MLIPs, respectively. Select PECs are shown in Figure S1. Detailed definitions and implementation details are available in Section A.1.

Figure S1. Potential energy curves (PECs) of selected homonuclear diatomic molecules, representing four different chemical characteristics—organics, alkali/alkaline earth metals, transition metals, and metalloids—are presented. The curves from different methods are shifted and aligned to zero at the largest separation distance.

Stability and reactivity from molecular dynamics simulation

Stability and Reactivity: Isochoric–isothermal (NVT) molecular dynamics (MD) simulations with a temperature ramp from 300 K to 3000 K over 10 ps, and isobaric–isothermal (NPT) MD simulations from 300 K at 0 GPa to 3000 K at 500 GPa over 10 ps, are performed on random mixture structures (RM24). Reactivity is exemplified by annealing MD simulations of hydrogen combustion. Benchmarking metrics include the fraction of valid runs and runtime speed performance (MD steps per second on a single A100 GPU).

Stable and accurate MD simulations are essential for atomistic modeling. As their name suggests, MLIPs should serve as reliable interatomic potentials for running MD simulations. We benchmark MLIPs for stability under extreme temperature and/or pressure conditions and record their runtime performance using the random amorphous mixture structure database RM24 (section A.5).

Stability under extreme conditions. We perform MD simulations on RM24 structures with a linear temperature schedule from 300 K to 3000 K for 10 ps using Nosé-Hoover NVT thermostats [19]. The number of valid runs and asymptotic speed scaling with the system size are presented in Figure 3a. Because many MLIPs exhibit short-range holes or require increasingly large neighbor lists under high pressure, we additionally apply a linearly increasing pressure from 0 GPa to 500 GPa over 10 ps using Nosé–Hoover NPT barostats (Figure 3b). An MD step is considered valid if the atomic structure exists and has finite energy prediction. This is a relatively lenient criterion when treating MLIPs purely as autoregressive samplers, as it does not account for thermodynamic drifts, fluctuations, or potential structural instabilities. Our benchmark challenges the common belief that equivariant models such as MACE and SevenNet are generally slower than non-equivariant models such as CHGNet and M3GNet. In reality, model architecture, engineering optimizations, and checkpoint quality all contribute to overall MD runtime performance, while the speed and stability of MD trajectories also depend on the chemical system. This is illustrated by the fact that MatterSim shares the same architecture as M3GNet but is significantly more stable and performant. ORBv2 is the fastest and has the best scaling exponent among the tested MLIPs in both heating and compression simulations. However, without an explicit short-range core repulsive potential built in, ORBv2 and many earlier MLIPs could not sustain high-pressure conditions up to 500 GPa.

MD stability on RM24 structures

Figure 3. MD stability on RM24 structures. For NVT (a), we perform Nosé-Hoover thermostats with linearly increasing temperature from 300 K to 3000 K. The number of valid trajectories and the scaling of MD steps per second (SPS) with the number of atoms NN are shown. For NPT (b), Nosé-Hoover thermostats is performed with an additional pressure ramp from 0 GPa to 500 GPa. The size of each point represents the valid steps along each valid trajectory. The power law SPS=aNb\mathrm{SPS}=aN^{b} is used to determine the asymptotic performance of MLIPs (solid line). First 120 structures from RM24 are used for NVT, and first 80 structures are used for NPT. The target length of each trajectory is 10 ps. cuEquivariance kernel was disabled for MACE family models.

Chemical reactivity. Classical force fields are periled by the inaccurate description of chemical reactions. While a bouquet of reactive force fields [3] has been parametrized to mitigate this limitation, they have shown limited transferability from one system to another. Although MLIPs hold the promise to bypass the limitation, one should not assume the reactivity to be guaranteed from pretraining. As an example to test the reactivity, we perform annealing MD simulation to emulate hydrogen combustion. Hydrogen combustion is a challenging out-of-distribution (OOD) test since there are multiple bond breaking and formation events that are poorly represented in most of the available MLIP training sets to date [20]. We evaluate the select models on 1 ns annealing MD simulations (2×1062 \times10^{6} steps with 0.5 fs timestep) by heating a system of hydrogen and oxygen molecules linearly from 300 K to 3000 K, holding at 3000 K, and then cooling back to 300 K. Temperature fluctuations, number of water molecules, and enthalpy change ΔH\Delta H are monitored along MD trajectories (Figure S3). Our results show that the model reactivity is uncorrelated with the prediction accuracy on bulk crystals. See section A.7 for detailed comparison between models.

Figure S3. Hydrogen combustion via annealing NVT MD simulation (128 H2+64 O2⟶128 H2O128\,\mathrm{H}_2 + 64\,\mathrm{O}_2 \longrightarrow128\,\mathrm{H}_2\mathrm{O}). Applied temperature schedule is illustrated in the top panel. The experimental reaction enthalpy of −68.31-68.31 kcal/mol is annotated in the bottom panel [60]. CHGNet, EquiformerV2(OC20), eSCN(OC2), and M3GNet could not finish 1 ns MD trajectories. Experimental adiabatic flame temperature of hydrogen ranges from 2380 K (air) to 3000 K (pure O2\mathrm{O}_2) [61]. Only MACE-MP(M) and EquiformerV2(OC20) ignite within this region. Runtime performance and center-of-mass drift are available in Figure S4.

Robustness to distribution shifts

Distribution shifts: The benchmarks assess violations of energy conservation in MLIP MD trajectories and of rotational equivariance in static force predictions under input distribution shifts, characterized by the differential entropy of atomic local environments. Energy drifts are monitored over eight MD trajectory windows to evaluate conservation, while rotation-induced force errors are computed and averaged within bins defined by differential entropy.

While model architectures that strictly adhere to known symmetries and physical laws have been the standard, recent models [21, 22, 23] have shown competitive performance with non-conservative and non-equivariant force predictions. While models with fewer constraints adhere to symmetries well in-distribution [21, 23], it is important to understand how these models generalize to out-of-distribution systems when considering them for practical use [24]. To this end, we propose an evaluation to measure robustness to symmetries in the face of out-of-distribution structures.

Measuring distribution shifts with differential entropy. To quantify how far a system is from the training distribution, we compute the differential entropy δH\delta H for each structure with respect to the training distribution, as implemented in the QUESTS descriptors proposed by Schwalbe-Koda et al. [25] (see section A.9 for details). The differential entropy provides a measure of uncertainty or “surprise” for one to probe how current MLIPs maintain energy conservation and rotational equivariance in the face of distribution shifts.

Energy conservation. We perform 5 ps NVE simulations with a 1 fs time step, initializing atomic velocities from a Maxwell-Boltzmann distribution at 1000 K. Simulations are conducted on random subsets of each model’s training set. Differential entropy is computed for structures along the simulation trajectories using a sliding window approach. For each 500 fs window, the differential entropy of the midpoint structure is calculated with respect to MPtrj, and the energy difference between the start and end of the window is recorded.

Figure 4 shows that direct force prediction models such as ORB and Equiformer demonstrate a significant correlation between higher differential entropy and greater energy deviation, indicating that non-conservative models tend to (increasingly) violate energy conservation on structures that are surprising in their training set. We also find that direct force prediction models reach more surprising regions of phase space over the course of simulation, indicated by the increasing window numbers as the differential entropy increases in Figure 4. However, gradient-based force prediction models show little correlation between differential entropy and energy conservation ability. Unlike non-conservative models, gradient-based models do not show increasing surprise as the simulation progresses.

Energy conservation under distribution shift

Figure 4. Energy conservation under distribution shift. Energy deviation is calculated for each sliding window during NVE MD simulations for 5 ps. Differential entropy of the structure in the middle of the window is calculated, and the energy deviation from the start to the end of the window is recorded. We report 95% confidence interval error bars and a line of best fit. The order in which windows appear during the simulation is annotated by the number on each point. For direct force prediction models, the simulated trajectories become increasingly surprising over time, as shown by the monotonically increasing numbers from left to right.

Force rotational equivariance. To evaluate the ability of models to learn rotational symmetries from data, we perform a test to quantify learned rotational equivariance. For a rotation matrix R\mathbf{R} and atomic positions as r\mathbf{r}, we measure the MAE between rotated force predictions MAE⁡(F)=13∑i=13∣RF(r)i−F(Rr)i∣\operatorname{MAE}(\mathbf{F}) = \frac{1}{3}\sum_{i=1}^{3} \left|\mathbf{R}\mathbf{F}(\mathbf{r})_i-\mathbf{F}(\mathbf{R}\mathbf{r})_i\right|, where F(r)\mathbf{F}(\mathbf{r}) represents the models’ force predictions for atomic positions r\mathbf{r}. Perfect equivariance would result in a MAE of 0.0 regardless of the rotation angle.

We evaluate models across a random subset of MPTrj [26], a dataset that consists of inorganic bulk materials. We uniformly sample 500 systems and their trajectories from the dataset. We then calculate the force MAE per frame averaged over 10 random rotation axes and 5 angles from 30° to 180°. Figure S6 shows that the non-rotationally equivariant ORB and ORBv2 models [23] exhibit strong correlation between greater differential entropy and higher rotational force MAE. This indicates that while current non-equivariant architectures can adhere to rotational equivariance on in-distribution structures, they may struggle to maintain symmetries for OOD structures with diverse orientations. The rest of the models, which have rotational equivariance built explicitly into the architecture [27, 28, 29, 30], achieve perfect rotational equivariance, as expected.

Rotational equivariance versus differential entropy

Figure S6. Rotational equivariance versus differential entropy. We calculate the mean absolute error (MAE) between each model’s predicted forces and the forces predicted for rotated structures after transforming them back to the original reference frame. We compare this to the bin averages of differential entropy and report 95% confidence interval error bars for 10 bins from low to high differential entropy. Perfect rotational equivariance corresponds to a constant MAE of 0.0. Architectures without explicit rotational equivariance struggle to adhere to rotational equivariance with structures farther from the training distribution.

Thermodynamic properties and phenomenological studies

Thermodynamic Properties: This section provides various benchmarks relevant for downstream property applications and phenomenological studies: vacancy formation and migration from nudged elastic band calculations [31], CO2_2 adsorption for metal-organic frameworks [32], second-order phase transition in perovskite [33], and dynamical stability screening of 2D materials from C2DB database [34].

Vacancy formation and migration energies. Defects, especially vacancies, play a key role in determining the properties of many functional materials used for photovoltaic, catalytic, thermoelectric, and optoelectronic applications [35, 36]. We evaluated six widely used MLIPs capable of predicting stress in elemental face-centered cubic (FCC) and hexagonal close-packed (HCP) crystals, leveraging the vacancy diffusion database by Angsten et al. [31]. The translational symmetry of crystal sites and vacancies requires that the paths and barriers for forward and backward vacancy migration be identical, making this a robust test of a model’s ability to respect crystal symmetry.

Climbing image nudged elastic band (CI-NEB) calculations were performed to analyze vacancy migration barriers. We define path asymmetry (eq. (8)) and barrier asymmetry (eq. (9)) of the migration energy profiles in Section A.8. We found the symmetry of NEB profiles has no strong correlation with built-in equivariance or not, and in general all models perform worse for HCP crystals. Figure 5 presents the NEB energy profiles of vacancy migration in FCC and HCP elemental solids. HCP pathways are chosen to be on basal plane to avoid asymmetrical migrations. We found that MACE-MP(M), MatterSim, and ORBv2 generally relax NEB more robustly than M3GNet, CHGNet, and SevenNet. MatterSim, MACE-MP(M), CHGNet, and SevenNet exhibit near-perfect mirror symmetry around the saddle point for most FCC paths, while MACE-MP(M) achieves the best balance between symmetry and robustness for HCP paths.

Figure 5. NEB profiles of vacancy migration in FCC (a) and HCP (b) elemental crystals. All path lengths are normalized to 1, and all energies are normalized by PBE vacancy migration energy barrier EvmPBEE_{\mathrm{vm}}^{\mathrm{PBE}} as given in [31]. Number of missing predictions, average path asymmetry, and MAPE of maximum energy barrier are annotated on top left.

Extended case studies. In Section A.10, we further assess the downstream utility of MLIPs for three extended case studies: CO2_2 adsorption in metal-organic frameworks (MOFs) (section A.10.1), dynamical stability of 2D materials (section A.10.2), and second-order phase transition in perovskite (section A.10.3). Each study exposes the certain weakness of modern MLIPs. We found possibly due to poorly described non-bonded interaction between CO2_2 molecule and MOFs, many MLIPs deviate at a large degree from experimental adsorption energies and may not be informative enough for MOF virtual screening (section A.10.1). Our results on the Landau-like second-order phase transition in BaZrO3_3 (section A.10.3) uncover subtle failure modes (energy degeneracy and asymmetrical PES) of octahedral tilts predicted by M3GNet and ORBv2. These limitations could cause the models unable to reproduce the correct transition behaviors important for exotic functional properties such as superconductivity [33, 37]. Furthermore, despite the recent saturation in the prediction of thermodynamic stability and lattice thermal conductivity of 3D bulk crystals, our results on the dynamical stability of 2D materials (section A.10.2) indicate that the discovery rate of stable 2D materials remains poor (highest macro F1 score of 0.420 and 0.412 by MACE family models) and there is still a large performance gap in 2D materials space that the success in 3D materials may not ostensibly translate over.

Related Work

Static DFT reference benchmarks. Benchmarking of MLIPs has largely centered around static DFT datasets such as QM9 [38], ANI-1 [39], MD17 [40], SPICE [6, 41], MPTrj [26], GMTKN55 [42], and more. While these have enabled rapid progress, they are tied to specific level of electronic structure theory and the data are generally incompatible with one another. Over time, the community has expanded benchmark domains, but the dependency on static DFT references remains [1, 9, 10, 11, 12, 43, 44]. The Matbench test suite introduced by Dunn et al. [43] compiles 13 tasks (e.g., formation energies, band gaps, elastic moduli) drawn largely from DFT-computed data. Matbench Discovery [1] leveraged the WBM database [14] as an extension beyond the MP [45] for crystal stability classification. Some other benchmarks rely on specific DFT reference while comparing models trained on incompatible dataset [12, 44, 46, 47]. While targeting to higher level of theory is desirable, the higher-level of theory however may not be equally transferable (e.g. coupled-cluster theory describes metallic solids poorly [48]), leading to misleading, non-cross-comparable assessments.

Many models now saturate these test metrics, yet fail to extrapolate to unseen chemistry, strained configurations, or finite-temperature behavior. MLIP Arena complements these efforts by introducing physically grounded tasks that probe model robustness beyond interpolation to a static reference.

Risk of regression error metrics. Standard metrics like MAE and RMSE are known to poorly reflect real-world MLIP utility. Bigi et al. [15], Fu et al. [49] show that models with low force errors may fail to conserve energy in MD, while Póta et al. [8], Loew et al. [50] demonstrate that good regression error metrics on energy and forces does not ensure accurate phonons or thermal transport. Direct force models often violate energy conservation due to the lack of a consistent potential energy surface [15, 51]. These inconsistencies could be due to a misalignment between energy and force pre-training objectives and physical properties of interest [52]. Moreover, average errors can obscure large failures in rare but critical configurations. MLIP Arena addresses these gaps by incorporating task-specific, granular evaluations to provide a more faithful measure of model reliability.

Discussion and Conclusion

Limitations. Traditionally, MLIP training and benchmarks rely on DFT references. This is a computationally cheap way to evaluate models since it only requires a few single-point predictions from the ML model, as opposed to autoregressive benchmarks (i.e. MD simulations). We acknowledge that moving away from DFT references makes it harder to directly compare models, at least on in-distribution data present in standard test datasets. However, the central promise of foundation MLIPs is to generalize to OOD systems and phases—in which case accuracy with respect to in-distribution DFT data becomes auxiliary rather than primary.

Analogously, many other areas of machine learning have moved beyond standard regression metrics [53]. Large language models, for example, are increasingly evaluated on practical, task-oriented performance rather than raw perplexity on training-like data. In the same spirit, MLIP Arena is our first attempt to prioritize qualities that are essential to atomic modeling beyond any DFT reference: symmetry, conservation laws, reasonable asymptotic behaviors and thermodynamic properties that any interatomic potential should satisfy for practical utility.

Opportunities. Reference-agnostic benchmarks like MLIP Arena could motivate and guide new directions in model development and training that explicitly tackle generalization and downstream utility, particularly through reinforcement learning [54], implicit differentiation [55], and test-time training [24] approaches. MLIP Arena provides reproducible workflows that can be scaled for high-throughput reward data generation across a broad range of practically relevant OOD tasks, facilitating the exploration of these training paradigms.

In summary, we present MLIP Arena, an open benchmarking platform that avoids simplistic regression metrics susceptible to error cancellation and instead focuses on evaluating physical awareness and practical utility. Our analysis uncovers some new insights: gradient-based force predictions may exhibit non-conservative behavior; alignment between training dataset size and better model performance is not always guaranteed but depends on design choice; and current MLIPs have not saturated in reactivity and robustness under distribution shifts. MLIP Arena serves as a transparent and reproducible workflow orchestrator, guiding the development of MLIPs with improved adherence to physical principles, runtime performance, and predictive capability.

Author Contribution Statement

Author contribution matrix showing the contributions of YC, TK, CV, MCK, EW, IA, HP, YJ, JK, AW, DC, SMB, MA, and ASK across Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, and Writing – review & editing

Figure 6 (PDF p. 10).

Acknowledgments

We acknowledge funding through the DOE, Office of Science, Office of Basic Energy Sciences, Materials Sciences and Engineering Division, under Contract No. DE-AC02-05-CH11231 within the Materials Project program (KC23MP). The benchmarks were developed and performed using resources of the National Energy Research Scientific Computing Center (NERSC), a Department of Energy Office of Science User Facility using NERSC award BES-ERCAP0032604. YC received the support from Taiwan-UC Berkeley Fellowship jointly offered by Ministry of Education in Taiwan and UC Berkeley. TK was supported by the Toyota Research Institute as part of the Synthesis Advanced Research Challenge. MCK was supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE-2146752. SMB was supported by the Energy Storage Research Alliance "ESRA" (DE-AC02-06CH11357), an Energy Innovation Hub funded by the U.S. Department of Energy, Office of Science, Basic Energy Sciences.

We thank Aaron Kaplan for advice and comments on the manuscript, and Janosh Riebesell, Philipp Benner, Patrick Huck, Rouxi Yang, Evan Walter Clark Spotte-Smith, and Bowen Deng for early discussions; and Jan Janssen, Rhys Goodall, Abhijeet Gangan, and Han Yang for valuable exchanges.

Supplementary Information

Details on metrics

Conservative field. Conservative forces are important for physical and stable MD simulations, as extra energy will be injected into or extracted from the system through non-conservative forces, degrading the stability of thermostats [15]. Some models use direct force prediction [22] or apply post-hoc correction [23] to achieve better prediction errors or smaller drifts during MD simulations. Despite enhanced speed performance, these non-conservative forces may violate the law of energy conservation, undermining the stability of phonon and MD simulations and the predictive power on finite-temperature thermodynamics quantities [8]. To quantify the deviation of force prediction from the conservative field, we compute the MAE between force and the central difference of energy along the homonuclear diatomic curves:

Conservation deviation=⟨∣F(r)⋅r∥r∥+∇rE∣⟩r=∥r∥(1)\text{Conservation deviation} = \left\langle\left| \mathbf{F}(\mathbf{r}) \cdot\frac{\mathbf{r}}{\lVert\mathbf{r} \rVert} + \nabla_{\mathbf{r}} E \right| \right\rangle_{\mathbf{r}=\lVert\mathbf{r} \rVert} \tag*{(1)}

The forces are projected onto the direction of interatomic vectors. Note that this definition is only valid for diatomic interaction but a well-defined, manageable alternative for the exploding combinatorics of hetero-nuclfgear, many-body interactions. Many modern MLIPs have many-body forces, and more careful decomposition of many-body contributions needs to be considered for those cases.

Short-range stiffness. Atoms at close distance should experience strong repulsion. Despite the inaccuracies of DFT calculations at short interatomic distances (section A.3), the well-behaved classical force fields and MLIPs should reproduce strong repulsive interactions between atoms at short range distances. In fact, Deng et al. [56] has indicated prominent softening across MLIPs trained on MPTrj, which consists of crystal relaxation trajectories close to equilibrium. Softened potentials often have early drop in energy and forces at the short range, leading to increased probability of instability. To quantify this behavior, we use Spearman’s coefficients to evaluate the repulsiveness of energy curves (E: repulsion in Table S1) at the distance range r∈[rmin⁡,req]r \in[r_{\min}, r_{\mathrm{eq}}], where req=arg⁡min⁡r∈[rmin⁡,rmax⁡] E(r)r_{\mathrm{eq}} = \underset{r \in[r_{\min},r_{\max}]}{\arg\min}\ E(r) is taken as the equilibrium internuclear distance. Force curves (F: descending) are evaluated at the distance range between rmin⁡r_{\min} and arg⁡min⁡r∈[rmin⁡,rmax⁡] F(r)\underset{r \in[r_{\min},r_{\max}]}{\arg\min}\ F(r) where the largest attractive (the most negative) force happens.

Smoothness. The smoothness of a PEC can be heuristically estimated by tortuosity as the ratio between total variation in energy TV⁡rmin⁡rmax⁡(E)\operatorname{TV}_{r_{\min}}^{r_{\max}}(E) and the sum of absolute energy differences between shortest separation distance rmin⁡r_{\min}, equilibrium distance reqr_{\mathrm{eq}}, and longest separation distance rmax⁡r_{\max}. This is essentially the arc-chord ratio projected in the energy dimension:

Tortuosity=∑ri∈[rmin⁡,rmax⁡]∣E(ri)−E(ri+1)∣∣E(rmin⁡)−E(req)∣+∣E(req)−E(rmax⁡)∣(2)\text{Tortuosity} = \frac{ \sum_{r_i \in[r_{\min},r_{\max}]} \left|E(r_i)-E(r_{i+1})\right|}{\left|E(r_{\min})-E(r_{\mathrm{eq}})\right|+\left|E(r_{\mathrm{eq}})-E(r_{\max})\right|} \tag*{(2)}

. The Lennard-Jones potential and any potentials with single repulsion-attraction transition or pure repulsion have tortuosity equal to 1. Note that the true PECs of some elements may have intermediate range energy barriers and thus ideally the elemental average across the periodic table should be slightly above one. For the simplicity of this metric, we rank the models by the absolute difference with 1.

We also identify the sign changes of energy gradients on PECs to extract the energy jump on both sides to the neighboring sampled points, which can be written down verbatim:

Energy jump=∑ri∈[rmin⁡,rmax⁡]∣sign⁡[E(ri+1)−E(ri)]−sign⁡[E(ri)−E(ri−1)]∣×(∣E(ri+1)−E(ri)∣+∣E(ri)−E(ri−1)∣)(3)\text{Energy jump} = \sum_{r_i \in[r_{\min},r_{\max}]} \left|\operatorname{sign}\left[E(r_{i+1})-E(r_i)\right]-\operatorname{sign}\left[E(r_i)-E(r_{i-1})\right]\right| \times\left(\left|E(r_{i+1})-E(r_i)\right|+\left|E(r_i)-E(r_{i-1})\right|\right) \tag*{(3)}

The smoother PEC has lower tortuosity and total energy jump.

Homonuclear diatomics

Pairwise interactions are the most important interactions in atomistic systems. PECs have the benefit of being less vulnerable to data leakage as DFT references for PECs are difficult to calculate due to multiple possible spin configurations, basis set incompleteness in local-orbital DFT codes, and convergence issues in plane-wave DFT codes. In Table S1, we compute six physical and geometric measures to rank the homonuclear PECs of MLIPs in three aspects: conservative field, short-range stiffness, and smoothness, as we discuss in the previous subsection (section A.1).

Two atoms are placed inside a vacuum box and the predictions are made with separation distances ranging from 0.9 covalent radius rcovr_{\mathrm{cov}} to 3.1 van der Waals radius rvdwr_{\mathrm{vdw}} or to 6 Å if van der Waals radius is not available. The range of interatomic distances is chosen heuristically by the fact that the equilibrium bond length is about the sum of covalent radii (for homonuclear diatomics this is 2rcov2r_{\mathrm{cov}}) and the interatomic energy and forces plateau around 2rvdw2r_{\mathrm{vdw}}. We increase the distance range by the factor of 50% of 2rcov2r_{\mathrm{cov}} and 2rvdw2r_{\mathrm{vdw}} and further extend both ends by 10% of radii. The shortest, equilibrium, and longest separation distances are denoted as rmin⁡r_{\min}, reqr_{\mathrm{eq}}, and rmax⁡r_{\max} respectively. Both energy and force curves are performed at 0.01 Å interval for dense samplings.

Inaccuracies of PAW DFT calculations at short interatomic distances

Due to the classical treatment of nuclei, frozen core approximation, and smoothed core electron wavefunctions in the projected-augmented wave (PAW) DFT formalism, when two atoms are too close to each other, electron wavefunctions start to overlap and oscillate significantly. In such cases, PAW projectors and plane-wave basis set may not accurately describe core electrons and their interactions with valence electrons, leading to large inaccuracies.

Equation of state and energy-volume scan

Structure selection. Structures were selected from the WBM dataset [14] with a slight bias to adjust for the elemental imbalance in the original paper. That is, each structure was assigned a probability for selection based on the prevalence of the elements it contains relative to the overall distribution of elements in the dataset. Elements with lower prevalence in the original dataset were assigned a higher probability of selection; then, 1000 structures from WBM were selected according to these assigned probabilities.

Equation of state. The EOS benchmark protocol in Arena includes first unconstrained structure optimization at 0 K and subsequently multiple energy calculations of isotropic deformations, including ionic relaxation at volumetric strain ranging from −20 % to 20 % of the optimized structure. After ionic relaxation of 21 deformed structures for each crystal, Birch–Murnaghan EOS is fitted with the following equation:

E=E0+9BV016[(η2−1)2(6+B′(η2−1)−4η2)],η=(VV0)13(4)E = E_0 + \frac{9BV_0}{16}\left[(\eta^2-1)^2\left(6+B'(\eta^2-1)-4\eta^2\right)\right], \qquad\eta= \left(\frac{V}{V_0}\right)^{\frac{1}{3}} \tag*{(4)}

where V0V_0 is the equilibrium volume after initial structure optimization, and BB and B′B' are the bulk modulus and its pressure derivative from the EOS fit. We calculate the reduced relative energy in Figure 2 by rearranging Equation (4) as:

ΔEBV0=E−E0BV0=916[(η2−1)2(6+B′(η2−1)−4η2)](5)\frac{\Delta E}{BV_0}=\frac{E-E_0}{BV_0}=\frac{9}{16}\left[(\eta^2-1)^2\left(6+B'(\eta^2-1)-4\eta^2\right)\right] \tag*{(5)}

Energy-volume scan. MLIPs should provide reasonable predictions at extreme deformations. In this benchmark, we take 1,000 structures selected from the WBM dataset [14] and uniformly deform them by ±20%\pm20\% along each lattice vector (i.e. from 0.51 to 1.73 of the initial volume). The energy of each deformed structure is evaluated without relaxation—preventing relaxation to another crystal system. Table S2 presents five metrics to evaluate the performance of MLIPs on the energy-volume scan benchmark. Unlike EOS benchmark in § 2.1.2, all of the select MLIPs have no missing predictions. Our result shows the saturation of all five metrics for top-ranked models.

ModelDerivative flips ↓\downarrowTortuosity ↓\downarrowE: compression ↓\downarrowSpearman’s coefficient dEdV\frac{dE}{dV}: compression ↑\uparrowE: tension ↑\uparrow
eSEN1.0000001.000403-0.9983391.0000000.999045
MACE-MPA1.0000001.000676-0.9983390.9993090.998718
CHGNet1.0000001.000629-0.9982790.9439640.999091
MatterSim1.0090001.000567-0.9980970.9997090.993754
eqV2(OMat)1.0350001.000835-0.9982060.9972240.998645
M3GNet1.0020001.002001-0.9975880.9974420.996468
ORBv21.0580001.004065-0.9977700.9707520.997600
SevenNet1.0340001.010025-0.9951640.9465580.994705
MACE-MP(M)1.1210001.080713-0.9438060.9011880.998745
ALIGNN<u>3.909000</u><u>1.375652</u><u>-0.889027</u><u>0.760271</u><u>0.862085</u>

Table S2. Energy-volume scan of 1,000 WBM structures [14]. Boldface and <u>underline</u> represent the best and the worst metrics across all MLIPs, respectively.

In comparison with EOS benchmark, energy-volume scan evaluates the orthogonal performance of MLIPs. Birch–Murnaghan EOS is theoretically defined properties based on finite elastic theory under isothermal condition. EOS benchmark evaluates both model’s capability to search for energy minima under relaxation protocol, thus mixing the consequences of energy minimum location and relaxation trajectory. Energy-volume scan tests more extreme condition and especially exposes short-range PES holes, penalizing the models with known corrugated short-range PES where DFT however may not converge as well. Energy-volume scan is also based on the assumption that the WBM structures are at local energy minima, but further investigation reveals that this assumption does not hold universally. There are a few structures with shifted energy local minima consistent across different MLIPs.

Expected behavior under compression. When subjected to significant compression, crystalline materials are expected to exhibit strong short-range repulsion. We evaluate this behavior using the Spearman’s rank correlation coefficient to quantify the monotonic increase in energy with decreasing volume. Additionally, the energy derivative dEdV\frac{dE}{dV} is expected to steepen in the high-compression regime. MLIPs that are physically consistent should yield Spearman’s coefficients approaching −1-1 in the compressive region of the energy–volume curve.

Expected behavior under tension. Under tensile strain, the system’s energy should also increase monotonically as atomic bonds are progressively stretched. However, as the crystal approaches dissociation into isolated atoms, the slope of the energy–volume curve (dEdV)\left(\frac{dE}{dV}\right) is expected to flatten. Thus, we evaluate only the monotonicity of the energy increase under tension using Spearman’s coefficients. Reliable MLIPs should produce coefficients close to +1+1 in the tensile regime.

Random mixture dataset (RM24)

New materials are often found by reacting two stable materials into a single phase. In the conceptually similar procedure, we generate 1,000 random mixture structures at arbitrary ratio of two stable materials from Materials Project (v2024.12.18). As the binary and ternary compounds are already well covered by MP, we aim at higher component systems of up to six elements from the mixture of binary and ternary systems. All stable binary and ternary materials are first retrieved, totaling 24,430 structures. The number of possible 2-combination is >298M. We randomly selected 1,000 pairs from possible combinations and generate the initial structures using Packmol [57] and Muse [58] to consider periodic boundary conditions. Each generated structure is then relaxed via FIRE optimizer and NVT MD simulation at 1500 K for 10 ps with Ziegler-Biersack-Littmark (ZBL) screened nuclear repulsion potential [59]. The final element count distribution of 1,000 structures is presented in Figure S2. The ASE DB file is available at https://huggingface.co/datasets/atomind/mlip-arena.

Figure S2. Element counts of random mixture dataset (RM24).

MD stability

We performed Nosé-Hoover thermostat and barostat on RM24 structures with linear scheduling of temperature from 300 K to 3000 K and/or pressure from 0 GPa to 500 GPa across 10 ps MD. The number of valid runs and asymptotic speed scaling with the system size are presented in Figure 3a. Using Prefect (https://github.com/PrefectHQ/prefect) utility, we ensure each run has access to the same resource of 1 AMD EPYC 7763 (Milan) CPU core and 1 NVIDIA A100 (Ampere) GPU. Each run has two retries, with timeout of 600 elapsed seconds for each retry. In § 2.2, the frames are marked as invalid if the simulation cannot reach the timestep or have non-numerical energy values.

Hydrogen combustion

CHGNet, EquiformerV2(OC20), eSCN(OC20), and M3GNet were not able to finish 1 ns MD trajectories (see Figure S3). As analyzed in Figure S4, the slow runtime performance of models without built-in equivariance, such as CHGNet and M3GNet, may seem surprising since equivariant models are often more expensive to use. However, we found that molecules condense into droplets at an early stage in CHGNet and M3GNet trajectories, drastically increasing the number of bond and angle edges and therefore slowing down the MD speed.

Figure S4. Hydrogen combustion. (Top left) The final reaction yield at the last MD step. (Top right) MD runtime speed measured in steps per second using single NVIDIA A100 GPU. cuEquivariance kernel was disabled for MACE-MP(M). (Bottom) The center-of-mass (COM) drift displacement during MD trajectory.

While ORB and ORBv2 were fastest in terms of MD steps per second (fig. S4), they could not react hydrogen and oxygen at the elevated temperature and keep the number of water molecules close to zero throughout the trajectories; they also have positive reaction enthalpies, contradicting experimental measurements [60]. Figure S4 also shows that direct force prediction models (EquiformerV2(OC20), ORB) have large center-of-mass drifts (>102> 10^{2} Å) during MD simulations by six orders of magnitude more than gradient-based models. Enforcing net zero forces as implemented by ORBv2 only decreases the drift to (∼2.4\sim2.4 Å), while other models keep drifts around 10−410^{-4} Å scales over 1 ns MD.

Here one should note that enforcing net zero forces does not guarantee zero center-of-mass (COM) drift during MD simulations under thermostats. For canonical ensemble like Nosé Hoover thermostats used here [62], the heat bath acts an extra correction on the equations of motion for the system of particles with coordinates qi\mathbf{q}_i, momenta pi\mathbf{p}_i, masses mim_i, and interaction potential VV

q˙i=pimi,(6)\dot{\mathbf{q}}_i = \frac{\mathbf{p}_i}{m_i}, \tag*{(6)}
p˙i=−∂V∂qi−pipξQ.(7)\dot{\mathbf{p}}_i = -\frac{\partial V}{\partial\mathbf{q}_i} - \mathbf{p}_i\frac{p_{\xi}}{Q}. \tag*{(7)}

where pξp_{\xi} and QQ are the artificial momentum and mass of the thermostat particle [13, 63]. Post-hoc correction to enforce net zero force from the model prediction will only correct the first term. The total momentum drift is not zero, as can be seen by the following simple proof:

∑ip˙i=∑i[−∂V∂qi−pipξQ]=−∑ipipξQ≠0.\sum_i \dot{\mathbf{p}}_i = \sum_i\left[-\frac{\partial V}{\partial\mathbf{q}_i} - \mathbf{p}_i\frac{p_{\xi}}{Q}\right] = -\sum_i \mathbf{p}_i\frac{p_{\xi}}{Q} \ne0.

Note that we in our test we have enforced net zero total momentum at the beginning of each MD simulation, but non-conservative forces and slight numerical errors may still accumulate over MD trajectory. This non-zero momentum drift will induce non-zero COM drift over time as the MD simulations progress. Models failed to interact with heat bath correctly may not reproduce correct thermodynamic ensembles and therefore yield COM drift and incorrect partitions of kinetic and potential energies, as kinetic energies might be taken largely by COM velocity.

Vacancy formation and migration in elemental solids

The benchmarking workflow included geometry optimization of pristine crystals, optimization of defective structure endpoints, and followed by climbing image nudged elastic band (CI-NEB) calculations [64] to identify transition states and determine vacancy migration barriers. Five intermediate images for NEB calculations were generated using the improved image-dependent pair potential (IDPP) method [65]. 57 FCC and 57 HCP crystals from Angsten et al. [31] consists of metallic, metalloid, and noble gas elements.

We define path asymmetry by calculating the mean difference between the left and right wings of normalized NEB profile ϵ(x)=EML(x)EVMPBE\epsilon(x) = \frac{E_{\mathrm{ML}}(x)}{E_{\mathrm{VM}}^{\mathrm{PBE}}} with respect to the middle point x=0.5x = 0.5:

path asymmetry=2∫00.5∣ϵ(0.5−x)−ϵ(0.5+x)∣ dx(8)\text{path asymmetry} = 2 \int_{0}^{0.5} \left|\epsilon(0.5-x)-\epsilon(0.5+x)\right|\,dx \tag*{(8)}

Barrier asymmetry is defined as the ratio of reaction energy to forward barrier height:

barrier asymmetry=ΔEEforward=Ef−EiETS−Ei(9)\text{barrier asymmetry} = \frac{\Delta E}{E_{\mathrm{forward}}} = \frac{E_f-E_i}{E_{\mathrm{TS}}-E_i} \tag*{(9)}

, where EiE_i, EfE_f are energies of initial and final endpoints, and ETSE_{\mathrm{TS}} is the transition state energy. Figure S5 demonstrates the distribution of barrier asymmetry (eq. (S9)) of the vacancy migrations in elemental FCC and HCP crystals. We found that the compliance to symmetry is not strongly correlated with the equivariance and non-equivariance of the underlying MLIPs. MACE-MP(M) and MatterSim produce symmetric pathways. In contrast, ORBv2 and SevenNet tend to have asymmetric migration pathways, possibly due to more corrugated PES with multiple local minima where relaxation trajectories converge to. This might unintentionally lead to more undesirable behaviors and broken symmetries for sophisticated PES and diverse chemistry.

Figure S5. Absolute barrier asymmetry of vacancy migration in FCC and HCP elemental crystals. Compliance to symmetry is not correlated with the (non-)equivariance of the underlying MLIPs. Non-equivariant MLIPs: ORBv2, MatterSim, CHGNet. Equivariant MLIPs: MACE-MP(M), SevenNet.

Details on robustness under distribution shifts

Descriptors consist of two concatenated components, XiRX_i^{R} and XiBX_i^{B}, that describe each central atom ii's radial distances to its kk-nearest neighbors and its bond angles, respectively. XiRX_i^{R} is a vector of length kk,

[w(ri1)ri1 ⋯ w(rik)rik]T,rij≤ri(j+1),(10)\left[\frac{w(r_{i1})}{r_{i1}}\ \cdots\ \frac{w(r_{ik})}{r_{ik}}\right]^{T},\quad r_{ij}\leq r_{i(j+1)}, \tag*{(10)}

where 1≤j≤k1 \le j \le k due to the kk-nearest neighbors approach, rijr_{ij} is the distance between ii and another atom jj, and w(r)w(r) is a smooth cutoff function:

w(r)={[1−(rrc)2]2,0≤r≤rc0,r>rc.(11)w(r) = \begin{cases} \left[1-\left(\frac{r}{r_c}\right)^2\right]^2, & 0 \le r \le r_c \\ 0, & r > r_c. \end{cases} \tag*{(11)}

To retain information not only about radial distances but also about bond angles, we use XijlBX_{ijl}^{B} given by

XijlB=w(rij)w(ril)rjl(12)X_{ijl}^{B}=\frac{\sqrt{w(r_{ij})w(r_{il})}}{r_{jl}} \tag*{(12)}

which describes each neighbor ll of atom jj in the neighborhood of ii. We represent the per-neighbor basis as

XijB=(Xij1B,…,XijkB),Xij1B≥⋯≥Xij(k−1)B.(13)X_{ij}^{B}=\left(X_{ij1}^{B},\ldots,X_{ijk}^{B}\right),\quad X_{ij1}^{B}\ge\cdots\ge X_{ij(k-1)}^{B}. \tag*{(13)}

For each atom, the bond angle descriptor then becomes

XiB=1k∑jXijB.(14)X_i^{B}=\frac{1}{k}\sum_j X_{ij}^{B}. \tag*{(14)}

To represent the training data, we compute these descriptors for every structure in each dataset. We hold out a subset of 50 structures in each dataset and estimate the Shannon information entropy of the remaining data using a kernel density estimate,

H({X})=−1n∑i=1nlog⁡[1n∑j=1nKh(Xi,Xj)],(15)\mathcal{H}(\{\mathbf{X}\})=-\frac{1}{n}\sum_{i=1}^{n}\log\left[\frac{1}{n}\sum_{j=1}^{n}K_h(\mathbf{X}_i,\mathbf{X}_j)\right], \tag*{(15)}

where we use a Gaussian kernel:

Kh(Xi,Xj)=exp⁡(−∥Xi−Xj∥22h2).(16)K_h(\mathbf{X}_i,\mathbf{X}_j)=\exp\left(-\frac{\lVert\mathbf{X}_i-\mathbf{X}_j\rVert^2}{2h^2}\right). \tag*{(16)}

The bandwidth hh was selected according to the default provided by QUESTS [25], which was chosen to rescale the metric space of X\mathbf{X} according to the distance between two FCC environments with a 1% strain. To quantify the surprise of a data point Y\mathbf{Y} compared to the existing observations {Xi}\{\mathbf{X}_i\}, we define the differential entropy δH\delta H as

δH(Y∣{X})=−log⁡[1n∑j=1nKh(Xi,Xj)].(17)\delta H(\mathbf{Y}|\{\mathbf{X}\})=-\log\left[\frac{1}{n}\sum_{j=1}^{n}K_h(\mathbf{X}_i,\mathbf{X}_j)\right]. \tag*{(17)}

Figure S7 shows a correlation between force MAE and differential entropy for each model trained on MPTrj, indicating that the differential entropy is a reasonable measure of distribution shifts for MLIPs. Although these models perform well on in-distribution data, their error on force predictions increases as structures become more surprising, indicating a potential weakness in ability to generalize to out-of-distribution structures.

Eight scatter plots showing force MAE with respect to DFT reference versus differential entropy for MACE-MP(M), CHGNet, MatterSim, ORBv2, SevenNet, MACE-MPA, DeepMD, and eqV2(OMat)

Figure S7. Force MAE with respect to DFT versus differential entropy. We compare the force MAE with respect to reference DFT values to the differential entropy of a random subset of 500 MPTrj structures. We report bin averages and 95% confidence interval error bars. Lines of best fit are provided. All models tend to predict forces less accurately when structures are more surprising.

Extended case studies

CO2 adsorption in metal-organic frameworks (MOFs)

Direct air capture (DAC) targets the removal of CO2 directly from ambient air (about 400 ppm), and is increasingly recognized as indispensable for achieving net-negative greenhouse-gas emissions[66]. In practical, DAC technologies mainly rely on aqueous KOH slurries or amine-based absorbents whose chemisorptive binding affords the requisite affinity but imposes large thermal regeneration cost and chemical degradation. MOFs offer a promising physisorptive alternative for CO2 capture. MOFs possess exceptionally high porosity and tunable structures that allow precise incorporation of functional groups such as open metal sites, diamines, thereby enhancing the affinity for CO2 within their pore to levels suitable for DAC applications. Furthermore, the combination of framework rigidity, high surface area, and chemical stability positions MOFs as highly attractive candidates for durable, high-performance sorbents capable of operating under under relatively mild regeneration conditions.

In this case study, we curated 20 MOFs with experimentally reported QstQ_{\mathrm{st}} values spanning three technologically relevant adsorption regimes and evaluated how accurately MLIPs classify them into correct categories: (1) General (Qst<35Q_{\mathrm{st}} < 35 kJ/mol), (2) post-combustion flue gas (35 kJ/mol <Qst<50< Q_{\mathrm{st}} < 50 kJ/mol), and (3) DAC (50 kJ/mol <Qst<100< Q_{\mathrm{st}} < 100 kJ/mol).

The general adsorption class includes seven representative MOFs such as MOF-5, HKUST-1, UiO-66, ZIF-8, MIL-177, MIL-53-Al, and MOF-74-Fe. They exhibit relatively low CO2 affinities and have not been prominently reported for CO2 capture applications. The post-combustion flue gas class corresponds to MOFs reported for capturing CO2 from power-plant exhaust, where the partial pressure of CO2 is much higher than in ambient air, thus requiring moderate adsorption strengths. This category includes CALF-20, Al-PyMOF, UTSA-16, MUF-16, and ZnH-MFU-4l. Finally, the DAC class comprises MOFs capable of capturing CO2 at extremely low partial pressures, demanding high binding affinities. DAC-relevant materials considered here include SIFSIX-3-Cu, NbOFFIVE-1-Ni, TIFSIX-3-Ni, SIFSIX-18-Ni-beta, en-Mg2(dobpdc), and CFA-1-OH-Zn, and SGU-29.

Common nameCO2 QstQ_{\mathrm{st}} (kJ/mol)CategoryReference
SIFSIX-3-Cu54DAC[67]
NbOFFIVE-1-Ni54DAC[68]
TIFSIX-3-Ni49DAC[68]
SIFSIX-18-Ni-β\beta52DAC[68]
en-Mg2(dobpdc)50DAC[69]
CFA-1-OH-Zn42 (71 in max)DAC[70]
SGU-2951.3DAC[71]
CALF-2039Post-combustion flue gas[72]
Al-PyrMOF28Post-combustion flue gas[73]
UTSA-1639.7Post-combustion flue gas[74]
MUF-1632.3Post-combustion flue gas[75]
MIL-120-Al-AP41Post-combustion flue gas[76]
ZnH-MFU-4l20 (93 in high T)Post-combustion flue gas[77]
Fe-MOF7433.2General[78]
MIL-53-Al26.3General[79]
HKUST-129General[80]
MOF-515General[81]
UiO-6628.6General[82]
ZIF-827General[83]
MIL-17714General[84]

Table S3. Experimentally reported CO2 QstQ_{\mathrm{st}} values used in MLIP-arena, including three categories: DAC, post-combustion flue gas, normal MOFs.

The heat of adsorption is calculated from the statistical average of interaction energies EintE_{\mathrm{int}} using Widom insertion method [32, 85]. The interaction energy EintE_{\mathrm{int}} is determined by energy difference between gas-inserted MOF and individual gas and MOF system:

Eint=EMOF+gas−EMOF−Egas.(18)E_{\mathrm{int}} = E_{\mathrm{MOF+gas}} - E_{\mathrm{MOF}} - E_{\mathrm{gas}}. \tag*{(18)}

The heat of adsorption is then determined from ensemble average:

Qst=−⟨Einte−βEint⟩⟨e−βEint⟩+kBT,(19)Q_{\mathrm{st}}=-\frac{\left\langle E_{\mathrm{int}}e^{-\beta E_{\mathrm{int}}}\right\rangle}{\left\langle e^{-\beta E_{\mathrm{int}}}\right\rangle}+k_{\mathrm{B}}T, \tag*{(19)}

where β=(kBT)−1\beta=(k_{\mathrm{B}}T)^{-1}, kBk_{\mathrm{B}} is Boltzmann constant, and TT is temperature.

All MLIP models are used in combination with D3 Becke-Johnson dispersion correction [86] with cutoff of 40 Bohr radius. Initial MOF structures were first alternately optimized with fixed and relaxed cell protocols until the final maximum atomic force is smaller than 0.05 eV/Å. The Widom insertion of CO2_2 was then performed at 300 K for three rounds for each MOF, with 5,000 insertion trials in each round. The grid spacing between gas insertion points was set at 0.15 Å.

Figure S8 shows distribution of predicted heat of CO2_2 adsorption and average misclassification margin across 20 MOFs. The misclassification margin is defined as the distance between misclassified point to the closest decision boundary. Our results show that MatterSim is the strong MOF classifiers with misclassification margin of 11.30 kJ/mol and misclassification count only 4, while MACE-MP(M) and SevenNet have severe overestimation, possibly due to short-range PES holes.

Figure S8. Classification of MOFs based on predicted heat of CO2_2 adsorption. (Top) Three classes of MOFs based on experimental CO2_2 QstQ_{\mathrm{st}} measurements: (1) general (green area and points), (2) flue gas (blue area and blue points), and (3) DAC (red area and points). The perfect classifiers should predict QstQ_{\mathrm{st}} of CO2_2 in the corresponding regions. (Bottom) Mean misclassification margin and count of misclassified and missing MOFs.

Dynamical stability of 2D materials

Two-dimensional materials are vital to emerging technologies due to their exceptional physical properties and chemical tunability. To evaluate the ability of MLIPs to predict dynamical stability, we randomly selected 505 monolayers from the C2DB database [34, 87] and computed elastic tensors and phonon band structures using Pymatgen [88] and Phonopy [89, 90]. Following the C2DB protocol, a material is labeled dynamically stable if both the elastic tensor eigenvalues and lowest phonon frequencies are non-negative (specifically, both values should be greater than −10−7-10^{-7} to be labeled as stable).

F1 scores (Figure S9) indicate that MACE-MP(M), MACE-MPA, and MatterSim perform best, with macro F1 scores of 0.420, 0.412, and 0.411, respectively. In contrast, CHGNet, ORBv2, and ALIGNN perform significantly worse, with macro F1 scores below 0.30 and stable F1 scores of 0. All models show higher F1 scores for the unstable class—e.g., 0.596 for MACE-MP(M) vs. 0.245 for stable—highlighting a bias toward detecting instability. Confusion matrices (Figure S10) confirm this trend: most models heavily misclassify stable materials as unstable, with CHGNet, ORBv2, and ALIGNN failing to identify any stable structures. These findings reflect current limitations in MLIP generalization to vibrational stability and emphasize the need for improved training strategies that target phonon-related properties.

Figure S9. F1 scores of dynamical stability classification for 2D materials from C2DB database.

Figure S10. Confusion matrices of dynamical stability classification for 2D materials from C2DB database.

Second-order dynamical phase transition in perovskite

Perovskites are a versatile class of materials exhibiting diverse properties, including ferroelectricity, magnetoresistance, ionic conductivity, piezoelectricity, and superconductivity. Barium zirconate (BaZrO3\mathrm{BaZrO}_{3}, BZO) has been predicted and observed to have a second-order phase transition due to dynamical instability in the cubic polymorph [33, 37]. In Figure S11, we probe the anharmonic PES of different MLIPs along the octahedral-tilting phonon mode with different unit cell lattice constants. Energy differences are calculated with respect to the undeformed structures at the respective lattice constants. We observe Landau-like second-order phase transition from quartic to quadratic polynomials in MACE-MP(M), MatterSim, CHGNet, and SevenNet. M3GNet remains in quadratic PES across all structures with close degeneracies. ORBv2 has an asymmetrical PES and multiple energy crossings.

Figure S11. Landau-like second-order phase transition of octahedral-tilting mode in BaZrO3\mathrm{BaZrO}_{3} (BZO). (a) Undeformed 4×4×44 \times4 \times4 supercell of BZO with cubic unit cell lattice constant of 4 A˚4\,\text{\AA}. (b) R-tilt phonon mode with maximum displacement of 0.5 A˚0.5\,\text{\AA}. Ba atoms are transparent for better visualization. (c) Transitional behavior from quadratic to quartic Landau-like potential energy landscape as a function of largest modal displacement for different lattice constants from 3.70 A˚3.70\,\text{\AA} to 4.15 A˚4.15\,\text{\AA}.

Supported models

*Modified Apache-2.0 (Meta)

Additional DFT reference benchmarks

Bulk modulus from equation of state (EOS) calculations. In the vacancy migration task (section A.8), the geometry optimization of each pristine structure is then followed by an EOS fit to compare with GGA-PBE data from Angsten et al. [31]. Figure S12 shows that most of the model can capture the trend up to 400 GPa well, with serious underestimation on a few FCC and several HCP structures.

Figure S12. Bulk modulus of FCC and HCP elemental solids compared with GGA-PBE calculations [31].

modelMAE (GPa)MAPE (%)nNA
MACE-MP(M)18.87828.72
MatterSim19.14228.11
ORBv232.58331.51
M3GNet21.86737.04
CHGNet19.81525.56
SevenNet14.50021.13

Table S5. Bulk modulus of FCC elemental crystals. nNA denotes the number of missing predictions out of 57 entries except for noble gases.

modelMAE (GPa)MAPE (%)nNA
MACE-MP(M)35.96936.35
MatterSim45.86535.55
ORBv241.11636.44
M3GNet21.32122.016
CHGNet21.48426.316
SevenNet21.92517.015

Table S6. Bulk modulus of HCP elemental crystals. nNA denotes the number of missing predictions out of 57 entries except for noble gases.

References

  1. [1]Janosch Riebesell, Rhys EA Goodall, Philipp Benner, Yuan Chiang, Bowen Deng, Gerbrand Ceder, Mark Asta, Alpha A Lee, Anubhav Jain, and Kristin A Persson. A framework to evaluate machine learning crystal stability predictions. Nature Machine Intelligence, 7(6):836–847, 2025.DOI
  2. [2]Xiang Fu, Zhenghao Wu, Wujie Wang, Tian Xie, Sinan Keten, Rafael Gomez-Bombarelli, and Tommi Jaakkola. Forces are not enough: Benchmark and critical evaluation for machine learning force fields with molecular simulations. arXiv preprint arXiv:2210.07237, 2022.arxiv.org/abs/2210.07237
  3. [3]Thomas P Senftle, Sungwook Hong, Md Mahbubul Islam, Sudhir B Kylasa, Yuanxia Zheng, Yun Kyung Shin, Chad Junkermeier, Roman Engel-Herbert, Michael J Janik, Hasan Metin Aktulga, Toon Verstraelen, Ananth Grama, and Adri C T van Duin. The reaxff reactive force-field: development, applications and future directions. npj Comput. Mater., 2(1):1–14, 2016.DOI
  4. [4]Jonathan Schmidt, Tiago FT Cerqueira, Aldo H Romero, Antoine Loew, Fabian Jäger, Hai-Chen Wang, Silvana Botti, and Miguel AL Marques. Improving machine-learning models in materials science through large datasets. Materials Today Physics, page 101560, 2024.
  5. [5]Luis Barroso-Luque, Muhammed Shuaibi, Xiang Fu, Brandon M Wood, Misko Dzamba, Meng Gao, Ammar Rizvi, C Lawrence Zitnick, and Zachary W Ulissi. Open materials 2024 (omat24) inorganic materials dataset and models. arXiv preprint arXiv:2410.12771, 2024.
  6. [6]Peter Eastman, Benjamin P Pritchard, John D Chodera, and Thomas E Markland. Nutmeg and spice: models and data for biomolecular machine learning. Journal of chemical theory and computation, 20(19):8583–8593, 2024.arxiv.org/abs/2406.13112
  7. [7]Aaron D Kaplan, Runze Liu, Ji Qi, Tsz Wai Ko, Bowen Deng, Janosh Riebesell, Gerbrand Ceder, Kristin A Persson, and Shyue Ping Ong. A foundational potential energy surface dataset for materials. arXiv preprint arXiv:2503.04070, 2025.arxiv.org/abs/2503.04070
  8. [8]Balázs Póta, Paramvir Ahlawat, Gábor Csányi, and Michele Simoncelli. Thermal conductivity predictions with foundation atomistic models. arXiv preprint arXiv:2408.00755, 2024.arxiv.org/abs/2408.00755
  9. [9]Haochen Yu, Matteo Giantomassi, Giuliana Materzanini, Junjie Wang, and Gian-Marco Rignanese. Systematic assessment of various universal machine-learning interatomic potentials. Materials Genome Engineering Advances, 2(3):e58, 2024.arxiv.org/abs/2403.05729
  10. [10]Bruno Focassio, Luis Paulo M. Freitas, and Gabriel R Schleder. Performance assessment of universal machine learning interatomic potentials: Challenges and directions for materials’ surfaces. ACS Applied Materials & Interfaces, 2024.DOI
  11. [11]Siya Zhu, Doğuhan Sarıtürk, and Raymundo Arróyave. Accelerating calphad-based phase diagram predictions in complex alloys using universal machine learning potentials: Opportunities and challenges. Acta Materialia, page 120747, 2025.
  12. [12]Daniel Wines and Kamal Choudhary. Chips-ff: Evaluating universal machine learning force fields for material properties. arXiv preprint arXiv:2412.10516, 2024.arxiv.org/abs/2412.10516
  13. [13]Ask Hjorth Larsen, Jens Jørgen Mortensen, Jakob Blomqvist, Ivano E Castelli, Rune Christensen, Marcin Dułak, Jesper Friis, Michael N Groves, Björk Hammer, Cory Hargus, et al. The atomic simulation environment—a python library for working with atoms. Journal of Physics: Condensed Matter, 29(27):273002, 2017.DOI
  14. [14]Hai-Chen Wang, Silvana Botti, and Miguel A. L. Marques. Predicting stable crystalline compounds using chemical similarity. 7(1):1–9. ISSN 2057-3960. doi: 10.1038/s41524-020-00481-6. URL https://www.nature.com/articles/s41524-020-00481-6. Publisher: Nature Publishing Group.
  15. [15]Filippo Bigi, Marcel Langer, and Michele Ceriotti. The dark side of the forces: assessing non-conservative force models for atomistic machine learning. arXiv preprint arXiv:2412.11569, 2024.arxiv.org/abs/2412.11569
  16. [16]Francis Dominic Murnaghan. The compressibility of media under extreme pressures. Proceedings of the National Academy of Sciences, 30(9):244–247, 1944.pubmed.ncbi.nlm.nih.gov/16588651
  17. [17]Francis Birch. Finite elastic strain of cubic crystals. Physical review, 71(11):809, 1947.DOI
  18. [18]Katherine Latimer, Shyam Dwaraknath, Kiran Mathew, Donald Winston, and Kristin A Persson. Evaluation of thermodynamic equations of state across chemistry and structure in the materials project. npj Computational Materials, 4(1):40, 2018.DOI
  19. [19]Denis J Evans and Brad Lee Holian. The nose-hoover thermostat. Journal of Chemical Physics, 83(8):4069–4074, 1985.DOI
  20. [20]Xingyi Guan, Joseph P Heindel, Taehee Ko, Chao Yang, and Teresa Head-Gordon. Using machine learning to go beyond potential energy surface benchmarking for chemical reactivity. Nature Computational Science, 3(11):965–974, 2023.DOI
  21. [21]Eric Qu and Aditi S. Krishnapriyan. The importance of being scalable: Improving the speed and accuracy of neural network interatomic potentials across chemical domains. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=Y4mBaZu4vy.arxiv.org/abs/2410.24169
  22. [22]Yi-Lun Liao, Brandon Wood, Abhishek Das, and Tess Smidt. Equiformer2: Improved equivariant transformer for scaling to higher-degree representations. arXiv preprint arXiv:2306.12059, 2023.arxiv.org/abs/2306.12059
  23. [23]Mark Neumann, James Gin, Benjamin Rhodes, Steven Bennett, Zhiyi Li, Hitarth Choubisa, Arthur Hussey, and Jonathan Godwin. Orb: A fast, scalable neural network potential. arXiv preprint arXiv:2410.22570, 2024.arxiv.org/abs/2410.22570
  24. [24]Tobias Kreiman and Aditi S Krishnapriyan. Understanding and mitigating distribution shifts for machine learning force fields. arXiv preprint arXiv:2503.08674, 2025.DOI
  25. [25]Daniel Schwalbe-Koda, Sebastien Hamel, Babak Sadigh, Fei Zhou, and Vincenzo Lordi. Model-free quantification of completeness, uncertainties, and outliers in atomistic machine learning using information theory. arXiv:2404.12367, 2024. URL https://arxiv.org/abs/2404.12367.
  26. [26]Bowen Deng, Peichen Zhong, KyuJung Jun, Janosh Riebesell, Kevin Han, Christopher J. Bartel, and Gerbrand Ceder. CHGNet as a pretrained universal neural network potential for charge-informed atomistic modelling. Nature Machine Intelligence, 5(9):1031–1041, September 2023. ISSN 2522-5839. doi: 10.1038/s42256-023-00716-3. URL https://www.nature.com/articles/s42256-023-00716-3. Publisher: Nature Publishing Group.DOI
  27. [27]Ilyes Batatia, Philipp Benner, Yuan Chiang, Alin M. Elena, Dávid P. Kovács, Janosh Riebesell, Xavier R. Advincula, Mark Asta, Matthew Aylavon, William J. Baldwin, Fabian Berger, Noam Bernstein, Arghya Bhowmik, Samuel M. Blau, Vlad Cărare, James P. Darby, Sandip De, Flaviano Della Pia, Volker L. Deringer, Rokas Elijošius, Zakariya El-Machachi, Fabio Falcioni, Edvin Fako, Andrea C. Ferrari, Annalena Genreith-Schriever, Janine George, Rhys E. A. Goodall, Clare P. Grey, Petr Grigoriev, Shuang Han, Will Handley, Hendrik H. Heenen, Kersti Hermansson, Christian Holm, Jad Jaafar, Stephan Hofmann, Konstantin S. Jakob, Hyunwook Jung, Venkat Kapil, Aaron D. Kaplan, Nima Karimitari, James R. Kermode, Namu Kroupa, Jolla Kullgren, Matthew C. Kuner, Domantas Kuryla, Guoda Liepionute, Johannes T. Margraf, Ioan-Bogdan Magdău, Angelos Michaelides, J. Harry Moore, Aakash A. Naik, Samuel P. Niblett, Sam Walton Norwood, Niamh O’Neill, Christoph Ortner, Kristian A. Persson, Karsten Reuter, Andrew S. Rosen, Lars L. Schaaf, Christoph Schran, Benjamin X. Shi, Eric Sivonxay, Tamás K. Stenczel, Viktor Svahn, Christopher Sutton, Thomas D. Swinburne, Jules Tilly, Cas van der Oord, Eszter Varga-Umbrich, Tejs Vegge, Martin Vondrák, Yangshuai Wang, William C. Witt, Fabian Zills, and Gábor Csányi. A foundation model for atomistic materials chemistry, March 2024. URL http://arxiv.org/abs/2401.00096. arXiv:2401.00096 [cond-mat, physics:physics].
  28. [28]Chi Chen and Shyue Ping Ong. A universal graph deep learning interatomic potential for the periodic table. Nature Computational Science, 2(11):718–728, November 2022. ISSN 2662-8457. doi: 10.1038/s43588-022-00349-3. URL https://www.nature.com/articles/s43588-022-00349-3. Publisher: Nature Publishing Group.DOI
  29. [29]Han Yang, Chenxi Hu, Yichi Zhou, Xixian Liu, Yu Shi, Jielan Li, Guanzhi Li, Zekun Chen, Shuizhou Chen, Claudio Zeni, et al. Mattersim: A deep learning atomistic model across elements, temperatures and pressures. arXiv preprint arXiv:2405.04967, 2024.arxiv.org/abs/2405.04967
  30. [30]Yutack Park, Jaesun Kim, Seungwoo Hwang, and Seungwu Han. Scalable parallel algorithm for graph neural network interatomic potentials in molecular dynamics simulations. Journal of Chemical Theory and Computation, 2024.DOI
  31. [31]Thomas Angsten, Tam Mayeshba, Henry Wu, and Dane Morgan. Elemental vacancy diffusion database from high-throughput first-principles calculations for fcc and hcp structures. New Journal of Physics, 16(1):015018, 2014.
  32. [32]Yunsung Lim, Hyunsoo Park, Aron Walsh, and Jihan Kim. Accelerating co2 direct air capture screening for metal-organic frameworks with a transferable machine learning force field. 2024.DOI
  33. [33]Erik Fransson, Petter Rosander, Paul Erhart, and G"oran Wahnstr"om. Understanding correlations in bazro3: Structure and dynamics on the nanoscale. Chemistry of Materials, 36(1):514–523, 2023.DOI
  34. [34]Morten Niklas Gjerding, Alireza Taghizadeh, Asbjørn Rasmussen, Sajid Ali, Fabian Bertoldo, Thorsten Deilmann, Nikolaj Rørbæk Knøsgaard, Mads Kruse, Ask Hjorth Larsen, Simone Manti, et al. Recent progress of the computational 2d materials database (c2db). 2D Materials, 8(4):044002, 2021.
  35. [35]Irea Mosquera-Lois, Seán R Kavanagh, Alex M Ganose, and Aron Walsh. Machine-learning structural reconstructions for accelerated point defect calculations. npj Computational Materials, 10(1):121, 2024.
  36. [36]Kamal Choudhary and Bobby G Sumpter. Can a deep-learning model make fast predictions of vacancy formation in diverse materials? AIP Advances, 13(9), 2023.DOI
  37. [37]Petter Rosander, Erik Fransson, Cosme Milesi-Brault, Constance Toulouse, Frédéric Bourdarot, Andrea Piovano, Alexei Bossak, Mael Guennou, and Göran Wahnström. Anharmonicity of the antiferroristortive soft mode in barium zirconate bazro 3. Physical Review B, 108(1):014309, 2023.DOI
  38. [38]Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1–7, 2014.
  39. [39]Justin S Smith, Oxana Isayev, and Adrian E Roitberg. Ani-1, a data set of 20 million calculated off-equilibrium conformations for organic molecules. Scientific data, 4(1):1–8, 2017.
  40. [40]Stefan Chmiela, Alexandre Tkatchenko, Huziel E Sauceda, Igor Poltavsky, Kristof T Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances, 3(5):e1603015, 2017.DOI
  41. [41]Peter Eastman, Pavan Kumar Behara, David L Dotson, Raimondas Galvelis, John E Herr, Josh T Horton, Yuezhi Mao, John D Chodera, Benjamin P Pritchard, Yuanqing Wang, et al. Spice, a dataset of drug-like molecules and peptides for training machine learning potentials. Scientific Data, 10(1):11, 2023.
  42. [42]Lars Goerigk, Andreas Hansen, Christoph Bauer, Stephan Ehrlich, Asim Najibi, and Stefan Grimme. A look at the density functional theory zoo with the advanced gmtkn55 database for general main group thermochemistry, kinetics and noncovalent interactions. Physical Chemistry Chemical Physics, 19(48):32184–32215, 2017.DOI
  43. [43]Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. npj Computational Materials, 6(1):138, 2020.DOI
  44. [44]Anyang Peng, Chun Cai, Mingyu Guo, Duo Zhang, Chengqian Zhang, Antoine Loew, Linfeng Zhang, and Han Wang. Lambench: A benchmark for large atomic models. arXiv preprint arXiv:2504.19578, 2025.arxiv.org/abs/2504.19578
  45. [45]Anubhav Jain, Shyue Ping Ong, Geoffrey Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, and Kristin A. Persson. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL Materials, 1(1):011002, July 2013. ISSN 2166-532X. doi: 10.1063/1.4812323. URL https://doi.org/10.1063/1.4812323.
  46. [46]Rebecca R Brew, Ian A Nelson, Meruyert Binayeva, Amlan S Nayak, Wyatt J Simmons, Joseph J Gair, and Corin C Wagen. Wiggle150: Benchmarking density functionals and neural network potentials on highly strained conformers. Journal of Chemical Theory and Computation, 2025.DOI
  47. [47]Ari Wagen. Nnp arena, 2025. URL https://ariwagen.com/nnp-arena.
  48. [48]Nikolaos Masios, Andreas Irmler, Tobias Schäfer, and Andreas Grü­neis. Averting the infrared catastrophe in the gold standard of quantum chemistry. Physical Review Letters, 131(18):186401, 2023.DOI
  49. [49]Xiang Fu, Zhenghao Wu, Wujie Wang, Tian Xie, Sinan Keten, Rafael Gomez-Bombarelli, and Tommi Jaakkola. Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations, August 2023. URL http://arxiv.org/abs/2210.07237. arXiv:2210.07237 [physics].
  50. [50]Antoine Loew, Dewen Sun, Hai-Chen Wang, Silvana Botti, and Miguel AL Marques. Universal machine learning interatomic potentials are ready for phonons. arXiv preprint arXiv:2412.16551, 2024.arxiv.org/abs/2412.16551
  51. [51]Xiang Fu, Brandon M Wood, Luis Barroso-Luque, Daniel S Levine, Meng Gao, Misko Dzamba, and C Lawrence Zitnick. Learning smooth and expressive interatomic potentials for physical property prediction. arXiv preprint arXiv:2502.12147, 2025.arxiv.org/abs/2502.12147
  52. [52]Yunsheng Liu, Xingfeng He, and Yifei Mo. Discrepancies and error evaluation metrics for machine learning interatomic potentials. npj Computational Materials, 9(1):174, 2023.DOI
  53. [53]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024. URL https://arxiv.org/abs/2403.04132.
  54. [54]Aditya Koneru, Henry Chan, Sukriti Manna, Troy D Loeffler, Debdas Dhabal, Andressa A Berto-lazzo, Valeria Molinero, and Subramanian KRS Sankaranarayanan. Multi-reward reinforcement learning based development of inter-atomic potential models for silica. npj Computational Materials, 9(1):125, 2023.DOI
  55. [55]Sanjeev Raja, Ishan Amin, Fabian Pedregosa, and Aditi S Krishnapriyan. Stability-aware training of machine learning force fields with differentiable boltzmann estimators. arXiv preprint arXiv:2402.13984, 2024.arxiv.org/abs/2402.13984
  56. [56]Bowen Deng, Yunyeong Choi, Peichen Zhong, Janosh Riebesell, Shashwat Anand, Zhuohan Li, Kyujung Jun, Kristin A Persson, and Gerbrand Ceder. Overcoming systematic softening in universal machine learning interatomic potentials by fine-tuning. arXiv preprint arXiv:2405.07105, 2024.arxiv.org/abs/2405.07105
  57. [57]Leandro Martínez, Ricardo Andrade, Ernesto G Birgin, and José Mario Martínez. Packmol: A package for building initial configurations for molecular dynamics simulations. Journal of computational chemistry, 30(13):2157–2164, 2009.DOI
  58. [58]Yuan Chiang. muse, December 2023. URL https://github.com/chiang-yuan/muse.
  59. [59]James F Ziegler and Jochen P Biersack. The stopping and range of ions in matter. In Treatise on heavy-ion science: volume 6: astrophysics, chemistry, and condensed matter, pages 93–129. Springer, 1985.DOI
  60. [60]David R Lide. CRC handbook of chemistry and physics, volume 85. CRC press, 2004.
  61. [61]Anna Hasche, Ali Navid, Hartmut Krause, and Sven Eckart. Experimental and numerical assessment of the effects of hydrogen admixtures on premixed methane-oxygen flames. Fuel, 352:128964, 2023.DOI
  62. [62]William G Hoover. Canonical dynamics: Equilibrium phase-space distributions. Physical review A, 31(3):1695, 1985.pubmed.ncbi.nlm.nih.gov/9895674
  63. [63]Glenn J Martyna, Mark E Tuckerman, Douglas J Tobias, and Michael L Klein. Explicit reversible integrators for extended systems dynamics. Molecular Physics, 87(5):1117–1157, 1996.DOI
  64. [64]Graeme Henkelman, Blas P Uberuaga, and Hannes Jónsson. A climbing image nudged elastic band method for finding saddle points and minimum energy paths. The Journal of chemical physics, 113(22):9901–9904, 2000.DOI
  65. [65]Søren Smidstrup, Andreas Pedersen, Kurt Stokbro, and Hannes Jónsson. Improved initial guess for minimum energy path calculations. The Journal of chemical physics, 140(21), 2014.DOI
  66. [66]Eloy S Sanz-Pérez, Christopher R Murdock, Stephanie A Didas, and Christopher W Jones. Direct capture of co2 from ambient air. Chemical reviews, 116(19):11840–11876, 2016.pubmed.ncbi.nlm.nih.gov/27560307
  67. [67]Osama Shekhah, Youssef Belmabkhout, Zhijie Chen, Vincent Guillerm, Amy Cairns, Karim Adil, and Mohamed Eddaoudi. Made-to-order metal–organic frameworks for trace carbon dioxide removal and air capture. Nature communications, 5(1):4228, 2014.DOI
  68. [68]Soumya Mukherjee, Nividita Sikdar, Daniel O’Nolan, Douglas M Franz, Victoria Gascón, Amrit Kumar, Naveen Kumar, Hayley S Scott, David G Madden, Paul E Kruger, et al. Trace co2 capture by an ultramicroporous physisorbent with low water affinity. Science advances, 5 (11):eaax9171, 2019.
  69. [69]Woo Ram Lee, Sang Yeon Hwang, Dae Won Ryu, Kwang Soo Lim, Sang Soo Han, Dohyun Moon, Jungkyu Choi, and Chang Seop Hong. Diamine-functionalized metal–organic framework: exceptionally high co 2 capacities from ambient air and flue gas, ultrafast co 2 uptake rate, and adsorption mechanism. Energy & Environmental Science, 7(2):744–751, 2014.DOI
  70. [70]Caitlin E Bien, Kai K Chen, Szu-Chia Chien, Benjamin R Reiner, Li-Chiang Lin, Casey R Wade, and WS Winston Ho. Bioinspired metal–organic framework for trace co2 capture. Journal of the American Chemical Society, 140(40):12662–12666, 2018.DOI
  71. [71]Shuvo Jit Datta, Chutharat Khumnoon, Zhen Hao Lee, Won Kyung Moon, Son Docao, Thanh Huu Nguyen, In Chul Hwang, Dohyun Moon, Peter Oleynikov, Osamu Terasaki, et al. Co2 capture from humid flue gases and humid atmosphere using a microporous coppersilicate. Science, 350(6258):302–306, 2015.DOI
  72. [72]Jian-Bin Lin, Tai TT Nguyen, Ramanathan Vaidhyanathan, Jake Burner, Jared M Taylor, Hana Durekova, Farid Akhtar, Roger K Mah, Omid Ghaffari-Nik, Stefan Marx, et al. A scalable metal–organic framework as a durable physisorbent for carbon dioxide capture. Science, 374 (6574):1464–1469, 2021.DOI
  73. [73]Peter G Boyd, Arunraj Chidambaram, Enrique García-Díez, Christopher P Ireland, Thomas D Daff, Richard Bounds, Andrzej Gladysiak, Pascal Schouwink, Seyed Mohamad Moosavi, M Mercedes Maroto-Valer, et al. Data-driven design of metal–organic frameworks for wet flue gas co2 capture. Nature, 576(7786):253–256, 2019.DOI
  74. [74]Alessio Masala, Jenny G Vitillo, Francesca Bonino, Maela Manzoli, Carlos A Grande, and Silvia Bordiga. New insights into utsa-16. Physical Chemistry Chemical Physics, 18(1):220–227, 2016.DOI
  75. [75]Omid T Qazvini and Shane G Telfer. Muf-16: A robust metal–organic framework for pre- and post-combustion carbon dioxide capture. ACS applied materials & interfaces, 13(10):12141–12148, 2021.DOI
  76. [76]Bingbing Chen, Dong Fan, Rosana V Pinto, Iurii Dovgaliuk, Shyamapada Nandi, Debanjan Chakraborty, Nuria García-Moncada, Alexandre Vimont, Charles J McMonagle, Marta Bordonhos, et al. A scalable robust microporous al-mof for post-combustion carbon capture. Advanced Science, 11(21):2401070, 2024.DOI
  77. [77]Rachel C Rohde, Kurtis M Carsch, Matthew N Dods, Henry ZH Jiang, Alexandra R McIsaac, Ryan A Klein, Hyunchul Kwon, Sarah L Karstens, Yang Wang, Adrian J Huang, et al. High-temperature carbon dioxide capture in a porous material with terminal zinc hydride sites. Science, 386(6723):814–819, 2024.DOI
  78. [78]Wendy L Queen, Matthew R Hudson, Eric D Bloch, Jarad A Mason, Miguel I Gonzalez, Jason S Lee, David Gygi, Joshua D Howe, Kyuhо Lee, Tamim A Darwishi, et al. Comprehensive study of carbon dioxide adsorption in the metal–organic frameworks m 2 (dobdc)(m= mg, mn, fe, co, ni, cu, zn). Chemical Science, 5(12):4569–4581, 2014.DOI
  79. [79]Prashant Mishra, Hari Prasad Uppara, Bishnupada Mandal, and Sasidhar Gumma. Adsorption and separation of carbon dioxide using mil-53 (al) metal-organic framework. Industrial & Engineering Chemistry Research, 53(51):19747–19753, 2014.DOI
  80. [80]Lukáš Grajciar, Andrew D Wiersum, Philip L Llewellyn, Jong-San Chang, and Petr Nachtigall. Understanding co2 absorption in cubt mof: comparing combined dft–ab initio calculations with microcalorimetry experiments. The Journal of Physical Chemistry C, 115(36):17925–17933, 2011.
  81. [81]Jason M Simmons, Hui Wu, Wei Zhou, and Taner Yildirim. Carbon capture in metal–organic frameworks—a comparative study. Energy & Environmental Science, 4(6):2177–2185, 2011.DOI
  82. [82]Hussein Rasool Abid, Huyong Tian, Ha-Ming Ang, Moses O Tade, Craig E Buckley, and Shaobin Wang. Nanosize zr-metal organic framework (uio-66) for hydrogen and carbon dioxide storage. Chemical Engineering Journal, 187:415–420, 2012.DOI
  83. [83]Zhijuan Zhang, Shikai Xian, Qibin Xia, Haihui Wang, Zhong Li, and Jing Li. Enhancement of co2 adsorption and co2/n2 selectivity on zif-8 via postsynthetic modification. AIChE Journal, 59(6):2195–2206, 2013.DOI
  84. [84]Jarad A Mason, Kenji Sumida, Zoey R Herm, Rajamani Krishna, and Jeffrey R Long. Evaluating metal–organic frameworks for post-combustion carbon dioxide capture via temperature swing adsorption. Energy & Environmental Science, 4(8):3030–3040, 2011.DOI
  85. [85]Ben Widom. Some topics in the theory of fluids. The Journal of Chemical Physics, 39(11):2808–2812, 1963.DOI
  86. [86]Stefan Grimme, Stephan Ehrlich, and Lars Goerigk. Effect of the damping function in dispersion corrected density functional theory. Journal of computational chemistry, 32(7):1456–1465, 2011.DOI
  87. [87]Sten Haastrup, Mikkel Strange, Mohnish Pandey, Thorsten Deilmann, Per S Schmidt, Nicki F Hinsche, Morten N Gjerding, Daniele Torelli, Peter M Larsen, Anders C R iis-Jensen, et al. The computational 2d materials database: high-throughput modeling and discovery of atomically thin crystals. 2D Materials, 5(4):042002, 2018.DOI
  88. [88]Jimmy-Xuan Shen and Joel Varley. pymatgen-analysis-defects: A python package for analyzing point defects in crystalline materials. Journal of Open Source Software, 9(93):5941, 2024.DOI
  89. [89]Atsushi Togo. First-principles phonon calculations with phonopy and phono3py. Journal of the Physical Society of Japan, 92(1):012001, 2023.DOI
  90. [90]Atsushi Togo, Laurent Chaput, Terumasa Tadano, and Isao Tanaka. Implementation strategies in phonopy and phono3py. Journal of Physics: Condensed Matter, 35(35):353001, 2023.DOI
  91. [91]Saro Passaro and C Lawrence Zitnick. Reducing so (3) convolutions to so (2) for efficient equivariant gnns. In International Conference on Machine Learning, pages 27420–27438. PMLR, 2023.
  92. [92]Duo Zhang, Xinzijian Liu, Xiangyu Zhang, Chengqian Zhang, Chun Cai, Hangrui Bi, Yiming Du, Xuejian Qin, Anyang Peng, Jiameng Huang, et al. Dpa-2: a large atomic model as a multi-task learner. npj Computational Materials, 10(1):293, 2024.DOI
  93. [93]Kamal Choudhary and Brian DeCost. Atomistic line graph neural network for improved materials property predictions. npj Computational Materials, 7(1):185, 2021.DOI
  94. [94]Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, et al. Open catalyst 2020 (oc20) dataset and community challenges. Acs Catalysis, 11(10):6059–6072, 2021.DOI
  95. [95]Richard Tran, Janice Lan, Muhammed Shuaibi, Brandon M Wood, Siddharth Goyal, Abhishek Das, Javier Heras-Domingo, Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, et al. The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysts. ACS Catalysis, 13(5): 3066–3084, 2023.DOI

Paper details

vlnc:W_u96k4hhk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents