betaFeedback

WorksPhysics and Astronomy

Training a Foundation Model for Materials on a Budget

Abstract

Foundation models for materials modeling are advancing quickly, but their training remains expensive, often placing state-of-the-art methods out of reach for many research groups. We introduce Nequix, a compact E(3)-equivariant potential that pairs a simplified NequIP design with modern training practices, including equivariant root-mean-square layer normalization and the Muon optimizer, to retain accuracy while substantially reducing compute requirements. Nequix has 700K parameters and was trained in 100 A100 GPU-hours. On the Matbench-Discovery and MDR Phonon benchmarks, Nequix ranks third overall while requiring a 20 times lower training cost than most other methods, and it delivers two orders of magnitude faster inference speed than the current top-ranked model. We release model weights and fully reproducible codebase at https://github.com/atomicarchitects/nequix.

Introduction

Machine learned inter-atomic potentials (MLIPs) are rapidly improving in capability and scope, with foundation models trained on broad datasets of atomistic materials offering the promise of augmenting or replacing expensive ab initio density functional theory (DFT) calculations [1]. While performance on community benchmarks such as Matbench-Discovery [2] is rising, the computational costs of both data generation and curation as well as the training of MLIP models on these datasets remain prohibitively expensive for many labs.

We pursue an orthogonal goal to scaling: a lower computational cost recipe that preserves strong downstream accuracy. Concretely, we revisit a simplified E(3)E(3)-equivariant architecture based on NequIP [3] with modern training practices: root-mean-square layer normalization for stability, latest custom CUDA kernels [4], and optimizer choices inspired by “speedrunning” deep learning workflows [5]. The resulting model, Nequix, has 700K parameters and can be trained in 100 GPU hours, while remaining competitive with larger and more costly to train models on Matbench-Discovery and other phonon prediction tasks.

Our contributions are threefold: (1) a simplified NequIP architecture featuring an equivariant layer normalization and efficient JAX and PyTorch implementations; (2) a budget-conscious training pipeline leveraging the Muon optimizer [6], achieving fast convergence; and (3) evaluations on the Matbench-Discovery and MDR phonon [7] benchmarks. Compared to prior MPtrj-trained models [8, 9, 1, 10, 11, 12, 13, 14, 15], we rank third (as of August 2025) on both benchmarks at 1/20 the training cost of any other published model and with 100×100\times faster inference than the current top-ranking model.

Preprint.

Methods

Architecture Nequix follows a simplified version of the NequIP [3] architecture, as shown in Figure 1a. We adopt two modifications suggested by Park et al. [ [16]: the species-specific self-connection layer within the interaction block is replaced with a single linear layer, and unused non-scalar representations are discarded from the final layer. Lastly, we add an equivariant root-mean-square layer normalization (RMSNorm) Liao et al. [ [17], which we find improves performance in our optimization setting. We document the full architecture hyper-parameters and the rationale behind each decision in Table A.1.

Figure 1. (a) Nequix architecture, a simplified version of NequIP [3], with a species-independent residual connection and layer normalization. (b) Combined performance scores of compliant models on the Matbench-Discovery (unique prototypes subset), collected on 2025-08-17. (c) Available published training times of current compliant models.

Implementation Nequix is implemented in both JAX [18, 19] and PyTorch [20, 21], taking advantage of just-in-time compilation and efficient automatic differentiation. Following standard energy-conserving MLIP practice [13], forces are obtained as the negative energy gradient with respect to atomic positions, −∇rE-\nabla_{\mathbf{r}} E, and stresses as the energy derivative with respect to strain, normalized by volume, σ=V−1∂E/∂ε\boldsymbol{\sigma} = V^{-1} \partial E/\partial\boldsymbol{\varepsilon}, where EE is the predicted total energy of the system, r\mathbf{r} is an atom position, ε\boldsymbol{\varepsilon} is the strain tensor, VV is the simulation cell volume, and σ\boldsymbol{\sigma} denotes the stress tensor.

Dynamic batching Materials radius graphs vary widely in their numbers of nodes and edges due to differences in unit-cell size and atom count. With fixed-size batching, the largest graphs dictate memory usage, leaving GPUs underutilized for most batches. To keep batch workloads more uniform while respecting memory limits, we use dynamic batching [22]: each batch is filled up to caps on the total nodes and edges. We set these caps to 1.1×1.1\times (ideal batch size) ×\times (dataset-average nodes-per-graph and edges-per-graph), respectively. Batches are then padded to these caps to fulfill the static shape requirements of JAX [23]. This capability is only used for the JAX implementation.

Optimization and normalization We compare the widely used Adam optimizer Kingma and Ba [2014] with the recently proposed Muon optimizer Jordan et al. [ [6], which uses the Newton-Schulz algorithm to orthogonalize weight updates. Using a smaller version of Nequix on MPtrj [25, 9] with hidden irreps of 128x0e+64x1o128\mathbf{x}0e + 64\mathbf{x}1o, we sweep learning rates of {0.03,0.01,0.003,0.001}\{0.03, 0.01, 0.003, 0.001\} for Adam and Muon, each with and without RMSNorm. The lowest validation error runs for each optimizer are shown in Figure 2. We find that the Muon configuration achieves comparable energy/force errors to Adam in 60-70% of the epochs, and results in a 7% reduction in energy MAE. We also find a significant reduction in the variance of stress error, which we notice in runs that use RMSNorm. Notably, the presence of the RMSNorm layer generally resulted in lower validation error for Muon-based training configurations, and higher for those using Adam.

Validation metrics during training

Figure 2. Validation metrics during training of a smaller version of Nequix configuration with Adam and Muon, trying learning rates in {0.03,0.01,0.003,0.001}\{0.03, 0.01, 0.003, 0.001\} and with/without RMSNorm. This model configuration uses the same hyperparameters as the final model, except with hidden irreps of 128×0e+64×1o128 \times0e + 64 \times1o. The dotted horizontal line shows the best validation performance reached during the Adam training.

GPU kernels There has been recent work [4, 26, 27] (Lee et al. 2025) on writing custom GPU kernels for the expensive equivariant tensor product operation [Xie et al., 2025]. These kernels fuse the tensor product and the outer gather-scatter from message passing step into a single GPU kernel. This avoids storing costly edge-based intermediates in GPU memory, improving both runtime and memory usage.

Training procedure The final Nequix model is trained for 100 epochs on MPtrj [25, 9], of which we hold out 5% for validation. More details on the training settings are provided in Sec. A.1. The model was trained on 2 NVIDIA A100 80 GB GPUs in 50 hours, for a total cost of 100 GPU hours.

Experiments

Matbench-Discovery benchmark

Matbench-Discovery [2] provides a standard framework for evaluating interatomic potentials in a high-throughput materials screening task consisting of geometry optimization and energy prediction on a set of 257,487 generated structures, and thermal conductivity prediction on a set of 103 structures. Ground truth is calculated with DFT/PBE level of theory, the same as MPtrj. The primary metrics include: 1) the F1 for stable/unstable classification after relaxation; 2) root mean squared displacement (RMSD) between predicted and reference structures after relaxation; and 3) symmetric relative mean error in predicted phonon mode contributions to thermal conductivity κ\kappa (κSRME\kappa_{\mathrm{SRME}}). A normalized and weighted combination of these metrics are then used to compute a combined performance score (CPS-1), which is used for ranking.

Following Riebesell et al. [ [2], we integrate our interatomic potential as Atomic Simulation Environment (ASE) calculator, which is then used to perform structure relaxation and phonon calculations with the default settings of the benchmark. For comparison, we consider only models in the compliant subset of the benchmark. This consists only of models that are trained on MPtrj or subsets, which limits data leakage and offers a more fair comparison among methods. Table 1 contains the performance of Nequix along with all current compliant models at the time of writing. We also include the reported training cost for the models when available, visualized in Figure 1. We find that Nequix ranks third by CPS-1, outperforming most models at a fraction of the training cost. It is noteworthy that this high ranking is due to high performance in the thermal conductivity task, however, the F1 score is still comparable to many of the other methods.

ModelParamsTrain costRMSD↓κSRME\kappa_{\mathrm{SRME}} ↓F1↑CPS-1↑
eSEN-30M-MP30.1M-0.0750.3400.8310.797
Eqnorm MPtrj1.31M20000.0840.4080.7860.756
Nequix708K1000.0850.4460.7500.729
DPA-3.1-MPtrj4.81M-0.0800.6500.8030.718
SevenNet-l3i51.17M-0.0850.5500.7600.714
HIENet7.51M28880.0800.6420.7770.707
MatRIS v0.5.0 MPtrj5.83M-0.0770.8610.8090.681
GRACE-2L-MPtrj15.3M-0.0900.5250.6910.681
MACE-MP-04.69M26000.0920.6470.6690.644
eqV2 S DeNS31.2M-0.0761.6760.8150.522
ORB v2 MPtrj25.2M-0.1011.7250.7650.470
M3GNet228K-0.1121.4120.5690.428
CHGNet413K-0.0951.7170.6130.400

Table 1. Matbench-Discovery v1 compliant leaderboard, sorted by combined performance score (CPS-1). Metrics are shown for the unique prototypes subset. Train cost is measured in A100 hours. Data as of 2025-08-17.

MDR phonon benchmark

Performance is also evaluated on the MDR phonon benchmark [7], a set of 10,000 phonon calculations also done with DFT/PBE level of theory. We follow the identical procedure to Loew et al. [2025], first performing a geometry relaxation, then phonon calculations using displacements of supercells. We report the mean absolute error (MAE) of properties derived from the phonon calculation: maximum phonon frequency ωmax⁡\omega_{\max}, vibrational entropy SS, Helmholtz free energy FF, and heat capacity at constant volume CVC_V. Table 2 demonstrates the performance of Nequix compared to other MPtrj-trained models. Similarly to Matbench-Discovery, we achieve performance within the top three of models, with a fraction of the parameter count of other methods.

ModelMAE(ωmax⁡\omega_{\max})MAE(SS)MAE(FF)MAE(CVC_V)
eSEN-30M211354
SevenNet-l3i52628105
Nequix2633126
SevenNet-04048199
GRACE-2L (r6)402595
MACE61602413
CHGNet891144521
M3GNet981505622

Table 2. Model performance of MPtrj-trained models on the MDR phonon benchmark, sourced from Loew et al. [ [7] and Fu et al. [ [13]. Metrics are MAE of maximum phonon frequency ωmax⁡\omega_{\max} (K), vibrational entropy SS (J/K/mol), Helmholtz free energy FF (kJ/mol) and heat capacity at constant volume CVC_V (J/K/mol).

Inference speed

Finally, we compare inference speed with existing interatomic potentials by using ASE Larsen et al. [2017] to run calculations on diamond across varying unit cell sizes, timing the calculations for each model.1 We run each model in the default configuration in which it used within its ASE calculator, with compilation and kernels wherever specified in the documentation. See Section A.2 for more information on benchmarking setup.

Figure 3 compares performance of each model in terms of steps per day vs. number of atoms. In this study, Nequix is about 100× faster than eSEN, offering a new option in the accuracy vs. speed Pareto frontier at a fraction of the training cost.

Figure 3. Inference speed of various models in steps per day.

Conclusion

We presented Nequix, an E(3)-equivariant interatomic potential that pairs a simplified NequIP architecture with modern training practices. Our results show that Nequix achieves competitive accuracy on Matbench-Discovery and the MDR phonon benchmark at less than one quarter of the reported training cost of many competitors. This resource-efficient recipe provides a practical alternative to large-scale foundation models and helps broaden access to high-quality atomistic modeling in settings with more limited compute. We release trained weights and a JAX/PyTorch codebase to streamline reuse and extension.

Looking ahead, we see several promising directions: scaling training duration and data while maintaining budget discipline, exploring pretraining and fine-tuning regimes across broader datasets, and pushing cost even lower through model distillation, pruning, quantization, kernel implementations, or more data-efficient training. We hope Nequix serves as a strong, efficient baseline for future work on accessible materials foundation models.

Acknowledgments and Disclosure of Funding

This work was supported by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org/) NSF Graduate Research Fellowship program under Grant No. DGE-1745302, and by DOE ICDI grant DE-SC0022215. This research used resources of the National Energy Research Scientific Computing Center (NERSC), a Department of Energy User Facility using NERSC award ERCAP0033254.

Appendix

Training and model configuration

Table A.1 shows the hyper-parameters used to train Nequix. The model is trained for 100 epochs, using an MAE loss function on energy and stress, and l2l_2 loss on forces. We use a linear warmup with cosine decay learning rate schedule. Figure A.1 shows the energy, force, and stress MAE on the validation set throughout training. The final MAEs are 10.05 meV/atom, 32.79 meV/Å, and 0.22 meV/Å3^3/atom for energy, forces, and stress respectively.

Hyper-parameterValueNotes/Rationale
Radial cutoff6 ÅMost models use 5 or 6 Å; 6 performed slightly better in preliminary validation performance.
Hidden irreps128×0e+64×1o+32×2e+32×3o128 \times0e + 64 \times1o + 32 \times2e + 32 \times3oFrom SevenNet-l3i5.
Lmax⁡L_{\max}3Consistent with hidden irreps.
NlayersN_{\mathrm{layers}}4Balance of performance and efficiency.
Radial basis size8From NequIP and analysis from Sec. 5.2 of Fu et al. [2025]
Radial MLP size64From NequIP.
Radial MLP layers2From NequIP.
Polynomial cutoff pp6.0From NequIP.
Radial basis functionBesselFrom NequIP. Also tried Gaussian, which had minimal difference on validation performance.
Learning rate0.01Selected from {0.03,0.01,0.003,0.001}\{0.03, 0.01, 0.003, 0.001\} based on validation performance early in training.
Warmup epochs0.1From eSEN.
Warmup factor0.2From eSEN.
OptimizerMuonSee § 2.
Weight decay0.001From eSEN. Also tried 0.0, which led to worse validation performance.
Energy weight20From eSEN.
Force weight20From eSEN.
Stress weight5From eSEN.
Batch size256 (dynamic)See § 2
Number of epochs100Standard training duration.

Table A.1. Hyper-parameters used and rationale behind selection

Figure A.1. Validation curves for Nequix training on MPtrj.

Inference Configuration

We summarize all of the inference configurations in Table A.2. We use MPTrj versions for all of the models except NequIP where the MPtrj model did not have kernel support, so we instead use the OMat24 model after confirming that they have the same hyperparams. For compatibility with eSEN, we use the older fairchem OCP calculator, and expect the latest version to have less overhead.

ModelCompileKernels
eSEN-30M×\times×\times
MACE-MP0torch.compilecuEquivariance
Nequixtorch.compileopenEquivariance

Table A.2. Inference configuration for the ASE benchmarking setup

References

  1. [1]Ilyes Batatia, Philipp Benner, Yuan Chiang, Alin M Elena, Dávid P Kovács, Janosh Riebesell, Xavier R Advincula, Mark Asta, Matthew Avoylon, William J Baldwin, et al. A foundation model for atomistic materials chemistry. arXiv preprint arXiv:2401.00096, 2023.arxiv.org/abs/2401.00096
  2. [2]Janosh Riebesell, Rhys EA Goodall, Philipp Benner, Yuan Chiang, Bowen Deng, Gerbrand Ceder, Mark Asta, Alpha A Lee, Runabhav Jain, and Kristin A Persson. A framework to evaluate machine learning crystal stability predictions. Nature Machine Intelligence, 7(6):836–847, 2025.DOI
  3. [3]Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):2453, 2022.DOI
  4. [4]Vivek Bharadwaj, Austin Glover, Aydin Buluc, and James Demmel. An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networks. Society for Industrial and Applied Mathematics, 2025. URL https://arxiv.org/abs/2501.13986.
  5. [5]Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @fernbear.bsky.social, Boza Vlado, You Jiacheng, Franz Cesista, Braden Koszarsky, and @Grad62304977. modded-nanogpt: Speedrunning the nanogpt baseline, 2024a. URL https://github.com/KellerJordan/modded-nanogpt.
  6. [6]Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024b. URL https://kellerjordan.github.io/posts/muon/.
  7. [7]Antoine Loew, Dewen Sun, Hai-Chen Wang, Silvana Botti, and Miguel AL Marques. Universal machine learning interatomic potentials are ready for phonons. npj Computational Materials, 11(1):178, 2025.arxiv.org/abs/2412.16551
  8. [8]Chi Chen and Shyue Ping Ong. A universal graph deep learning interatomic potential for the periodic table. Nature Computational Science, 2(11):718–728, 2022.DOI
  9. [9]Bowen Deng, Peichen Zhong, KyuJung Jun, Janosh Riebesell, Kevin Han, Christopher J Bartel, and Gerbrand Ceder. Chgnet as a pretrained universal neural network potential for charge-informed atomistic modelling. Nature Machine Intelligence, 5(9):1031–1041, 2023.DOI
  10. [10]Anton Bochkarev, Yury Lysogorskiy, and Ralf Drautz. Graph atomic cluster expansion for semilocal interactions beyond equivariant message passing. Physical Review X, 14(2):021036, 2024.arxiv.org/abs/2311.16326
  11. [11]Mark Neumann, James Gin, Benjamin Rhodes, Steven Bennett, Zhiyi Li, Hitarth Choubisa, Arthur Hussey, and Jonathan Godwin. Orb: A fast, scalable neural network potential. arXiv preprint arXiv:2410.22570, 2024.arxiv.org/abs/2410.22570
  12. [12]Luis Barroso-Luque, Muhammed Shuaibi, Xiang Fu, Brandon M Wood, Misko Dzamba, Meng Gao, Ammar Rizvi, C Lawrence Zitnick, and Zachary W Ulissi. Open materials 2024 (omat24) inorganic materials dataset and models. arXiv preprint arXiv:2410.12771, 2024.
  13. [13]Xiang Fu, Brandon M Wood, Luis Barroso-Luque, Daniel S Levine, Meng Gao, Misko Dzamba, and C Lawrence Zitnick. Learning smooth and expressive interatomic potentials for physical property prediction. arXiv preprint arXiv:2502.12147, 2025.arxiv.org/abs/2502.12147
  14. [14]Duo Zhang, Anyang Peng, Chun Cai, Wentao Li, Yuanchang Zhou, Jinzhe Zeng, Mingyu Guo, Chengqian Zhang, Bowen Li, Hong Jiang, et al. Graph neural network model for the era of large atomistic models. arXiv preprint arXiv:2506.01686, 2025.arxiv.org/abs/2506.01686
  15. [15]Keqiang Yan, Montgomery Bohde, Andrii Kryvenko, Ziyu Xiang, Kaiji Zhao, Siya Zhu, Saagar Kolachina, Doğuhan Sartürk, Jianwen Xie, Raymundo Arróyave, et al. A materials foundation model via hybrid invariant-equivariant architectures. arXiv preprint arXiv:2503.05771, 2025.arxiv.org/abs/2503.05771
  16. [16]Yutack Park, Jaesun Kim, Seungwoo Hwang, and Seungwu Han. Scalable parallel algorithm for graph neural network interatomic potentials in molecular dynamics simulations. Journal of chemical theory and computation, 20(11):4857–4868, 2024.DOI
  17. [17]Yi-Lun Liao, Brandon Wood, Abhishek Das, and Tess Smidt. EquiformerV2: Improved equivariant transformer for scaling to higher-degree representations. arXiv preprint arXiv:2306.12059, 2023.arxiv.org/abs/2306.12059
  18. [18]James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/jax-ml/jax.
  19. [19]Patrick Kidger and Cristian Garcia. Equinox: neural networks in JAX via callable PyTrees and filtered transformations. Differentiable Programming workshop at Neural Information Processing Systems 2021, 2021.arxiv.org/abs/2111.00254
  20. [20]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019. URL https://arxiv.org/abs/1912.01703.
  21. [21]Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burkovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazors, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, C. K. Luk, Bert Maher, Yunjie Pan, Christian Puh rsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Shunting Zhang, Michael Suo, Phil Tillet, Xu Zhao, Eikan Wang, Keren Zhou, Richard Zou, Xiaodong Wang, Ajit Mathews, William Wen, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Fast machine learning through dynamic python bytecode transformation and graph compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’24, page 929–947, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798407043050. doi: 10.1145/3620665.3640366. URL https://doi.org/10.1145/3620665.3640366.
  22. [22]Daniel T Speckhard, Tim Bechtel, Sebastian Kehl, Jonathan Godwin, and Claudia Draxl. Analysis of static and dynamic batching algorithms for graph neural networks. arXiv preprint arXiv:2502.00944, 2025.
  23. [23]Jonathan Godwin*, Thomas Keck*, Peter Battaglia, Victor Bapst, Thomas Kipf, Yujia Li, Kimberly Stachenfeld, Petar Veličković, and Alvaro Sanchez-Gonzalez. Jraph: A library for graph neural networks in jax., 2020. URL http://github.com/deepmind/jraph.
  24. [25]Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL materials, 1(1), 2013.DOI
  25. [26]NVIDIA. cuequivariance: Cuda-accelerated library for equivariant neural networks, 2024. URL https://github.com/NVIDIA/cuEquivariance.
  26. [27]Chuin Wei Tan, Marc L. Descoteaux, Mit Kotak, Gabriel de Miranda Nascimento, Sean R. Kavanagh, Laura Zichi, Menhang Wang, Aadit Saluja, Yizhong R. Hu, Tess Smidt, Anders Johansson, William C. Witt, Boris Kozinsky, and Albert Musae lian. High-performance training and inference for deep equivariant interatomic potentials, 2025. URL https://arxiv.org/abs/2504.16068.

Paper details

vlnc:W_rqzxhdeu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents