WorksPhysics and Astronomy
CHIPS-FF: Evaluating Universal Machine Learning Force Fields for Material Properties
Abstract
In this work, we introduce CHIPS-FF (Computational High-Performance Infrastructure for Predictive Simulation-based Force Fields), a universal, open-source benchmarking platform for machine learning force fields (MLFFs). This platform provides robust evaluation beyond conventional metrics such as energy, focusing on complex properties including elastic constants, phonon spectra, defect formation energies, surface energies, and interfacial and amorphous phase properties. Utilizing 16 graph-based MLFF models including ALIGNN-FF, CHGNet, MatGL, MACE, SevenNet, ORB, MatterSim and OMat24, the CHIPS-FF workflow integrates the Atomic Simulation Environment (ASE) with JARVIS-Tools to facilitate automated high-throughput simulations. Our framework is tested on a set of 104 materials, including metals, semiconductors and insulators representative of those used in semiconductor components, with each MLFF evaluated for convergence, accuracy, and computational cost. Additionally, we evaluate the force-prediction accuracy of these models for close to 2 million atomic structures. By offering a streamlined, flexible benchmarking infrastructure, CHIPS-FF aims to guide the development and deployment of MLFFs for real-world semiconductor applications, bridging the gap between quantum mechanical simulations and large-scale device modeling.
Multiscale modeling in materials science[1, 2] is an essential tool to bridge the gap between atomistic properties, device performance, and manufacturing. Recently, there has been a renewed focus on modeling semiconductor materials, devices, and components. Various approaches to semiconductor modeling can be applied at different scales, including quantum mechanical tools such as density functional theory (DFT),[3] classical atomistic simulations such as molecular dynamics (MD),[4] technology computer-aided design (TCAD)[5] and various machine learning (ML) models.[6, 7, 8, 9, 10] [11, 12]
In order to better inform semiconductor device models, accurate structure-to-property relationships must be established, which often require more computationally expensive quantum mechanical approaches. Accurate structural relaxations of defect structures and semiconductor interfaces are key to uncovering the underlying physics that govern device performance. For example, defects in semiconductors can introduce energy levels within the band gap, which can impact carrier mobility and concentration and create traps, leading to non-radiative recombination of holes and electrons.13–18 Semiconductor surfaces and interfaces are especially important to consider since they are the building blocks of devices, impacting charge carrier behavior (band alignment), electrical properties (i.e., Schottky vs. Ohmic), thermal management and device efficiency.19–24 In addition, the combination of semiconductor materials and high-dielectric amorphous structures are essential for building Metal-Oxide-Semiconductor Field-Effect Transistors (MOSFETs).25–30
Due to the costly scaling of DFT (, where is the number of electrons), high-throughput calculations of larger simulation cells required to study defects, surfaces and interfaces are not feasible. Unfortunately, many of the current classical atomistic interatomic potentials do not have the capability to achieve near-DFT accuracy and capture these complex interactions. In addition, most classical potentials are limited in their transferability to various systems and experimental conditions. Early attempts to create a universal force field for the entire periodic table were limited in their accuracy. 31 For this reason, many researchers have explored machine learning force fields (MLFFs) trained on highly-accurate DFT calculations as a viable route to bridge the gap between quantum mechanical accuracy and large-scale atomistic simulations.
One of the pioneering MLFF architectures was developed by Behler and Parrinello in 2007 using neural networks. 32 Other MLFF methods include Gaussian processes-based Gaussian approximation potentials (GAP), 33 spectral neighbor analysis potential (SNAP) 34 and Allegro.35 Graph neural network (GNN)-based machine learning force fields (MLFFs) are now considered state-of-the-art due to their superior accuracy and transferability. With the public availability of large DFT databases (such as the Materials Project, 36 JARVIS-DFT, 37,38 OQMD, 39,40 and Alexandria 41–44), several researchers have trained GNN force fields on full DFT databases, encompassing the entire periodic table. These so-called universal/unified/foundational machine learning force fields (uMLFFs) have been successful at reproducing near-DFT accuracy for a wide-range of systems.
A major advantage of utilizing these pretrained uMLFF architectures is that expensive DFT dataset generation is not an initial requirement. These pretrained uMLFFs can be used in place of DFT or fine tuned on more tailored datasets or tasks. Such uMLFFs have shown remarkable potential for numerous applications including modeling of defects and surfaces, 45,46 disordered alloys, 47 high-pressure superconductors, 48 solid-solution energetics and ion migration barriers, 45 catalysis, 49 thermal conductivity, 50 semiconductor interfaces, 20 guiding inverse materials design 11,51,52 and many other applications in chemistry, physics and materials science. 53–55 Important quantities such as optimized structure, elastic properties, vibrational properties (phonons), stability, thermal properties, defect formation energy and surface energy can be easily computed from uMLFFs at a significantly lower cost than DFT. Additionally, uMLFF structural relaxations of bulk semiconductors, point defects, surfaces and interfaces can be coupled with other low-cost tools (direct prediction with GNNs, 16,16,20,56,57 tight binding58,59) to predict electronic properties. uMLFFs can also be used to perform structure searches (replacing hundreds to tens of thousands of DFT relaxation calculations), which can be followed by a single DFT calculation for electronic properties of the target structure.
One of the first graph-based uMLFF architectures introduced in 2022 was M3GNet from Chen and Ong. 60 M3GNet has since transformed into MatGL (Materials Graph Library), 61 a re-implementation of M3GNet built on the Deep Graph Library (DGL) 62 and PyTorch. 63 M3GNet evolved from the MEGNet (MatErials Graph Network) property prediction model introduced in Ref. 64 M3GNet was trained on inorganic materials ( relaxation steps) in the Materials Project (MP), covering 89 elements of the periodic table. 60 This dataset is commonly referred to as MPF. 60 In M3GNet, the angles (representing three-body interactions) are incorporated by aggregating to bonded atoms within the graph convolution steps to update atoms, bonds and properties. 60 The Atomistic LIne Graph Neural Network (ALIGNN)-FF model65 was introduced in 2022 and was trained on the JARVIS-DFT dataset of materials ( relaxation steps, ALIGNN-FF DB). ALIGNN-FF evolved from the ALIGNN property prediction model released in 2021. 57 The ALIGNN model achieved significant improvement in property prediction tasks due to the inclusion of bond-angles in terms of line graphs, which previous GNN models lacked. 57 The earlier versions of ALIGNN-FF utilized a -nearest neighbor graph, 65 while the updated version of ALIGNN-FF (2024.12.2) utilizes a radius graph, making it faster and more robust. Another uMLFF that utilizes a crystal graph (atom graph and bond graph) is the Crystal Hamiltonian Graph Neural Network (CHGNet), 66 which was proposed in 2023. This model was trained on the Materials Project Trajectory (MPtrj) dataset, 66 which includes DFT calculations for materials ( million relaxation steps). CHGNet also incorporates magnetic moments into the training process to enhance the description of chemical reactions. 66
MACE,53,67,68 an equivariant message passing neural network potential, uses higher order messages (higher than two body) combined with atomic cluster expansion (ACE), 69,70 which is a method for deriving an efficient body-ordered symmetric polynomial basis to represent functions of atomic neighborhoods. A pre-trained model for the entire periodic table (trained on MPTrj) for MACE was released in late 2022 53 (referred to as MACE-MP-0). In late 2024, a new pretrained version of MACE (known as MACE-MPA-0) was released, which is trained on a combination of MPTrj and Alexandria. 71 SevenNet (Scalable EquiVariance Enabled Neural Network), 72 another equivariant message passing uMLFF, is based on the NequIP 73 architecture for building E(3)-equivariant force fields. SevenNet trained on the MPtrj dataset and was released in 2024. In addition to these open source models, there have been proprietary uMLFF models developed including MatterSim (Microsoft), 74 GNoME (Google DeepMind), 75 and PFP (Preferred Networks Inc). 76 A version of MatterSim (MatterSim-v1), which has a similar but slightly modified architecture to M3GNet and is trained on over 17 million structures sampled from Materials Project, Alexandria, and Microsoft’s own dataset (consisting of additional structures and MD trajectories at ambient and extreme conditions), was made open source to the community in December 2024. 77 The Orb78 pretrained uMLFF, released in fall 2024, was developed by the startup company Orbital Materials, where superior performance for force and energy predictions were demonstrated at significantly reduced computational cost. 54,79,80 Orb utilizes an attention augmented Graph Network-based Simulator (GNS), 81 which is a type of Message Passing Neural Network (MPNN). 82 In contrast to equivariant models such as MACE and SevenNet, Orb does not utilize equivariant message passing. The open source Orb model (orb-v2) is trained on MPtrj and Alexandria (over 3 million materials and 32 million relaxation steps). 54 In late 2024, Meta FAIR Chemistry released the Open Materials 2024 (OMat24) 83 models and dataset, which contains over 110 million DFT calculations and pretrained uMLFF models based on the EquiformerV2 84 architecture, which is a highly accurate equivariant transformer model. OMat24 offers various sized models trained on OMat, MPTrj and Alexandria.
Recently, interest in the field of uMLFFs has grown immensely (from academia, government and industry), as seen by the several models above and various use cases of new model architectures. Given the current state of the field, benchmarking the performance of uMLLFs is imperative. Large-scale efforts to benchmark the performance of various pretrained uMLFF architectures have emerged such as Matbench Discovery80,85 and the JARVIS-Leaderboard,86,87 which calculate error metrics and allow for new models to be uploaded as time progresses. Matbench Discovery is an interactive leaderboard platform where uMLFFs are ranked based on accuracy of properties (such as energy and more recently thermal conductivity) and the ability to simulate high-throughput discovery of stable new inorganic materials. These uMLFFs are evaluated on targeted large-scale test datasets (i.e., a test set of 200,000 Materials Project entries) to calculate error metrics.80,85 The JARVIS-Leaderboard is a flexible benchmarking platform that allows for the evaluation of various machine learning, electronic structure, force-field and quantum computing, and experimental methods. So far, there are over 300 benchmarks and over 2,000 user contributions.86,87 A subset of these benchmarks and user contributions are focused on uMLFFs for various test datasets.86,87 There have been focused benchmarking efforts for more involved properties beyond energy. Yu et al.88 performed a systematic assessment of uMLFFs (M3GNet, ALIGNN-FF, CHGNet and MACE), calculating properties such as optimized structure, formation energy, bulk modulus, and phonon band structure, comparing directly to DFT to calculate error metrics. Benchmarking of more advanced properties such as surface energies, defect formation energies, solid-solution energetics, and ion migration barriers have recently been performed with M3GNet, CHGNet and MACE.45,46 In addition, a high-level assessment of uMLFF computed thermal conductivity has been conducted with M3GNet, CHGNet, MACE, SevenNet and ORB by Póta et al.50 More recently, Zhu et al. utilized M3GNet, CHGNet, MACE, SevenNet, and ORB to accelerate the prediction of CALPHAD-based phase diagrams for complex alloys89 and Lowe et al. performed in-depth benchmarking of phonons for over 10,000 materials using all of the recent state-of-the-art uMLFFs.90 These recent efforts have highlighted the successes and limitations of various uMLFF architectures, and have emphasized the need for a comprehensive publicly available benchmarking platform. Such a platform allows the community to make informed decisions on regarding uMLFFs, outweighing factors such as computational cost versus accuracy. In addition, researchers may find systematic trends in the uMLFF results which can help others to understand the applicability of each uMLFF type. Well-understood systematic trends in DFT functionals (i.e., the underbinding and lattice constant errors of GGA) have been exploited to identify new exfoliable van der Waals (vdW) materials. 91 In contrast to the Jacob’s Ladder metaphor of DFT exchange-correlation functionals, 92 where the “rungs” of the ladder refer to the increasing complexity (and computational cost) and accuracy of the functionals, there (as of yet) exists no systematic assessment or generalizable statements for uMLFFs. As uMLFFs begin to replace DFT calculations, such analysis is needed to help the community better develop new force fields or fine-tune existing force fields for downstream applications.
In this work, we present CHIPS-FF (Computational High-performance Infrastructure for Predictive Simulation-based Force Fields): a generalized user-friendly open-source benchmarking package to test various uMLFFs for a number of properties beyond energy. We have implemented a streamlined workflow to use uMLFFs for structural relaxation, bulk modulus, elastic properties, point defect formation energy, surface energy, interfacial properties (work of adhesion), molecular dynamics and creation of amorphous structures, all with automated error metrics from JARVIS-DFT. In contrast to large scale benchmarking efforts such as Matbench Discovery, 80,85 which requires users to submit contributions covering large-scale test sets (on the order of 200,000), CHIPS-FF allows for robust benchmarking on smaller datasets and more complex properties. Although our framework is agnostic to the type of material, we chose a set of 104 materials that are most relevant for semiconductor devices as a case study. Additionally, we evaluate force-prediction accuracy of these models for close to 2 million atomic structures from Materials Project and JARVIS-DFT databases. We intend this codebase and benchmarking dataset to benefit the MLFF community and aid in the development of future models.
The CHIPS-FF workflow mainly utilizes the Atomic Simulation Environment (ASE) 93 and JARVIS-Tools 37,38 to perform atomistic simulations in Python. In terms of pretrained uMLFF models, we utilized ALIGNN-FF: 2.12.2024, CHGNet: 0.3.8, MatGL: 1.1.2 (using both the M3GNet-MP-2021.2.8-PES and M3GNet-MP-2021.2.8-DIRECT-PES 94 models), MACE: 0.3.10 (using MACE-MP-0 and MACE-MPA-0), SevenNet-0: 0.9.2 (using the 11July2024 model), orb-models: 0.4.1 (orb-v2 and orb-d3-v2, which utilizes the D3 dispersion correction 95), and MatterSim-v1 (MatterSim-v1.0.0-5M). In addition, we benchmarked a MACE force field trained on the newly released Alexandria dataset presented in Ref. 44 (this model was trained on over 200,000 2D material calculations with the PBEsol 96 functional and is referred to as MACE in that work 97). In our work, we refer to this model as mace-alexandria (which is distinct from mace-mpa). For the OMat24 models from Meta, we tested the three EquiformerV2 (eqV2) models trained on OMat (Small: 31M (million), medium: 86M, and large: 153M in terms of model parameters) and two eqV2 models trained on OMat+MPTrj+Alexandria (Small: 31M and medium: 86M). 55,83 These models are named as: eqV2_31M_omat, eqV2_86M_omat, eqV2_153M_omat, eqV2_31M_omat_mp_salex, and eqV2_86M_omat_mp_salex. 55,83
Initial structures were obtained from the JARVIS-DFT database (but not limited by it) and then optimized using the FIRE 98 algorithm within ASE. For bulk relaxations, atomic coordinates and lattice vectors were allowed to relax, but for surfaces and vacancy calculations, cell volume was fixed and atomic positions were allowed to relax. The maximum force value (stopping criterion for relaxation) was set to 0.05 eV/. If a calculation did not reach the stopping criterion by 200 steps, it was considered unconverged. If a structure did not reach convergence within 200 steps, the final structure and energy at 200 steps was logged and used for subsequent portions of the workflow. We tested both the FrechetCellFilter (main text) and ExpCellFilter (in SI) within ASE. 99 The bulk modulus was obtained by applying isotropic strain to the system from -6% to +6% and using a Murnaghan fit for the equation of state. The elastic tensor was computed using the Elastic package. 100–102 For bulk modulus and elastic tensor calculations, the conventional cell was used. We utilized the phonopy103,104 package to perform simulations of the phonon spectrum using the finite displacement method,105 where a supercell was constructed from the initial unit cell. Displacement values of Å, Å, Å, and Å were benchmarked. From these phonopy results, we also extracted the Helmholtz free energy (and zero point energy at K), entropy and heat capacity as a function of temperature. We have added capabilities to CHIPS-FF to compute the thermal expansion of each material at constant pressure and the lattice thermal conductivity, utilizing the quasi-harmonic approximation106 implemented in phonopy and phono3py103,104 respectively (these simulations are ongoing and will be discussed in future work). JARVIS-Tools was used to automatically generate supercells for neutral point defects (vacancies) and non-polar surfaces (both of which were later relaxed with uMLFFs). In this work, we obtained initial defect structures from the relaxed structures in the JARVIS-DFT vacancy database from Ref.15 This allowed us to make a 1:1 comparison between vacancy formation energies (calculated with DFT and uMLFFs) for the exact same crystal structure and supercell size (avoiding finite-size scaling effects). In Ref.,15 these defect supercells were constructed from the conventional cell and enforced to have a lattice vector c of at least Å, where single vacancies were created based on Wyckoff-position information for each atomic species in the crystal. The following formula is used to obtain the vacancy formation energy ():
where is the total energy of the defect supercell, is the energy of the bulk structure (no defect) and is the chemical potential of the missing atom. The lowest energy crystal structure of the elemental solid (from JARVIS-DFT) is used to calculate the chemical potential with each respective uMLFF (where the structure is fully re-relaxed). Surfaces were generated for miller indices of [1, 0, 0], [1, 1, 1], [1, 1, 0], [0, 1, 1], [0, 0, 1], and [0, 1, 0], skipping any polar surfaces in the surface relaxation. Each surface contained at least 4 layers and 18 Å of vacuum. The surface energy of each non-polar surface was calculated as:
where is the energy of the surface structure, is the energy of the bulk structure, is the number of bulk unit cells in the surface, and is the cross sectional area of the surface. In order to study amorphous materials, we have implemented finite-temperature molecular dynamics into our workflow to perform melt/quench simulations using Berendsen NVT (Number of particles, Volume, Temperature ensemble) dynamics within ASE. For our case study of amorphous Si, we used a timestep of 1 femtosecond and ran the simulation at 3500 K for 10 ps (melt) and 300 K for 20 ps (quench). For benchmarking calculations to compare uMLFF results to, we performed similar ab initio MD (AIMD) calculations with the Vienna Ab initio Simulation Package (VASP) code, using projector augmented wave (PAW) pseudopotentials 107,108 and the vdW-DF-optB88 functional. These gamma-point AIMD calculation used a plane wave cutoff energy of 500 eV and the Nosé-Hoover 109–111 thermostat to run NVT simulations at 2000 K for 5 ps (melting), and 300 K for 5 ps (at a timestep of 1 femtosecond). In order to perform calculations for material interfaces, we utilized InterMat,20 a newly developed package for the generation and calculation of interface structures (substrate plus film). The initial interface is obtained by creating a superlattice or alternating slab junction (ASJ) structure (without vacuum padding) 112,113 and then using the Zur algorithm114 to obtain the best candidate interface. From here, we computed the relative alignment between the film and substrate in the in-plane (xy) direction by performing a grid search with a 0.05 fractional spacing interval with each respective uMLFF model. In addition to determining the optimal in-plane orientation between the substrate and film, this gives us a qualitative estimate of how smooth the potential energy surface is at the interface for each pretrained uMLFF. From this calculation, we also determined the work of adhesion () at the interface with:
where is the surface energy of the film, is the surface energy of the substrate and is the interfacial energy calculated as:
where is the total energy of the interface, is the total energy of the film, is the total energy of the substrate and is the cross sectional area.
A majority of the reference data used to compare our uMLFF results to were obtained from the JARVIS-DFT database, which contains relaxed structural data (primitive lattice vectors, volume), elastic properties (bulk modulus, elastic tensor) and phonon band structure. 37,38 Within JARVIS-DFT, there exists a database of single vacancy calculations 15 and surfaces20 that were both used for benchmarking these uMLFFs. All of the DFT data in JARVIS was computed with the vdW-DF-optB88115 functional, which is a modified Generalized Gradient Approximation (GGA) functional designed to explicitly include van der Waals (vdW) interactions. This is slightly different than other materials repositories such as the Materials Project (MPtrj), which mostly contains DFT calculations performed with the Perdew Burke Ernzerhof (PBE)116 GGA functional. Despite mostly comparing to JARVIS-DFT results (vdW-DF-optB88), the CHIPS-FF workflow has a flexible framework to use any DFT dataset as a ground truth (i.e., the Materials Project, Alexandria, a users own DFT dataset, etc.). In addition, the platform is flexible enough to compare uMLFF results directly to available experimental values. Of course, comparing the results of a quantity computed with a uMLFF to a DFT result with an arbitrary exchange-correlation functional and varying convergence criteria may result in biased or inconclusive error metrics. In this case, it may be suitable to compare uMLFF results directly to available experimental data, especially if the uMLFF prediction is closer to “reality” than the DFT calculation it is being evaluated against. Although it is possible to assess the accuracy of MLFFs by comparing results directly to DFT or experiment, there is a lack of explicit uncertainty quantification (UQ) for most MLFFs, including the universal pretrained MLFFs mentioned in this work. A robust UQ method should account for uncertainty arising from measurement noise (aleatoric) and uncertainty in predictions arising from model error (epistemic). Due to the fact that the DFT training data does not possess aleatoric error, most of the uncertainty for MLFFs is epistemic, which can be due to scarcity of data, limitations of model architecture, or poor parameter optimization within the model. Recently, different epistemic UQ methods such as ensemble-based uncertainty,117 deep evidential regression,118 mean-variance estimation,119 Gaussian mixture models120 have been implemented and benchmarked for atomistic MLFFs.121,122 It is evident that low-cost and widespread implementations of UQ for universal pretrained uMLFFs is a necessary future development.
Figure 1 depicts a full schematic of the CHIPS-FF workflow. In this workflow, initial structures are taken from the JARVIS-DFT database. JARVIS-Tools is used to pull structures from the JARVIS-DFT database and generate supercell surface and defect structures prior to atomistic simulations. Various uMLFF calculators are accessed through ASE. So far, we have implemented MatGL, CHGNet, ALIGNN-FF, MACE, SevenNet, ORB, MatterSim, and eqV2 (OMat24). As new pretrained force fields are developed over time, they will be added to the workflow. In addition, CHIPS-FF offers the flexibility for the user to add their own force field or property to the pipeline. Once each uMLFF is loaded and run through the ASE calculator, a variety of properties can be computed including: optimized structure, bulk modulus (equation of state fitting), elastic tensor (elastic package), phonons (phonopy), thermal expansion (phonopy), thermal conductivity (phono3py), molecular dynamics, vacancy formation energy, surface energy, and interface properties (InterMat). As these calculations are carried out, detailed logging information is saved, including the computational timing for each stage of the calculation and whether or not each stage converged. Once these material properties are computed with each uMLFF, they are cross checked with entries in the JARVIS-DFT database and the mean absolute error (MAE) for each property is computed. For spectral quantities such as the phonon band structure, we computed the

Figure 1. A full schematic of the CHIPS-FF workflow.
MAE for each band along the q-point path using:
where is the phonon energy (in ) at each q-point and each branch index () (calculated with uMLFF and DFT from JARVIS-DFT) and represents the number of wavevectors along the q-path. In addition to automatically calculating the error metrics, our workflow will create entries for the JARVIS-Leaderboard in the appropriate format (.json.zip and .csv.zip files with the proper naming convention). From here, these entries can be directly uploaded to the leaderboard and compared to other benchmarks. Although we use JARVIS-DFT for the initial structures and calculation of the error metrics, our CHIPS-FF platform is flexible to use other datasets for initial structures and ground truth comparison. Our CHIPS-FF package offers easy-to-use command line tools, which can easily be parallelized
to simultaneously run hundreds or even thousands of materials/uMLFFs on CPU and GPU.
For our benchmarking study, we chose a set of 104 materials most commonly found in semiconductor devices and interfaces. This test set contains metals, semiconductors and insulators to be representative of the various parts and interfaces of integrated circuits (semiconductor/insulator interfaces, semiconductor/metal contacts, etc.). Figure S1 depicts various distributions of properties of the test set, including space group, band gap (calculated with vdW-DF-optB88), atomic number, chemical formula type, crystal system, and Wyckoff site, and dimensionality, all obtained from JARVIS-DFT. In addition, Table S1 and Table S2 give detailed information of the chemical composition, crystal structure and band gap for each material, where we see that the set contains 28 metals (zero band gap), 14 insulators (band gap above 3 eV) and 62 semiconductors (band gap between 0 eV and 3 eV). In addition, we see from Figure S1 that our test set represents a diverse set of materials ideal for testing our uMLFF workflow.
| JID | Formula | SPG Number | SPG Symbol | optb88vdw Band Gap (eV) |
| JVASP-1002 | 227 | \mathrm{Fd\mathchar`-3m} | 0.731 | |
| JVASP-10036 | 136 | 1.769 | ||
| JVASP-10037 | 136 | 0.894 | ||
| JVASP-1008 | 227 | \mathrm{Fd\mathchar`-3m} | 0 | |
| JVASP-1023 | 152 | 0.165 | ||
| JVASP-1029 | 191 | 0 | ||
| JVASP-103127 | 115 | \mathrm{P\mathchar`-4m2} | 0.242 | |
| JVASP-104 | 141 | 2.023 | ||
| JVASP-104764 | 6 | 3.505 | ||
| JVASP-105410 | 216 | \mathrm{F\mathchar`-43m} | 0.694 | |
| JVASP-10591 | 186 | 2.102 | ||
| JVASP-106363 | 82 | \mathrm{I\mathchar`-4} | 0.022 | |
| JVASP-106686 | 115 | \mathrm{P\mathchar`-4m2} | 0 | |
| JVASP-1067 | 166 | \mathrm{R\mathchar`-3m} | 0.319 | |
| JVASP-107 | 186 | 2.495 | ||
| JVASP-10703 | 224 | \mathrm{Pn\mathchar`-3m} | 0 | |
| JVASP-108770 | 115 | \mathrm{P\mathchar`-4m2} | 0 | |
| JVASP-110 | 99 | 1.749 | ||
| JVASP-110231 | 156 | 0.217 | ||
| JVASP-1103 | 225 | \mathrm{Fm\mathchar`-3m} | 1.119 | |
| JVASP-1109 | 62 | 1.018 | ||
| JVASP-111005 | 166 | \mathrm{R\mathchar`-3m} | 0.488 | |
| JVASP-1112 | 225 | \mathrm{Fm\mathchar`-3m} | 0.564 | |
| JVASP-1115 | 225 | \mathrm{Fm\mathchar`-3m} | 0.513 | |
| JVASP-113 | 14 | 3.622 | ||
| JVASP-1174 | 216 | \mathrm{F\mathchar`-43m} | 0.085 | |
| JVASP-1177 | 216 | \mathrm{F\mathchar`-43m} | 0 | |
| JVASP-1180 | 186 | 0 | ||
| JVASP-1183 | 216 | \mathrm{F\mathchar`-43m} | 0.331 | |
| JVASP-1186 | 216 | \mathrm{F\mathchar`-43m} | 0 | |
| JVASP-1189 | 216 | \mathrm{F\mathchar`-43m} | 0 | |
| JVASP-1192 | 216 | \mathrm{F\mathchar`-43m} | 0.455 | |
| JVASP-1195 | 186 | 0.965 | ||
| JVASP-1198 | 216 | \mathrm{F\mathchar`-43m} | 1.057 | |
| JVASP-1201 | 216 | \mathrm{F\mathchar`-43m} | 0.729 | |
| JVASP-1216 | 224 | \mathrm{Pn\mathchar`-3m} | 0.644 | |
| JVASP-1222 | 225 | \mathrm{Fm\mathchar`-3m} | 0 | |
| JVASP-1240 | 161 | 3.339 | ||
| JVASP-131 | 164 | \mathrm{P\mathchar`-3m1} | 1.224 | |
| JVASP-1312 | 216 | \mathrm{F\mathchar`-43m} | 1.515 | |
| JVASP-1327 | 216 | \mathrm{F\mathchar`-43m} | 1.794 | |
| JVASP-133719 | 216 | \mathrm{F\mathchar`-43m} | 1.296 | |
| JVASP-1372 | 216 | \mathrm{F\mathchar`-43m} | 1.681 | |
| JVASP-1408 | 216 | \mathrm{F\mathchar`-43m} | 1.322 | |
| JVASP-14616 | 229 | \mathrm{Im\mathchar`-3m} | 0 | |
| JVASP-14968 | 70 | 0 | ||
| JVASP-14970 | 139 | 0 | ||
| JVASP-149871 | 122 | \mathrm{I\mathchar`-42d} | 0.89 | |
| JVASP-149906 | 122 | \mathrm{I\mathchar`-42d} | 0.68 | |
| JVASP-149916 | 225 | \mathrm{Fm\mathchar`-3m} | 0.479 | |
| JVASP-18983 | 61 | 2.254 |
Table S1. A detailed description of the materials in the test set (part one), including JARVIS-ID, space group number and symbol and band gap (vdW-DF-optB88).
| JID | Formula | SPG Number | SPG Symbol | optb88vdw Band Gap (eV) |
| JVASP-1915 | InSe | 160 | 0.174 | |
| JVASP-19780 | SiW | 139 | 0 | |
| JVASP-20092 | CdO | 225 | 0 | |
| JVASP-21211 | Se | 152 | 0.898 | |
| JVASP-22694 | NiO | 225 | 0 | |
| JVASP-23 | CdTe | 216 | 0.498 | |
| JVASP-2376 | ZnSiP | 122 | 1.428 | |
| JVASP-25 | BiTe | 166 | 0.351 | |
| JVASP-29539 | PbI | 186 | 2.259 | |
| JVASP-30 | GaN | 186 | 1.943 | |
| JVASP-32 | AlO | 167 | 6.43 | |
| JVASP-34249 | HfO | 225 | 4.037 | |
| JVASP-34674 | SiO | 20 | 5.673 | |
| JVASP-3510 | BiI | 148 | 2.365 | |
| JVASP-36018 | GeC | 216 | 1.927 | |
| JVASP-36408 | SnC | 216 | 0.748 | |
| JVASP-36873 | BSb | 216 | 0.945 | |
| JVASP-39 | AlN | 186 | 4.474 | |
| JVASP-41 | SiO | 154 | 5.986 | |
| JVASP-4282 | CrBr | 148 | 1.299 | |
| JVASP-43367 | HfO | 61 | 4.033 | |
| JVASP-5224 | HgI | 137 | 0.969 | |
| JVASP-54 | MoS | 194 | 0.922 | |
| JVASP-58349 | SiO | 152 | 5.985 | |
| JVASP-62940 | BN | 194 | 4.46 | |
| JVASP-7836 | BN | 216 | 4.813 | |
| JVASP-79522 | CuO | 131 | 0 | |
| JVASP-8003 | CdS | 216 | 0.993 | |
| JVASP-802 | Hf | 194 | 0 | |
| JVASP-8082 | SrTiO | 221 | 1.814 | |
| JVASP-8118 | SiC | 186 | 2.618 | |
| JVASP-813 | Ag | 225 | 0 | |
| JVASP-8158 | SiC | 216 | 1.62 | |
| JVASP-816 | Al | 225 | 0 | |
| JVASP-8184 | GaP | 186 | 1.263 | |
| JVASP-825 | Au | 225 | 0 | |
| JVASP-85416 | AgS | 14 | 1.11 | |
| JVASP-85478 | CuS | 96 | 0.59 | |
| JVASP-8554 | InCuSe | 122 | 0.006 | |
| JVASP-8559 | TlBr | 221 | 2.023 | |
| JVASP-861 | Cr | 229 | 0 | |
| JVASP-867 | Cu | 225 | 0 | |
| JVASP-890 | Ge | 227 | 0 | |
| JVASP-90668 | ZnCuSnS | 82 | 0.115 | |
| JVASP-91 | C | 227 | 4.457 | |
| JVASP-9117 | FeS | 205 | 0.433 | |
| JVASP-9147 | HfO | 14 | 4.123 | |
| JVASP-9166 | BAs | 166 | 2.739 | |
| JVASP-943 | Ni | 225 | 0 | |
| JVASP-96 | ZnSe | 216 | 1.224 | |
| JVASP-963 | Pd | 225 | 0 | |
| JVASP-972 | Pt | 225 | 0 | |
| JVASP-99732 | CdHgTe | 215 | 0 |
Table S2. A detailed description of the materials in the test set (part two), including JARVIS-ID, space group number and symbol and band gap (vdW-DF-optB88).

Figure S1. The data distribution of the 104 materials in the test set for a) space group number, b) band gap (vdW-DF-optB88), c) atomic number, d) chemical formula, e) crystal system, f) Wychkoff site, g) dimensionality.
When considering the use of a pretrained uMLFF, the most important considerations should be accuracy, rate of convergence, scalability, and computational cost. After running the full CHIPS-FF workflow for the test set of 104 materials, we performed an analysis of which structural relaxation calculations reached convergence (force-maximum threshold of within 200 steps) for bulk materials, surfaces and vacancies. This analysis is presented in Table 1 (FrechetCellFilter) and Table S3 (ExpCellFilter), where the percentage of unconverged results is depicted for each calculation type. Aside from ALIGNN-FF, all other uMLFF have a near-perfect convergence rate for bulk structures. With regards to surfaces and defects, the convergence rate is more variable when using the ExpCellFilter, but near-perfect when using the FrechetCellFilter (with the exception of ALIGNN-FF). Interestingly, the OMat24 models significantly vary in terms of convergence for surfaces depending on training data (for the ExpCellFilter). Specifically, the OMat24 models trained on OMat+MPTrj+sAlexandria have a significantly higher rate of convergence when compared to the models solely trained on OMat (see Table S3). These results (Table 1 vs. Table S3) demonstrate the robustness of the FrechetCellFilter in ASE for these uMLFF models.
| uMLFF Type | Bulk | Surface | Vacancy |
| alignn_ff | 6 | 44 | 35 |
| chgnet | 0 | 0 | 0 |
| eqV2_153M_omat | 1 | 2 | 0 |
| eqV2_31M_omat | 0 | 0 | 0 |
| eqV2_31M_omat_mp_salex | 0 | 0 | 0 |
| eqV2_86M_omat | 0 | 1 | 0 |
| eqV2_86M_omat_mp_salex | 0 | 0 | 0 |
| mace | 0 | 1 | 0 |
| mace-alexandria | 0 | 1 | 0 |
| mace-mpa | 0 | 0 | 0 |
| matgl | 0 | 0 | 0 |
| matgl-direct | 0 | 0 | 0 |
| mattersim | 0 | 0 | 0 |
| orb-d3-v2 | 0 | 0 | 0 |
| orb-v2 | 0 | 0 | 0 |
| sevennet | 0 | 0 | 0 |
Table 1. The percentage of unconverged structural relaxations for bulk, surface, and vacancy calculations for each uMLFF (relaxed using the FrechetCellFilter).
For each of the 104 structures, we performed structural relaxations, fitting of the energy vs. volume curve (equation of state), calculation of the elastic tensor, calculation of the phonon band structure, relaxation of all nonpolar surfaces (among [1, 0, 0], [1, 1, 1], [1, 1, 0], [0, 1, 1], [0, 0, 1], and [0, 1, 0]), and relaxation of each type of point defect in the material. The computational timing and errors in the lattice constants, and elastic properties for each uMLFF model are depicted in Table 2 (computed with the FrechetCellFilter). An interactive version of these results is available on the JARVIS-Leaderboard platform (see Figure 2b for Pearson Correlation Coefficient metric), where it will be continuously updated as new uMLFF models are developed and released. In addition, structural results computed using the ExpCellFilter are depicted in Table S4. Unsurprisingly, we observe more accurate lattice parameters computed with the FrechetCellFilter due to its robustness in optimizing structural parameters (see Table 2 vs. Table S4). In Table 2, we see that all uMLFF models are able to compute reasonably accurate structural parameters. ALIGNN-FF does an excellent job of simultaneously capturing , , and . This is expected due to the fact that ALIGNN-FF was trained on the JARVIS-DFT dataset (vdW-DF-optB88) and the target/“ground truth”, in addition to the initial structures, are from JARVIS-DFT. The OMat24 models perform exceedingly well, but are the most computationally expensive. The ORB models also perform exceedingly well for structural relaxation and have the advantage of being close to an order of magnitude more computationally efficient. One interesting trend we observe in our results for this set of 104 materials is that the error for lattice constant is substantially higher than and . This can be due in part to the test set containing a significant portion ( %) of vdW bonded (“2D-bulk-like”) layered structures (see Figure S1). Since most of these uMLFF were trained on PBE data without vdW corrections (with the exception of ALIGNN-FF), we can expect larger errors for vdW systems. For example, we observe significant errors for hexagonal BN (JVASP-62940) when relaxed with orb-v2, where we observe over 10 % error in . These errors are drastically more significant when using the ExpCellFilter along with orb-v2, where we observed a 130 % error in for JVASP-62940 (see Table S4 for total MAE). Fortunately, some of these models such as MACE and ORB have explicit dispersion corrections that can be added to more accurately address vdW interactions.53,54,67,68 To test this, we ran the workflow for dispersion-corrected orb-d3-v2. For orb-d3-v2, we observe a reduction in error for and volume (see Table 2). For JVASP-62940 relaxed with orb-d3-v2, the percent error in is reduced to 0.02 %. This emphasizes the importance of dispersion corrections in uMLFF architectures for vdW bonded materials.
Table 2: The mean absolute error (MAE) and total computational time for a variety of material properties calculated with each uMLFF type (relaxed using the FrechetCellFilter). The MAE is calculated with respect to JARVIS-DFT data. Properties include (in order) lattice constants a, b, c, volume, bulk modulus, and C11 and C44 components of the elastic tensor. The computational time is measured per CPU. An interactive and up-to-date version of the table is available in the JARVIS-Leaderboard. The green indicates the best performing model.
| uMLFF Type | err_a | err_b | err_c | err_vol | err_kv | err_c11 | err_c44 | Time |
| GPa | GPa | GPa | seconds | |||||
| alignn_ff | 0.011 | 0.011 | 0.014 | 0.42 | 135 | 196 | 74 | 33,081.6 |
| chgnet | 0.036 | 0.038 | 0.072 | 3.29 | 89 | 59 | 46 | 14,102.6 |
| eqV2_153M_omat | 0.022 | 0.025 | 0.055 | 2.54 | 117 | 75 | 32 | 304,173.9 |
| eqV2_31M_omat | 0.015 | 0.016 | 0.052 | 2.25 | 114 | 78 | 37 | 81,420.3 |
| eqV2_31M_omat_mp_salex | 0.016 | 0.017 | 0.043 | 1.73 | 112 | 46 | 41 | 65,971.6 |
| eqV2_86M_omat | 0.022 | 0.025 | 0.047 | 2.26 | 118 | 86 | 35 | 218,168.4 |
| eqV2_86M_omat_mp_salex | 0.016 | 0.017 | 0.054 | 2.23 | 113 | 43 | 35 | 146,318.1 |
| mace | 0.027 | 0.027 | 0.075 | 2.85 | 94 | 42 | 38 | 27,620.4 |
| mace-alexandria | 0.068 | 0.072 | 0.191 | 5.65 | 78 | 69 | 45 | 62,932.5 |
| mace-mpa | 0.040 | 0.042 | 0.084 | 3.58 | 110 | 35 | 34 | 27,255.2 |
| matgl | 0.043 | 0.046 | 0.106 | 4.23 | 83 | 64 | 41 | 10,446.1 |
| matgl-direct | 0.043 | 0.042 | 0.089 | 3.00 | 84 | 72 | 50 | 13,420.6 |
| mattersim | 0.027 | 0.028 | 0.089 | 3.11 | 110 | 32 | 32 | 19,047.3 |
| orb-d3-v2 | 0.025 | 0.025 | 0.032 | 1.60 | 130 | 77 | 74 | 15,118.9 |
| orb-v2 | 0.015 | 0.016 | 0.044 | 1.91 | 118 | 81 | 74 | 12,184.7 |
| sevennet | 0.028 | 0.030 | 0.094 | 3.39 | 103 | 42 | 37 | 17,739.3 |
Table 2.

Figure 2. a) Scaling analysis of various uMLFF up to 10,000 atoms (for a supercell of Cu), b) an example JARVIS-Leaderboard entry for the force MAE of the MLEARN dataset, c) an example of the interactive error metrics within the JARVIS-Leaderboard (Pearson correlation coefficient).
| uMLFF Type | err_a (Å) | err_b (Å) | err_c (Å) | err_vol (Å) |
| alignn_ff | 0.087 | 0.102 | 0.137 | 9.95 |
| chgnet | 0.046 | 0.049 | 0.109 | 3.58 |
| eqV2_153M_omat | 0.036 | 0.040 | 0.111 | 2.90 |
| eqV2_31M_omat | 0.029 | 0.032 | 0.099 | 3.05 |
| eqV2_31M_omat_mp_salex | 0.028 | 0.030 | 0.096 | 3.17 |
| eqV2_86M_omat | 0.033 | 0.040 | 0.094 | 3.41 |
| eqV2_86M_omat_mp_salex | 0.027 | 0.030 | 0.100 | 3.14 |
| mace | 0.035 | 0.038 | 0.084 | 3.00 |
| mace-alexandria | 0.081 | 0.086 | 0.206 | 6.05 |
| mace-mpa | 0.044 | 0.046 | 0.098 | 3.57 |
| matgl | 0.052 | 0.057 | 0.128 | 3.50 |
| matgl-direct | 0.044 | 0.046 | 0.106 | 2.88 |
| mattersim | 0.031 | 0.033 | 0.110 | 3.18 |
| orb-d3-v2 | 0.031 | 0.030 | 0.048 | 1.90 |
| orb-v2 | 0.023 | 0.025 | 0.159 | 3.38 |
| sevennet | 0.035 | 0.038 | 0.100 | 3.43 |
Table S4. The mean absolute error (MAE) for lattice constants , , , and volume calculated with each uMLFF type (relaxed using the ExpCellFilter), compared to JARVIS-DFT data. The green highlights the best performing models.
Fitting the equation of state to obtain bulk modulus and calculating the elastic tensor prove to be a more challenging task for these uMLFF models, emphasizing difficulties in modeling the potential energy landscape far from equilibrium. We found that models such as MACE-Alexandria and MatGL have superior performance in predicting the energy vs. volume curves compared to the OMat and ORB models. For contributions to the elastic tensor ( and ), we also observe relatively high errors. We see that only a few models can simultaneously predict and with reasonable accuracy (MAE less than 50 GPa). Interestingly, we found that eqV2 31M omat, eqV2 86M omat and eqV2 153M omat can predict C44 with reasonable accuracy, but have much higher prediction errors for C11, while eqV2_31M_omat_mp_salex and eqV2_86M_omat_mp_salex do a better job at predicting C11 and C44 simultaneously. We observe that MACE-MPA-0 and MatterSim give the best results for simultaneous C11 and C44 predictions.
Out of the 104 materials in our test set, there were 84 matching entries for DFT phonon calculations in JARVIS, from which the composite MAE of the phonon band structure was computed in Table 3. We performed these phonon calculations with the uMLFFs at various displacements in the finite-displacement calculation of force constants. We found that models such as ALIGNN-FF and MatGL have relatively large errors in phonon predictions. For certain uMLFFs, we observe a consistent phonon MAE as the displacements get smaller (MACE-MP-0, MACE-MPA-0, MACE-Alexandria, Mattersim, SevenNet). For ORB and OMat models, we see a drastic increase in phonon MAE as the displacements become smaller. This signifies that ORB and OMat models could possibly perform poorly in the low-force regime due to increased noise from the small displacements. This is consistent with recent results, where it was reported that OMat and ORB models had substantial errors in phonon predictions (for small displacements) when compared to models such as MatterSim, MACE, CHGNet, and SevenNet for 10,000 materials. 90 This work hypothesized that the errors are due to ORB and OMat computing forces as a direct output of the neural network as opposed to calculating the forces by evaluating the derivative of the energy with respect to the atomic positions (through back-propagation). 90 MatterSim offers similar performance (and consistency across atomic displacements) in terms of predicting the phonon band structure when compared to accurate equivariant uMLFFs such as MACE and SevenNet. This is especially interesting since MatterSim-v1 is not an equivariant uMLFF. In addition, we see improved phonon results for the newer MACE-MPA-0 when compared to MACE-MP-0.
| Method | Displacements (Å) | 0.2 | 0.05 | 0.01 | 0.001 |
| alignn_ff | 155 | 155 | 157 | 164 | |
| chgnet | 70 | 75 | 83 | 91 | |
| eqV2_153M_omat | 49 | 47 | 49 | 75 | |
| eqV2_31M_omat | 48 | 46 | 53 | 73 | |
| eqV2_31M_omat_mp_salex | 48 | 46 | 112 | 219 | |
| eqV2_86M_omat | 51 | 47 | 49 | 77 | |
| eqV2_86M_omat_mp_salex | 48 | 46 | 103 | 208 | |
| mace | 58 | 60 | 60 | 60 | |
| mace-alexandria | 82 | 84 | 85 | 85 | |
| mace-mpa | 50 | 50 | 50 | 50 | |
| matgl | 82 | 93 | 94 | 94 | |
| matgl-direct | 86 | 101 | 102 | 101 | |
| mattersim | 48 | 47 | 47 | 47 | |
| orb-d3-v2 | 53 | 50 | 85 | 181 | |
| orb-v2 | 49 | 50 | 108 | 201 | |
| sevennet | 54 | 55 | 56 | 56 |
Table 3. MAE for phonon band structure (in ) computed with different values of displacement.
In addition to summing the computational timing for the entire workflow, we also examined the scaling behavior of each uMLFF (on CPU). These results for supercells of Cu (up to 10,000 atoms) are depicted in Figure 2a). Unsurprisingly, we see that OMat model scales much more drastically with system size as compared to the other uMLFF models. In addition, we found that the invariant models have more favorable scaling than the equivariant models such as MACE and SevenNet, with SevenNet having slightly better scaling than MACE. The scalability of ORB and MatterSim is a huge advantage due to their accuracy of various material property predictions. Similar scalability studies were conducted in Ref. 54
The quantities of surface energy and vacancy formation energy are difficult to predict with machine learning methods and provide a rigorous benchmarking test for uMLFF architectures. Out of the 104 materials, there exists 85 entries for surface DFT calculations and 48 entries for defect DFT calculations within JARVIS-DFT. We used these calculations to compare our uMLFF results. Figure 3a) and Figure 3b) depict the parity plots and corresponding MAE for surface energy and vacancy formation energy of our test sets. The very low error for surface energy () and vacancy formation energy () highlight some of the successes of recent state-of-the-art models such as OMat24, ORB, MACE-MPA-0 and MatterSim. We also observe reasonably accurate results for MACE-MP-0 and SevenNet when predicting vacancy formation energy and surface energies. Recently, the proprietary model PFP from Preferred Networks Inc. was evaluated for our CHIPS-FF surface energy benchmark and achieved an MAE of 0.19 J/m, 124 achieving similar accuracy to the OMat24 and ORB models. The low cost of ORB, high rate of convergence and high accuracy make it a viable tool to relax larger surfaces and defect supercells.

Figure 3. Parity plots for a) surface energy (in J/m) and b) vacancy formation energy (in eV) for each uMLFF type with respect to JARVIS-DFT data (MAE is given in the inset and the best performing models are included in the plots). c) Depicts the parity plot for work of adhesion ( in J/m) for several material interfaces and MAE with respect to previous calculated and experimental data (taken from Ref. 123). The parity plot is zoomed in to focus on a subset of calculations, but the MAE is computed for the entire test set. To the right of the parity plot, we focus on the best performing prediction for the Al(111)/AlO(001) interface, showing the xy grid search and optimal in-plane interface configurations with each uMLFF.
In addition to benchmarking the properties of bulk materials, we utilized the CHIPS-FF workflow for interface calculations. Figure 3c depicts the parity plot for work of adhesion () for various material interfaces. The MAE for is computed with respect to “ground truth” data taken from Ref. 123 (from experiment and theory). As seen in Figure 3c, predictions for are extremely poor for each uMLFF type. The parity plot in Figure 3c is zoomed in to focus on a subset of calculations, but the MAE is computed for the entire test set of interfaces, which include significant outliers. This is not surprising due to the fact that none of these uMLFF are trained on interface data. We went on to analyze one of the best performing predictions for an Al(111)AlO(001) interface (see red box on Figure 3c). From here, we performed an in-plane (xy) grid search to find the optimal energy configuration of the interface. In addition, this in-plane scan can be a test of how smooth the potential energy surface is. On the right panel of Figure 3c we see the results of this interface scan for a number of uMLFFs. We observe a smoother potential energy surface for MACE, SevenNet, MatterSim and OMat24 models. Although the MAE is extremely high for all uMLFF models for , it is the lowest for ALIGNN-FF. This can be due in part to ALIGNN-FF being trained on vdW-corrected DFT calculations and having a substantial amount of exfoliable materials in the dataset.
We chose to benchmark the performance of uMLFF models for the generation of amorphous Si (a-Si) by performing melt/quench simulations. Figure 4 depicts a summary of these results, where the radial distribution function (RDF) is shown as a function of pair separation distance. We benchmarked these uMLFF RDF curves against the RDF curve obtained from computationally expensive AIMD simulations for a-Si. The inset of Figure 4 depicts the MAE of each RDF curve with respect to AIMD. We observe that architectures such as MACE, SevenNet, and OMat24 (eqV2), CHGNet, ALIGNN-FF, MatterSim and ORB have excellent accuracy, while MatGL has a slightly higher MAE and a lower value. The final amorphous structures for AIMD, MatterSim, eqV2 and ORB (best performing uMLFF models) are given in the inset of Figure 4. The exceptional performance of ORB and MatterSim (both invariant uMLFFs) for a-Si is significant due to the fact that it is much less costly than other similarly performing equivariant uMLFF models. The relatively high accuracy of ALIGNN-FF with regards to a-Si is surprising due to the fact that ALIGNN-FF had inferior performance for a number of properties and is also an invariant model. A more thorough assessment of how ALIGNN-FF and other uMLFFs can model amorphous materials will be the subject of future work, which will also utilize the CHIPS-FF infrastructure. These results bring into question whether or not equivariance is necessary to achieve accurate results for amorphous materials with uMLFFs, and more rigorous benchmarking will need to be conducted to bring clarity to this.

Figure 4. The calculated radial distribution function () as a function of pair separation distance computed with several uMLFF models for amorphous Si (by performing melt/quench simulations). Ab initio (AIMD) results are given as a ground truth benchmark (black dotted line) and the MAE and with respect to AIMD is reported in the inset. The atomic structure for amorphous Si is given for AIMD and the best performing uMLFF models (MatterSim, eqV2_31M_omat and ORB).
In addition to testing these uMLFF models on a smaller and more focused dataset, we also tested on a larger and more diverse dataset for force predictions. Unlike total energy, forces are more independent of exchange-correlation functional and the accuracy of force predictions on various datasets can give valuable insight into how a particular model performs. We chose to benchmark each uMLFF on the MLEARN dataset from Ref., 125 which consists of face-centered cubic (Cu, Ni) and body-centered cubic (Li, Mo) metals and diamond group IV semiconductors (Si, Ge) which span a wide range of crystal structures and bonding environments ( data points for each element). These results are depicted in Table 4. In addition, we benchmarked the accuracy of force predictions on very large datasets that were used to train uMLFFs. These datasets included ALIGNN FF DB (307,000 used to train ALIGNN-FF), MPF (188,000 used to train M3GNet), and MPTrj (1.58 million used to train CHGNet, MACE, SevenNet and used in the training of ORB and OMat models). These results are depicted in Table 5. Unsurprisingly, superior accuracy is obtained across all datasets when the OMat and ORB models are used to predict forces. We also observe highly accurate forces for MatterSim and a substantial reduction in force error when going from MACE-MP-0 to MACE-MPA-0. The CHIPS-FF package has built in functions to perform these benchmarking calculations on each respective dataset, with a flexible framework to add additional datasets (such as OMat and Alexandria) and new uMLFFs as they develop over time.
| uMLFF Type | Mo Error (eV/Å) | Mo Time (s) | Li Error (eV/Å) | Li Time (s) | Si Error (eV/Å) | Si Time (s) | Ni Error (eV/Å) | Ni Time (s) | Cu Error (eV/Å) | Cu Time (s) | Ge Error (eV/Å) | Ge Time (s) |
| alignn_ff | 0.869 | 0.122 | 0.324 | 0.303 | 0.216 | 0.395 | ||||||
| chgnet | 0.361 | 0.045 | 0.188 | 0.059 | 0.058 | 0.207 | ||||||
| eqV2 153M_omat | 0.096 | 0.004 | 0.012 | 0.009 | 0.006 | 0.031 | ||||||
| eqV2 31M_omat | 0.098 | 0.004 | 0.016 | 0.010 | 0.005 | 0.033 | ||||||
| eqV2 31M_omat_mp_salex | 0.056 | 0.007 | 0.021 | 0.011 | 0.006 | 0.028 | ||||||
| eqV2 86M_omat | 0.094 | 0.004 | 0.014 | 0.009 | 0.006 | 0.032 | ||||||
| eqV2 86M_omat_mp_salex | 0.048 | 0.005 | 0.016 | 0.009 | 0.005 | 0.030 | ||||||
| mace | 0.292 | 0.035 | 0.149 | 0.055 | 0.059 | 0.203 | ||||||
| mace-alexandria | 0.343 | 0.025 | 0.202 | 0.099 | 0.055 | 0.161 | ||||||
| mace-mpa | 0.130 | 0.014 | 0.083 | 0.026 | 0.013 | 0.063 | ||||||
| matgl | 0.333 | 0.042 | 0.226 | 0.069 | 0.124 | 0.299 | ||||||
| matgl-direct | 0.300 | 0.031 | 0.214 | 0.114 | 0.054 | 0.260 | ||||||
| mattersim | 0.121 | 0.024 | 0.067 | 0.021 | 0.012 | 0.068 | ||||||
| orb-v2 | 0.130 | 0.018 | 0.068 | 0.026 | 0.015 | 0.060 | ||||||
| sevennet | 0.365 | 0.051 | 0.129 | 0.089 | 0.055 | 0.131 |
Table 4. A comparison of force errors (eV/Å) and timings (s) for the mlearn DFT dataset for Mo, Li, Si, Ni, Cu, and Ge. The green color highlights the best performing model for each element.
| MLFF Type | ALIGNN_FF_DB (307k) Error (eV/) | ALIGNN_FF_DB (307k) Time (s) | MPF (188k) Error (eV/) | MPF (188k) Time (s) | MPTrj (1.58M) Error (eV/) | MPTrj (1.58M) Time (s) |
| alignn_ff | 0.472 | 0.639 | 0.581 | |||
| chgnet | 0.088 | 0.101 | 0.060 | |||
| eqV2_153M_omat | 0.050 | - | - | - | - | |
| eqV2_31M_omat | 0.051 | 0.046 | - | - | ||
| eqV2_31M_omat_mp_salex | 0.055 | 0.032 | - | - | ||
| eqV2_86M_omat | 0.052 | 0.043 | - | - | ||
| eqV2_86M_omat_mp_salex | 0.056 | - | - | - | - | |
| mace | 0.104 | 0.125 | 0.059 | |||
| mace-alexandria | 0.221 | 0.395 | - | - | ||
| mace-mpa | 0.072 | 0.061 | 0.039 | |||
| matgl | 0.091 | 0.066 | 0.100 | |||
| matgl-direct | 0.104 | 0.095 | 0.118 | |||
| mattersim | 0.071 | 0.047 | 0.059 | |||
| orb-v2 | 0.062 | 0.049 | 0.028 | |||
| sevennet | 0.092 | 0.081 | 0.041 |
Table 5. A comparison of force errors (eV/) and timings (s) for different datasets: ALIGNN_FF_DB, MPF, and MPTrj. The green color highlights the best performing model.
We have introduced CHIPS-FF, an open-source benchmarking platform for MLFF architectures, which has the capability to test properties beyond the standard total energy, including forces, phonons, elastic properties, surface energy, vacancy formation energy and properties of interfaces. We benchmarked several recent state-of-the-art uMLFF architectures on a subset of 104 materials most relevant for the semiconductor industry, taking into account accuracy with respect to DFT, convergence and computational cost. As MLFFs continue to develop over time, we expect the CHIPS-FF benchmarking platform to be critical in terms of testing the quality of uMLFFs.
Data Availability Statement
The CHIPS-FF package can be found at https://github.com/usnistgov/chipsff. All data specific to this work will be made available on Figshare upon publication. Related benchmarks for this work can be found at https://pages.nist.gov/jarvis_leaderboard/Special/CHIPS_FF/.
Conflicts of Interest
The authors declare no competing interests.
Supporting Information
Description of software used, detailed description of test set, results for ExpCellFilter, CHIPS-FF tutorial
Acknowledgments
All authors thank the National Institute of Standards and Technology for funding, computational, and data-management resources. This work was performed with funding from the CHIPS Metrology Program, part of CHIPS for America, National Institute of Standards and Technology, U.S. Department of Commerce. Certain commercial equipment, instruments, software, or materials are identified in this paper in order to specify the experimental procedure adequately. Such identifications are not intended to imply recommendation or endorsement by NIST, nor it is intended to imply that the materials or equipment identified are necessarily the best available for the purpose. The authors would like to acknowledge John Bonini (NIST), Orbital Materials, and Preferred Networks for fruitful discussions regarding the manuscript and software package.
Supporting Information: CHIPS-FF: Evaluating Universal Machine Learning Force Fields for Material Properties
Daniel Wines; Kamal Choudhary
CHIPS-FF mainly relies on several Python-based packages: Atomic Simulation Environment (ASE), [1] JARVIS-Tools, [2, 3] phonopy, [4, 5] elastic[6, 7, 8] and Intermat. [9] These packages work in conjunction to design and manipulate atomic structures, access materials databases and compute physically observable quantities. ASE is a general toolkit for setting up, running and analyzing atomistic calculations for a variety of electronic structure and molecular dynamics codes. In addition, it allows for the integration of machine learning force fields (MLFFs) as calculators within the atomistic workflows. JARVIS-Tools, which utilizes ASE for certain functionalities, is an infrastructure to perform high-throughput atomistic calculations, manipulate structures, and parse data and results from different electronic structure and molecular dynamics codes. The main feature of JARVIS-Tools is the seamless integration of large-scale materials databases, such as JARVIS-DFT, which can be used to generate initial structures, filter materials based on specific properties and compare data with density functional theory (DFT) benchmarks. In this work, we utilize the JARVIS-DFT datasets of bulk materials, surfaces, and defects. The phonopy package is used to determine phonon and lattice dynamics based on computed force constants, which can be obtained from the finite-displacement method. These force constants can be computed via DFT calculations or via MLFF. The phonon dispersion, phonon density of states and derived properties such as heat capacity and zero-point energy can be calculated from phonopy. The elastic package (which utilizes routines in ASE) is based on standard elasticity theory and the finite deformation approach to compute the elastic tensor of a crystal. The Intermat package is a toolkit for the generation of material interfaces that utilizes ASE and JARVIS-Tools to carry out high-throughput simulations. Interface configurations can be generated using the algorithm of Zur et al. [10] and the - (in-plane) and (out-of-plane) parameters of the interface can be optimized with DFT or MLFFs (passed through ASE).
| uMLFF Type | Bulk | Surface | Vacancy |
| alignn_ff | 7 | 38 | 38 |
| chgnet | 0 | 1 | 0 |
| eqV2_153M_omat | 0 | 15 | 2 |
| eqV2_31M_omat | 0 | 17 | 0 |
| eqV2_31M_omat_mp_salex | 0 | 6 | 0 |
| eqV2_86M_omat | 2 | 18 | 2 |
| eqV2_86M_omat_mp_salex | 0 | 6 | 0 |
| mace | 0 | 7 | 4 |
| mace-alexandria | 3 | 16 | 2 |
| mace-mpa | 0 | 7 | 0 |
| matgl | 0 | 5 | 10 |
| matgl-direct | 4 | 7 | 13 |
| mattersim | 0 | 5 | 2 |
| orb-d3-v2 | 0 | 1 | 0 |
| orb-v2 | 1 | 0 | 0 |
| sevennet | 0 | 2 | 0 |
Table S3. The percentage of unconverged structural relaxations for bulk, surface, and vacancy calculations for each uMLFF (relaxed using the ExpCellFilter). The green highlights which models have the highest convergence rate.
Tutorial
Installation and Setup
To install CHIPS-FF, first clone the repository and set up a conda environment.
Cloning and Setting Up the Environment
git clone https://github.com/usnistgov/chipsff
conda env create -f environment.yml -n chipsff
conda activate chipsff
cd chipsff
pip install -e .
Input Configuration
CHIPS-FF utilizes a JSON input file that specifies various parameters such as the material identifier, calculator type, and simulation settings. Below is an example input file (input.json) for a single material:
Example input.json
{
"jid": "JVASP-1002",
"calculator_type": "chgnet",
"chemical_potentials_file": "chemical_potentials.json",
"properties_to_calculate": [
"relax_structure",
"calculate_ev_curve",
"calculate_formation_energy",
"calculate_elastic_tensor",
"run_phonon_analysis",
"analyze_surfaces",
"analyze_defects",
"run_phonon3_analysis",
"general_melter",
"calculate_rdf"
],
"bulk_relaxation_settings": {
"filter_type": "FrechetCellFilter",
"relaxation_settings": {
"fmax": 0.05,
"steps": 200,
"constant_volume": false
}
},
"phonon_settings": {
"dim": [2, 2, 2],
"distance": 0.01
},
"use_conventional_cell": false
}
Here is an example input file for material interfaces (e.g., interface_input.json):
Example interface_input.json
{
"film_id": ["JVASP-1002"],
"substrate_id": ["JVASP-816"],
"calculator_type": "alignn_ff",
"chemical_potentials_file": "chemical_potentials.json",
"film_index": "1_1_0",
"substrate_index": "1_1_0",
"properties_to_calculate": [
"analyze_interfaces"
]
}
Running the Simulations
The main run_chipsff.py script allows for a command-line interface for executing the simulations. For a single material analysis:
Running Single Material Analysis
python run_chipsff.py --input_file input.json
For interface analysis:
Running Interface Analysis
python run_chipsff.py --input_file interface_input.json
Overview of Key Methods
The CHIPS-FF framework includes several key functions:
relax_structure(): Optimizes atomic structures.calculate_formation_energy(): Computes formation energies using relaxed structures and chemical potential data.calculate_elastic_tensor(): Evaluates elastic properties.calculate_ev_curve(): Fits energy-volume curves to determine equilibrium parameters.run_phonon_analysis(): Performs phonon band structure and thermal property calculations.analyze_defects()andanalyze_surfaces(): Analyze defect and surface energies.analyze_defects_from_db()Analyze defect energies with initial structures directly from the JARVIS-DFT vacancy database.run_phonon3_analysis(): Calculates third order force constants for thermal conductivity.general_melter(): Executes MD simulations for melting and quenching processes.analyze_interfaces(): Conducts interface analysis between film and substrate materials.
References
- [1]Hjorth Larsen, A. et al. The atomic simulation environment—a Python library for working with atoms. Journal of Physics: Condensed Matter 2017, 29, 273002.DOI
- [2]Wines, D.; Gurunathan, R.; Garrity, K. F.; DeCost, B.; Biacchi, A. J.; Tavazza, F.; Choudhary, K. Recent progress in the JARVIS infrastructure for next-generation data-driven materials design. Applied Physics Reviews 2023, 10, 041302.DOI
- [3]Choudhary, K. et al. The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design. npj Computational Materials 2020, 6, 173.DOI
- [4]Togo, A.; Chaput, L.; Tanaka, I. Distributions of phonon lifetimes in Brillouin zones. Phys. Rev. B 2015, 91, 094306.DOI
- [5]Togo, A.; Chaput, L.; Tadano, T.; Tanaka, I. Implementation strategies in phonopy and phono3py. J. Phys. Condens. Matter 2023, 35, 353001.DOI
- [6]Jochym, P. T. Module for calculating elastic tensor of crystals. https://github.com/jochym/Elastic/, 2022.
- [7]Jochym, P. T.; Parlinski, K.; Sternik, M. TiC lattice dynamics from ab initio calculations. The European Physical Journal B - Condensed Matter and Complex Systems 1999, 10, 9–13.DOI
- [8]Jochym, P. T.; Parlinski, K. Ab initio lattice dynamics and elastic constants of ZrC. The European Physical Journal B - Condensed Matter and Complex Systems 2000, 15, 265–268.DOI
- [9]Choudhary, K.; Garrity, K. F. InterMat: accelerating band offset prediction in semiconductor interfaces with DFT and deep learning. Digital Discovery 2024, 3, 1365–1377.DOI
- [10]Zur, A.; McGill, T. C. Lattice match: An application to heteroepitaxy. Journal of Applied Physics 1984, 55, 378–386.DOI