WorksPhysics and Astronomy
Overcoming systematic softening in universal machine learning interatomic potentials by fine-tuning
Abstract
Machine learning interatomic potentials (MLIPs) have introduced a new paradigm for atomic simulations. Recent advancements have seen the emergence of universal MLIPs (uMLIPs) that are pre-trained on diverse materials datasets, providing opportunities for both ready-to-use universal force fields and robust foundations for downstream machine learning refinements. However, their performance in extrapolating to out-of-distribution complex atomic environments remains unclear. In this study, we highlight a consistent potential energy surface (PES) softening effect in three uMLIPs: M3GNet, CHGNet, and MACE-MP-0, which is characterized by energy and force under-prediction in a series of atomic-modeling benchmarks including surfaces, defects, solid-solution energetics, phonon vibration modes, ion migration barriers, and general high-energy states. We find that the PES softening behavior originates from a systematic underprediction error of the PES curvature, which derives from the biased sampling of near-equilibrium atomic arrangements in uMLIP pre-training datasets. We demonstrate that the PES softening issue can be effectively rectified by fine-tuning with a single additional data point. Our findings suggest that a considerable fraction of uMLIP errors are highly systematic, and can therefore be efficiently corrected. This result rationalizes the data-efficient fine-tuning performance boost commonly observed with foundational MLIPs. We argue for the importance of a comprehensive materials dataset with improved PES sampling for next-generation foundational MLIPs.
Introduction
Artificial intelligence (AI) is increasingly shifting the paradigm of scientific discovery to accelerate research and solve real-world scientific challenges [1]. While ab-initio quantum mechanical simulation methods, such as density functional theory (DFT), offer the theoretical foundation to investigate material and chemical science problems at the atomic scale, their computational demands limit their applicability in both spatial and temporal scales. Recent advancements in machine learning interatomic potentials (MLIPs) [2, 3] have enabled the opportunity to scale up quantum mechanical methods to million atoms simulations such as water, copper [4], and biomolecules [5].
Alongside improvements in atomic environment descriptors and graph neural networks that enhance the expressivity of MLIP models [3, 6], universal machine learning interatomic potentials (uMLIPs) have demonstrated another avenue by taking advantage of pre-training on large and comprehensive material datasets [7, 8, 9, 10, 11, 12, 13]. These uMLIPs enable out-of-box atomic modeling covering the entire periodic table as well as providing robust machine-learning foundations for fine-tuning downstream tasks. While uMLIPs hold considerable promise, a critical challenge lies in their ability to reliably generalize to complex and diverse chemical environments, particularly those that deviate significantly from the pre-training data distribution. Several recent benchmark efforts have tested the uMLIPs’ ability to identify stable materials [14], surface energies [15], lattice relaxations and vibrational properties [16], etc. A systematic understanding of the ability of uMLIPs to extrapolate to common atomic-modeling tasks, especially those with atomic environments that are out of distribution (OOD), remains an open question with implications for their real-world applicability in material discovery and design.
In this work, we systematically investigate the extrapolative capabilities of three foundational MLIPs – M3GNet [7], CHGNet [8], and MACE-MP-0 [10] (hereafter referred to as MACE) – across a diverse suite of OOD material modeling tasks, including surface energies, defect energies, solid-solution energetics, phonon vibrational modes, ion migration barriers, and high-energy transition states. Across all benchmark tests for all uMLIP models, our analysis reveals a consistent potential energy surface (PES) softening behavior, characterized by a systematic underprediction of energies and forces, as illustrated in Figure 1. We attribute the PES softening issue to the combination of the biased sampling of near-ground-state configurations in the uMLIP pre-training datasets [17], which primarily comprise DFT ionic relaxation trajectories near PES local energy minima. The uMLIPs trained predominantly on small energy and force labels suffer from distribution shifts and experience increased but systematic prediction errors in high-energy PES regions which are important for the kinetics of rare events, such as ion migrations, and for the energy of defects with undercoordinated atoms, such as vacancies and surfaces.

Figure 1. Potential energy surface softening in uMLIPs. Left: schematic representation of the potential energy surface (PES) described in density functional theory (DFT), with two arbitrary coordinate axes. Right: PES described by universal machine learning interatomic potentials (uMLIPs), which well describes the PES regions sampled by near-equilibrium states in the pre-training dataset (orange), but experience larger errors in high-energy regions (red) with under-prediction of energies and forces. The softening behavior is largely systematic and, therefore can be efficiently fixed by a linear correction with a small amount of data augmentation.
We demonstrate that this systematic PES softening can be effectively mitigated by fine-tuning with a minimal amount of data points. We find that a simple linear correction derived from a single DFT reference label is sufficient to remove much of the PES softening issue, significantly enhancing the performance and robustness of uMLIPs. We rationalize this observation by arguing that a considerable amount of prediction errors in pre-trained uMLIPs are highly systematic, and therefore can be efficiently corrected by modifying a limited fraction of the model parameters with only a small amount of data augmentation. Our work provides a theoretical foundation for the widely observed data-efficient performance boosts achieved by fine-tuning uMLIPs and highlights the advantage of atomic modeling with large and comprehensive foundational AI models.
Results
Machine Learning Interatomic Potentials Framework
MLIPs approximate the total energy of a system as a sum of atomic contributions, each dependent on the positions and chemical identities of the atoms in their local environment:
is a learnable function that maps the set of position vectors and chemical species of the neighboring atoms to the energy contribution of atom . The force acting on each atom is calculated as the derivative of the total energy with respect to its position. In the training process, the parameters of the MLIP model are optimized to minimize the discrepancy between the predicted energies and forces and the corresponding reference values from the DFT labels.
The design of the atomic environment descriptor function is crucial to developing accurate and efficient MLIPs. To capture the essential physics and chemistry of the system, should be informative and satisfy proper translational and rotational symmetries. This is typically achieved through the use of graph representations [18], high-order interactions [6, 7], the preservation of SE(3)/E(3)-equivariance using tensor products based on spherical harmonics [3, 10], Fourier basis [19], or Cartesian-coordinates-based atomic density expansion [20]. Additionally, the incorporation of chemical information, such as charge [21] or atomic magnetic moment [8], has been shown to enhance the predictive power of MLIPs.
In addition, recent efforts have been made to pre-train MLIPs on large open-sourced materials datasets such as the Materials Project [17], which primarily consists of DFT ionic relaxation trajectories of various compounds and elements across the periodic table. While initial benchmarks have shown the promising applicability of universal MLIPs in predicting bulk materials energetics [14, 16], their performance and limitations in OOD atomic configurations require more benchmarking as the energy of these configurations is often directly relevant for practical materials behavior. The following sections present a systematic assessment of the uMLIPs’ ability to extrapolate to low-symmetry OOD atomic configurations that are crucial for atomic-modeling tasks.
Surface Energies
Surface energies play an important role in determining the stability and morphology of materials, especially at the nano-scale where the surface-to-volume ratio is significant. Accurate prediction of surface energies is crucial for various applications such as catalysis [22], corrosion [23], adhesion [24], nucleation [25], and thin film growth [26]. In this section, we assess uMLIP’s performance in predicting surface energies, which are calculated as
where are the relaxed energies of the slab/bulk structures that can be obtained independently using DFT or MLIP methods in a large supercell approach. denotes the surface area of the slab.
The energies of 147 surfaces with multiple Miller indices of 29 elements and binary compounds are evaluated, including Si, Cu, AlO, LiF, ZnS, etc. The DFT and uMLIP calculation details are listed in the Methods section and Supplementary Table 1 lists the full set of elements and compounds with their corresponding prediction errors. Figure 2(a) shows the uMLIP surface energies versus the DFT surface energies for the three uMLIPs tested, where MAE stands for the model’s mean absolute error. MACE exhibits relatively better performance compared to CHGNet and M3GNet, achieving a MAE of 0.032 eV/Å. All three uMLIPs consistently underestimate the surface energies compared to DFT, except for a few predictions made by MACE and M3GNet. The trend in our result is consistent with the recent evaluation of Focassio et al. [15] on the surface energies of elemental crystals.
Figure 2. uMLIP performance on surfaces, defects, and solid solutions. (a) Comparison of DFT surface energies and MLIP surface energies, evaluated on 147 surfaces from 29 chemical systems. (b) Comparison of DFT defect energies and MLIP defect energies, evaluated on 134 point defects from 32 chemical systems. (c) Formation energies in solid solution from DFT and uMLIPs. Each point corresponds to the energy of a specific Ca-Mg cation arrangement at a given Ca fraction. The distributions of energies are collectively underestimated, which would lead to an underprediction of the miscibility gap temperature in uMLIPs compared to DFT.
Defect Energies
We also analyze the accuracy of uMLIPs in calculating point defect energies, which is crucial for understanding a material’s vacancy formation [27], dopabilities [28], mechanical properties [29], and ionic mobilities [30]. Specifically, we perform benchmarks for point defects including vacancies, interstitials, and anti-site defects. In metallic systems, the point defect energy can be calculated from the energy of a defect structure referenced to the corresponding perfect structure and the external chemical potential of the species added or removed
where is the chemical potential of the species forming the defect and is the number of atoms of added (+1) or removed (−1) at the defect. To avoid additional errors in the defect energy introduced by the equilibrium chemical potentials determined from the phase diagram, we used the energy of the pure elemental phases for this benchmark section. This choice does not affect the benchmark, but only shifts the value of the point defect energy.
Figure 2(b) presents a comparison between uMLIP and DFT defect energies for 129 point defects across 32 chemical systems, including AlNi, CaSn, CuAu, NaPb, NaAg, etc. Calculation details are listed in the Methods section and the complete list of materials is provided in Supplementary Table 2. Interestingly, the uMLIP calculated defect energies are mostly underestimated, similar to the trend observed in the surface energies in Figure 2(a).
Solid-Solution Energetics
Thermodynamic modeling of solubility in solid state systems such as metallic alloys [31] and high-entropy ceramics [32] requires accurate energetics to capture the dependence of the energy on substitutional arrangements [33, 34]. This dependence, relative to , determines the temperature scale at which mixing or order/disorder transitions occur [35]. In this section, we use the mixing of Ca and Mg in the CaMgO rocksalt as an example to examine the ability of uMLIPs to predict the behavior of the solid solution. The end members of the system, MgO and CaO are both rocksalt and the phase diagram has been previously studied both experimentally [36] and computationally [37].
We explore different possible Ca-Mg cation arrangements in the rocksalt at various CaO-MgO ratios and evaluate the corresponding energies (see Methods). These OK formation energies are shown in Figure 2(c), where each point corresponds to the energy of a specific Ca-Mg cation arrangement at a given Ca fraction. The predicted formation energies from all uMLIPs are positive, consistent with the low T immiscibility of CaO and MgO [37]. We observe a systematic underprediction of the mixing energies and the energy difference between the uMLIPs and DFT at a specific Ca fraction. Among the uMLIPs, CHGNet’s predictions closely approximate those of DFT, followed by those of M3GNet and MACE. We note that an underprediction of the formation energy would lead to an underestimation of the solubilization temperature in phase diagram calculations and an over-estimation of the solubility limits at a given temperature [35].
Phonon Properties
Accurate descriptions of vibrational properties and phonon spectra are crucial for understanding a wide range of material characteristics, such as thermodynamic [38], mechanical [39], and thermal transport properties [40]. Predicting phonon frequencies represents a stringent test of the MLIPs’ ability to capture the subtle energy and force landscape around equilibrium configurations. In this section, we benchmark the uMLIPs’ performance on phonon frequencies by applying the finite displacement method [41] to calculate harmonic phonons.
Figure 3(a) shows an example of uMLIP and DFT calculated phonon frequency on (Materials Project ID mp-1784), where the solid red lines represent DFT phonon frequencies and the dashed lines show uMLIP phonon frequencies. While the overall shapes of the phonon bands are generally well-captured by the uMLIPs, a systematic reduction of the vibrational frequencies (i.e., the frequency magnitude difference of the branches at a given wave vector) is observed across all models compared to the DFT reference, particularly for the optical modes predicted by M3GNet (blue dashed line). The reduced vibrational frequency is an indication that the forces described by uMLIPs are systematically lower than the DFT values.

Figure 3. Softened phonon vibration modes in uMLIPs. (a) the phonon dispersion relation and density of states (DOS) of CsF(mp-1784) calculated with DFT and uMLIPs. Systematic underpredictions of phonon vibration frequencies are observed with all uMLIPs. (b) Distribution of ratios between uMLIP maximum frequency to DFT maximum frequency for 229 different compounds.
To quantify this softening behavior, we evaluate the ratio between the maximum phonon frequencies predicted by the uMLIPs and the corresponding DFT value for a diverse set of 229 materials (see Supplementary Table 3) from the PhononDB [42, 43]. The distribution of these ratios is shown in Figure 3(b), which demonstrates that the majority () of materials are found to be softened in uMLIPs compared to DFT, with the phonon frequency under-predicted. The result suggests that both the energy and force described by uMLIPs are softened for almost all chemical systems.
Ion Migration Barriers
The migration barrier for an ion to move through a crystal structure forms the basis for evaluating the diffusion constants in a material and as such is critical to understand its functional or processing behavior. An accurate description of ion mobility is directly relevant in various applications, such as lithium-ion conductors for battery technologies [44], and proton conductors for fuel cells [45], etc. Because the migration barrier is determined by the extrapolation of the energy along the path between two stable sites, it is by definition also a poorly sampled configuration when uMLIPs are only fitted to local equilibrium configurations.
We employ uMLIPs and DFT to conduct a comprehensive assessment of 470 Mg-ion migration pathways in 110 distinct structures including oxides, halides, and sulfides [46]. For all ion migration paths, we generate an initial guess of the minimum energy pathway based on the DFT charge density [47] and subsequently evaluate it with the approximate nudged elastic band (Approx-NEB) method [48] (see method section). ApproxNEB is different from regular NEB in that it does not perform a relaxation of the pathway but solely evaluates the energy along the predefined trajectory [48]. Figure 4(a) presents the energy landscape of one Mg ion migration path in VO(SO) (mp-28207), where the energies of each image have been referenced against the energy of image 0. The migration barrier is defined as the energy difference between the highest and lowest energies along the reaction coordinate. While all three uMLIPs are shown to capture the overall shape of the DFT energy along the path, we observe systematic energy under-predictions of uMLIPs resulting in under-predictions of migration barriers. MACE achieves the best performance with a 0.30 eV MAE against DFT, followed by CHGNet (0.39 eV) and M3GNet (0.49 eV). The parity plot of uMLIP barriers vs. DFT barriers is provided in Supplementary Figure 1 and shows that the majority of uMLIP barriers are under-predicted, similar to the result of the surface and defect benchmarks. Figure 4(b) presents the distribution of the energy barrier difference between uMLIPs and DFT, from which we observe that all three uMLIPs show negative shifts in barrier predictions.

Figure 4. Underpredicted ion migration barriers in DFT and uMLIPs. (a) An example of a Mg-ion migration path in (mp-28207) with 6 intermediate images. The migration barrier is defined as the energy difference between the highest and lowest image. (b) The distribution of 477 energy barrier differences between uMLIPs and DFT, showing uMLIPs’ tendency to underestimate the ion migration barriers.
PES softening scale for high-energy states
By definition, a machine learning model with only random errors should have its prediction error distribution centered at 0. However, all three uMLIPs are shown to not satisfy such criterion in our OOD benchmarks. The negative distribution shifts in Figure 3(b) and Figure 4(b) indicate the existence of systematic prediction errors in the uMLIPs. These systematic underpredictions of energies and forces act as a softening effect of the PES, creating systematic errors in the calculation of important material quantities. To quantify the magnitude of softening observed in our benchmark tasks, we define a softening scale parameter, which is calculated as the linear fitted slope of uMLIPs vs. DFT forces in a material. As an example, Figure 5(a) shows an exemplary parity plot of uMLIPs vs. DFT forces from sampled high-energy OOD atomic configurations of (Materials Project ID mp-1020015). These OOD atomic
Figure 5. The PES softening scale from shifted force predictions. (a) uMLIP forces vs. DFT forces in high-energy states, sampled from high-temperature MDs of (mp-1020015). Systematic softening of PES is indicated by the tilted distribution of forces from the diagonal. The softening scale is defined as the slope of the distribution, where softening is indicated by slope < 1. cMAE stands for corrected mean absolute error, which is the MAE if the softening scale is corrected to 1, equivalent to having the force distribution rotated back to diagonal. (b) Distribution of softening scales of 1000 compounds sampled from the WBM dataset, showing the PES softening behavior is universal across various chemical systems.
configurations are sampled away from the energy minimum in the PES, by applying high-temperature molecular dynamics (MD) simulations (see method section). The corresponding forces of each sampled state are subsequently evaluated using static calculations with uMLIPs and DFT.
The systematic PES softening effect shows up in Figure 5(a) by the clockwise tilting of the distribution away from the diagonal. The slope of this distribution, extractable by linear regression, can be defined to be the PES softening scale. In Figure 5(a), we provide the fitted slopes and force MAEs of the three uMLIPs. When the softening scale is 1, the MLIP’s force distribution aligns with the diagonal, indicating that the curvature of the MLIP-PES systematically agrees with DFT with only random errors present. A softening scale smaller than 1.00, which is observed for all the benchmark tasks, indicates a systematic underprediction of energy and forces that leads to an overall smoother PES curvature as illustrated in Figure 1.
To investigate how broadly across chemistry the PES softening occurs, we collected 1000 different compounds from the WBM materials dataset by Wang et al. [49], which was generated by elemental substitution of Materials Project compounds and therefore contains only crystalline structures that are not included in the pre-training dataset of the three uMLIPs. For each of these compounds, 10 high-energy states away from the PES energy minimum are sampled with a 1000K MD simulation, and the softening scale is extracted from a linear fit with uMLIP and DFT forces. Figure 5(b) presents the distribution of the PES softening scale for these 1000 WBM compounds, and shows that for the majority (> 90%) of the compounds, the softening scale is smaller than 1 for all 3 uMLIPs we have tested. This result indicates the systematic softening behavior is universal across all chemical systems in current uMLIP models.
Data-efficient fine-tuning
The PES softening issue appears as a tilted distribution of forces in the parity plot as shown in Figure 5(a). Intuitively, one can rotate the distribution back to the diagonal to reset the softening scale to 1 thereby reducing the prediction error. In this scenario, we define cMAE as the linearly corrected mean absolute error if the uMLIP force distributions were rotated back to align with the diagonal. As shown in Figure 5(a), the cMAEs are considerably reduced from the original MAE from 0.220/0.190/0.176 eV/Å to 0.184, 0.162, 0.155 eV/Å for M3GNet/CHGNet/MACE, respectively. This observation suggests that a considerable fraction of force errors from uMLIP are likely to be systematic and can be easily corrected to reduce force errors.
Mathematically, rotating the force distribution is equivalent to multiplying every force value by a scalar, which can be realized by multiplying the MLIP energy by a scalar term
It is noted that the above formulation is equivalent to fine-tuning a MLIP by fixing all model weights except a scalar linear layer, which essentially modifies only the scalar parameter c in (4). Since only a scalar parameter requires modification, only one single label (1 force component) is needed for the training. In the left part of Figure 6(a), we show the result when pre-trained CHGNet is fine-tuned with an added hypothetical scalar linear layer (see Methods), trained on only a single high-energy configuration of LiBPO (mp-1020015). The test forces, which originate from the same set of atomic arrangements in Figure 5(a), are labeled in orange and the training forces from the single additional configuration are labeled in red. The linear corrected CHGNet exhibits a softening scale of 0.965 and a force MAE of 0.166 eV/Å, improved from 0.859 and 0.190 eV/Å in the pre-trained CHGNet as shown in Figure 5. The estimated cMAE is 0.162 eV/Å when the softening is corrected to 1, which is close to the force MAE of 0.166 eV/Å that is achieved by fine-tuning the scalar linear layer. Hence, a linear correction with one high-energy OOD configuration indeed operates as a rotation of the force distribution back to the diagonal, substantially eliminating the systematic softening error and considerably reducing the force MAE.
Figure 6. Linear correction with one label solves PES softening. (a) Parity plots of fine-tuned CHGNet predictions on LiBPO (mp-1020015), with the training force labels plotted in red and pre-excluded test force labels plotted in orange. Left: fine-tuned CHGNet with a linear correction and a single DFT label solves the softening issue and greatly reduces force MAE from 0.190 eV/Å to 0.166 eV/Å. Right: a more realistic fine-tuning example that optimizes all model parameters with 10 DFT labels, which further decreases the force MAE. (b) Distribution of force MAEs and linear corrected MAEs (cMAEs) for 1000 WBM compounds, showing uMLIP force errors can be greatly reduced by fine-tuning with a single data point. Quartiles are labeled by dashed lines.
We propose that the cMAE derived from the linear correction serves as an approximate lower bound for the expected error reduction from fine-tuning uMLIPs. While a linear correction with one label stands as an extreme case, a typical fine-tuning process involves hundreds and thousands of structure labels that can further reduce the MAE of the model. We tested fine-tuning the pretrained CHGNet by optimizing all model parameters with 10 training structures, and the resulting force parity plot is shown on the right of Figure 6(a). Compared to the linear correction with only one configuration, the right panel in Figure 6(a) shows that a very small dataset of 10 training structures further reduces the MAE to 0.125 eV/Å. By statistically evaluating the distribution of force MAEs and cMAEs for the 1000 WBM structures, we present their fine-tuning error-reduction lower-bounds in Figure 6(b). From the observed distribution, considerable error reduction (~ 15%) can be adequately achieved with a simple linear correction.
These results suggest a theoretical explanation for the commonly observed data-efficient performance boost that is achievable by fine-tuning foundational uMLIPs compared to training randomly initialized MLIPs. The data efficiency arises from the observation that a significant part of the MAEs in pre-trained uMLIPs are highly systematic, which can be efficiently amended by optimizing a fraction of model parameters with a small amount of data. In Supplementary Fig. 2, we present a comparison between the force error in the fine-tuned CHGNet models to those trained from scratch. The result demonstrates that the data-efficient error reduction is a unique advantage that only applies to foundational uMLIPs that have been well-pre-trained, and randomly initialized MLIPs will require substantially more data points to converge to the same performance.
Discussion
The design and discovery of novel materials raises the need for advanced simulation tools capable of efficiently and accurately describing the intricate details of atomic interactions. MLIPs offer a potential solution to bridge the gap between quantum mechanical accuracy and affordable computation cost by learning and emulating complex atomic interactions. Recent work on pre-training foundational MLIPs with comprehensive material datasets has opened up the possibility for out-of-box use of robust universal interatomic potentials [7, 8, 10, 12, 13].
Unlike DFT, MLIPs cannot by default be expected to perform well in a configurational space where they have not been trained. We therefore benchmark the performance of three uMLIPs for multiple OOD modeling tasks including surfaces, defects, solid-solution energetics, phonon vibration modes, ion migration barriers, and more general high energy states. These states are under-represented in the widely-used pre-training dataset [7, 8, 17] that only consists of bulk crystalline materials. For the properties tested in this work, we observe a universal softening of the PES, characterized by the uMLIPs’ under-prediction of energies and forces. The uMLIP datasets are primarily drawn from Materials Project [17] ionic relaxation trajectories and are therefore largely distributed around the energy minima of the PES. Consequently, the uMLIPs are exposed to a limited range of atomic configurations and force gradients, leading to difficulties in accurately capturing the energy landscapes and steep gradients associated with OOD states and processes like ion migrations and phase transformations.
We found similar signs of softening in the published literature, though less attention was dedicated to an in-depth examination of the softening issue. Pandey et al. [50] and Bartel [51] presented an extrapolation issue arising from a distribution shift when training a CGCNN [18] energy predictor with ICSD data [52]. The CGCNN model trained with only experimental stable materials experienced a six-fold increased prediction MAE when applied to hypothetical crystal structures in the Materials Project [17]. Furthermore, the Google DeepMind’s GNoME uMLIP exhibited pronounced softening tendencies when trained on the M3GNet dataset [7], as evidenced in the Supplementary Information of Ref. [12], similar to our observation in Figure 5(a). After being trained on the expanded dataset of 89 million structures, the softening issue in GNoME was shown to be mitigated but not fully eliminated, which is shown in the Supplementary fig:S34–S37 from Ref. [12]. These examples underscore the universality of the PES softening issue across various models and datasets, highlighting the importance of the systematic benchmark and analysis undertaken by our study to address this challenge.
The observed limitations of current uMLIPs raise questions about the effect of model size and expressive capacity on their ability to capture the intricate details of the PES [53]. The MACE model with 4.69 Million parameters, which is around 11 times the size of the CHGNet and 21 times the size of M3GNet, shows improved MAE and decreased softening compared to the smaller uMLIPs. The better performance of larger uMLIPs aligns with the previous study by Frey et al. [54] on the scaling of model performance as a function of MLIP capacity. The observed relationship between model capacity and performance prompts further inquiry into the extent to which the parameter size of current uMLIPs influences the PES softening issue, and whether the softening can be minimized by scaling to a larger, yet reasonable model size without expanding the dataset. In Supplementary Figure 4 and 5, we show the distribution of softening scale and force MAEs for two additional uMLIPs: CHGNet-matgl and M3GNet-matgl, which were also pre-trained using Materials Project database. The CHGNet-matgl with increased model size and M3GNet-matgl with enhanced sampling [55] demonstrate decreased softening effect and improved force predictions. While the scope of current work does not explicitly investigate the effect of model size, further studies could provide a better explanation of the number of model parameters needed to describe a universal potential energy surface.
Fortunately, we demonstrate the softening issue can be effectively resolved by including a minimal amount of high-energy OOD training points in fine-tuning. Our result not only provides a guideline to avoid softening issues when applying uMLIPs to atomic modeling, but more importantly, derives an explanation for the frequently observed data-efficient fine-tuning of foundational MLIPs. Our result suggests that a significant portion of the MAE in uMLIPs is highly systematic and therefore can be efficiently corrected by a small amount of data. In addition to the robustness of uMLIPs that has been acknowledged as an advantage obtained from pre-training [8, 12], our study elucidates another benefit of fine-tuning foundational MLIPs – the data-efficient systematic error correction that is unavailable for training a randomly initialized MLIP. Our study serves as a guideline for researchers attempting to fit interatomic potentials for their systems of interest.
In summary, our work presents an in-depth analysis of the softening effect of uMLIPs observed in a series of OOD materials benchmarks, from which we provide guidelines for the fine-tuning effects of uMLIPs. With the observed limitation of current uMLIPs, we advocate the need for an improved next-generation dataset for training foundational atomic models, and more investigation in the role of model complexity. Despite significant efforts dedicated to model design and training strategies, less emphasis has been placed on constructing comprehensive and well-curated open-source materials datasets. The current foundational models still rely on datasets that were not originally generated for machine learning purposes. Apart from diversifying the chemical space, our findings highlight the importance of ensuring a comprehensive sampling of the PES in generating a reliable MLIP dataset. We believe a next-generation foundational atomic dataset with improved sampling will be pivotal for the development of MLIP and atomistic simulations.
Acknowledgments
This work was funded by the U.S. Department of Energy, Office of Science, Office of Basic Energy Sciences, Materials Sciences and Engineering Division under Contract No. DE-AC0205CH11231 (Materials Project program KC23MP). The work was also supported by the computational resources provided by the Extreme Science and Engineering Discovery Environment (XSEDE), supported by National Science Foundation grant number ACI1053575; the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility located at Lawrence Berkeley National Laboratory; and the Swift Cluster resource provided by the National Renewable Energy Laboratory (NREL). The authors would also like to thank Tsz Wai Ko and Yuanqi Du for helpful discussions.
Methods
uMLIP versions
The table below shows the details and versions of the uMLIPs tested.
Table 1. uMLIP Model Specifications
Materials Modeling Tasks
For surface energy calculations, stoichiometric and symmetric slabs are generated with up to a maximum Miller index of 2 in three directions. Minimum slab thickness of 10 Å and minimum vacuum length of 10 Å are used for DFT to ensure convergence of surface energy [56]. When relaxing the slab, in-plane lattice vectors are fixed to their bulk value. The ionic relaxations are converged to a maximum interatomic force criteria of 0.05 eV/Å for all uMLIPs.
For defect energy calculations, defects in elemental phases as well as binary metallic compounds are considered. The defect structures are fully relaxed and refreferenced to the bulk energy. The off-stoichiometric defect energies (ex: vacancy defect) are referenced to the chemical potential of the pure elemental phase, instead of any chemical potential corresponding to multi-phase equilibria in the phase diagram. This is done deliberately to avoid additional errors associated with calculating the phase diagram using the uMLIPs. For all uMLIPs, the ionic relaxations are converged while the a maximum interatomic force is 0.05 eV/Å.
For solid-solution calculation in , we randomly select different Ca-Mg orderings (up to 52 number of configuration) at each Ca concentration and evaluate the energy of the configuration with ionic relaxation with DFT or uMLIPs.
For phonon calculations, we use the phonopy workflow as implemented in atomate2 [57] with relaxation convergence and supercell settings identical to those used in Batatia et al. [10]. The DFT referenced data are taken from the PhononDB [42, 43]. We restrict benchmarking materials without magnetism and U-corrections. Moreover, we removed the non-analytic corrections (NAC) to PBE phonons which are derived from the Born effective charges as these are unavailable from uMLIPs which have no concept of electronic structure. In practice, a future hybrid uMLIP-DFT workflow could perform a single DFT static at the uMLIP relaxed structure to obtain Born charges and post-hoc apply non-analytic corrections to the uMLIP phonon spectrum. However, such a hybrid workflow while necessary in practice, would not affect the results of this benchmark concerned specifically with the ML-obtainable parts of the spectrum.
The ion migration barrier DFT data are collected from the work of Rutt et al. [46], in which the ApproxNEB algorithm [48] was used to evaluate ion migration barriers. The key difference between ApproxNEB with regular NEB [58] is that ApproxNEB relaxes each image along the migration path independently, while NEB relaxes the migration path collectively. In the ApproxNEB method, an initial guess of the ion migration path is interpolated based on the charge density of the host structure. The energies associated with suggested image structures are calculated by the constrained relaxation that fixes the moving ion and lattice vectors. The ApproxNEB method was shown to provide a comparable barrier within 20 meV error of NEB and reduce the computational time significantly for materials where the path is not too complex[48].
To sample the high-energy states, we randomly select 1000 structures from the WBM dataset [49]. For each structure selected, a 20 ps,1000K molecular dynamics simulation is performed under constant number of particles, volume, and temperature (NVT) ensemble with the pre-trained CHGNet, and 10 structures are subsequently sampled from each MD trajectory [55]. +3% strain and a −3% strain are applied along three lattice dimensions for 4 out of the 10 structures to sample strained configurations. All the force MAEs and fine-tuning are calculated with the three-dimensional force components rather than the absolute magnitude of forces.
Fine-tuning
For the fine-tuning of CHGNet uMLIP, the models are trained with energy, force, and stress labels with 0.1-100-0.1 loss fractions under the mean squared error (MSE) loss criterion. The structures and labels are taken from a DFT ab-initio MD trajectory data of (mp-1020015), where 100 structures are reserved for the test set, as shown by the orange points in Figure 6(a). The train-validation ratio is set to be 9:1. As a result, 9 out of the 10 training structures in the right panel of Figure 6(a) are actually used for gradient back-propagations. The Adam optimizer [59] is used with a learning rate of that cosine decays to 1e-5 in 100 epochs. The model checkpoint of best validation force MAE is collected for test set predictions. For the model trained with only 1 structure, the last-epoch checkpoint is used instead.
The linear correction of CHGNet is realized by adding a hypothetical scalar linear before the energy prediction. The weight of the scalar linear layer is initialized to be 1, therefore not influencing the energy prediction before being optimized. During the linear correction, all CHGNet model parameters are frozen except for the added scalar linear layer.
DFT calculations
DFT calculations were performed with the Vienna ab initio simulation package (VASP) using the projector-augmented wave method [60, 61]. All calculation settings are generated using pymatgen MPRelaxSet to ensure all DFT results are compatible with Materials Project DFT calculations [62]. All the calculations were converged to at least eV in total energy for electronic steps and 0.02 eV/Å in interatomic forces for ionic steps.
Supplementary Information
The supplementary material is available at URL-When-Published.
References
- [1]H. Wang, T. Fu, Y. Du, W. Gao, K. Huang, Z. Liu, P. Chandak, S. Liu, P. V. Katwyk, A. Deac, A. Anankumar, K. Bergen, C. P. Gomes, S. Ho, P. Kohli, J. Lasenby, J. Leskovec, T.-Y. Liu, A. Manrai, D. Marks, B. Ramsun- dar, L. Song, J. Sun, J. Tang, P. Veličković, M. Welling, L. Zhang, C. W. Coley, Y. Bengio, and M. Zitnik, Scientific discovery in the age of artificial intelligence, Nature 620, 47–60 (2023).
- [2]H. Wang, L. Zhang, J. Han, and W. E, DeepMD-kit: A deep learning package for many-body potential energy representation and molecular dynamics, Computer Physics Communications 228, 178 (2018).DOI
- [3]S. Batzner, A. Musaelian, L. Sun, M. Geiger, J. P. Mailoa, M. Kornbluth, N. Molinari, T. E. Smidt, and B. Kozinsky, E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials, Nature Communications 13, 2453 (2022).DOI
- [4]Z. Guo, D. Lu, Y. Yan, S. Hu, R. Liu, G. Tan, N. Sun, W. Jiang, L. Liu, Y. Chen, L. Zhang, M. Chen, H. Wang, and W. Jia, Extending the limit of molecular dynamics with ab initio accuracy to 10 billion atoms, Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming , 205–218 (2022).DOI
- [5]A. Musaelian, A. Johansson, S. Batzner, and B. Kozinsky, Scaling the leading accuracy of deep equivariant models to biomolecular simulations of realistic size, SC23: International Conference for High Performance Computing, Networking, Storage and Analysis 00, 1–12 (2023).arxiv.org/abs/2304.10061
- [6]J. Gasteiger, J. Groß, and S. Günnemann, Directional message passing for molecular graphs, ICLR 10.48550/arxiv.2003.03123 (2020).arxiv.org/abs/2003.03123
- [7]C. Chen and S. P. Ong, A universal graph deep learning interatomic potential for the periodic table, Nature Computational Science 2, 718–728 (2022).DOI
- [8]B. Deng, P. Zhong, K. Jun, J. Riebesell, K. Han, C. J. Bartel, and G. Ceder, Chgnet as a pretrained universal neural network potential for charge-informed atomistic modelling, Nature Machine Intelligence 5, 1031 (2023).DOI
- [9]K. Choudhary, B. DeCost, L. Major, K. Butler, J. Thiyagalingam, and F. Tavazza, Unified graph neural network force-field for the periodic table: solid state applications, Digital Discovery 2, 346–355 (2023).
- [10]I. Batatia, P. Benner, Y. Chiang, A. M. Elena, D. P. Kovács, J. Riebesell, X. R. Advincula, M. Asta, W. J. Baldwin, N. Bernstein, et al., A foundation model for atomistic materials chemistry, arXiv preprint arXiv:2401.00096 (2023).arxiv.org/abs/2401.00096
- [11]S. Takamoto, C. Shinagawa, D. Motoki, K. Nakago, W. Li, I. Kurata, T. Watanabe, Y. Yayama, H. Iriguchi, Y. Asano, T. Onodera, T. Ishii, T. Kudo, H. Ono, R. Sawada, R. Ishitani, M. Ong, T. Yamaguchi, T. Kataoka, A. Hayashi, N. Charoenphakdee, and T. Ibuka, Towards universal neural network potential for material discovery applicable to arbitrary combination of 45 elements, Nature Communications 13, 2991 (2022).DOI
- [12]A. Merchant, S. Batzner, B. S. Schoenholz, M. Aykol, G. Cheon, and E. D. Cubuk, Scaling deep learning for materials discovery, Nature , 1–6 (2023).DOI
- [13]D. Zhang, H. Bi, F.-Z. Dai, W. Jiang, X. Liu, L. Zhang, and H. Wang, Pretraining of attention-based deep learning potential model for molecular simulation, npj Computational Materials 10, 94 (2024).DOI
- [14]J. Riebesell, R. E. A. Goodall, P. Benner, Y. Chiang, B. Deng, A. A. Lee, A. Jain, and K. A. Persson, Matbench discovery – an evaluation framework for machine learning crystal stability prediction, arXiv preprint arXiv:2308.14920 (2023).
- [15]B. Focassio, L. P. M. Freitas, and G. R. Schleder, Performance assessment of universal machine learning interatomic potentials: Challenges and directions for materials’ surfaces, arXiv preprint arXiv:2403.04217 (2024).DOI
- [16]H. Yu, M. Giantomassi, G. Maternizani, and G.-M. Rignanese, Systematic assessment of various universal machine-learning interatomic potentials, arXiv preprint arXiv:2403.05729 (2024).arxiv.org/abs/2403.05729
- [17]A. Jain, S. P. Ong, G. Hautier, W. Chen, W. D. Richards, S. Dacek, S. Cholia, D. Gunter, D. Skinner, G. Ceder, and K. A. Persson, Commentary: The materials project: A materials genome approach to accelerating materials innovation, APL Materials 1, 011002 (2013).DOI
- [18]T. Xie and J. C. Grossman, Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties, Physical Review Letters 120, 145301 (2018).DOI
- [19]S. Luo, T. Chen, and A. S. Krishnapriyan, Enabling efficient equivariant operations in the fourier basis via gaunt tensor products, arXiv preprint arXiv:2401.10216 (2024).arxiv.org/abs/2401.10216
- [20]B. Cheng, Cartesian atomic cluster expansion for machine learning interatomic potentials, arXiv preprint arXiv:2402.07472 (2024).arxiv.org/abs/2402.07472
- [21]T. W. Ko, J. A. Finkler, S. Goedecker, and J. Behler, Accurate fourth-generation machine learning potentials by electrostatic embedding, Journal of Chemical Theory and Computation 19, 3567–3579 (2023).arxiv.org/abs/2305.10692
- [22]L. Chanussot, A. Das, S. Goyal, T. Lavril, M. Shuaibi, M. Riviere, K. Tran, J. Heras-Domingo, C. Ho, W. Hu, A. Palizhati, A. Sriram, B. Wood, J. Yoon, D. Parikh, C. L. Zitnick, and Z. Ulissi, Open catalyst 2020 (oc20) dataset and community challenges, ACS Catalysis 11, 6059–6072 (2021).
- [23]I. Obot, D. Macdonald, and Z. Gasem, Density functional theory (dft) as a powerful tool for designing new organic corrosion inhibitors. Part 1: An overview, Corrosion Science 99, 1–30 (2015).
- [24]Y. Han, K. C. Lai, A. Lii-Rosales, M. C. Tringides, J. W. Evans, and P. A. Thiel, Surface energies, adhesion energies, and exfoliation energies relevant to copper-graphene and copper-graphite systems, Surface Science 685, 48–58 (2019).DOI
- [25]W. Sun, D. A. Kitchaev, D. Kramer, and G. Ceder, Non-equilibrium crystallization pathways of manganese oxides in aqueous solution, Nature Communications 10, 573 (2019).DOI
- [26]K. A. Fichthorn and M. Scheffler, Island nucleation in thin-film epitaxy: A first-principles investigation, Physical Review Letters 84, 5371–5374 (2000).DOI
- [27]V. Gurylev and T. P. Perng, Defect engineering of zno: Review on oxygen and zinc vacancies, Journal of the European Ceramic Society 41, 4977–4996 (2021).DOI
- [28]D. Broberg, K. Bystrom, S. Srivastava, D. Dahliah, B. A. D. Williamson, L. Weston, D. O. Scanlon, G.-M. Rignanese, S. Dwaraknath, J. Varley, K. A. Persson, M. Asta, and G. Hautier, High-throughput calculations of charged point defect properties with semi-local density functional theory—performance benchmarks for materials screening applications, npj Computational Materials 9, 72 (2023).DOI
- [29]M. G. A. Ahangari, A. H. Mashhadzadeh, M. Fathalian, A. Dadrasi, Y. Rostamiyan, and A. Mallahi, Effect of various defects on mechanical and electronic properties of zinc-oxide graphene-like structure: A dft study Vac-\nuum 165, 26–34 (2019).
- [30]K. Kang and G. Ceder, Factors that affect li mobility in layered lithium transition metal oxides, Physical Review B 74, 094105 (2006).DOI
- [31]A. van de Walle, Multicomponent multisublattice alloys, nonconfigurational entropy and other additions to the Alloy Theoretic Automated Toolkit, Calphad 33, 266 (2009).DOI
- [32]Cation-disordered rocksalt-type high-entropy cathodes for Li-ion batteries, Nature Materials 20, 214 (2021).
- [33]L. Barroso-Luque, P. Zhong, J. H. Yang, F. Xie, T. Chen, B. Ouyang, and G. Ceder, Cluster expansions of multicomponent ionic materials: Formalism and methodology, Physical Review B 106, 144202 (2022).
- [34]P. Zhong, F. Xie, L. Barroso-Luque, L. Huang, and G. Ceder, Modeling intercalation chemistry with multi-redox reactions by sparse lattice models in disordered rocksalt cathodes, PRX Energy 2, 043005 (2023).DOI
- [35]G. Ceder, A derivation of the ising model for the computation of phase diagrams, Computational Materials Science 1, 144–150 (1993).DOI
- [36]R. C. DOMAN, J. B. BARR, R. N. McNALLY, and A. M. ALPER, Phase equilibria in the system cao–mgo, Journal of the American Ceramic Society 46, 313–316 (1963).DOI
- [37]I.-H. Jung, S. A. Decterov, and A. D. Pelton, Critical thermodynamic evaluation and optimization of the cao–mgo–sio2 system, Journal of the European Ceramic Society 25, 313–333 (2005).DOI
- [38]A. v. d. Walle and G. Ceder, The effect of lattice vibrations on substitutional alloy thermodynamics, Reviews of Modern Physics 74, 11–45 (2002).DOI
- [39]A. Togo and I. Tanaka, First principles phonon calculations in materials science, Scripta Materialia 108, 1–5 (2015).arxiv.org/abs/1506.08498
- [40]S. Yue, B. Deng, Y. Liu, Y. Quan, R. Yang, and B. Liao, Phonon softening near topological phase transitions, Physical Review B 102, 235428 (2020).DOI
- [41]K. Parlinksi, Z. Q. Li, and Y. Kawazoe, First-Principles Determination of the Soft Mode in Cubic ZrO2, Physical Review Letters 78, 4063–4066 (1997).
- [42]A. Togo, First-principles Phonon Calculations with Phonopy and Phono3py, Journal of the Physical Society of Japan 92, 012001.
- [43]A. Togo, L. Chaput, T. Tadano, and I. Tanaka, Implementation strategies in phonopy and phono3py, Journal of Physics: Condensed Matter 35, 353001.
- [44]Z. Deng, B. Radhakrishnan, and S. P. Ong, Rational Composition Optimization of the Lithium-Rich Li3OCl1−x Br x Anti-Perovskite Superionic Conductors, Chemistry of Materials 27, 3749–3755 (2015).DOI
- [45]P. Du, Q. Chen, Z. Fan, P. Fan, G. Haibach, M. A. Gomez, and A. Braun, Cooperative origin of proton pair diffusivity in yttrium substituted barium zirconate, Communications Physics 3, 200 (2020).DOI
- [46]A. Rutt, J.-X. Shen, M. Horton, J. Kim, J. Lin, and K. A. Persson, Expanding the material search space for multivalent cathodes, ACS Applied Materials & Interfaces 14, 44367–44376 (2022).DOI
- [47]J.-X. Shen, M. Horton, and K. A. Persson, A charge-density-based general cation insertion algorithm for generating new Li-ion cathode materials, npj Computational Materials 6, 161 (2020).DOI
- [48]Z. Rong, D. Kitchaev, P. Canepa, W. Huang, and G. Ceder, An efficient algorithm for finding the minimum energy path for cation migration in ionic materials, The Journal of Chemical Physics 145, 074112 (2016).DOI
- [49]H.-C. Wang, S. Botti, and M. A. L. Marques, Predicting stable crystalline compounds using chemical similarity, npj Computational Materials 7, 12 (2021).DOI
- [50]S. Pandey, J. Qu, V. Stevanović, P. S. John, and P. Gorai, Predicting energy and stability of known and hypothetical crystals using graph neural network, Patterns 2, 100361 (2021).DOI
- [51]C. J. Bartel, Data-centric approach to improve machine learning models for inorganic materials, Patterns 2, 100382 (2021).DOI
- [52]A. Belsky, M. Hellenbrandt, V. L. Karen, and P. Luksch, New developments in the Inorganic Crystal Structure Database (ICSD): accessibility in support of materials research and design, Acta Crystallographica Section B 58, 364–369 (2002).pubmed.ncbi.nlm.nih.gov/12037357
- [53]T. W. Ko and S. P. Ong, Recent advances and outstanding challenges for machine learning interatomic potentials, Nature Computational Science 1, 1–3 (2023).DOI
- [54]N. C. Frey, R. Soklaski, S. Axelrod, S. Samsi, R. Gómez-Bombarelli, C. W. Coley, and V. Gadepally, Neural scaling of deep chemical models, Nature Machine Intelligence , 1–9 (2023).
- [55]J. Qi, T. W. Ko, B. C. Wood, T. A. Pham, and S. P. Ong, Robust training of machine learning interatomic potentials with dimensionality reduction and stratified sampling, npj Computational Materials 10, 43 (2024).arxiv.org/abs/2307.13710
- [56]W. W. Sun and G. Ceder, Efficient creation and convergence of surface slabs, Surface Science 617, 53–59 (2013).DOI
- [57]A. Ganose, J. Riebesell, J. George, J. Shen, A. S. Rosen, A. Ashok Naik, winner, M. Wen, rdguha1995, M. Kuner, G. Petretto, Z. Zhu, M. Horton, H. Sahasrabuddhe, A. Kaplan, J. Schmidt, C. Ertural, R. Kingsbury, M. McDermott, R. Goodall, A. Bonkowski, T. Purcell, D. Zügner, and J. Qi, atomate2 (2024).
- [58]G. Henkelman, B. P. Uberuaga, and H. Jónsson, A climbing image nudged elastic band method for finding saddle points and minimum energy paths, The Journal of Chemical Physics 113, 9901–9904 (2000).DOI
- [59]D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, arXiv 10.48550/arxiv.1412.6980 (2014).
- [60]G. Kresse and J. Furthmüller, Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set, Computational Materials Science 6, 15 (1996).DOI
- [61]G. Kresse and D. Joubert, From ultrasoft pseudopotentials to the projector augmented-wave method, Physical Review B 59, 1758 (1999).DOI
- [62]S. P. Ong, W. D. Richards, A. Jain, G. Hautier, M. K. Ch. ..., D. Gunter, V. L. Chevrier, K. A. Persson, and G. Ceder, Python materials genomics (pymatgen): A robust, open-source python library for materials analysis, Computational Materials Science 68, 314–319 (2013).DOI


