WorksPhysics and Astronomy
Learning Together: Towards foundational models for machine learning interatomic potentials with meta-learning
Abstract
The development of machine learning models has led to an abundance of datasets containing quantum mechanical (QM) calculations for molecular and material systems. However, traditional training methods for machine learning models are unable to leverage the plethora of data available as they require that each dataset be generated using the same QM method. Taking machine learning interatomic potentials (MLIPs) as an example, we show that meta-learning techniques, a recent advancement from the machine learning community, can be used to fit multiple levels of QM theory in the same training process. Meta-learning changes the training procedure to learn a representation that can be easily re-trained to new tasks with small amounts of data. We then demonstrate that meta-learning enables simultaneously training to multiple large organic molecule datasets. As a proof of concept, we examine the performance of a MLIP refit to a small drug-like molecule and show that pre-training potentials to multiple levels of theory with meta-learning improves performance. This difference in performance can be seen both in the reduced error and in the improved smoothness of the potential energy surface produced. We therefore show that meta-learning can utilize existing datasets with inconsistent QM levels of theory to produce models that are better at specializing to new datasets. This opens new routes for creating pre-trained, foundational models for interatomic potentials.
Introduction
Machine learning is fundamentally changing and expanding our capabilities for modeling chemical and materials systems [1, 2, 3, 4, 5, 6, 7, 8]. A growing array of properties have been successfully predicted with machine learning models from materials’ band gaps and formation energies to molecular energies and bond orders [9, 10, 11, 12]. The development of machine learning models for various applications has involved the creation of a large number of datasets containing quantum-mechanical calculations at different fidelities (levels of theory) [13, 14, 15, 16, 17, 18]. However, incorporating this multi-fidelity information into machine learning models remains challenging. In this work, we show that multiple datasets can be used to fit a single learning model, even if the datasets were calculated with many varying QM levels of theory. To overcome this challenge, we incorporate meta-learning techniques into the training process and subsequently demonstrate improvements in accuracy for multiple applications. The aim of meta-learning is to use a wide collection of data to train a machine learning model that can then be easily re-trained to specialized tasks and we demonstrate the applicability of the meta-learning method to MLIPs.
In the landscape of broader efforts to incorporate machine learning and molecular and material modelling, a particular attention has been paid to MLIPs [10, 19, 20, 21, 22, 23, 24]. Accurate atomistic simulations rely on interatomic potentials that closely recreate the interactions present between atoms and molecules [25, 26]. Recreating these interactions involves a trade-off between accuracy and computational cost, with quantum mechanical techniques offering highly accurate simulations whilst classical force fields are fast and capable of modelling much larger systems over long timescales [27, 28, 29]. Within the last decade, MLIPs have increasingly been seen as a method that could provide a model that is both fast and accurate [10, 19, 20]. However, the development of MLIPs that are transferable to unseen organic molecules requires datasets that cover a large fraction of chemical space. This requirement has lead to the production of numerous datasets [13, 14, 15, 16, 17, 18]. These datasets contain the quantum mechanical (QM) energies and forces of millions of structures spanning large regions of chemical space. However, the QM methods used to calculate the energies and forces vary considerably. As different QM methods result in different potential energy surfaces, this inconsistency in QM techniques limits the extent that datasets can used together to fit potentials.
Numerous organic molecule datasets have been created for training MLIPs [13, 14, 15, 16, 17, 18]. However, a consensus on the best QM techniques to employ to create these datasets has never been reached as a compromise between accuracy and computational cost must always be considered when performing QM calculations. This lack of consensus has led to a variety of different software, methods, basis sets and exchange-correlation functionals being used. For example, the QM7-x and ANI-1x datasets both contain energies and forces for millions of small organic molecules. However, QM7-x was calculated using the PBE0 exchange-correlation functional with many body dispersion whilst ANI-1x was calculated with the B97x functional and 6-31G* basis set [14, 16] and does not include dispersion effects. Therefore, these two datasets describe similar, but slightly different potential energy surfaces. If both datasets were joined together to train a potential then problems would likely arise as contradictory information is present. For example, identical structures at different levels of theory can have different energy and forces. Whilst datasets from different sources have been fit together without further refinement [30], this approach does not account for differences in the interactions described. Techniques exist in the machine learning literature to address the difference in the potential energy surface.
Previous work on fitting MLIPs to multiple datasets is limited. In Ref. 31, a transferable molecular potential was first trained to million density functional theory (DFT) training points before being refit, with frozen parameters, to 0.5 million CCSD(T)* energies. This technique, known as transfer learning has been used in several works [32, 33, 34, 35, 36]. The advantage of using transfer learning for training MLIPs is that it requires fewer calculations at a higher, and more expensive, level of theory. However, this kind of transfer learning technique, freezing neural network (NN) parameters, is limited to just two datasets. If we want to use multiple existing datasets, and expand the size and variety of training data, then new methods must be found.
Fortunately, this problem is being explored in a branch of machine learning research known as meta-learning [37, 38, 39, 40]. Meta-learning seeks to build a model that, although not specialized to any particular task, can be quickly re-trained to many new tasks - where a task is a specific learning problem. Furthermore, this retraining can be effective even if the amount of new data is limited [37, 39].
For transferable MLIPs, the concepts of tasks naturally lends itself to quantum mechanical datasets calculated with different methods. By using meta-learning techniques, we will show how information from multiple levels of theory can be incorporated together. We begin by investigating training data with multiple levels of theory for an individual aspirin molecule and for the QM9 dataset (which contains over 100,000 molecules in their equilibrium configuration). With these systems, the problems associated with naively combining datasets together are seen and the benefits of meta-learning are clearly observed in the test set errors. We then move on to combining several large molecule datasets to pre-train an MLIP. Combining large organic datasets to fit MLIPs has never previously been attempted. Subsets, chosen using active learning, of six existing datasets (ANI-1x, GEOM, QMugs, QM7-x, Transition-1x and the QM9 dataset from Ref. 13) were used to fit an adaptable potential using meta-learning – see Figure 1 for a visualization of the space the datasets cover [13, 14, 15, 16, 17, 18]. Figure 1 demonstrates the increase in chemical space possible when multiple datasets are combined together. The benefits of pre-training are then shown by retraining to the 3BPA molecule and testing various properties. These tests show that pre-training models using meta-learning produces a more accurate and smoother potential. The benefits of pre-training include enhanced accuracy and generalization capabilities in modelling intramolecular potentials.
Figure 1. A diverse collection of datasets, with varying levels of theory, molecule sizes, and energies, will be incorporated into a single meta-learned potential. The distributions of the number of atoms and energy of the structures contained in the datasets used for training a potential in this work are shown. The structures included contain only C,H,N,O. Energies are made comparable using linear scaling as detailed in § 2 C 3.
Training machine learning models to large amounts of data before re-training to a specific task is related to the concept of foundational models [41]. This concept has been used to create large language models, ie. GPT-4, which have been pre-trained to extremely large datasets before being fine-tuned to specific tasks, i.e. ChatGPT which is fine-tuned for conversational usage [42]. Creat
| Algorithm 1 Reptile |
| Initialize , the initial parameter vector |
| for iteration 1,2,3... do |
| Randomly sample a task T |
| Perform steps in task T, starting with parameters , resulting in parameters W |
| Update: |
| Return |
Figure 2. This work uses Reptile to build a potential that incorporates information from multiple molecular datasets, calculated at different levels of theory. This meta-learned potential adapts well to new tasks, and outperforms potentials that were trained only to the data for a single task.
ing foundational models allows a wide range of information to be encoded before specialisation. With meta-learning techniques, we can now pre-train interatomic potentials to numerous large datasets and this is a step towards foundational models for MLIPs – MLIPs that could be quickly re-trained to diverse molecular systems.
The number of QM datasets has grown rapidly over the last few years. However, a major bottleneck in exploiting this information has been the absence of methods that can effectively combine all of this information. In this work, we have overcome this limitation by exploiting techniques which enable the incorporation of datasets with different fidelities. Whilst we focus on MLIPs, these techniques are applicable to the wide range of predictive models that exist for material and molecular property prediction. By showing how meta-learning can be applied, we aim to encourage researchers to fully utilize the vast amount of existing data that the scientific community has already collected.
Methods
Meta-Learning Algorithm
Meta-learning is an area of machine learning concerned with improving the learning process to produce models that can easily adapt to new problems [37, 38, 39, 40]. A key component of meta-learning is the concept of different ‘tasks’. Tasks are datasets with similar properties but slight differences. For example, if we were interested in animal classification of a cat and a dog, a similar task might be to classify a lion and a bear. The task is not the same but we would expect fundamental similarities in the model needed to perform the classification. By using a meta-learning algorithm to learn multiple different tasks, less data will be required when a new learning problem is introduced.
The objective of meta-learning algorithms is to train a model that can generalize more easily to new data[38, 40]. We will use meta-learning to fit multiple different QM datasets with slightly different properties. To our knowledge, meta-learning for MLIPs has not been previously carried out, although it has been used in other areas of science [43, 44, 45]. The meta-learning algorithm we have chosen to fit multiple datasets for MLIPs is called Reptile [39]. Reptile works by repeatedly sampling a task (a dataset), performing a limited number of optimization steps on the task and then updating the weights of the machine learning model towards the new weights. Reptile was chosen over other meta-learning algorithms such as MAML [37] as Reptile is simpler to implement and therefore more likely to be adopted by the wider community. A comparison of methods such as MAML for interatomic potentials will therefore be left to future work.
Reptile is described in Algorithm 2 with a visual illustration also given. The algorithm works by separating the training data into distinct learning problems (tasks). An individual task is selected and multiple optimization steps are performed. The parameters of the model are then updated. A new task is then selected and the procedure is repeated multiple times. This moves the model to a region of parameter space where it can readily move between the different datasets present.
Throughout this work, the result is used as comparison point. This is because when the algorithm becomes equivalent to stochastic gradient descent on the expected loss over all the training tasks [39]. This is referred to as joint training in Ref. 39 At , the algorithm is not expected to account for differences in the QM theory but still uses all the information present from the datasets.
Interatomic Potential
In this work, we have used the NN architecture implemented in torchANI with the same structure as the ANI-1x model [10, 31]. However, the meta-learning techniques described are not specific to this form of model and there is no reason that they could not be applied to other machine learning models that employ similar iterative solvers. The hyperparameters used for the ANI potential are the same as those used for previous training to the ANI-1x and ANI-1ccx datasets, see Ref. 31 for more details.
Datasets
Aspirin
Aspirin structures were produced by molecular dynamic simulations at 300K, 600K and 900K. Density Functional based Tight Binding (DFTB) was used to perform the MD simulations and a total of 400 structures were created for each temperature. QM calculations of the energies and forces were then performed on these structures with three levels of theory: DFT with the B97x exchange-correlation function and 6-31G* basis set, DFT with Becke, 3-parameter, Lee–Yang–Parr (B3LYP) exchange-correlation functions and def2-TZVP basis set and Hartree-Fock with the def2-SVP basis set for 300K, 600K and 900K respectively. These datasets were used to pre-train a molecular potential. The pre-trained potential was then refit to a new dataset of MD configuration at the Møller–Plesset (MP2) level of theory with the def2-SVP basis set (a more accurate level of theory). The training dataset for refitting used 400 MD configurations sampled at 300K whilst the test set contained structures at 300K,600K and 900K. A batch size of 8 was used for training.
QM9
The QM9 dataset contains over 100,000 equilibrium structures for small organic molecules with up to 9 heavy atoms [46]. In Ref. 13, the QM9 dataset was recalculated with 76 different exchange-correlation functionals and 3 basis sets [13].
Multiple Organic Molecules
Seven separate datasets were chosen to fit a potential to organic molecule potential that could be easily re-trained to new data. The seven datasets used for meta-learning were chosen to cover both diverse regions of chemical space and multiples levels of theory – including the accurate recreation of dispersion effects. The chemical space covered included reactive paths and biologically and pharmacologically relevant structures. Whilst ANI1x does cover a large number of conformations for organic molecules, it has limitations. This is demonstrated by Figure 1 and Figure S1. Figure 1 demonstrates how the additional datasets increase the size of the molecules and range of energies included. The energy is calculated using linear fitting an then subtracted from each dataset. The minimum energy for each dataset is then shifted to zero. Whilst it is not covered in this work as we use the ANI potential, including larger molecules in datasets may be increasingly important for newer generations of interatomic potentials that include message passing and describe longer length scales [47, 48]. Figure S1 shows the distribution of uncertainty for the ANI-1x potential across the dataset space. Whilst ANI-1x dz, ANI-1x tz, GEOM and QMugs have similar probability distributions, QM7-x and Transition-1x contain larger uncertainties. Transition-1x contains reactive structures that are not contained in the original dataset and therefore higher uncertainties are expected. For QM7-x, there are also higher uncertainties and this may be due to the different sampling techniques used.
Figure S1. The distribution of uncertainties with the ANI-1x potential for 100,000 structures taken from the different datasets included in this work.
A property that is not shown in Table 1 is the software used for the DFT calculations. Even when the same level of theory is used, we can expect different software to give slightly different results. This will cause further discrepancies between the datasets as a variety of codes are employed. For example, although Transition-1x and ANI-1x are calculated at the same level of theory, Transition-1x is calculated with the ORCA program whilst ANI-1x is calculated with Gaussian [49, 50].
| Dataset | Unique Compounds | Total Conformers | Heavy Atoms Max | Conformer Generation | Method | Dispersion | Transition Paths |
| QM9 | 133,885 | 133,885 | 9 | None | 76 DFT Functionals | Yes | No |
| ANI-1x | ∼64,000 | 4,956,005 and 4,617,229 | 8 | Normal Mode Sampling, MD sampling, Torsional Sampling, Active Learning | B97x/6-31G* and B97x/def2-TZVPP | No | No |
| QMugs | 665,911 | 1,992,984 | 100 | Meta Dynamics with xTB | B97X-D/def2-SVP | Yes | No |
| GEOM | 437,724 | 32,657,609 | 91 | Meta Dynamics with CREST | r2scan-3c/mTZVPP | Yes | No |
| QM7-x | 41,537* | 4,195,237 | 7 | Normal-mode Sampling with DFTB | PBE0+MBD | Yes | No |
| Transition-1x | 10,073 Reactions | 9,644,740 | 7 | Nudged Elastic Band | B97x/6-31G(d) | No | Yes |
| ANI-1ccx | ∼64,000 | 489,571 | 8 | Active Learning from ANI-1x | CCSD(T)*/CBS | Yes | No |
Table 1. The properties of the datasets used in this work are shown in the table. (*including conformational isomers)
The individual description and justification for including each dataset used is as follows:
QM9 - This dataset contains a diverse range of 76 functionals and 3 basis sets for small equilibrium organic molecules [13].
ANI-1x - This is a large dataset of small (up to 8 heavy atoms) organic molecules generated with active learning methods [14].
QMugs - This dataset includes the largest molecules with up to 100 heavy atoms. It specializes in including drug-like molecules [17]
GEOM - This is the largest dataset and contains both large molecules and drug-like molecules [15].
QM7-x - This is also a large dataset of small (up to 7 heavy atoms) organic molecules but has dispersion accurately described with many-body dispersion [16]
Transition-1x - This datasets includes minimum energy paths for 12,000 reactions [18].
ANI-1ccx - This dataset contains coupled cluster level theory calculations for a subset of the ANI-1x dataset [14].
Other datasets considered for inclusion include SPICE, PubChemQC-PM6 and TensorMol [51, 52, 53]. However, with the existing datasets a sufficient representation of chemical space is covered. It is also worth noting that retraining to recreate the specific properties of the excluded datasets would also be quickly possible with the meta-learning potential.
Meta-learning Hyperparameter Optimization
There are three parameters in the Reptile algorithm. These control the number of steps () taken at each optimization step, how the parameters are updated () from the task’s individual NN parameters and the maximum number of epochs used for retraining. The number of epochs was investigated to see whether restricting the training improved accuracy by ensuring the potential remained close to the meta-learned potential or if longer retraining improved results. For a detailed discussion of the hyper parameters chosen when fitting to the seven separate datasets, see § 9.2. The value used throughout this work is whilst the value is changed depending on the problem. The maximum number of epochs used for retraining for the meta-learning algorithm with is restricted to 150 epochs.
Stages of Fitting for the Organic Molecule datasets
In the first iteration, 100,000 structures were taken randomly from the ANI-1x, QMugs, GEOM, QM7-x and Transition-1x datasets. For QM9, 10,000 structures were used for each level of theory. This is restricted as 276 levels of theory exist, and each theory level samples different structures in the QM9 dataset. After the first iteration, the highest error structures were added to the next iteration [54]. The cutoffs used for adding structures are described in SI 1.6. This process was repeated 3 times. A diagram of the process is show in Figure S3.
Figure S3. The stages of fitting in the meta-learning process. *The QM9 is the exception with only 10,000 structures.
Results
A Simple Case Study on Aspirin
As the initial test case we investigate the performance of meta-learning on a dataset containing a single aspirin molecule. Aspirin structures were produced by molecules dynamic simulations at 300K, 600K and 900K. The QM energies and forces were then calculated at three different levels of theory: two distinct DFT functionals, and Hartree-Fock. This created three different datasets, with each temperature corresponding to a different level of theory. These three datasets were used to pre-train a molecular potential to the energy and forces of 1,200 structures. The pre-trained potential was then refit to a new dataset of 400 MD configuration at the MP2 level of theory from the 300K simulation.
The change in the RMSE error for the forces is shown with the value of used in the meta-learning algorithm in Figure 3. The parameter controls the number of steps taken towards each dataset. As is increased the speed of the algorithm also increases and this is an additional consideration in choosing the optimal value. In the limit of the algorithm would correspond to iterative training to each dataset and then transfer learning to a new task. However, while this may work for small problems, this approach is impractical for large datasets.
Figure 3. The error as a function of the value of used in the meta-learning algorithm for an aspirin molecule. The potential is first pre-trained to multiple levels of theory before retraining to 400 structures at MP2 level of theory. When no pre-training is used the root mean squared error (RMSE) is 5.35 0.41 kcal/mol/Å.
Figure 3 shows that as the parameter is increased the error in the test set decreases with the minimum error at around . There is therefore an improvement in test set error in comparison to both no pre-training (5.35 0.41 kcal/mol / Å) and (3.38 0.16 kcal/mol / Å). Note that exactly corresponds to simultaneous training to all tasks. Therefore, when we attempt to combine multiple datasets at different levels of theory an improvement in performance can be seen when meta-learning is incorporated into the training process.
Meta-learning many levels of theory using QM9
Next, we move onto the QM9 dataset that contains multiple different small organic molecules in their equilibrium structures. The QM9 dataset has been calculated at 228 different levels of theory and therefore provides an ideal dataset for analysing meta-learning techniques. We can use this dataset to test whether meta-learning can develop a potential which can be refit to a new level of theory encountered for the QM9 dataset with less data. In order to do this, a subset of the QM9 dataset was used to train a potential to 10,000 molecules, 50 different exchange-correlation functionals and three different basis set. The potential was then refit to a new exchange-correlation functional, that had not been previously encountered, and the performance of this new model was assessed and compared to no pre-training and meta-learning.
The test set error for the meta-learning potential refit to a new level of theory in the QM9 dataset is shown in Figure 4. Pre-training the potential greatly improves the test set error for this case. In Figure S9 a comparison between meta-learning and is shown and we see that does not perform as well as . This is because it does not account for the discrepancy in the interaction present. These results show that even when the number of levels of theory is relatively large, at 150, and multiple molecules are present that meta-learning improves test set error over .
Figure 4. The error as a function of the number of structures averaged across 5 different DFT functions and 3 basis sets, for different fitting procedures for the QM9 dataset. The meta-learning algorithm fits the different levels of theory together with used. The standard deviation of the error bars corresponds to the variation across different, randomly selected, specialization tasks. Freezing the parameters before retraining was also attempted for 400 structures, however, no improvement in accuracy was observed.

Figure S9. The error as a function of the number of structures averaged across 5 different DFT functions and 3 basis sets, for different fitting procedures for the QM9 dataset. The meta-learning algorithm fits the different levels of theory together with used and is compared to . The standard deviation of the error bars corresponds to the variation across different levels of theory.
Making the most of scarce data at CCSD(T) level
We will now move to the datasets used to train transferable interatomic potentials. As a starting example, we will look at pre-training to the multiple levels of theory (B97x/ 6-31G* and B97x/ def2-TZVPP) contained in the ANI-1x dataset [14]. We will then retrain to the ANI-1ccx dataset [14]. Figure 5 shows the distribution in error when pre-training to multiple levels of theory with meta-learning and . The RMSE is 3.30 0.10 kcal/mol and 2.39 0.00 kcal/mol for and meta-learning respectively. Therefore, we can again see that meta-learning with a higher values improves results compared to . The comparative results for direct training to B97x/ 6-31G* and B97x/ def2-TZVPP and then transfer learning to CCSD(T) is 2.20 0.01 kcal/mol and 2.090.02 kcal/mol respectively. Therefore, in this case fitting to multiple datasets does not improve results over fitting to just one. This is in part because both datasets contain the same structures and cover the same chemical and configurational space. The potential trained to multiple organic datasets was also refit to the CCSD(T) dataset and the benefits of meta-learning over were also seen with errors of 2.89 and 3.32 respectively. However, this is notably higher than training to the ANI-1x dataset alone. The CCSD(T) dataset is a subset of the ANI-1x dataset and contains identical structures. For these cases, adding additional data in other areas of chemical space may not improve results.
Figure 5. The error distribution for the CCSD(T) specialization task after pre-training to ANI-1x with meta-learning () using and . The RMSE is 3.47 kcal/mol and 2.39 kcal/mol.
Training to multiple transferable organic molecule datasets
Numerous datasets have been created that contain quantum mechanical calculations for organic molecules. However, as these datasets use different levels of theory and software, combining the information from different datasets requires advanced training techniques. By using meta-learning, a pre-trained model was created that uses information from seven different datasets. This is the first instance, to our knowledge, of combining information from multiple organic molecule datasets in this manner.
We have already seen that meta-learning can improve results compared to when multiple datasets are used. We will now use the pre-trained model to explore the benefits of pre-training with meta-learning in comparison to no pre-training, and when retraining to a single molecular system. The pre-trained model was re-trained to the 3BPA dataset taken from Ref. 36 and various properties explored [56].
The first properties we will analyze are the energy and force RMSE errors. The force errors for a dataset taken from MD at 1200K is shown in Figure 6 with the energy and force learning curves for datasets at 300K, 600K and 1200K given in Figure S4. From these graphs, the improved performance of pre-training using the meta-learning approach (with three passes through the dataset) to both and no pre-training can be seen for energies and forces. Therefore, just by adapting the training scheme, with no change in the model architecture or the dataset itself, consistent improvements in accuracy can be seen with meta-learning. The importance of the training method used has previously been seen in Ref. 57. Here we see how it can improve performance for fitting multiple datasets together. In comparison to when the ANI-1x model is used for pre-training, meta-learning performs
Figure 6. The a) 3BPA molecule, b) the force error versus the number of structures used for the final training and c) hydrogen bond dissociation for 3BPA with the frequency of C–H bonds in the 3BPA training set. The results shown in part b) demonstrate the difference in performance with varying pre-training approaches – with meta-learning producing the lowest energy results. The bond dissociation calculated using DFT is 119.6kcal/mol and 111.2kcal/mol with the bond dissociation energies estimator from Ref. 55. All other atoms are fixed with the removal of the hydrogen atom. The frequency of the C–H bond is for the 3BPA dataset used for re-training and the maximum length of the bond is 1.27Å.

Figure S4. The learning curve for the energy RMSE for the 3BPA molecule at three temperatures.
slightly better at force errors but slightly worse for energy predictions. Given that the ANI-1x model is fit to the same level of theory as the 3BPA dataset, the performance of the meta-learning potential is encouraging. However, it is known that RMSE errors alone are not enough to verify the performance of a potential [36, 58]. We will therefore examine additional properties. The 3BPA molecule has three central dihedral angles which are illustrated in Figure 6. The energy scans along these dihedral angles are shown in Figure 7 with the model refit to the energies and forces of just 62 3BPA conformations. When no pre-training is used, the surface at significantly over-estimates the high energy point and lacks smoothness. A similar shape is seen for the potential. However, when meta-learning is used for pre-training the surface remains noticeably smoother with significantly less over prediction. When is used, multiple different potential energy surfaces are combined together in a nonphysical way which destroys the smoothness of the underlying potential. The error in the gradient of the 2D energy surface is shown in Figure 7 b and emphasizes this difference in smoothness. When meta-learning is used, the contradiction in the potential energy surface described is corrected resulting in a smoother model. When no pre-training or is used, an additional problem can occur for the high energy regions at failing to be recreated for the and scan respectively. In contrast, both the meta-learning pre-training model correctly recreate this behaviour. The results for ANI-1x pre-training are given in Figure S6.
Figure 7. a) Torsional energy scans (of and ) for potentials with different pre-training approaches. The angle is set to three different values. The potentials are re-trained to 62 structures from the 3BPA dataset. An ensemble of 8 models is used. Part b) shows the gradient of the torsional energy surface with respect to . The units are kcal/mol/degrees.

Figure S6. The dihedral angle energy scans for the 3BPA molecule at three different angles for a potential pre-trained to ANI-1x and the QM results.
One advantage of pre-training with multiple datasets over ANI-1x or QM7-x, is that reactive systems can be added that are not contained in ANI-1x. To test if this information has been effectively passed to the meta-learning potential, hydrogen bond dissociation for the 3BPA molecules was performed. There is no reactive information contained within the 3BPA training set and so this test relies entirely on the information contained in the pre-training.
Figure 6 shows the change in energy as a hydrogen molecule is removed from the 3BPA. The potential pre-trained with meta-learning recreates the smooth dissociation curve expected. In contrast, when no pre-training, or ANI-1x is used the curve lacks smoothness and has an additional barrier present. In Figure S7, the bond dissociation energy when just 31 structures are used for retraining. Even in this low data limit the smooth dissociation curves for the meta-learning potential remain. To demonstrate that this is not unique to 3BPA, the hydrogen bond dissociation for ethanol is shown in Figure S8. Again, fails to recreate the smooth curve expected whilst the meta-learning potential captures the correct shape.

Figure S7. The bond dissociation energy for the removal of a hydrogen atom when trained to 31 structures of 3BPA.
Figure S8. The bond dissociation energy when trained to the rMD17 structures for an ethanol molecule. Again, smooth dissociation is not seen when joint fitting is used. The potential was refit for a maximum of 150 epochs.
We have therefore shown how meta-learning can be used to combine multiple datasets and the resulting improvements in the error, torsion energy scans and bond dissociation. Joint-fitting can improve on no-pretraining. However, not accounting for the difference in QM level of theory causes a reduction in performance that can be seen in the test set errors, smoothness of the potential and performance in extrapolation regions.
Conclusion
The quantum mechanical properties of millions of molecular species and many materials systems have already been calculated and composed into extended datasets [13, 14, 15, 16, 17, 18]. However, the varying levels of theory
used to perform the QM calculations has previously prevented different datasets being used together to make machine learning models, for example for MLIPs. In this work, we have shown that meta-learning techniques can be used to jointly fit multiple datasets and demonstrated the improvement in performance that results from including a diverse selection of datasets.
We show the wide applicability of meta-learning by creating MLIPs for a variety of systems, from a single aspirin molecule to the ANI-1ccx dataset. By pre-training a model to multiple large organic molecule datasets we show that these datasets (QM7-x, QMugs, ANI-1x, Transition-1x and GEOM) can be combined together to pre-train a model. The benefits of using a pre-trained model are then shown for the 3BPA molecule, with a more accurate and smoother potential produced. Meta-learning greatly expands the variety of fitting data available for MLIPs and establishes the possibility of creating readily pre-trained, foundational models for MLIPs.
Pre-training machine learning models has been extensively discussed in the machine learning literature in recent years [59, 60, 61]. Whilst pre-training has been carried out for MLIPs, its use has been limited to training from one dataset to another [31, 32, 36]. With techniques such as meta-learning, this pre-training does not need to be limited to one specific dataset but can include large numbers of existing datasets. In this work, we added only a single reactive dataset to pre-train a model. However, many different reactive datasets exist and combining this large amount of information could help build a general transferable potentials for reactions in both condensed and gas phase without the need for millions of new QM calculations. Additionally, datasets have been created for many different combinations of elements. Meta-learning techniques could help build more transferable MLIPs over a wider range of elements with fewer calculations required.
However, combining multiple datasets together and training with meta-learning will not always improve results. This was seen with the CCSD(T) results where fitting straight from ANI-1x to CCSD(T) resulted in the lowest error. Therefore, adding more data when there is a specific application in mind is not always the best approach, particularly if the additional data is far from the final application. For specific applications, transfer learning from one dataset to another may yield the best training and test set errors. However, if multiple data sets need to be incorporate together, or a general model is desired which can be specialized to multiple different tasks, meta-learning methods are preferable.
With the techniques described in this work, multiple datasets can be fit at once. However, this advancement has exposed a more practical problem with the datasets currently published. There is not a standard format for storing information. Manual manipulation of datasets to a standard format is extremely time-consuming. The need for uniformity in the structure of datasets produced is therefore becoming increasingly important.
The growth of available datasets containing quantum mechanical information for molecular and material structures has given researchers unprecedented levels of QM information. However, combining data from multiple data-sources is a major challenge. We have shown how meta-learning can be used to combine information from multiple datasets generated with varying levels of theory. This advancement changes the way that existing datasets should be viewed, and opens up new avenues for MLIP fitting. Beyond this, the results suggest that meta-learning can be seen as a general approach for combining training datasets for the broad array of chemical and materials processes where data science models can benefit.
Acknowledgments
This work was supported by the United States Department of Energy (US DOE), Office of Science, Basic Energy Sciences, Chemical Sciences, Geosciences, and Biosciences Division under Triad National Security, LLC (“Triad”) contract grant no. 89233218CNA000001 (FWP: LANLE3F2). A. E. A. Allen and S. Matin also acknowledge the Center for Nonlinear Studies. Computer time was provided by the CCS-7 Darwin cluster at LANL.
Supporting Information: Learning Together: Towards foundational models for machine learning interatomic potentials with meta-learning
Alice E. A. Allen; Nicholas Lubbers; Sakib Matin; Justin Smith; Richard Messerly; Sergei Tretiak; Kipton Barros
E-mail:
Supplementary Methods
Uncertainty of ANI-1x across Datasets
Hyperparameter Optimization
The meta-learning hyperparameters which control:
The number of steps taken at each optimization step, ,
How the parameters are updated given from the task individual NN parameters
The maximum number of epochs () for fitting to the final dataset
were optimized. These are hyperparameters which specific to the Reptile meta-learning algorithm and not the molecular potential itself.
The change in the test set error with , and is shown in Figure S2. These test were performed on a subset of 10,000 structures from each DFT dataset and 7,650 structures from the CCSD(T) dataset. The number of steps chosen (32,64,96) correspond to the number of complete passes through training data. This batch size used is 250. The optimal hyperparameters for the Reptile algorithm are and , however very similar performance is seen for and and as fewer steps results in faster convergence this was chosen as the optimal value. The value of used in this work was increased as the dataset size increases so that it consistently corresponds to 3 complete passes through each dataset. The performance does not change as the maximum number of epochs are increased and the CCSD(T) training tends to converge within 150 epochs. The default retraining used for the meta-learning potential throughout this work is 150 epochs.
Figure S2. The change in RMSE for the CCSD(T) test set for varying meta-learning hyperparameters. By 15,000 epochs the training has fully converged. If no training to DFT is performed and the potential is fit directly to the CCSD(T) data the error is 13.90 kcal/mol.
The CCSD(T) error from individual transfer learning from data-sets are shown in Table S1. The ANI-1x datasets produce the best results, this is to be expected as they contain the ANI-1ccx configurations. GEOM and QMugs do not help fitting, this may be because both only contain energies and not forces. An improvement of around 20% for the CCSD(T) be seen when multiple datasets are used with 10,000 datapoints from each dataset.
| CCSD(T) Error kcal/mol | |
| AN1-1x dx | 6.44 |
| ANI-1x tz | 6.98 |
| GEOM | 16.46 |
| Transition1x | 11.73 |
| QM7-x | 8.26 |
| Qmugs | 14.69 |
| ANI-1ccx | 13.9 |
Table S1. The error for the CCSD(T) data after first fitting the six datasets. The ANI-1ccx result is when no previous training is carried out.
Stages of fitting
QM9 Weighting
The QM9 dataset contains the configurations of 133,885 organic molecules. The energy of these configurations was calculated with 76 different exchange-correlation functionals and 3 basis sets. Therefore, this dataset contains 228 separate tasks. If all of these tasks were included in every loop of the Reptile algorithm this would heavily bias the potential to the equilibrium QM9 structures. To overcome this, we added a weighting parameter . At every loop of the Reptile cycle a task is included with probability . This allows over-represented tasks to be included less frequently. The weighting for the QM9 tasks were set to 228, all other datasets had a weighting of 1 - meaning that they were always included in the fitting process.
Validation and Learning Rate Schedule
As multiple datasets are being fit at once, there are validation errors measured for each dataset. These errors are calculated after steps have been taken towards the dataset. The validation error therefore follows the error after ‘fine-tuning’.
The learning rate schedule starts at 0.001 and decreases by 50% after 150 epochs have not produced a decrease in error. This continues until the learning rate reaches 0.00001, at this stage the training is stopped.
Adding New Structures
For iteration 1/2/3 the following criteria is used:
For QMugs, ANI-1x-dz, ANI-1x-tz, GEOM and QM7-x structures with an error greater than 3.5/2.5/1.5 kcal/mol per atom were added to the datasets.
For Transition-1x, the structures are highly correlated as they all originate an approximation to the minimum energy path. Therefore, structures were separated into their reaction paths. The five highest error structures for each path were then added only if the energy differs from the existing structures added in that path by at least 0.1 kcal/mol.
The final number of structures present are given in Figure S2.
| Number of Structures | |
| ANI-1x dz | 855,028 |
| ANI-1x tz | 732,154 |
| Qmugs | 103,374 |
| QM7-x | 1,085,249 |
| GEOM | 122,552 |
| Transition-1x | 251,095 |
| Total | 3,149,452 |
Table S2. The number of structures included from each dataset.
3BPA
Energy RMSE
Force RMSE
Figure S5. The learning curve for the force RMSE for the 3BPA molecule at three temperatures.
Dihedral Scans
Hydrogen Dissociation
Fitting to QM9
References
- [1]O. Isayev, C. Oses, C. Toher, E. Gossett, S. Curtarolo, and A. Tropsha, Nat. Commun. 8, 15679 (2017).
- [2]T. Xie and J. C. Grossman, Phys. Rev. Lett. 120, 145301 (2018).
- [3]P. Mikulskis, M. R. Alexander, and D. A. Winkler, Adv. Intell. Syst 1, 1900045 (2019).
- [4]G. Pilania, Comput. Mater. Sci. 193, 110360 (2021).
- [5]R. Ouyang, S. Curtarolo, E. Ahm etcik, M. Scheffler, and L. M. Ghiringhelli, Phys. Rev. Materials 2, 083802 (2018).
- [6]D. Jha, L. Ward, A. Paul, W.-k. Liao, A. Choudhary, C. Wolverton, and A. Agrawal, Sci. Rep. 8, 17593 (2018).
- [7]A. Nandy, C. Duan, M. G. Taylor, F. Liu, A. H. Steeves, and H. J. Kulik, Chem. Rev. 121, 9927 (2021), pMID: 34260198, https://doi.org/10.1021/acs.chemrev.1c00347.
- [8]J. A. Keith, V. Vassilev-Galindo, B. Cheng, S. Chmiela, M. Gastegger, K.-R. Müller, and A. Tkatchenko, Chem. Rev. 121, 9816 (2021).
- [9]Y. Zhuo, A. Mansouri Tehrani, and J. Brgoch, J. Phys. Chem. 9, 1668 (2018).
- [10]T. J. Smith, O. Isayev, and A. E. Roitberg, Chem. Sci. 8, 3192 (2017).
- [11]F. A. Faber, A. Lindmaa, O. A. von Lilienfeld, and R. Armiento, Phys. Rev. Lett. 117, 135502 (2016).
- [12]S. Magedov, C. Koh, W. Malone, N. Lubbers, and B. Nebgen, J. Appl. Phys. 129, 064701 (2021).
- [13]S. Nandi, T. Vegge, and A. Bhowmik, ChemRxiv (2022), 10.26434/chemrxiv-2022-fs70n.
- [14]J. S. Smith, R. Zubatyuk, B. Nebgen, N. Lubbers, K. Barros, A. E. Roitberg, O. Isayev, and S. Tretiak, Sci. Data 7, 134 (2020).
- [15]S. Axelrod and R. Gómez-Bombarelli, Sci. Data 9, 185 (2022).
- [16]J. Hoja, L. Medrano Sandonas, B. G. Ernst, A. Vazquez-Mayagoitia, R. A. DiStasio Jr., and A. Tkatchenko, Sci. Data 8, 43 (2021).
- [17]C. Isert, K. Atz, J. Jiménez-Luna, and G. Schneider, Sci. Data 9, 273 (2022).
- [18]M. Schreiner, A. Bhowmik, T. Vegge, J. Busk, and O. Winther, Sci. Data 9, 779 (2022).
- [19]K. T. Schütt, F. Arbabzadah, S. Chmiela, K.-R. Müller, and A. Tkatchenko, Nat. Commun. 8, 13890 (2017).
- [20]A. P. Bartók, M. C. Payne, R. Kondor, and G. Csányi, Phys. Rev. Lett. 104, 134603 (2010).
- [21]J. Behler, J. Chem. Phys 134, 074106 (2011).
- [22]O. T. Unke, S. Chmiela, H. E. Sauceda, M. Gastegger, I. Poltavsky, K. T. Schütt, A. Tkatchenko, and K.-R. Müller, Chem. Rev. 121, 10142 (2021).
- [23]T. Mueller, A. Hernandez, and C. Wang, J. Chem. Phys. 152, 050902 (2020).
- [24]A. P. Bartók, J. Kermode, N. Bernstein, and G. Csányi, Phys. Rev. X 8, 041048 (2018).
- [25]F. Noé, A. Tkatchenko, K.-R. Müller, and C. Clementi, Ann. Rev. Phys. Chem. 71, 361 (2020).
- [26]D. Frenkel and B. Smit, Understanding Molecular Simulation, 2nd ed. (Academic Press, Inc., USA, 2001).
- [27]S. J. Weiner, P. A. Kollman, D. A. Case, U. C. Singh, C. Ghio, G. Alagona, S. Profeta, and P. Weiner, J. Am. Chem. Soc. 106, 765 (1984).
- [28]W. L. Jorgensen and J. Tirado-Rives, J. Am. Chem. Soc. 110, 1657 (1988).
- [29]R. J. Bartlett and M. Musial, Rev. Mod. Phys. 79, 291 (2007).
- [30]R. Tran, J. Lan, M. Shuaibi, S. Goyal, B. M. Wood, A. Das, J. Heras-Domingo, A. Kolluru, A. Rizvi, N. Shoghi, A. Sriram, Z. Ulissi, and C. L. Zitnick, “The Open Catalyst 2022 (OC22) dataset and challenges for oxide electrocatalysis,” (2022).DOI
- [31]J. S. Smith, B. T. Nebgen, R. Zubatyuk, N. Lubbers, C. Devereux, K. Barros, S. Tretiak, O. Isayev, and A. E. Roitberg, Nat. Commun. 10, 2903 (2019).
- [32]M. S. Chen, J. Lee, H.-Z. Ye, T. C. Berkelbach, D. R. Reichman, and T. E. Markland, “Machine learning potentials from transfer learning of periodic correlated electronic structure methods: Application to liquid water with afqmc, ccsd, and ccsd(t),” (2022), arXiv:2211.16619 [physics.chem-ph].arxiv.org/abs/2211.16619
- [33]M. E. Taylor and P. Stone, J. Mach. Learn. Res. 10, 1633 (2009).
- [34]S. J. Pan and Q. Yang, IEEE Trans. Knowl. Data Eng. 22, 1345 (2010).
- [35]J. Zaverkin, D. Holzmüller, L. Bonfirraro, and J. Kästner, Phys. Chem. Chem. Phys. 25, 5383 (2023).
- [36]D. P. Kovács, C. v. d. Oord, J. Kucera, A. E. A. Allen, D. J. Cole, C. Ortner, and G. Csányi, J. Chem. Theory Comput. 17, 7696 (2021).
- [37]C. Finn, P. Abbeel, and S. Levine, in Proceedings of the 34th International Conference on Machine Learning (2017) p. 1126–1135.
- [38]T. Hospedales, A. Antoniou, P. Micaelli, and A. Storey, IEEE Trans. Pattern Anal. Mach. Intell. 44, 5149 (2022).
- [39]A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,” (2018), arXiv:1803.02999 [cs.LG].arxiv.org/abs/1803.02999
- [40]M. Huisman, J. N. van Rijn, and A. Plaat, Artif. Intell. Rev. 54, 4483 (2021).
- [41]R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khatib, P. W. Koh, M. Krass, R. Krishna, R. Kudtipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramer, R. E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang, “On the opportunities and risks of foundation models,” (2022), arXiv:2108.07258 [cs.LG].arxiv.org/abs/2108.07258
- [42]OpenAI, “GPT-4 technical report,” (2023), arXiv:2303.08774 [cs.CL].arxiv.org/abs/2303.08774
- [43]Y. Sun, R. F. DeJaco, Z. Li, D. Tang, S. Glante, D. S. Sholl, C. M. Colina, R. Q. Shurr, M. Thommes, M. Hartmann, and J. I. Siepmann, Sci. Adv. 7, eabg3983 (2021).
- [44]J. Nie, N. Wang, J. Li, K. Wang, and H. Wang, Plant Methods 17, 119 (2021).
- [45]J. Wang, S. Zheng, J. Chen, and Y. Yang, J. Chem. Inf 61, 1627 (2021).
- [46]R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. von Lilienfeld, Sci. Data 1, 140022 (2014).
- [47]S. Batzner, A. Musaelian, L. Sun, M. Geiger, J. P. Mailoa, M. Kornbluth, N. Molinari, T. E. Smidt, and B. Kozinsky, Nat. Commun. 13, 2453 (2022).
- [48]N. Lubbers, J. S. Smith, and K. Barros, J. Chem. Phys. 148 (2018).
- [49]F. Neese, F. Wennmohs, U. Becker, and C. Riplinger, J. Chem. Phys. 152 (2020).
- [50]M. J. Frisch, G. W. Trucks, H. B. Schlegel, G. E. Scuseria, M. A. Robb, J. R. Cheeseman, G. Scalmani, V. Barone, G. A. Petersson, H. Nakatsuji, X. Li, M. Caricato, A. V. Marenich, J. Bloino, B. G. Janesko, R. Gomperts, B. Mennucci, H. P. Hratchian, J. V. Ortiz, A. F. Izmaylov, J. L. Sonnenberg, D. Williams-Young, F. Ding, F. Lipparini, F. Egidi, J. Goings, B. Peng, A. Petrone, T. Henderson, D. Ranasinghe, V. G. Zakrzewski, J. Gao, N. Rega, G. Zheng, W. Liang, M. Hada, M. Ehara, K. Toyota, R. Fukuda, J. Hasegawa, M. Ishida, T. Nakajima, Y. Honda, O. Kitao, H. Nakai, T. Vreven, K. Throssell, J. A. Montgomery, Jr., J. E. Peralta, F. Ogliaro, M. J. Bearpark, J. J. Heyd, E. N. Brothers, K. N. Kudin, V. N. Staroverov, T. A. Keith, R. Kobayashi, J. Normand, K. Raghavachari, A. P. Rendell, J. C. Burant, S. S. Iyengar, J. Tomasi, M. Cossi, J. M. Millam, C. Adamo, R. Cammi, J. W. Ochterski, R. L. Martin, K. Morokuma, O. Farkas, J. B. Foresman, and D. J. Fox, “Gaussian~16 Revision C.01,” (2016), Gaussian Inc. Wallingford CT.
- [51]P. Eastman, P. K. Behar, D. L. Dotson, R. Galvelis, J. E. Herr, J. T. Horton, Y. Mao, J. D. Chodera, B. P. Pritchard, Y. Wang, G. De Fabritiis, and T. E. Mark- land, “Spice, a dataset of drug-like molecules and peptides for training machine learning potentials,” (2022).
- [52]M. Nakata, T. Shimazaki, M. Hashimoto, and T. Maeda, J. Chem. Inf 60, 5891 (2020).
- [53]K. Yao, J. E. Herr, D. Toth, R. McIntyre, and J. Parkhill, Chem. Sci. 9, 2261 (2018).
- [54]J. S. Smith, B. Nebgen, N. Lubbers, O. Isayev, and A. E. Roitberg, J. Chem. Phys. 148, 241733 (2018).
- [56]D. Cole, L. Mones, and G. Csányi, Faraday Discuss. 224, 247 (2020).
- [58]land, “Forces are not enough: Benchmark and critical evaluation for machine learning force fields with molecular simulations,” (2022), arXiv:2210.07237 [physics.comp-ph].arxiv.org/abs/2210.07237
- [59]X. Han, Z. Zhang, N. Ding, Y. Gu, X. Liu, Y. Huo, J. Qiu, Y. Yao, A. Zhang, L. Zhang, W. Han, M. Huang, Q. Jin, Y. Lan, Y. Liu, Z. Liu, Z. Lu, X. Qiu, R. Song, J. Tang, J.-R. Wen, J. Yuan, W. X. Zhao, and J. Zhu, AI Open 2, 225 (2021).
- [60]W. Hu*, B. Liu*, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovec, in International Conference on Learning Representations (2020).
- [61]D. Hendrycks, K. Lee, and M. Mazeika, “Using pre-training can improve model robustness and uncertainty,” (2019), arXiv:1901.09960 [cs.LG].