WorksPhysics and Astronomy
Kernel Ridge Regression for conformer ensembles made easy with Structured Orthogonal Random Features
Abstract
A computationally efficient protocol for machine learning in chemical space using Boltzmann ensembles of conformers as input is proposed; the method is based on rewriting Kernel Ridge Regression expressions in terms of Structured Orthogonal Random Features, yielding physics-motivated trigonometric neural networks. To evaluate the method's utility for materials discovery, we test it on experimental datasets of two quantities related to battery electrolyte design, namely oxidation potentials in acetonitrile and hydration energies, using several popular molecular representations to demonstrate the method's flexibility. Despite only using computationally cheap forcefield calculations for conformer generation, we observe systematic decrease of machine learning error with increased training set size in all cases, with experimental accuracy reached after training on hundreds of molecules and prediction errors being comparable to state-of-the-art machine learning approaches. We also present novel versions of Huber and LogCosh loss functions that made hyperparameter optimization of the new approach more convenient.
Introduction
Optimization in chemical compound space is an important part of chemical industry, with machine learning (ML) models emerging at the forefront of accelerating it [1] [2]. This has led to creation of a large number of representation functions encoding geometrical [3, 4, 5] or orbital [6, 7, 8] information about conformers, which can then be combined with standard regressors for accurate ML of computational [9] or experimental [10] data. However, it makes more physical sense to think of molecules in terms of not single conformers, but their Boltzmann ensembles, which was the motivation behind a family of algorithms that perform Kernel Ridge Regression (KRR) [11] while using either representations of averaged conformers [12] or representations averaged over molecular configurations [13, 14]. The downside of such methods is the information loss implicit to averaging, and while defining a loss-less KRR protocol operating on ensembles rather than conformers is straightforward (see Subsection 2.3), it would be highly computationally inefficient. It could be possible to augment representation averages with other information about their distributions (e.g. medians and higher distribution moments) as is done in Molecular Dynamics Fingerprints approach [15], but systematically including higher-order moments beyond dispersion can be problematic due to statistical errors. Instead we turned to efficient kernel approximation schemes [16, 17, 18, 19, 20], basing our method on Structured Orthogonal Random Feature (SORF) formalism [17], which was previously used to speed up KRR calculations of molecular forces and energies [21]. Rewriting a possible KRR protocol for ML based on Boltzmann ensembles in terms of SORF yields numerically efficient neural networks (NN) characterized by using trigonometric activation functions and fast Walsh-Hadamard transform [22]. NN architectures sharing such characteristics also arise [23, 24, 25, 26] when SORF or related feature expressions are combined with Deep Gaussian Processes [27] or Multilayer Kernel Machines [28]. Since our NNs end up with several SORF layers corresponding to different levels of molecular structure (e.g. conformer ensemble, individual conformer, conformer’s atoms), the proposed method will be from now on referred to as Multilevel SORF (MSORF).
MSORF was tested on experimental data pertaining to two compound properties fundamentally important to battery electrolyte design. The first one is electrolyte component’s electrochemical stability window, which fundamentally affects battery’s operating voltage and energy capacity [29]; this work used the collection of experimental oxidation potentials of molecules in acetonitrile compiled in Ref. [10] from peer-reviewed scientific literature using Large Language Models. Since the authors did not provide an abbreviated name for their dataset, we will refer to it as “literature electrochemical stability” (LES) dataset for short. The second property of interest is compound’s solvation energy, which is useful for approximating trends in solubility [30], which in turn is fundamentally important for how usable a compound is as an electrolyte additive. We therefore also tested MSORF on experimental hydration energies collected in the FreeSolv dataset [31, 32, 33]. Importantly, and are examples of an intensive and an extensive property, allowing to explore how this could be reflected in MSORF architectures. We also demonstrate MSORF’s flexibility by presenting results with a diverse set of molecular representations, which will be discussed in Subsection 2.1.
The rest of the paper is structured as follows. The theory behind MSORF is explained in § 2, numerical details of computational experiments are presented in § 3, with the latter’s results discussed in § 4. Summary of the paper’s results along with potential further research directions are discussed in § 5. As will become clear in Subsection 2.3, applying MSORF to conformer ensembles leads to considering more hyperparameters than needed for standard KRR, making their optimization an issue; hence we document the protocol we used in this work in Appendix § 9. The protocol relies on a novel version of the LogCosh [34] loss function that is documented in Appendix B along with its counterpart for the Huber [35] loss function.
Theory
Kernel Ridge Regression in space of molecular conformers
Given a training set of molecules (), which we for now assume to be represented by atomic coordinates and nuclear charges, and the corresponding quantities of interest , the KRR estimate of the quantity of interest for a query molecule is
where is the kernel function and are regression coefficients solving the equation
where is a matrix such that , is the unity matrix, is a hyperparameter introduced for regularization. Equations (1) and (2) can be viewed as linear interpolation of quantities in a vector space of infinite dimensionality, with being the dot product of vectors in that space where and are mapped [36].
We will now outline three strategies for representing molecular conformers and defining the kernel function in terms of those representations that were used in this work. The first strategy is to represent each molecule with a vector and then calculate the kernel function using the (global) Gaussian kernel function
where is a vector’s norm and is a hyperparameter. Examples of such molecular representations considered in this work are Coulomb Matrix (CM) [37, 38] and Spectrum of London and Axilrod-Teller-Muto potential (SLATM) [4].
The second strategy is to represent each atom in molecule with a vector, the resulting molecular representation being a matrix . We consider two popular choices for defining the corresponding kernel function, the first one being the local Gaussian kernel function
where and are the atomic representation vectors of atoms with indices and in molecules and , while and are numbers of atoms in molecules and . is particularly useful for ML of extensive molecular quantities (e.g. total potential energy); an example of a representation that can be used with is Smooth Overlap of Atomic Orbitals (SOAP) descriptors [39, 3]. The kernel expression can be modified as
where and are nuclear charges of atoms and in molecules A and B, and is the Kroneker delta function. Examples of representations used with considered in this work are revised version of Faber-Christensen-Huang-Lilienfeld representation (FCHL19) [40, 41], atomic version of SLATM (aSLATM) [4], and convolutional version of Many Body Distribution Functional (cMBDF) [42, 43].
The third strategy we will consider is specific to the FJK representation [44] (named so due to Fock , Coulomb , and exchange used to generate it). The representation of molecule A takes results of cheap Hartree Fock calculations, among them properties of localized orbitals (, where is the number of occupied orbitals in molecule A), then assigns a weight and vector representation to each contribution to the localized orbital from atomic orbitals centered on an atom with index (). The corresponding expression for kernel function between molecules A and B can be written as
where and are linear and Gaussian kernel functions for localized orbitals, is a hyperparameter, is a vector of hyperparameters of the same dimensionality as (the implied division by this vector is element-wise), and stands for element-wise multiplication.1 The number of weights and that are not exactly zero does not grow with system size, meaning the same can be said about the computational cost of evaluating (8). consists of entries corresponding to momentum distribution, , , and matrices; in this work these four different types of entries were given a separate value each (which differs from the used in [44]).
While was used for ML of extensive quantities, Ref. [44] additionally considered mapping properties associated with changes of energy due to changes of electronic structure (such as HOMO and LUMO energies) onto pairs of Slater determinants, each given a separate FJK representation. The kernel function for such Slater determinant pairs A and B reads
where and are the Slater determinants constituting and . The expression features cancellation between localized orbitals that are not affected by the change of electronic structure represented by the Slater determinant pair, which makes physical sense.
Rewriting kernels in terms of Structured Orthogonal Random Features
SORF is based on Random Fourier Features (RFF) formalism [16], which approximates the infinite dimensional vectors on which molecules are mapped in KRR with finite dimensional feature vectors of dimensionality , resulting in the kernel function
where “” is dot product. Rewriting Equations (1) and (2) in terms of results in the following expression for the SORF estimate of a quantity
with the regression coefficients found by solving the equation
where is a matrix such that . Though the expression implies finding by solving a system of linear equations, in our work we avoided that by rewriting Equation (12) in terms of Singular Value Decomposition (SVD) [45] of the matrix, making the cost of solving (12) if (which was always the case in this work). There are two significant advantages of Equations (11) and (12) over their KRR versions (1) and (2). The first one is that for larger datasets it is possible to choose smaller than , decreasing the computational time cost and memory requirements of finding . The second one is most relevant for kernel functions whose evaluation scales bilinearly w.r.t. and , such as (4), (5), and (6); as will be demonstrated later evaluating the corresponding feature vectors is linear w.r.t. the size of molecule of interest, eliminating computational cost scaling w.r.t. training set system size for calculating ML predictions and improving said scaling for finding the regression coefficients.
If the system of interest is represented by a single vector of size (which is a power of 2) and the global Gaussian kernel (3) is used, the vectors are generated with the standard SORF expression
Here is Hadamard matrix [22], is an array of diagonal matrices (, ), each diagonal element randomly chosen between 1 and , is an array of long vectors , each value randomly chosen from the interval , stands for concatenating the vectors into a single vector; the implied calculation of the cosine function is element-wise. Products of the Hadamard matrix with a vector are evaluated with fast Walsh-Hadamard transform (hence the name of the parameter) whose computational cost is . Note that . It can be demonstrated [17] that
with the approximation introducing a statistical error which can be decreased by increasing and a systematic bias which can be decreased by increasing [17] (e.g. by padding the input vector with zeros). We note at this point that all the applications discussed in this work in the context of SORF can be analogously performed with other procedures [19, 20] for generating feature vectors that approximate analogously to Equation (14). However, in this work we used SORF due to its balance of good numerical performance and algorithmic simplicity.
Using SORF can be considered [26] using a single-layer NN (the corresponding layer will be called from now on), and we will now add layers to it to accommodate more complex kernel expressions. We first introduce the sum layer , its combination with , which we will refer to as , is defined as
It is evident that dot product of features reproduces (4). To reproduce (5) we additionally introduce the switch layer , applicable if a vector corresponding to an atom is augmented with the atom’s nuclear charge . Writing means a separate layer is initialized for each possible nuclear charge, with written as
where and are tensors analogous to and in Equation (13) that have been initialized specifically for atoms with nuclear charge . features reproduce (5) as their dot product and were previously used in Ref. [21].
To reproduce the nested exponential expressions appearing in (6)-(8) we use (13) to generate vectors whose dot product reproduces (8) and note that applying (3) to a pair of such vectors (after normalization) approximates (7). We introduce weighted sum layer that allows us to write features corresponding to (8) as
Additionally defining normalization layer that divides a vector by its norm allows us to reproduce (6) with features generated by , formally written as
features allow to reproduce (9) as
We calculated by assigning representations of localized orbitals in and weights of and and applying to them. Note that contributions to the feature vectors from localized orbitals that do not change between and cancel out, which makes physical sense. To simplify discussion, we will assume that for the single determinant case is used as well, with each localized orbital given a weight of 1 (which is also the way FJK representations were processed in our code implementation).
Applications to ensembles of conformers
Consider doing ML of an extensive quantity of a molecule A about which we know conformers with indices , each assigned a matrix of atomic representations and a nonnegative Boltzmann weight . A natural way to extend (4) to ensembles would be with the kernel function
The cost of evaluating this kernel function is , making the motivation to instead use SORF even stronger than for ; in terms of Subsection 2.2, the combination of layers corresponding to is . Using though leaves the ML algorithm unable to correct for potentially inaccurately representing relative importance of different conformers. This motivated introduction of an alternative expression, , where the mixed-extensive layer is defined as
where is a nonnegative hyperparameter. controls switching between (21) and a different kernel that is related to in a way similar to how (7) is related to (8). Note that for
meaning that while quantity predictions of are not truly extensive, they demonstrate correct scaling with molecule’s size. The case of representations using (5) is completely analogous and yields and as possible layer combinations.
Since global representations do not allow to implicitly account for a quantity’s extensivity during model construction, for them we simply used , normalization layer added to remove unphysical dependence of the result on how the importance weights are normalized. For FJK, considerations analogous to the case of local representations led us to consider and .
For intensive properties, we aim to create models whose prediction is invariant w.r.t. “molecule multiplication”, i.e. creating several copies of a molecule far away from each other. For global representations the requirement again cannot be satisfied implicitly, thus simple was used. For local representations or generates a feature vector for a conformer invariant w.r.t. molecule multiplication; treating it as a global representation vector yields and as the final layer combinations.
Treatment of intensive quantities with FJK depends on whether the molecule is represented by a single determinant or a pair of Slater determinants. For single Slater determinant case, analogously to the case of local representations, we obtain . For pairs of Slater determinants, we note that the differences of features appearing in are already invariant w.r.t. “molecule multiplication,” making normalization of conformer features excessive. The final expression is therefore . All layer combinations considered in this work for different representations and quantities are summarized in Table 1 in a way more illustrative of how they are connected to each other.
| method | max. MAE | max. RMSE | est. for target MAE 0.3 eV | est. for target MAE 0.2 eV |
| aSLATM | 91 | 409 | ||
| FCHL19 | 100 | 321 | ||
| cMBDF | 116 | 442 | ||
| SOAP | 165 | - | ||
| FJK | 291 | - | ||
| FJK (s. det.) | 259 | - | ||
| CM | - | - | ||
| SLATM | 117 | - |
Table 1. Summary of results for machine learning from Ref. [1], namely MAE for maximum training set size of 474, and values of needed to reach MAE of 0.3 and 0.2 eV as estimated from linear interpolation of the log-log plots of MAE vs. . Unlike Table 2 of the main text, the results were obtained with shifting by a constant.
We make two closing notes regarding how ensemble conformers were processed by our workflow. Firstly, in all examples considered in this work instead of Boltzmann weights we used their “cut” versions defined as
with such that
where is a parameter chosen by the user; note that changes continuously with .‡ The procedure was introduced to avoid calculating features of conformers whose contribution is irrelevant. Secondly, our algorithm’s implementation allows performing conformer generation not once, but several times to potentially account for statistical noise. Though we did not use the latter option and conformers were generated only
| rep. type | quantity | ||
| global | intensive: | extensive: (same as for intensive) | extensive (m.-ext.): (same as for intensive) |
| local (w. ) | |||
| local (w. ) | |||
| FJK | |||
| FJK (s.-det.) | (same as FJK) | (same as FJK) |
¶ - summation over atoms
∥ - summation over localized orbitals
∗ - summation over conformers
Table 1. Combinations of layers used in this work for different representations and quantities. “Extensive (m.-ext.)” means using the “mixed-extensive” layers (22) to calculate an extensive property, “s.-det.” means generating FJK representation using only one Slater determinant.
once for each molecule, code-wise we were still performing “summation over a single conformer generation.” As a result in the actual scripts used in this work each layer corresponding to summation over conformers (see Table 1) was preceded by an additional layer corresponding to summation over conformer generations.
Numerical details
For each learning curve presented in this work we divided a dataset into a training set (approximately 80% of the total number of points, making it 474 for LES and 514 for FreeSolv) and a test set. For each training size we picked a random subset of the training set, used it to optimize hyperparameters as described in Appendix § 9, trained the model using these hyperparameters, and then calculated Mean Absolute Error (MAE) of model’s predictions for the test set molecules. For each dataset and method the procedure was repeated 4 times, providing both the average MAEs and their dispersions presented in § 4. Since the datasets used in this work a collections of literature data gathered over inhomogeneous experimental reports, we consider it more natural to measure ML model accuracy in terms of MAE rather than Root Mean Square Error (RMSE), which is more punishing towards outliers. However, for future reference we will still present RMSEs observed for largest training set sizes.
By default we ran calculations with at all layers, since the value had performed well in Ref. [21], and chose to minimize the non-statistical bias error introduced in the SORF product approximation (14) (the “extra” input vector values were padded with zeros). This could not be done straightforwardly for SLATM, aSLATM, and (in the case of LES) SOAP since their representation vectors corresponded to larger than 32768. For SLATM, the issue was resolved by projecting all representation vectors onto linear space spawned by representation vectors of conformers in the training set, and then appending to the resulting vectors the distance between the projected and non-projected representation vector; note that using thus defined new representation vectors would not affect training and predictions of a KRR model. The procedure decreased to 4096 for SLATM, but could not be used for aSLATM and SOAP due to large number of atomic environments in the training set. Ref. [21] used Principal Component Analysis to fit atomic representations into a small enough , but we wanted to avoid that in order to accurately observe prediction errors of aSLATM and SOAP relative to other methods. Therefore, for aSLATM we used , for SOAP we used for LES and for FreeSolv.
We generated molecular conformers with the algorithm from Ref. [46], as implemented in the Morfeus package [47], at with the Merck molecular force field (MMFF94) [48, 49, 50, 51, 52, 53, 54] as implemented [55] in RDKit [56]; unimportant conformers were cut off with . Morfeus calculations require a parameter determining the number of conformers the package attempts to generate; we checked convergence of the matrix generated for one of the 4 training sets chosen for the learning curves with cMBDF representations by scanning and observed that the values were converged at for both LES and FreeSolv, hence was used throughout this work. In both cases, the and hyperparameters used in layers were taken to be the initial guess defined in Appendix A. We used the same convergence criterion for optimizing by calculating a training set’s for each representation and combination of MSORF layers considered. We settled on using throughout the work (which is also consistent with observations in Refs. [17] and [21]).
Almost all representation functions used in this work were calculated using the QML2 code [57], the exception being SOAP which was calculated with DScribe package [58, 59]. While aSLATM, SLATM, FCHL19, and cMBDF were generated using default parameters present in QML2, SOAP was generated using , , and , which were the parameters used in Ref. [60] for ML on FreeSolv with SOAP descriptors and KRR. FJK representations were generated from ab initio calculations performed with the PySCF package [61, 62, 63]. While the original paper where FJK was proposed [44] used STO-3G basis set [64, 65] and intrinsic bond orbitals [66], we found both to be ill-suited for the datasets considered in this work: Hartree-Fock STO-3G calculations sometimes failed to converge for charged molecules (which we wanted to consider as part of Slater determinant pairs), while using intrinsic bond orbitals in PySCF is complicated with some elements. Taking inspiration from previous work on using Huckel calculations for ML [7, 67], we switched to Huckel calculations with 6-311G basis [68, 69, 70, 71] and Boys localized orbitals [72] to generate FJK representations. Presence of solvent was simulated with Solvation Model based on Density (SMD) [73], an ab initio approach that has proven to be accurate in predicting experimental hydration energies [12]. When referring to “single Slater determinant” FJK we will mean representations calculated from ground state Huckel calculations for conformers in solvent (acetonitrile for LES and water for FreeSolv); for “pair Slater determinant” FJK for LES we took ground states of the molecule at charges 0 and 1 (both solvated in acetonitrile), while for FreeSolv we took ground states of the molecule in vacuum and solvated in water. For FJK we also used and applied to atomic contributions to localized orbitals (see [44]).
While the full procedure used to optimize hyperparameters is described in Appendix § 9, here we note that apart from optimizing value and different hyperparameters, we also considered optimizing quantity of interest by shifting it by a constant if it is intensive or by a stoichiometry-dependent shift [see Eq. (A.4)] if it is extensive; the latter procedure was inspired by and is a more sophisticated variation of the “dressed” atom approach [74, 75]. Such optimization would mean, for example, that ML of oxidation potentials is invariant w.r.t. the choice of the reference electrode, while ML of a compound’s free energy of formation and atomization are equivalent since they only differ in reference state for different elements. While the procedure could prove useful for larger datasets, it did not improve prediction accuracy for smaller datasets used in this work, not changing the results significantly for LES and becoming numerically unstable for smaller training dataset sizes considered for FreeSolv (probably due to the large number of atom types present, leading to a large number of additional shift-related hyperparameters). Therefore all data presented in § 4 have been obtained without using the quantity shift procedure, while results obtained with it are left for Supplementary Data.
Results and discussion
Oxidation potentials in acetonitrile
Figure 1 presents MAEs of predictions on LES as a function of training set size for all representations considered in this work. We compared the resulting learning curves by three metrics: MAE at maximum training set size considered (474 datapoints) and training set sizes required to reach experimental error (which the authors of Ref. [10] estimate to be of the order of 0.2 eV) and error which is larger than experimental by 50% (i.e. 0.3 eV). Estimates for the three metrics are gathered in Table 2. CM performs the worst out of all representations considered, which is unsurprising given its simplicity. Of the remaining representations, cMBDF, FCHL19, SLATM and aSLATM seem to perform better than SOAP and FJK, the latter becoming significantly less accurate if a conformer is represented with a pair of Slater determinants corresponding to electron removal. It might mean that the ab initio calculations used to generate FJK representations were not good enough to reflect the changes of orbital structure relevant to oxidation even qualitatively correctly, thus making representations based on molecular geometry, which is qualitatively correct, significantly more efficient. The remaining geometric representations are similar in terms of MAE, all reaching accuracy close to experimental. In Ref. [10] several combinations of different molecular representations and regressors were tested on the dataset (with molecular representations generated from single conformers optimized at xTB level of theory), with the best performer being combination of SLATM and KRR reaching MAE of 0.2 eV at an estimated number of 414 training points; we observe a similar (up to statistical error) performance for combining MSORF with aSLATM, SLATM, FCHL19, and cMBDF.
Figure 1. Mean Absolute Errors (MAE) of predictions of oxidation potentials in acetonitrile from Ref. [10] obtained by combining MSORF with several molecular representations for different training set sizes . “FJK (s. det.)” indicates FJK was used with a single Slater determinant.
Hydration energies
Figure 2 presents MAEs as a function of training set size for all representations considered in this work, with Table 3 summarizing MAEs at the largest training set size (514 datapoints) and estimated training sizes needed to reach chemical accuracy of 1 kcal/mol and FreeSolv’s “default” experimental error3 of 0.6 kcal/mol. Relative accuracies of different representations are mostly the same as for LES and can be explained analogously to the discussion in Subsec. 4.1; the only difference is pair-determinant FJK being as inaccurate as CM in this setup, possibly meaning that the way FJK captures changes of orbitals induced by SMD is disproportionately affected by numerical noise. We also note that introduction of the layer (22) only noticeably affected prediction errors in the case of cMBDF. Unlike LES, FreeSolv has been used as a benchmark in numerous ML studies, of which several have reported prediction errors of the order of experimental uncertainty [15, 76, 77, 78, 79, 80, 81, 60, 82, 83]. In the context of this work, particularly interesting are the MAEs of 0.417 and 0.359 kcal/mol of [79] and [81] (which are also the lowest MAEs obtained with a ML algorithm for FreeSolv so far) obtained by first training a model on abundant computational data and then finetuning it [84] for FreeSolv. While similar capitalization on finetuning is impossible in the current MSORF framework, we note a possible avenue to circumvent the problem in § 5. Our results also improve on errors previously reported for KRR-based approaches: Ref. [12] reported MAE of 0.68 kcal/mol for FML [14] approach combined with FCHL19 representation, while Ref. [60] reported MAE of 1.7 kcal/mol for combination of SOAP descriptors with KRR. Possible reasons for the improvement are discussed in Subsection 4.3.
| method | max. MAE | max. RMSE | est. for target MAE 0.3 eV | est. for target MAE 0.2 eV |
| aSLATM | 89 | 365 | ||
| FCHL19 | 74 | 292 | ||
| cMBDF | 88 | 448 | ||
| SOAP | 151 | - | ||
| FJK | 322 | - | ||
| FJK (s. det.) | 240 | - | ||
| CM | - | - | ||
| SLATM | 116 | - |
Table 2. Summary of results for machine learning from [10], namely MAE and RMSE for maximum training set size of 474, and values of needed to reach MAE of 0.3 and 0.2 eV as estimated from linear interpolation of the log-log plots of MAE vs. .
Comparison of machine learning with ensembles or minimum energy conformers
On a closing note, we evaluate how much representing a molecule with a conformer ensemble rather than a single conformer actually affected ML results presented in this work, using as examples combinations of MSORF representations that had demonstrated best and worst performance, namely aSLATM, FCHL19, and CM. For these representations we reran the same workflow as in the previous subsections, but for each ensemble we only considered the conformer with the lowest MMFF94 energy. For CM we used single layer (i.e. we used simple SORF), for aSLATM and FCHL we used with LES and with FreeSolv. The resulting learning curves are presented in Figure 3, their properties summarized in Tables 4 and Table 5. We observe that the difference between representing a molecule with an ensemble and one optimal conformer is of the order of statistical error, and is significantly less pronounced than differences between MAEs observed in this work with different conformer representations. This implies that, at least for the quantities considered in this work, choosing how to represent molecular geometry is more important than how many molecular geometries are used. Another implication is that our demonstration of lower MAEs for predicting of FreeSolv compared to KRR values obtained with SOAP in Ref. [60] is likely due to a different hyperparameter optimization procedure and a different choice of kernel function. For the minor improvement over MAEs obtained with FML [14] and FCHL19 (as reported in Ref. [12]), the cause is likely a combination of a different hyperparameter optimization procedure and statistical noise introduced to the ensemble representation in FML due to averaging over a molecular dynamics trajectory.
| method | max. MAE | max. RMSE | est. for target MAE 1.0 kcal/mol | est. for target MAE 0.6 kcal/mol |
| CM | 1.62 0.20 | 2.41 0.40 | - | - |
| aSLATM | 0.50 0.06 | 0.88 0.18 | 77 | 366 |
| FCHL19 | 0.49 0.05 | 0.81 0.14 | 78 | 301 |
Table 5. Summary of results for machine learning of the FreeSolv dataset while representing molecules with minimum energy conformers labeled analogously to Table 2.
Figure 2. Mean Absolute Error (MAE) of predictions of of the FreeSolv dataset obtained by combining MSORF with several molecular representations for different training set sizes . “S. det.” next to FJK entries indicates single Slater determinant was used, “m.-ext.” indicates using the “mixed-extensive” layer (22).
Conclusions and outlook
We have devised a general SORF-based procedure for generating physically motivated molecular representations from Boltzmann ensembles of conformers and demonstrated its utility for predicting experimental values for molecular properties. The method both demonstrates systematic decrease of prediction error with increased training set size and reaches experimental accuracy with hundreds of molecules as training set. At the
| method | max. MAE | max. RMSE | est. for target MAE 1.0 kcal/mol | est. for target MAE 0.6 kcal/mol |
| aSLATM | 77 | 332 | ||
| aSLATM (m.-ext.) | 79 | 336 | ||
| FCHL19 | 71 | 309 | ||
| FCHL19 (m.-ext.) | 79 | 277 | ||
| cMBDF | 100 | - | ||
| cMBDF (m.-ext.) | 145 | 425 | ||
| SOAP | 228 | - | ||
| SOAP (m.-ext.) | 232 | - | ||
| FJK | - | - | ||
| FJK (s. det.) | 272 | - | ||
| FJK (m.-ext.) | - | - | ||
| FJK (m.-ext., s. det.) | 306 | - | ||
| CM | - | - | ||
| SLATM | 153 | 466 |
Table 3. Summary of results for machine learning of the FreeSolv dataset labeled analogously to Table 2; note that for FreeSolv the maximum training set size was 514 molecules.
Figure 3. Comparison of MAEs obtained by representing molecules either with a conformer ensemble or the lowest energy conformer (“min. conf.”) for different representations and training set sizes . and are obtained from [10] and FreeSolv dataset.
| method | max. MAE | max. RMSE | est. for target MAE 0.3 eV | est. for target MAE 0.2 eV |
| CM | 0.39 0.03 | 0.50 0.04 | - | - |
| aSLATM | 0.19 0.01 | 0.28 0.04 | 85 | 383 |
| FCHL19 | 0.17 0.01 | 0.25 0.03 | 74 | 301 |
Table 4. Summary of results for machine learning from Ref. [10] while representing molecules with minimum energy conformers labeled analogously to Table 2.
same time, we did not observe a significant difference between representing a molecule with an ensemble of conformers or a single conformer, implying that, at least for the quantities considered in this work, how conformers are represented is a more important question than how many of them are used.
Prediction errors observed in this work were comparable to state-of-the-art achieved with other ML approaches, despite generating the underlying conformer representations using parameters that (with the exception of SOAP) had not been optimized for the quantities of interest. One possible way to improve on this is to generate conformer representations with another NN, train the combined model on computational data, and then re-optimize the MSORF part of the model on experimental data, essentially finetuning the lowest layers of the combined model. This would bridge the gap between MSORF and transfer learning techniques, which have shown promise when tested on FreeSolv in Refs. [85, 79, 86, 81]. Another possible direction could be going beyond the simple expressions for weighting contributions of different conformers, atoms, or localized orbitals used in this work to more complicated expressions gleaned from explainable artificial intelligence applied to ab initio calculations [87]. In both cases, it would be possible to capitalize on additional ab initio data while keeping MSORF grounded in KRR formalism. Lastly, while we focused on combining representations of conformers into representations of molecular ensembles, MSORF could analogously combine representations of molecular ensembles into representations of mixtures of molecules. This could make MSORF models useful for exploring chemical spaces of mixtures, a problem relevant for several industrial applications that involve using organic liquids [88, 89, 90].
Code availability
The MSORF algorithm has been made part of the QML2 code [57], with the implementation being based on Numpy [91], Numba [92], and SciPy [93]. The version used in this work together with the ML scripts has been uploaded as the 0.1.6 release at the QML2’s repository.
Supplementary Data
Contains data analogous to the one presented in § 4, but with LES results obtained with shifting by a constant and FreeSolv results obtained with shifting by a stoichiometry-dependent term. Also present are text files containing all MAEs and RMSEs for different representations, training set sizes, and random generator seeds.
Acknowledgments
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957189 (BIG-MAP) and No 957213 (BATTERY 2030+).
https://github.com/qml2code/qml2
Hyperparameter optimization
Our hyperparameter optimization procedure is based on optimizing leave-one-out cross-validation errors, i.e. prediction errors for a point given a model had been trained on all other points in the dataset. Leave-one-out errors can be efficiently calculated using the expression derived in [94], which in our context reads
where are predictions for training points of the model trained on the entire training set, . Note that
Ideally we would want to find hyperparameters that minimize MAE corresponding to , but MAE’s discontinuous derivatives would make it difficult to use the gradient during optimization. We thus considered instead using Huber or LogCosh loss functions, both of which can be thought of as “smoothed” versions of MAE with continuous derivatives, but in both cases the smoothness is controlled by parameters that are not straightforward to estimate without knowing beforehand the expected magnitude of model’s errors. We therefore introduced , a self-consistent version of LogCosh loss function whose smoothness is controlled by a smoothness parameter , and focused on optimizing
While the full definition and derivation of is provided in Appendix B, here we note that is a convex loss function w.r.t. the error values and exhibits a global minimum whose MAE has a relative difference from the global minimum of MAE that does not exceed .
We perform optimization of hyperparameters that can be divided into three groups. The first one are the hyperparameters mentioned throughout Subsec. 2.2 that affect calculation of . The second is that for fixed affects calculation of . The third and last, which was mentioned in § 3, are shifts that can be applied to quantities . For the latter, we would attempt to decrease by adding to a constant factor of if the quantity of interest is intensive. If the quantity is extensive, for molecule the quantity will be modified by adding
is hyperparameter shift specific to nuclear charge . Note that for fixed hyperparameters and , can be written as a linear function of or that can be evaluated with operations. Linearity in this situation means that (A.3) can be minimized with a quasi-Newton [95] method; in this work the implementation of Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm (L-BFGS) [96, 97] in SciPy was used.
Optimizing for fixed hyperparameters can also be done with relatively small computational cost. To see that we start with the SVD of
where is the matrix of left singular vectors, is diagonal matrix whose diagonal elements are the singular values, is the matrix of right singular vectors. Using Equations (A.2) and (A.5) to rewrite (A.1) yields
where is the matrix of features transformed into the basis of right singular vectors
With and pre-calculated with operations from , calculating (A.6) for a given costs ; the same scaling holds for calculating matrices later used in quantity shift optimization. As a result, for each evaluation of we optimize (or, to be more exact, to enforce ’s positivity). Unlike the case of quantity shifts we cannot guarantee the optimization problem is convex and thus approach it in two steps. In the first we use Bayesian Optimization Structure Search (BOSS) package [98] to approximately find the minimum of , inversion done to give minimized function a well-defined limit should a value in go to infinity due to numerical instability.
The second step is to perform gradient optimization with ”Sequential Least-Squares Quadratic Programming” (SLSQP)4 algorithm [100] as implemented in SciPy.
We thus define as minimum obtained by optimizing quantity shifts and values for a set of hyperparameters [i.e. hyperparameters appearing in the definition of (13) and (22)] defined by and smoothness parameter . The general idea is to find global minimum of for with steepest descent, but setting from the beginning will likely make the optimization loop stuck in a local minimum. Hence we used several gradient descent steps and values to avoid the problem, as documented in Algorithm 1.
Algorithm 1 Gradient optimization of .
Require: Initial guess for hyperparameters ;
for do
for do
loop
;
;
if then
break loop;
else
;
end if
end loop
end for
end for
For initial guesses, we always set from (22) as unity. Initial guesses for values appearing in each (13) were calculated as root mean square distance from training set vectors used as procedure’s input to their average. For example, for FJK features that meant that first values are guessed, then used to generate the corresponding features (17), which are in turn used to estimate the initial guess for used to generate the features (19). Also, as mentioned in Subsec. 2.1, FJK applies four different values of to four different types of entries in the vectors; each of these values was calculated analogously to the case of single hyperparameter, but with only entries rescaled by the corresponding considered and the final result multiplied by .
Lastly, in this work hyperparameter optimization always ran with consequitive values of 0.5, 0.25, and 0.125 and gradient step values of 0.5, 0.25, and 0.125. During L-BFGS and SLSQP algorithm calls we set gradient tolerance parameter as . While performing BOSS optimization the search bounds for were and shifted by , which is approximately the logarithm of average diagonal element of the training kernel matrix in the SORF approximation. Each BOSS run started with putting 16 points on a uniform grid in the search bounds, then doing 16 more evaluations according to the Bayesian optimization algorithm.
Self-consistent versions of Huber and LogCosh loss functions
We start with a general statement about regression loss functions operating on error vectors of dimensionality .
Theorem 1. Suppose is a function that
(i) for a fixed is convex and first-order smooth w.r.t. , as well as strictly positive for any , while for it equals zero and its gradient w.r.t. equals ;
(ii) is positively homogenous of order 1 w.r.t its arguments, i.e. for all and
with ;
(iii) and satisfies the limiting behavior
In this case we can define as the unique solution to the equation
where is a dimensionless parameter between 0 and 1. is a convex function of that is zero at , strictly positive everywhere else, and is related to MAE by
Proof. We first rewrite Euler’s theorem [101] for as
and use convexity of to write for
We now define
and rewrite Equation (B.3) as
For a fixed is a monotonously decreasing function of [see Equation (B.6)] with a positive limit at [see Equation (B.2)], hence Equation (B.8) has a single positive solution, making well-defined. We also note that is homogeneous and convex w.r.t. for a fixed . By virtue of Equation (B.2) it is straightforward to check that at is continuous and its left and right derivatives w.r.t. both equal . According to Ref. [102], these properties of guarantee that it is convex w.r.t. all arguments, and convex w.r.t. for fixed values of . To check convexity of we consider and differentiate (B.8) w.r.t. twice to write
This completes the proof that is convex.
To prove Equation (B.4) recall convexity of w.r.t. for a given and Equation (B.2), which together imply
and when combined with yields
The inequalities (B.10) and (B.11) together with Equation (B.3) prove the inequalities in (B.4).
There are two closing remarks on Theorem 1. Firstly, since is order 1 positively homogeneous w.r.t. , its -th derivative is positively homogeneous of order , implying that at the first and second derivatives of have a discontinuity and a singularity. This is the reason we instead use ( squared), as it is order 2 positively homogeneous, making its first derivative continous for all and its second derivative bound, though discontinuous at . Secondly, Equation (B.4) means that using or in hyperparameter optimization yields a minimum whose MAE does not differ from the global minimal MAE by more than in relative terms.
To derive we differentiate Equation (B.3) w.r.t. and use Equation (B.5) to write
This expression was used in steepest descent optimization as described in Appendix § 9.
Though it is not, in general, possible to analytically solve equation (B.8), the latter’s solvability with Newton method [95] can be guaranteed by requiring second derivative of w.r.t. to be continuous and non-zero for and fixed . Combined with convexity of w.r.t. for fixed , this makes second derivative of for fixed strictly positive and thus having positive lower and upper bounds for . Therefore, a Newton method run starting at will only consider points in where second derivative of the r.h.s. of (B.8) has positive lower and upper bounds, giving well-defined upper bound on the number of steps needed to converge the solution to a given accuracy [95]. The derivative required by the Newton method can be obtained by differentiating (B.7) and applying Euler’s theorem to the case of
We now consider two loss functions satisfying the conditions of Theorem 1. The first one is Huber loss function [35], which we rescale to satisfy (B.2)
Substitution of Equation (B.14) into Equation (B.3) yields the equation for self consistent Huber loss function
where and are shorthands for summation over such that and , is the number of values such that . The sets of for both summation signs can be found by sorting values and finding the interval between two consecutive where r.h.s of (B.16) as a function of changes sign. Finding the only positive root of (B.16) yields the final expression
An alternative way to find without sorting is derived by considering the arising in this case as a function defined as equaling on each interval of the form , where consists of values of sorted by increasing absolute value and augmented with , and reads
For fixed each is a convex monotonously decreasing function, with differing from by a nonnegative function. This means each has only one positive root w.r.t. , which is also smaller than the positive root of . Uniqueness of also implies that of all the roots of , only one actually lies in the corresponding interval . That means can be evaluated similarly to the Newton method, by starting with the initial guess and calculating each successive guess with Equation (B.17) with and evaluated based on the previous guess. The procedure will yield increasing estimates of until the exact value is reached, as signified by and staying the same as in the previous iteration.
Secondly, we considered the LogCosh loss function [34] which, when rescaled to satisfy (B.2), reads
It is straightforward to check that for fixed the second derivative of w.r.t. is never zero, allowing to calculate the self-consistent version of , dubbed , numerically via the Newton method as discussed previously.
We implemented both and in the QML2 code. Though is significantly more expensive to calculate than , we chose the former for our work since its cost is small relative to the cost of finding the SVD of the matrix (A.5) and its derivatives are infinitely continuous for all [as can be verified by differentiating (B.3)], with extremely unlikely to be encountered during hyperparameter optimization. We underline that Equation (B.3) implies that the operates similarly to the way operates in the definitions of (B.14) and (B.19). Since the parameter in this context is sometimes described as an absolute error value beyond which a point is considered an outlier, can be described as how large an error is compared to (i.e., to the “average smoothed error”) to be considered an outlier. As mentioned in Appendix § 9, we found to be a more convenient parameter to regulate in our applications.
As a closing remark, we note that and cannot be straightforwardly used with Stochastic Gradient Descent [103] algorithms since these loss functions are not sums of contributions from different elements of . While it should be possible to bypass the issue by using or and enforcing Equation (B.3) with a Lagrange multiplier, investigating feasibility and usefulness of such a procedure was beyond the scope of this work.
Supplementary Data: Kernel Ridge Regression for Boltzmann ensembles made easy with Structured Orthogonal Random Features
Konstantin Karandashev
Figure 1 (PDF p. 30). Mean Absolute Errors (MAE) of predictions of oxidation potentials in acetonitrile from Ref. [1] obtained by combining MSORF with several molecular representations for different training set sizes . “FJK (s. det.)” indicates FJK was used with a single Slater determinant. Unlike Figure 1 of the main text, the results were obtained while shifting by a constant.
Additional results for Section 4 of the main text
Here we present results analogous to the ones in § 4 of the main text, but with LES oxidation potentials shifted by a constant and FreeSolv hydration energies shifted using as defined in Equation (A.4) of the main text. The results for LES are presented in Figures 1 and 3 and Table 1 and 3, demonstrating that shifting the quantities did not affect the learning curves significantly. The results for FreeSolv are presented in Figures 2 and 3 and Tables 2 and 4; comparing the table values demonstrates that applying the stochiometry-dependent shift often increased MAE and RMSE for different representations, and the large statistical errors occasionally appearing on the learning curves correspond to cases when the procedure proved numerically unstable.
Figure 2 (PDF p. 31). Mean Absolute Error (MAE) of predictions of of the FreeSolv dataset obtained by combining MSORF with several molecular representations for different training set sizes . “S. det.” next to FJK entries indicates single Slater determinant was used, “m.-ext.” indicates using the “mixed-extensive” layer [see Equation (22) in the main text]. Unlike Figure 2 of the main text, the results were obtained with shifting by a stochiometry-dependent term.
| method | max. MAE | max. RMSE | est. for target MAE 1.0 kcal/mol | est. for target MAE 0.6 kcal/mol |
| aSLATM | 128 | - | ||
| aSLATM (m.-ext.) | 153 | 412 | ||
| FCHL19 | 93 | 366 | ||
| FCHL19 (m.-ext.) | 98 | 357 | ||
| cMBDF | 140 | - | ||
| cMBDF (m.-ext.) | 156 | 466 | ||
| SOAP | 262 | - | ||
| SOAP (m.-ext.) | 287 | - | ||
| FJK | - | - | ||
| FJK (s. det.) | 312 | - | ||
| FJK (m.-ext.) | - | - | ||
| FJK (m.-ext., s. det.) | 347 | - | ||
| CM | - | - | ||
| SLATM | 173 | - |
Table 2 (PDF p. 32). Summary of results for machine learning of the FreeSolv dataset labeled analogously to Table 1; note that for FreeSolv the maximum training set size was 514 molecules. Unlike Table 3 of the main text, the results were obtained with shifting by a stoichiometry-dependent term.
| method | max. MAE | max. RMSE | est. for target MAE 0.3 eV | est. for target MAE 0.2 eV |
| CM | - | - | ||
| aSLATM | 92 | 438 | ||
| FCHL19 | 85 | 314 |
Table 3 (PDF p. 32). Summary of results for machine learning from Ref. [1] while representing molecules with minimum energy conformers labeled analogously to Table 1. Unlike Table 4 of the main text, the results were obtained with shifting by a constant.
| method | max. MAE | max. RMSE | est. for target MAE 1.0 kcal/mol | est. for target MAE 0.6 kcal/mol |
| CM | - | - | ||
| aSLATM | 132 | - | ||
| FCHL19 | 100 | 372 |
Table 4 (PDF p. 32). Summary of results for machine learning of the FreeSolv dataset while representing molecules with minimum energy conformers labeled analogously to Table 1. Unlike Table 5 of the main text, the results were obtained with shifting by a stoichiometry-dependent term.
Figure 3 (PDF p. 33). Comparison of MAEs obtained by representing molecules either with a conformer ensemble or the lowest energy conformer (“min. conf.”) for different representations and training set sizes . and are obtained from Ref. [1] and FreeSolv dataset. Unlike Figure 3 of the main text, results for were obtained with shifting the quantities by a constant and results for were obtained with shifting them by a stoichiometry-dependent term.
References
- [1]Lee, S, Heinen, S, Khan, D, & Anatole von Lilienfeld, O. (2024) Autonomous data extraction from peer reviewed literature for training machine learning models of oxidation potentials. Mach. Learn.: Sci. Technol. 5, 015052.