betaFeedback

WorksPhysics and Astronomy

Kernel Ridge Regression for conformer ensembles made easy with Structured Orthogonal Random Features

Abstract

A computationally efficient protocol for machine learning in chemical space using Boltzmann ensembles of conformers as input is proposed; the method is based on rewriting Kernel Ridge Regression expressions in terms of Structured Orthogonal Random Features, yielding physics-motivated trigonometric neural networks. To evaluate the method's utility for materials discovery, we test it on experimental datasets of two quantities related to battery electrolyte design, namely oxidation potentials in acetonitrile and hydration energies, using several popular molecular representations to demonstrate the method's flexibility. Despite only using computationally cheap forcefield calculations for conformer generation, we observe systematic decrease of machine learning error with increased training set size in all cases, with experimental accuracy reached after training on hundreds of molecules and prediction errors being comparable to state-of-the-art machine learning approaches. We also present novel versions of Huber and LogCosh loss functions that made hyperparameter optimization of the new approach more convenient.

Introduction

Optimization in chemical compound space is an important part of chemical industry, with machine learning (ML) models emerging at the forefront of accelerating it [1] [2]. This has led to creation of a large number of representation functions encoding geometrical [3, 4, 5] or orbital [6, 7, 8] information about conformers, which can then be combined with standard regressors for accurate ML of computational [9] or experimental [10] data. However, it makes more physical sense to think of molecules in terms of not single conformers, but their Boltzmann ensembles, which was the motivation behind a family of algorithms that perform Kernel Ridge Regression (KRR) [11] while using either representations of averaged conformers [12] or representations averaged over molecular configurations [13, 14]. The downside of such methods is the information loss implicit to averaging, and while defining a loss-less KRR protocol operating on ensembles rather than conformers is straightforward (see Subsection 2.3), it would be highly computationally inefficient. It could be possible to augment representation averages with other information about their distributions (e.g. medians and higher distribution moments) as is done in Molecular Dynamics Fingerprints approach [15], but systematically including higher-order moments beyond dispersion can be problematic due to statistical errors. Instead we turned to efficient kernel approximation schemes [16, 17, 18, 19, 20], basing our method on Structured Orthogonal Random Feature (SORF) formalism [17], which was previously used to speed up KRR calculations of molecular forces and energies [21]. Rewriting a possible KRR protocol for ML based on Boltzmann ensembles in terms of SORF yields numerically efficient neural networks (NN) characterized by using trigonometric activation functions and fast Walsh-Hadamard transform [22]. NN architectures sharing such characteristics also arise [23, 24, 25, 26] when SORF or related feature expressions are combined with Deep Gaussian Processes [27] or Multilayer Kernel Machines [28]. Since our NNs end up with several SORF layers corresponding to different levels of molecular structure (e.g. conformer ensemble, individual conformer, conformer’s atoms), the proposed method will be from now on referred to as Multilevel SORF (MSORF).

MSORF was tested on experimental data pertaining to two compound properties fundamentally important to battery electrolyte design. The first one is electrolyte component’s electrochemical stability window, which fundamentally affects battery’s operating voltage and energy capacity [29]; this work used the collection of experimental oxidation potentials of molecules in acetonitrile EoxE_{\mathrm{ox}} compiled in Ref. [10] from peer-reviewed scientific literature using Large Language Models. Since the authors did not provide an abbreviated name for their dataset, we will refer to it as “literature electrochemical stability” (LES) dataset for short. The second property of interest is compound’s solvation energy, which is useful for approximating trends in solubility [30], which in turn is fundamentally important for how usable a compound is as an electrolyte additive. We therefore also tested MSORF on experimental hydration energies EhydE_{\mathrm{hyd}} collected in the FreeSolv dataset [31, 32, 33]. Importantly, EoxE_{\mathrm{ox}} and EhydE_{\mathrm{hyd}} are examples of an intensive and an extensive property, allowing to explore how this could be reflected in MSORF architectures. We also demonstrate MSORF’s flexibility by presenting results with a diverse set of molecular representations, which will be discussed in Subsection 2.1.

The rest of the paper is structured as follows. The theory behind MSORF is explained in § 2, numerical details of computational experiments are presented in § 3, with the latter’s results discussed in § 4. Summary of the paper’s results along with potential further research directions are discussed in § 5. As will become clear in Subsection 2.3, applying MSORF to conformer ensembles leads to considering more hyperparameters than needed for standard KRR, making their optimization an issue; hence we document the protocol we used in this work in Appendix § 9. The protocol relies on a novel version of the LogCosh [34] loss function that is documented in Appendix B along with its counterpart for the Huber [35] loss function.

Theory

Kernel Ridge Regression in space of molecular conformers

Given a training set of molecules Ai\mathrm{A}_i (i=1,…,Ntraini = 1,\ldots,N_{\mathrm{train}}), which we for now assume to be represented by atomic coordinates and nuclear charges, and the corresponding quantities of interest qiq_i, the KRR estimate of the quantity of interest for a query molecule B\mathrm{B} is

qKRR(B)=∑t=1NtrainαiKRRk(Ai,B),(1)q_{\mathrm{KRR}}(\mathrm{B}) = \sum_{t=1}^{N_{\mathrm{train}}} \alpha_i^{\mathrm{KRR}} k(\mathrm{A}_i,\mathrm{B}), \tag*{(1)}

where kk is the kernel function and α⃗KRR\vec{\alpha}^{\mathrm{KRR}} are regression coefficients solving the equation

(K+λINtrain)α⃗KRR=q⃗,(2)(\mathbf{K} + \lambda\mathbf{I}_{N_{\mathrm{train}}})\vec{\alpha}^{\mathrm{KRR}} = \vec{q}, \tag*{(2)}

where K\mathbf{K} is a matrix such that Kij=k(Ai,Aj)K_{ij} = k(\mathrm{A}_i,\mathrm{A}_j), INtrain\mathbf{I}_{N_{\mathrm{train}}} is the Ntrain×NtrainN_{\mathrm{train}} \times N_{\mathrm{train}} unity matrix, λ>0\lambda> 0 is a hyperparameter introduced for regularization. Equations (1) and (2) can be viewed as linear interpolation of quantities qiq_i in a vector space of infinite dimensionality, with k(A,B)k(\mathrm{A},\mathrm{B}) being the dot product of vectors in that space where A\mathrm{A} and B\mathrm{B} are mapped [36].

We will now outline three strategies for representing molecular conformers and defining the kernel function kk in terms of those representations that were used in this work. The first strategy is to represent each molecule A\mathrm{A} with a vector X⃗(A)\vec{X}^{(\mathrm{A})} and then calculate the kernel function k(A,B)k(\mathrm{A},\mathrm{B}) using the (global) Gaussian kernel function

kglobal(A,B):=exp⁡(−∣X⃗(A)−X⃗(B)∣22σ2),(3)k_{\mathrm{global}}(\mathrm{A},\mathrm{B}) := \exp\left(-\frac{\left\lvert\vec{X}^{(\mathrm{A})}-\vec{X}^{(\mathrm{B})}\right\rvert^2}{2\sigma^2}\right), \tag*{(3)}

where ∣…∣\lvert\ldots\rvert is a vector’s l2l^2 norm and σ\sigma is a hyperparameter. Examples of such molecular representations considered in this work are Coulomb Matrix (CM) [37, 38] and Spectrum of London and Axilrod-Teller-Muto potential (SLATM) [4].

The second strategy is to represent each atom in molecule A\mathrm{A} with a vector, the resulting molecular representation being a matrix X(A)\mathbf{X}^{(\mathrm{A})}. We consider two popular choices for defining the corresponding kernel function, the first one being the local Gaussian kernel function

klocal(A,B):=∑j′=1Na(A)∑j′′=1Na(B)kglobal(X⃗j′(A),X⃗j′′(B)),(4)k_{\mathrm{local}}(\mathrm{A},\mathrm{B}) := \sum_{j'=1}^{N_{\mathrm{a}}^{(\mathrm{A})}} \sum_{j''=1}^{N_{\mathrm{a}}^{(\mathrm{B})}} k_{\mathrm{global}}\left(\vec{X}_{j'}^{(\mathrm{A})},\vec{X}_{j''}^{(\mathrm{B})}\right), \tag*{(4)}

where X⃗j′(A)\vec{X}_{j'}^{(\mathrm{A})} and X⃗j′′(B)\vec{X}_{j''}^{(\mathrm{B})} are the atomic representation vectors of atoms with indices j′j' and j′′j'' in molecules A\mathrm{A} and B\mathrm{B}, while Na(A)N_{\mathrm{a}}^{(\mathrm{A})} and Na(B)N_{\mathrm{a}}^{(\mathrm{B})} are numbers of atoms in molecules A\mathrm{A} and B\mathrm{B}. klocalk_{\mathrm{local}} is particularly useful for ML of extensive molecular quantities (e.g. total potential energy); an example of a representation that can be used with klocalk_{\mathrm{local}} is Smooth Overlap of Atomic Orbitals (SOAP) descriptors [39, 3]. The kernel expression can be modified as

kdlocal(A,B):=∑j′=1Na(A)∑j′′=1Na(B)δ(nj′(A)−nj′′(B))kglobal(X⃗j′(A),X⃗j′′(B)),(5)k_{\mathrm{dlocal}}(A,B) := \sum_{j'=1}^{N_a^{(A)}} \sum_{j''=1}^{N_a^{(B)}} \delta\left(n_{j'}^{(A)}-n_{j''}^{(B)}\right)k_{\mathrm{global}}\left(\vec{X}_{j'}^{(A)},\vec{X}_{j''}^{(B)}\right), \tag*{(5)}

where nj′(A)n_{j'}^{(A)} and nj′′(B)n_{j''}^{(B)} are nuclear charges of atoms j′j' and j′′j'' in molecules A and B, and δ\delta is the Kroneker delta function. Examples of representations used with kdlocalk_{\mathrm{dlocal}} considered in this work are revised version of Faber-Christensen-Huang-Lilienfeld representation (FCHL19) [40, 41], atomic version of SLATM (aSLATM) [4], and convolutional version of Many Body Distribution Functional (cMBDF) [42, 43].

The third strategy we will consider is specific to the FJK representation [44] (named so due to Fock F\mathbf{F}, Coulomb J\mathbf{J}, and exchange K\mathbf{K} used to generate it). The representation of molecule A takes results of cheap Hartree Fock calculations, among them properties of localized orbitals ψo(A)\psi_o^{(A)} (o=1,…,Norb(A)o=1,\ldots,N_{\mathrm{orb}}^{(A)}, where Norb(A)N_{\mathrm{orb}}^{(A)} is the number of occupied orbitals in molecule A), then assigns a weight wj(ψo(A))w_j(\psi_o^{(A)}) and vector representation v⃗j(ψo(A))\vec{v}_j(\psi_o^{(A)}) to each contribution to the localized orbital from atomic orbitals centered on an atom with index jj (j∈1,…,Na(A)j\in1,\ldots,N_a^{(A)}). The corresponding expression for kernel function between molecules A and B can be written as

kFJK(A,B):=∑o′=1Norb(A)∑o′′=1Norb(B)kFJKGauss(ψo′(A),ψo′′(B)),(6)k_{\mathrm{FJK}}(A,B) := \sum_{o'=1}^{N_{\mathrm{orb}}^{(A)}} \sum_{o''=1}^{N_{\mathrm{orb}}^{(B)}} k_{\mathrm{FJK}}^{\mathrm{Gauss}}\left(\psi_{o'}^{(A)},\psi_{o''}^{(B)}\right), \tag*{(6)}
kFJKGauss(ψo′(A),ψo′′(B)):=exp⁡{−1σglobal2[1−kFJKlin(ψo′(A),ψo′′(B))kFJKlin(ψo′(A),ψo′(A))kFJKlin(ψo′′(B),ψo′′(B))]},(7)k_{\mathrm{FJK}}^{\mathrm{Gauss}}\left(\psi_{o'}^{(A)},\psi_{o''}^{(B)}\right) := \exp\left\{-\frac{1}{\sigma_{\mathrm{global}}^2}\left[1-\frac{k_{\mathrm{FJK}}^{\mathrm{lin}}\left(\psi_{o'}^{(A)},\psi_{o''}^{(B)}\right)}{\sqrt{k_{\mathrm{FJK}}^{\mathrm{lin}}\left(\psi_{o'}^{(A)},\psi_{o'}^{(A)}\right)k_{\mathrm{FJK}}^{\mathrm{lin}}\left(\psi_{o''}^{(B)},\psi_{o''}^{(B)}\right)}}\right]\right\}, \tag*{(7)}
kFJKlin(ψo′(A),ψo′′(B)):=∑j′=1Na(A)∑j′′=1Na(B)wj′(ψo′(A))wj′′(ψo′′(B))exp⁡[−∣v⃗(ψo′(A))−v⃗(ψo′′(B))∣22σ⃗⊙σ⃗],(8)k_{\mathrm{FJK}}^{\mathrm{lin}}\left(\psi_{o'}^{(A)},\psi_{o''}^{(B)}\right) := \sum_{j'=1}^{N_a^{(A)}} \sum_{j''=1}^{N_a^{(B)}} w_{j'}\left(\psi_{o'}^{(A)}\right)w_{j''}\left(\psi_{o''}^{(B)}\right)\exp\left[-\frac{\left\lvert\vec{v}\left(\psi_{o'}^{(A)}\right)-\vec{v}\left(\psi_{o''}^{(B)}\right)\right\rvert^2}{2\vec{\sigma}\mathbin{\odot}\vec{\sigma}}\right], \tag*{(8)}

where kFJKlink_{\mathrm{FJK}}^{\mathrm{lin}} and kFJKGaussk_{\mathrm{FJK}}^{\mathrm{Gauss}} are linear and Gaussian kernel functions for localized orbitals, σglobal\sigma_{\mathrm{global}} is a hyperparameter, σ⃗\vec{\sigma} is a vector of hyperparameters of the same dimensionality as v⃗(ψo(A))\vec{v}(\psi_o^{(A)}) (the implied division by this vector is element-wise), and ⊙\odot stands for element-wise multiplication.1 The number of weights wj′(ψo′(A))w_{j'}(\psi_{o'}^{(A)}) and wj′′(ψo′′(B))w_{j''}(\psi_{o''}^{(B)}) that are not exactly zero does not grow with system size, meaning the same can be said about the computational cost of evaluating kFJKlink_{\mathrm{FJK}}^{\mathrm{lin}} (8). v⃗(ψo(A))\vec{v}(\psi_o^{(A)}) consists of entries corresponding to momentum distribution, F\mathbf{F}, J\mathbf{J}, and K\mathbf{K} matrices; in this work these four different types of entries were given a separate σ\sigma value each (which differs from the σ⃗\vec{\sigma} used in [44]).

While kFJKk_{\mathrm{FJK}} was used for ML of extensive quantities, Ref. [44] additionally considered mapping properties associated with changes of energy due to changes of electronic structure (such as HOMO and LUMO energies) onto pairs of Slater determinants, each given a separate FJK representation. The kernel function for such Slater determinant pairs A1→2_{1\to2} and B1→2_{1\to2} reads

kFJKpair(A1→2,B1→2)=kFJK(A1,B1)+kFJK(A2,B2)−kFJK(A1,B2)−kFJK(A2,B1),(9)k_{\mathrm{FJK}}^{\mathrm{pair}}(A_{1\to2},B_{1\to2}) = k_{\mathrm{FJK}}(A_1,B_1) + k_{\mathrm{FJK}}(A_2,B_2) - k_{\mathrm{FJK}}(A_1,B_2) - k_{\mathrm{FJK}}(A_2,B_1), \tag*{(9)}

where A1/A2\mathrm{A}_1/\mathrm{A}_2 and B1/B2\mathrm{B}_1/\mathrm{B}_2 are the Slater determinants constituting A1→2\mathrm{A}_{1\to2} and B1→2\mathrm{B}_{1\to2}. The expression features cancellation between localized orbitals that are not affected by the change of electronic structure represented by the Slater determinant pair, which makes physical sense.

Rewriting kernels in terms of Structured Orthogonal Random Features

SORF is based on Random Fourier Features (RFF) formalism [16], which approximates the infinite dimensional vectors on which molecules are mapped in KRR with finite dimensional feature vectors z⃗\vec{z} of dimensionality NfN_{\mathrm{f}}, resulting in the kernel function

kSORF(A,B)=z⃗(A)⋅z⃗(B),(10)k_{\mathrm{SORF}}(\mathrm{A},\mathrm{B}) = \vec{z}(\mathrm{A}) \cdot\vec{z}(\mathrm{B}), \tag*{(10)}

where “⋅\cdot” is dot product. Rewriting Equations (1) and (2) in terms of kSORFk_{\mathrm{SORF}} results in the following expression for the SORF estimate of a quantity

qSORF(B)=α⃗⋅z⃗(B)(11)q_{\mathrm{SORF}}(\mathrm{B}) = \vec{\alpha} \cdot\vec{z}(\mathrm{B}) \tag*{(11)}

with the regression coefficients α⃗\vec{\alpha} found by solving the equation

(ZTZ+λINf)α⃗=ZTq⃗,(12)\left(\mathbf{Z}^{\mathsf{T}}\mathbf{Z} + \lambda\mathbf{I}_{N_{\mathrm{f}}}\right)\vec{\alpha} = \mathbf{Z}^{\mathsf{T}}\vec{q}, \tag*{(12)}

where Z\mathbf{Z} is a matrix such that Z⃗i=z⃗(Ai)\vec{Z}_i = \vec{z}(\mathrm{A}_i). Though the expression implies finding α⃗\vec{\alpha} by solving a Nf×NfN_{\mathrm{f}} \times N_{\mathrm{f}} system of linear equations, in our work we avoided that by rewriting Equation (12) in terms of Singular Value Decomposition (SVD) [45] of the Z\mathbf{Z} matrix, making the cost of solving (12) O(Ntrain2Nf)O(N_{\mathrm{train}}^2 N_{\mathrm{f}}) if Ntrain<NfN_{\mathrm{train}} < N_{\mathrm{f}} (which was always the case in this work). There are two significant advantages of Equations (11) and (12) over their KRR versions (1) and (2). The first one is that for larger datasets it is possible to choose NfN_{\mathrm{f}} smaller than NtrainN_{\mathrm{train}}, decreasing the computational time cost and memory requirements of finding α⃗\vec{\alpha}. The second one is most relevant for kernel functions whose evaluation scales bilinearly w.r.t. Na(A)N_{\mathrm{a}}^{(\mathrm{A})} and Na(B)N_{\mathrm{a}}^{(\mathrm{B})}, such as klocalk_{\mathrm{local}} (4), kdlocalk_{\mathrm{dlocal}} (5), and kFJKk_{\mathrm{FJK}} (6); as will be demonstrated later evaluating the corresponding feature vectors z⃗\vec{z} is linear w.r.t. the size of molecule of interest, eliminating computational cost scaling w.r.t. training set system size for calculating ML predictions and improving said scaling for finding the regression coefficients.

If the system of interest is represented by a single vector of size NinitN_{\mathrm{init}} (which is a power of 2) and the global Gaussian kernel (3) is used, the vectors z⃗\vec{z} are generated with the standard SORF expression

F⃗(X⃗(A),D,ϕ⃗,σ):=2Nf{cos⁡[Ninitσ(∏t=1NtrHNinitDst)X⃗(A)+ϕ⃗s]}s=1Nstack.(13)\vec{\mathcal{F}}\left(\vec{X}^{(\mathrm{A})}, \mathbf{D}, \vec{\phi}, \sigma\right) := \sqrt{\frac{2}{N_{\mathrm{f}}}} \left\{ \cos\left[ \frac{\sqrt{N_{\mathrm{init}}}}{\sigma} \left( \prod_{t=1}^{N_{\mathrm{tr}}} \mathbf{H}_{N_{\mathrm{init}}}\mathbf{D}_{st} \right) \vec{X}^{(\mathrm{A})} + \vec{\phi}_s \right] \right\}_{s=1}^{N_{\mathrm{stack}}}. \tag*{(13)}

Here HNinit\mathbf{H}_{N_{\mathrm{init}}} is Ninit×NinitN_{\mathrm{init}} \times N_{\mathrm{init}} Hadamard matrix [22], D\mathbf{D} is an Nstack×NtrN_{\mathrm{stack}} \times N_{\mathrm{tr}} array of Ninit×NinitN_{\mathrm{init}} \times N_{\mathrm{init}} diagonal matrices Dst\mathbf{D}_{st} (s=1,…,Nstacks = 1, \ldots, N_{\mathrm{stack}}, t=1,…,Ntrt = 1, \ldots, N_{\mathrm{tr}}), each diagonal element randomly chosen between 1 and −1-1, ϕ⃗\vec{\phi} is an NstackN_{\mathrm{stack}} array of NinitN_{\mathrm{init}} long vectors ϕ⃗s\vec{\phi}_s, each value randomly chosen from the interval [0,2π)[0,2\pi), {…}s=1Nstack\{\ldots\}_{s=1}^{N_{\mathrm{stack}}} stands for concatenating the NstackN_{\mathrm{stack}} vectors into a single vector; the implied calculation of the cosine function is element-wise. Products of the Hadamard matrix with a vector are evaluated with fast Walsh-Hadamard transform (hence the name of the NtrN_{\mathrm{tr}} parameter) whose computational cost is O(Ninit ⁣ln⁡Ninit)O(N_{\mathrm{init}\!}\ln N_{\mathrm{init}}). Note that Nf=NinitNstackN_{\mathrm{f}}=N_{\mathrm{init}}N_{\mathrm{stack}}. It can be demonstrated [17] that

F⃗(X⃗(A),D,ϕ,σ)⋅F⃗(X⃗(B),D,ϕ,σ)≈kglobal(A,B),(14)\vec{\mathcal{F}}(\vec{X}^{(A)},\mathbf{D},\boldsymbol{\phi},\sigma)\cdot\vec{\mathcal{F}}(\vec{X}^{(B)},\mathbf{D},\boldsymbol{\phi},\sigma)\approx k_{\mathrm{global}}(A,B), \tag*{(14)}

with the approximation introducing a statistical error which can be decreased by increasing NstackN_{\mathrm{stack}} and a systematic bias which can be decreased by increasing NinitN_{\mathrm{init}} [17] (e.g. by padding the input vector with zeros). We note at this point that all the applications discussed in this work in the context of SORF can be analogously performed with other procedures [19, 20] for generating feature vectors that approximate kglobalk_{\mathrm{global}} analogously to Equation (14). However, in this work we used SORF due to its balance of good numerical performance and algorithmic simplicity.

Using SORF can be considered [26] using a single-layer NN (the corresponding layer will be called F\mathcal{F} from now on), and we will now add layers to it to accommodate more complex kernel expressions. We first introduce the sum layer S\mathcal{S}, its combination with F\mathcal{F}, which we will refer to as SFS\mathcal{F}, is defined as

SF(X(A),D,ϕ,σ):=∑j=1Na(A)F(X⃗j(A),D,ϕ,σ).(15)S\mathcal{F}(\mathbf{X}^{(A)},\mathbf{D},\boldsymbol{\phi},\sigma):=\sum_{j=1}^{N_{\mathrm{a}}^{(A)}}\mathcal{F}(\vec{X}_{j}^{(A)},\mathbf{D},\boldsymbol{\phi},\sigma). \tag*{(15)}

It is evident that dot product of SFS\mathcal{F} features reproduces klocalk_{\mathrm{local}} (4). To reproduce kdlocalk_{\mathrm{dlocal}} (5) we additionally introduce the switch layer T\mathcal{T}, applicable if a vector X⃗\vec{X} corresponding to an atom is augmented with the atom’s nuclear charge nn. Writing TF\mathcal{T}\mathcal{F} means a separate F\mathcal{F} layer is initialized for each possible nuclear charge, with STFS\mathcal{T}\mathcal{F} written as

STF(X(A),σ):=∑j=1Na(A)F[X⃗j(A),D(nj(A)),ϕ(nj(A)),σ],(16)S\mathcal{T}\mathcal{F}(\mathbf{X}^{(A)},\sigma):=\sum_{j=1}^{N_{\mathrm{a}}^{(A)}}\mathcal{F}\left[\vec{X}_{j}^{(A)},\mathbf{D}\left(n_{j}^{(A)}\right),\boldsymbol{\phi}\left(n_{j}^{(A)}\right),\sigma\right], \tag*{(16)}

where D(nj(A))\mathbf{D}\left(n_{j}^{(A)}\right) and ϕ(nj(A))\boldsymbol{\phi}\left(n_{j}^{(A)}\right) are tensors analogous to D\mathbf{D} and ϕ\boldsymbol{\phi} in Equation (13) that have been initialized specifically for atoms with nuclear charge nj(A)n_{j}^{(A)}. STFS\mathcal{T}\mathcal{F} features reproduce kdlocalk_{\mathrm{dlocal}} (5) as their dot product and were previously used in Ref. [21].

To reproduce the nested exponential expressions appearing in kFJKk_{\mathrm{FJK}} (6)-(8) we use F\mathcal{F} (13) to generate vectors whose dot product reproduces kFJKlink_{\mathrm{FJK}}^{\mathrm{lin}} (8) and note that applying kglobalk_{\mathrm{global}} (3) to a pair of such vectors (after normalization) approximates kFJKGaussk_{\mathrm{FJK}}^{\mathrm{Gauss}} (7). We introduce weighted sum layer W\mathcal{W} that allows us to write features corresponding to kFJKlink_{\mathrm{FJK}}^{\mathrm{lin}} (8) as

WF(ψo(A)):=∑j=1Na(A)wj(ψo(A))F[v⃗j(ψo(A))/σ,Dorb,ϕorb,1].(17)\mathcal{W}\mathcal{F}(\psi_{o}^{(A)}):=\sum_{j=1}^{N_{\mathrm{a}}^{(A)}}w_{j}(\psi_{o}^{(A)})\mathcal{F}\left[\vec{v}_{j}(\psi_{o}^{(A)})/\sigma,\mathbf{D}^{\mathrm{orb}},\boldsymbol{\phi}^{\mathrm{orb}},1\right]. \tag*{(17)}

Additionally defining normalization layer N\mathcal{N} that divides a vector by its l2l^2 norm allows us to reproduce kFJKk_{\mathrm{FJK}} (6) with features generated by SFNWF\mathcal{S}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}, formally written as

SFNWF(A):=∑o=1Norb(A)FNWF(ψo(A))(18)\mathcal{S}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}(\mathrm{A}) := \sum_{o=1}^{N_{\mathrm{orb}}^{(\mathrm{A})}} \mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}(\psi_o^{(\mathrm{A})}) \tag*{(18)}
FNWF(ψo(A)):=F[WF(ψo(A))∣WF(ψo(A))∣,Dmol,ϕmol,σglobal](19)\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}(\psi_o^{(\mathrm{A})}) := \mathcal{F}\left[\frac{\mathcal{W}\mathcal{F}(\psi_o^{(\mathrm{A})})}{\left|\mathcal{W}\mathcal{F}(\psi_o^{(\mathrm{A})})\right|},\mathbf{D}^{\mathrm{mol}},\boldsymbol{\phi}^{\mathrm{mol}},\sigma_{\mathrm{global}}\right] \tag*{(19)}

SFNWF\mathcal{S}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F} features allow to reproduce kFJKpairk_{\mathrm{FJK}}^{\mathrm{pair}} (9) as

kFJKpair(A1→2,B1→2)≈[SFNWF(A2)−SFNWF(A1)]⋅[SFNWF(B2)−SFNWF(B1)].(20)k_{\mathrm{FJK}}^{\mathrm{pair}}(\mathrm{A}_{1\to2},\mathrm{B}_{1\to2}) \approx\left[\mathcal{S}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}(\mathrm{A}_2)-\mathcal{S}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}(\mathrm{A}_1)\right]\cdot\left[\mathcal{S}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}(\mathrm{B}_2)-\mathcal{S}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}(\mathrm{B}_1)\right]. \tag*{(20)}

We calculated SFNWF(A2)−SFNWF(A1)\mathcal{S}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}(\mathrm{A}_2)-\mathcal{S}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}(\mathrm{A}_1) by assigning representations of localized orbitals in A1\mathrm{A}_1 and A2\mathrm{A}_2 weights of −1-1 and 11 and applying WFNWF\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F} to them. Note that contributions to the feature vectors from localized orbitals that do not change between A1\mathrm{A}_1 and A2\mathrm{A}_2 cancel out, which makes physical sense. To simplify discussion, we will assume that for the single determinant case WFNWF\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F} is used as well, with each localized orbital given a weight of 1 (which is also the way FJK representations were processed in our code implementation).

Applications to ensembles of conformers

Consider doing ML of an extensive quantity of a molecule A about which we know conformers with indices c=1,…,Nconf(A)c=1,\ldots,N_{\mathrm{conf}}^{(\mathrm{A})}, each assigned a matrix of atomic representations Xc(A)\mathbf{X}_c^{(\mathrm{A})} and a nonnegative Boltzmann weight wcb(A)w_c^{\mathrm{b(A)}}. A natural way to extend klocalk_{\mathrm{local}} (4) to ensembles would be with the kernel function

klocalens=∑c′=1NconfA∑c′′=1NconfBwc′b(A)wc′′b(B)klocal(Xc′(A),Xc′′(B)).(21)k_{\mathrm{local}}^{\mathrm{ens}}=\sum_{c'=1}^{N_{\mathrm{conf}}^{\mathrm{A}}}\sum_{c''=1}^{N_{\mathrm{conf}}^{\mathrm{B}}}w_{c'}^{\mathrm{b(A)}}w_{c''}^{\mathrm{b(B)}}k_{\mathrm{local}}\left(\mathbf{X}_{c'}^{(\mathrm{A})},\mathbf{X}_{c''}^{(\mathrm{B})}\right). \tag*{(21)}

The cost of evaluating this kernel function is O(Nconf(A)Nconf(B)Na(A)Na(B))\mathcal{O}\left(N_{\mathrm{conf}}^{(\mathrm{A})}N_{\mathrm{conf}}^{(\mathrm{B})}N_{\mathrm{a}}^{(\mathrm{A})}N_{\mathrm{a}}^{(\mathrm{B})}\right), making the motivation to instead use SORF even stronger than for klocalensk_{\mathrm{local}}^{\mathrm{ens}}; in terms of Subsection 2.2, the combination of layers corresponding to klocalensk_{\mathrm{local}}^{\mathrm{ens}} is WSF\mathcal{W}\mathcal{S}\mathcal{F}. Using WSF\mathcal{W}\mathcal{S}\mathcal{F} though leaves the ML algorithm unable to correct for w⃗b\vec{w}^{\mathrm{b}} potentially inaccurately representing relative importance of different conformers. This motivated introduction of an alternative expression, EWESF\mathcal{E}\mathcal{W}\mathcal{E}\mathcal{S}\mathcal{F}, where the mixed-extensive layer E\mathcal{E} is defined as

E⃗(X⃗,D,ϕ,σ,σmix):=1Nstack(1+σmix−1){∑s=1NstackX⃗s+∣X⃗∣1+σmixF⃗(X⃗∣X⃗∣,D,ϕ,σ)}.(22)\vec{\mathcal{E}}(\vec{\mathbf{X}},\mathbf{D},\boldsymbol{\phi},\sigma,\sigma_{\mathrm{mix}}):=\frac{1}{\sqrt{N_{\mathrm{stack}}(1+\sigma_{\mathrm{mix}}^{-1})}}\left\{\sum_{s=1}^{N_{\mathrm{stack}}}\vec{\mathbf{X}}_s+\frac{|\vec{\mathbf{X}}|}{\sqrt{1+\sigma_{\mathrm{mix}}}}\vec{\mathcal{F}}\left(\frac{\vec{\mathbf{X}}}{|\vec{\mathbf{X}}|},\mathbf{D},\boldsymbol{\phi},\sigma\right)\right\}. \tag*{(22)}

where σmix\sigma_{\mathrm{mix}} is a nonnegative hyperparameter. σmix\sigma_{\mathrm{mix}} controls switching between klocalensk_{\mathrm{local}}^{\mathrm{ens}} (21) and a different kernel that is related to klocalensk_{\mathrm{local}}^{\mathrm{ens}} in a way similar to how kFJKGaussk_{\mathrm{FJK}}^{\mathrm{Gauss}} (7) is related to kFJKlink_{\mathrm{FJK}}^{\mathrm{lin}} (8). Note that for C>0C>0

E(CX⃗):=CE(X⃗),(23)\mathcal{E}(C\vec{\mathbf{X}}):=C\mathcal{E}(\vec{\mathbf{X}}), \tag*{(23)}

meaning that while quantity predictions of EWESF\mathcal{E}\mathcal{W}\mathcal{E}\mathcal{S}\mathcal{F} are not truly extensive, they demonstrate correct scaling with molecule’s size. The case of representations using kdlocalk_{\mathrm{dlocal}} (5) is completely analogous and yields WSTF\mathcal{W}\mathcal{S}\mathcal{T}\mathcal{F} and EWESTF\mathcal{E}\mathcal{W}\mathcal{E}\mathcal{S}\mathcal{T}\mathcal{F} as possible layer combinations.

Since global representations do not allow to implicitly account for a quantity’s extensivity during model construction, for them we simply used FNWF\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}, normalization layer added to remove unphysical dependence of the result on how the importance weights are normalized. For FJK, considerations analogous to the case of local representations led us to consider WWFNWF\mathcal{W}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F} and EWEWFNWF\mathcal{E}\mathcal{W}\mathcal{E}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}.

For intensive properties, we aim to create models whose prediction is invariant w.r.t. “molecule multiplication”, i.e. creating several copies of a molecule far away from each other. For global representations the requirement again cannot be satisfied implicitly, thus simple FNWF\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F} was used. For local representations NSF\mathcal{N}\mathcal{S}\mathcal{F} or NSTF\mathcal{N}\mathcal{S}\mathcal{T}\mathcal{F} generates a feature vector for a conformer invariant w.r.t. molecule multiplication; treating it as a global representation vector yields FNWFNSF\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{S}\mathcal{F} and FNWFNSTF\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{S}\mathcal{T}\mathcal{F} as the final layer combinations.

Treatment of intensive quantities with FJK depends on whether the molecule is represented by a single determinant or a pair of Slater determinants. For single Slater determinant case, analogously to the case of local representations, we obtain FNWFNWFNWF\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}. For pairs of Slater determinants, we note that the differences of features appearing in kFJKpairk_{\mathrm{FJK}}^{\mathrm{pair}} are already invariant w.r.t. “molecule multiplication,” making normalization of conformer features excessive. The final expression is therefore FNWFWFNWF\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}. All layer combinations considered in this work for different representations and quantities are summarized in Table 1 in a way more illustrative of how they are connected to each other.

methodmax. NtrainN_{\mathrm{train}} MAEmax. NtrainN_{\mathrm{train}} RMSEest. NtrainN_{\mathrm{train}} for target MAE 0.3 eVest. NtrainN_{\mathrm{train}} for target MAE 0.2 eV
aSLATM0.19±0.010.19 \pm0.010.28±0.030.28 \pm0.0391409
FCHL190.17±0.010.17 \pm0.010.24±0.030.24 \pm0.03100321
cMBDF0.19±0.020.19 \pm0.020.29±0.030.29 \pm0.03116442
SOAP0.23±0.010.23 \pm0.010.33±0.020.33 \pm0.02165-
FJK0.27±0.010.27 \pm0.010.37±0.020.37 \pm0.02291-
FJK (s. det.)0.24±0.020.24 \pm0.020.37±0.060.37 \pm0.06259-
CM0.36±0.020.36 \pm0.020.47±0.020.47 \pm0.02--
SLATM0.21±0.020.21 \pm0.020.31±0.050.31 \pm0.05117-

Table 1. Summary of results for machine learning EoxE_{\mathrm{ox}} from Ref. [1], namely MAE for maximum training set size NtrainN_{\mathrm{train}} of 474, and values of NtrainN_{\mathrm{train}} needed to reach MAE of 0.3 and 0.2 eV as estimated from linear interpolation of the log-log plots of MAE vs. NtrainN_{\mathrm{train}}. Unlike Table 2 of the main text, the results were obtained with shifting EoxE_{\mathrm{ox}} by a constant.

We make two closing notes regarding how ensemble conformers were processed by our workflow. Firstly, in all examples considered in this work instead of Boltzmann weights we used their “cut” versions w⃗cb\vec{w}^{\mathrm{cb}} defined as

wjcb:=max⁡(wjb−ρ−,0)∑j′=1Nconfmax⁡(wj′b−ρ−,0)(24)w_j^{\mathrm{cb}} := \frac{\max(w_j^{\mathrm{b}}-\rho_-,0)}{\sum_{j'=1}^{N_{\mathrm{conf}}}\max(w_{j'}^{\mathrm{b}}-\rho_-,0)} \tag*{(24)}

with ρ−\rho_- such that

∑j=1Nconfmax⁡(wjb−ρ−,0)∑j=1Nconfwjb=1−ρcut,(25)\frac{\sum_{j=1}^{N_{\mathrm{conf}}}\max(w_j^{\mathrm{b}}-\rho_-,0)}{\sum_{j=1}^{N_{\mathrm{conf}}}w_j^{\mathrm{b}}}=1-\rho_{\mathrm{cut}}, \tag*{(25)}

where ρcut\rho_{\mathrm{cut}} is a parameter chosen by the user; note that w⃗cb\vec{w}^{\mathrm{cb}} changes continuously with wbw^{\mathrm{b}}.‡ The procedure was introduced to avoid calculating features of conformers whose contribution is irrelevant. Secondly, our algorithm’s implementation allows performing conformer generation not once, but several times to potentially account for statistical noise. Though we did not use the latter option and conformers were generated only

rep. typequantity
globalintensive: FNWF\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}extensive: (same as for intensive)extensive (m.-ext.): (same as for intensive)
local (w. klocalk_{\mathrm{local}})FNWFNSF\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{S}\mathcal{F}WSF\mathcal{W}\mathcal{S}\mathcal{F}EWESF\mathcal{E}\mathcal{W}\mathcal{E}\mathcal{S}\mathcal{F}
local (w. kdlocalk_{\mathrm{dlocal}})FNWFNSTF\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{S}\mathcal{T}\mathcal{F}WSTF\mathcal{W}\mathcal{S}\mathcal{T}\mathcal{F}EWESTF\mathcal{E}\mathcal{W}\mathcal{E}\mathcal{S}\mathcal{T}\mathcal{F}
FJKFNWFWFNWF\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}WWFNWF\mathcal{W}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}EWEWFNWF\mathcal{E}\mathcal{W}\mathcal{E}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}
FJK (s.-det.)FNWFNWFNWF\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}\mathcal{N}\mathcal{W}\mathcal{F}(same as FJK)(same as FJK)

¶ - summation over atoms

∥ - summation over localized orbitals

∗ - summation over conformers

Table 1. Combinations of layers used in this work for different representations and quantities. “Extensive (m.-ext.)” means using the “mixed-extensive” layers E\mathcal{E} (22) to calculate an extensive property, “s.-det.” means generating FJK representation using only one Slater determinant.

once for each molecule, code-wise we were still performing “summation over a single conformer generation.” As a result in the actual scripts used in this work each W\mathcal{W} layer corresponding to summation over conformers (see Table 1) was preceded by an additional S\mathcal{S} layer corresponding to summation over conformer generations.

Numerical details

For each learning curve presented in this work we divided a dataset into a training set (approximately 80% of the total number of points, making it 474 for LES and 514 for FreeSolv) and a test set. For each training size we picked a random subset of the training set, used it to optimize hyperparameters as described in Appendix § 9, trained the model using these hyperparameters, and then calculated Mean Absolute Error (MAE) of model’s predictions for the test set molecules. For each dataset and method the procedure was repeated 4 times, providing both the average MAEs and their dispersions presented in § 4. Since the datasets used in this work a collections of literature data gathered over inhomogeneous experimental reports, we consider it more natural to measure ML model accuracy in terms of MAE rather than Root Mean Square Error (RMSE), which is more punishing towards outliers. However, for future reference we will still present RMSEs observed for largest training set sizes.

By default we ran calculations with Nf=32768N_{\mathrm{f}} = 32768 at all F\mathcal{F} layers, since the value had performed well in Ref. [21], and chose Ninit=NfN_{\mathrm{init}} = N_{\mathrm{f}} to minimize the non-statistical bias error introduced in the SORF product approximation (14) (the “extra” input vector values were padded with zeros). This could not be done straightforwardly for SLATM, aSLATM, and (in the case of LES) SOAP since their representation vectors corresponded to NinitN_{\mathrm{init}} larger than 32768. For SLATM, the issue was resolved by projecting all representation vectors onto linear space spawned by representation vectors of conformers in the training set, and then appending to the resulting vectors the l2l^2 distance between the projected and non-projected representation vector; note that using thus defined new representation vectors would not affect training and predictions of a KRR model. The procedure decreased NinitN_{\mathrm{init}} to 4096 for SLATM, but could not be used for aSLATM and SOAP due to large number of atomic environments in the training set. Ref. [21] used Principal Component Analysis to fit atomic representations into a small enough NinitN_{\mathrm{init}}, but we wanted to avoid that in order to accurately observe prediction errors of aSLATM and SOAP relative to other methods. Therefore, for aSLATM we used Ninit=131072N_{\mathrm{init}} = 131072, for SOAP we used Ninit=65536N_{\mathrm{init}} = 65536 for LES and Ninit=32768N_{\mathrm{init}} = 32768 for FreeSolv.

We generated molecular conformers with the algorithm from Ref. [46], as implemented in the Morfeus package [47], at T=298.15 KT = 298.15\ \mathrm{K} with the Merck molecular force field (MMFF94) [48, 49, 50, 51, 52, 53, 54] as implemented [55] in RDKit [56]; unimportant conformers were cut off with ρcut=0.05\rho_{\mathrm{cut}} = 0.05. Morfeus calculations require a parameter NconfMorfeusN_{\mathrm{conf}}^{\mathrm{Morfeus}} determining the number of conformers the package attempts to generate; we checked convergence of the ZZT\mathbf{ZZ}^{\mathsf{T}} matrix generated for one of the 4 training sets chosen for the learning curves with cMBDF representations by scanning NconfMorfeus=8,…,1024N_{\mathrm{conf}}^{\mathrm{Morfeus}} = 8,\ldots,1024 and observed that the values were converged at NconfMorfeus=32N_{\mathrm{conf}}^{\mathrm{Morfeus}} = 32 for both LES and FreeSolv, hence NconfMorfeus=32N_{\mathrm{conf}}^{\mathrm{Morfeus}} = 32 was used throughout this work. In both cases, the σ\sigma and σ⃗\vec{\sigma} hyperparameters used in F\mathcal{F} layers were taken to be the initial guess defined in Appendix A. We used the same convergence criterion for optimizing NtrN_{\mathrm{tr}} by calculating a training set’s ZZT\mathbf{ZZ}^{\mathsf{T}} Ntr=1,2,…,6,7N_{\mathrm{tr}} = 1,2,\ldots,6,7 for each representation and combination of MSORF layers considered. We settled on using Ntr=3N_{\mathrm{tr}} = 3 throughout the work (which is also consistent with observations in Refs. [17] and [21]).

Almost all representation functions used in this work were calculated using the QML2 code [57], the exception being SOAP which was calculated with DScribe package [58, 59]. While aSLATM, SLATM, FCHL19, and cMBDF were generated using default parameters present in QML2, SOAP was generated using rcutSOAP=9.53r_{\mathrm{cut}}^{\mathrm{SOAP}} = 9.53, nmaxSOAP=8n_{\mathrm{max}}^{\mathrm{SOAP}} = 8, and lmaxSOAP=8l_{\mathrm{max}}^{\mathrm{SOAP}} = 8, which were the parameters used in Ref. [60] for ML on FreeSolv with SOAP descriptors and KRR. FJK representations were generated from ab initio calculations performed with the PySCF package [61, 62, 63]. While the original paper where FJK was proposed [44] used STO-3G basis set [64, 65] and intrinsic bond orbitals [66], we found both to be ill-suited for the datasets considered in this work: Hartree-Fock STO-3G calculations sometimes failed to converge for charged molecules (which we wanted to consider as part of Slater determinant pairs), while using intrinsic bond orbitals in PySCF is complicated with some elements. Taking inspiration from previous work on using Huckel calculations for ML [7, 67], we switched to Huckel calculations with 6-311G basis [68, 69, 70, 71] and Boys localized orbitals [72] to generate FJK representations. Presence of solvent was simulated with Solvation Model based on Density (SMD) [73], an ab initio approach that has proven to be accurate in predicting experimental hydration energies [12]. When referring to “single Slater determinant” FJK we will mean representations calculated from ground state Huckel calculations for conformers in solvent (acetonitrile for LES and water for FreeSolv); for “pair Slater determinant” FJK for LES we took ground states of the molecule at charges 0 and 1 (both solvated in acetonitrile), while for FreeSolv we took ground states of the molecule in vacuum and solvated in water. For FJK we also used lmax⁡=1l_{\max}=1 and applied ρcut=0.05\rho_{\mathrm{cut}}=0.05 to atomic contributions to localized orbitals (see [44]).

While the full procedure used to optimize hyperparameters is described in Appendix § 9, here we note that apart from optimizing λ\lambda value and different σ\sigma hyperparameters, we also considered optimizing quantity of interest by shifting it by a constant if it is intensive or by a stoichiometry-dependent shift [see Eq. (A.4)] if it is extensive; the latter procedure was inspired by and is a more sophisticated variation of the “dressed” atom approach [74, 75]. Such optimization would mean, for example, that ML of oxidation potentials is invariant w.r.t. the choice of the reference electrode, while ML of a compound’s free energy of formation and atomization are equivalent since they only differ in reference state for different elements. While the procedure could prove useful for larger datasets, it did not improve prediction accuracy for smaller datasets used in this work, not changing the results significantly for LES and becoming numerically unstable for smaller training dataset sizes considered for FreeSolv (probably due to the large number of atom types present, leading to a large number of additional shift-related hyperparameters). Therefore all data presented in § 4 have been obtained without using the quantity shift procedure, while results obtained with it are left for Supplementary Data.

Results and discussion

Oxidation potentials in acetonitrile

Figure 1 presents MAEs of EoxE_{\mathrm{ox}} predictions on LES as a function of training set size for all representations considered in this work. We compared the resulting learning curves by three metrics: MAE at maximum training set size considered (474 datapoints) and training set sizes required to reach experimental error (which the authors of Ref. [10] estimate to be of the order of 0.2 eV) and error which is larger than experimental by 50% (i.e. 0.3 eV). Estimates for the three metrics are gathered in Table 2. CM performs the worst out of all representations considered, which is unsurprising given its simplicity. Of the remaining representations, cMBDF, FCHL19, SLATM and aSLATM seem to perform better than SOAP and FJK, the latter becoming significantly less accurate if a conformer is represented with a pair of Slater determinants corresponding to electron removal. It might mean that the ab initio calculations used to generate FJK representations were not good enough to reflect the changes of orbital structure relevant to oxidation even qualitatively correctly, thus making representations based on molecular geometry, which is qualitatively correct, significantly more efficient. The remaining geometric representations are similar in terms of MAE, all reaching accuracy close to experimental. In Ref. [10] several combinations of different molecular representations and regressors were tested on the dataset (with molecular representations generated from single conformers optimized at xTB level of theory), with the best performer being combination of SLATM and KRR reaching MAE of 0.2 eV at an estimated number of 414 training points; we observe a similar (up to statistical error) performance for combining MSORF with aSLATM, SLATM, FCHL19, and cMBDF.

Image not extracted

Figure 1. Mean Absolute Errors (MAE) of predictions of oxidation potentials EoxE_{\mathrm{ox}} in acetonitrile from Ref. [10] obtained by combining MSORF with several molecular representations for different training set sizes NtrainN_{\mathrm{train}}. “FJK (s. det.)” indicates FJK was used with a single Slater determinant.

Hydration energies

Figure 2 presents MAEs as a function of training set size for all representations considered in this work, with Table 3 summarizing MAEs at the largest training set size (514 datapoints) and estimated training sizes needed to reach chemical accuracy of 1 kcal/mol and FreeSolv’s “default” experimental error3 of 0.6 kcal/mol. Relative accuracies of different representations are mostly the same as for LES and can be explained analogously to the discussion in Subsec. 4.1; the only difference is pair-determinant FJK being as inaccurate as CM in this setup, possibly meaning that the way FJK captures changes of orbitals induced by SMD is disproportionately affected by numerical noise. We also note that introduction of the E\mathcal{E} layer (22) only noticeably affected prediction errors in the case of cMBDF. Unlike LES, FreeSolv has been used as a benchmark in numerous ML studies, of which several have reported prediction errors of the order of experimental uncertainty [15, 76, 77, 78, 79, 80, 81, 60, 82, 83]. In the context of this work, particularly interesting are the MAEs of 0.417 and 0.359 kcal/mol of [79] and [81] (which are also the lowest MAEs obtained with a ML algorithm for FreeSolv so far) obtained by first training a model on abundant computational data and then finetuning it [84] for FreeSolv. While similar capitalization on finetuning is impossible in the current MSORF framework, we note a possible avenue to circumvent the problem in § 5. Our results also improve on errors previously reported for KRR-based approaches: Ref. [12] reported MAE of 0.68 kcal/mol for FML [14] approach combined with FCHL19 representation, while Ref. [60] reported MAE of 1.7 kcal/mol for combination of SOAP descriptors with KRR. Possible reasons for the improvement are discussed in Subsection 4.3.

methodmax. NtrainN_{\mathrm{train}} MAEmax. NtrainN_{\mathrm{train}} RMSEest. NtrainN_{\mathrm{train}} for target MAE 0.3 eVest. NtrainN_{\mathrm{train}} for target MAE 0.2 eV
aSLATM0.19±0.010.19 \pm0.010.27±0.030.27 \pm0.0389365
FCHL190.17±0.020.17 \pm0.020.25±0.040.25 \pm0.0474292
cMBDF0.20±0.020.20 \pm0.020.29±0.040.29 \pm0.0488448
SOAP0.23±0.010.23 \pm0.010.33±0.030.33 \pm0.03151-
FJK0.27±0.040.27 \pm0.040.36±0.060.36 \pm0.06322-
FJK (s. det.)0.23±0.020.23 \pm0.020.34±0.030.34 \pm0.03240-
CM0.38±0.020.38 \pm0.020.49±0.030.49 \pm0.03--
SLATM0.20±0.010.20 \pm0.010.29±0.040.29 \pm0.04116-

Table 2. Summary of results for machine learning EoxE_{\mathrm{ox}} from [10], namely MAE and RMSE for maximum training set size NtrainN_{\mathrm{train}} of 474, and values of NtrainN_{\mathrm{train}} needed to reach MAE of 0.3 and 0.2 eV as estimated from linear interpolation of the log-log plots of MAE vs. NtrainN_{\mathrm{train}}.

Comparison of machine learning with ensembles or minimum energy conformers

On a closing note, we evaluate how much representing a molecule with a conformer ensemble rather than a single conformer actually affected ML results presented in this work, using as examples combinations of MSORF representations that had demonstrated best and worst performance, namely aSLATM, FCHL19, and CM. For these representations we reran the same workflow as in the previous subsections, but for each ensemble we only considered the conformer with the lowest MMFF94 energy. For CM we used single F\mathcal{F} layer (i.e. we used simple SORF), for aSLATM and FCHL we used FNSTF\mathcal{F}N S T\mathcal{F} with LES and STFS T\mathcal{F} with FreeSolv. The resulting learning curves are presented in Figure 3, their properties summarized in Tables 4 and Table 5. We observe that the difference between representing a molecule with an ensemble and one optimal conformer is of the order of statistical error, and is significantly less pronounced than differences between MAEs observed in this work with different conformer representations. This implies that, at least for the quantities considered in this work, choosing how to represent molecular geometry is more important than how many molecular geometries are used. Another implication is that our demonstration of lower MAEs for predicting EhydE_{\mathrm{hyd}} of FreeSolv compared to KRR values obtained with SOAP in Ref. [60] is likely due to a different hyperparameter optimization procedure and a different choice of kernel function. For the minor improvement over MAEs obtained with FML [14] and FCHL19 (as reported in Ref. [12]), the cause is likely a combination of a different hyperparameter optimization procedure and statistical noise introduced to the ensemble representation in FML due to averaging over a molecular dynamics trajectory.

methodmax. NtrainN_{\mathrm{train}} MAEmax. NtrainN_{\mathrm{train}} RMSEest. NtrainN_{\mathrm{train}} for target MAE 1.0 kcal/molest. NtrainN_{\mathrm{train}} for target MAE 0.6 kcal/mol
CM1.62 ±\pm 0.202.41 ±\pm 0.40--
aSLATM0.50 ±\pm 0.060.88 ±\pm 0.1877366
FCHL190.49 ±\pm 0.050.81 ±\pm 0.1478301

Table 5. Summary of results for machine learning EhydE_{\mathrm{hyd}} of the FreeSolv dataset while representing molecules with minimum energy conformers labeled analogously to Table 2.

Image not extracted

Figure 2. Mean Absolute Error (MAE) of predictions of EhydE_{\mathrm{hyd}} of the FreeSolv dataset obtained by combining MSORF with several molecular representations for different training set sizes NtrainN_{\mathrm{train}}. “S. det.” next to FJK entries indicates single Slater determinant was used, “m.-ext.” indicates using the “mixed-extensive” layer E\mathcal{E} (22).

Conclusions and outlook

We have devised a general SORF-based procedure for generating physically motivated molecular representations from Boltzmann ensembles of conformers and demonstrated its utility for predicting experimental values for molecular properties. The method both demonstrates systematic decrease of prediction error with increased training set size and reaches experimental accuracy with hundreds of molecules as training set. At the

methodmax. NtrainN_{\mathrm{train}} MAEmax. NtrainN_{\mathrm{train}} RMSEest. NtrainN_{\mathrm{train}} for target MAE 1.0 kcal/molest. NtrainN_{\mathrm{train}} for target MAE 0.6 kcal/mol
aSLATM0.50±0.050.50 \pm0.050.87±0.170.87 \pm0.1777332
aSLATM (m.-ext.)0.51±0.050.51 \pm0.050.97±0.200.97 \pm0.2079336
FCHL190.49±0.050.49 \pm0.050.83±0.130.83 \pm0.1371309
FCHL19 (m.-ext.)0.49±0.030.49 \pm0.030.83±0.130.83 \pm0.1379277
cMBDF0.63±0.050.63 \pm0.051.27±0.281.27 \pm0.28100-
cMBDF (m.-ext.)0.53±0.050.53 \pm0.050.92±0.120.92 \pm0.12145425
SOAP0.76±0.050.76 \pm0.051.15±0.101.15 \pm0.10228-
SOAP (m.-ext.)0.75±0.040.75 \pm0.041.16±0.111.16 \pm0.11232-
FJK1.47±0.151.47 \pm0.152.33±0.122.33 \pm0.12--
FJK (s. det.)0.75±0.060.75 \pm0.061.27±0.121.27 \pm0.12272-
FJK (m.-ext.)1.64±0.021.64 \pm0.022.36±0.092.36 \pm0.09--
FJK (m.-ext., s. det.)0.80±0.180.80 \pm0.181.33±0.371.33 \pm0.37306-
CM1.58±0.171.58 \pm0.172.37±0.482.37 \pm0.48--
SLATM0.55±0.050.55 \pm0.050.95±0.080.95 \pm0.08153466

Table 3. Summary of results for machine learning EhydE_{\mathrm{hyd}} of the FreeSolv dataset labeled analogously to Table 2; note that for FreeSolv the maximum training set size was 514 molecules.

Image not extracted

Figure 3. Comparison of MAEs obtained by representing molecules either with a conformer ensemble or the lowest energy conformer (“min. conf.”) for different representations and training set sizes NtrainN_{\mathrm{train}}. EoxE_{\mathrm{ox}} and EhydE_{\mathrm{hyd}} are obtained from [10] and FreeSolv dataset.

methodmax. NtrainN_{\mathrm{train}} MAEmax. NtrainN_{\mathrm{train}} RMSEest. NtrainN_{\mathrm{train}} for target MAE 0.3 eVest. NtrainN_{\mathrm{train}} for target MAE 0.2 eV
CM0.39 ±\pm 0.030.50 ±\pm 0.04--
aSLATM0.19 ±\pm 0.010.28 ±\pm 0.0485383
FCHL190.17 ±\pm 0.010.25 ±\pm 0.0374301

Table 4. Summary of results for machine learning EoxE_{\mathrm{ox}} from Ref. [10] while representing molecules with minimum energy conformers labeled analogously to Table 2.

same time, we did not observe a significant difference between representing a molecule with an ensemble of conformers or a single conformer, implying that, at least for the quantities considered in this work, how conformers are represented is a more important question than how many of them are used.

Prediction errors observed in this work were comparable to state-of-the-art achieved with other ML approaches, despite generating the underlying conformer representations using parameters that (with the exception of SOAP) had not been optimized for the quantities of interest. One possible way to improve on this is to generate conformer representations with another NN, train the combined model on computational data, and then re-optimize the MSORF part of the model on experimental data, essentially finetuning the lowest layers of the combined model. This would bridge the gap between MSORF and transfer learning techniques, which have shown promise when tested on FreeSolv in Refs. [85, 79, 86, 81]. Another possible direction could be going beyond the simple expressions for weighting contributions of different conformers, atoms, or localized orbitals used in this work to more complicated expressions gleaned from explainable artificial intelligence applied to ab initio calculations [87]. In both cases, it would be possible to capitalize on additional ab initio data while keeping MSORF grounded in KRR formalism. Lastly, while we focused on combining representations of conformers into representations of molecular ensembles, MSORF could analogously combine representations of molecular ensembles into representations of mixtures of molecules. This could make MSORF models useful for exploring chemical spaces of mixtures, a problem relevant for several industrial applications that involve using organic liquids [88, 89, 90].

Code availability

The MSORF algorithm has been made part of the QML2 code [57], with the implementation being based on Numpy [91], Numba [92], and SciPy [93]. The version used in this work together with the ML scripts has been uploaded as the 0.1.6 release at the QML2’s repository.

Supplementary Data

Contains data analogous to the one presented in § 4, but with LES results obtained with shifting EoxE_{\mathrm{ox}} by a constant and FreeSolv results obtained with shifting EhydE_{\mathrm{hyd}} by a stoichiometry-dependent term. Also present are text files containing all MAEs and RMSEs for different representations, training set sizes, and random generator seeds.

Acknowledgments

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957189 (BIG-MAP) and No 957213 (BATTERY 2030+).

https://github.com/qml2code/qml2

Hyperparameter optimization

Our hyperparameter optimization procedure is based on optimizing leave-one-out cross-validation errors, i.e. prediction errors for a point given a model had been trained on all other points in the dataset. Leave-one-out errors Δq⃗\Delta\vec{q} can be efficiently calculated using the expression derived in [94], which in our context reads

Δqi=qir−qi1−Z⃗iTK~−1Z⃗i(26)\Delta q_i = \frac{q_i^{r}-q_i}{1-\vec{Z}_i^{T}\widetilde{K}^{-1}\vec{Z}_i} \tag*{(26)}

where q⃗r\vec{q}^{r} are predictions for training points of the model trained on the entire training set, K~:=ZTZ+λINf\widetilde{K} := Z^{T}Z+\lambda I_{N_f}. Note that

q⃗r=ZK~−1ZTq⃗.(27)\vec{q}^{r}=Z\widetilde{K}^{-1}Z^{T}\vec{q}. \tag*{(27)}

Ideally we would want to find hyperparameters that minimize MAE corresponding to Δq⃗\Delta\vec{q}, but MAE’s discontinuous derivatives would make it difficult to use the gradient during optimization. We thus considered instead using Huber or LogCosh loss functions, both of which can be thought of as “smoothed” versions of MAE with continuous derivatives, but in both cases the smoothness is controlled by parameters that are not straightforward to estimate without knowing beforehand the expected magnitude of model’s errors. We therefore introduced LSCLCL_{\mathrm{SCLC}}, a self-consistent version of LogCosh loss function whose smoothness is controlled by a smoothness parameter γ\gamma, and focused on optimizing

Lhyp:=LSCLC2(Δq⃗,γ).(28)L_{\mathrm{hyp}} := L_{\mathrm{SCLC}}^{2}(\Delta\vec{q},\gamma). \tag*{(28)}

While the full definition and derivation of LSCLCL_{\mathrm{SCLC}} is provided in Appendix B, here we note that LSCLC2L_{\mathrm{SCLC}}^{2} is a convex loss function w.r.t. the error values and exhibits a global minimum whose MAE has a relative difference from the global minimum of MAE that does not exceed γ\gamma.

We perform optimization of hyperparameters that can be divided into three groups. The first one are the σ\sigma hyperparameters mentioned throughout Subsec. 2.2 that affect calculation of Z\mathbf{Z}. The second is λ\lambda that for fixed Z\mathbf{Z} affects calculation of K~\widetilde{\mathbf{K}}. The third and last, which was mentioned in § 3, are shifts that can be applied to quantities q⃗\vec{q}. For the latter, we would attempt to decrease LhypL_{\mathrm{hyp}} by adding to q⃗\vec{q} a constant factor of qshiftq^{\mathrm{shift}} if the quantity of interest is intensive. If the quantity is extensive, for molecule AiA_i the quantity qiq_i will be modified by adding

qishift,tot=∑j=1Naqshift(nj(Ai))(29)q_i^{\mathrm{shift,tot}} = \sum_{j=1}^{N_a} q^{\mathrm{shift}}(n_j^{(A_i)}) \tag*{(29)}

qjshift(ni(Ai))q_j^{\mathrm{shift}}(n_i^{(A_i)}) is hyperparameter shift specific to nuclear charge ni(Ai)n_i^{(A_i)}. Note that for fixed σ\sigma hyperparameters and λ\lambda, Δq⃗\Delta\vec{q} can be written as a linear function of qshiftq^{\mathrm{shift}} or q⃗shift,tot\vec{q}^{\mathrm{shift,tot}} that can be evaluated with O(Ntrain)O(N_{\mathrm{train}}) operations. Linearity in this situation means that LhypL_{\mathrm{hyp}} (A.3) can be minimized with a quasi-Newton [95] method; in this work the implementation of Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm (L-BFGS) [96, 97] in SciPy was used.

Optimizing λ\lambda for fixed σ\sigma hyperparameters can also be done with relatively small computational cost. To see that we start with the SVD of Z\mathbf{Z}

Z=UZDZVZT,(30)\mathbf{Z} = \mathbf{U}_{Z}\mathbf{D}_{Z}\mathbf{V}_{Z}^{\mathsf{T}}, \tag*{(30)}

where UZ\mathbf{U}_{Z} is the Ntrain×NtrainN_{\mathrm{train}} \times N_{\mathrm{train}} matrix of left singular vectors, DZ\mathbf{D}_{Z} is Ntrain×NtrainN_{\mathrm{train}} \times N_{\mathrm{train}} diagonal matrix whose diagonal elements are the NtrainN_{\mathrm{train}} singular values, VZ\mathbf{V}_{Z} is the Nf×NtrainN_f \times N_{\mathrm{train}} matrix of right singular vectors. Using Equations (A.2) and (A.5) to rewrite Δq⃗\Delta\vec{q} (A.1) yields

Δqi=(Z⃗itr)T[DZ2+λINtrain]−1(Ztr)Tq⃗−qi1−(Z⃗itr)T[DZ2+λINtrain]−1Z⃗itr(31)\Delta q_i = \frac{(\vec{Z}_i^{\mathrm{tr}})^{\mathsf{T}}[\mathbf{D}_{Z}^{2}+\lambda\mathbf{I}_{N_{\mathrm{train}}}]^{-1}(\mathbf{Z}^{\mathrm{tr}})^{\mathsf{T}}\vec{q}-q_i}{1-(\vec{Z}_i^{\mathrm{tr}})^{\mathsf{T}}[\mathbf{D}_{Z}^{2}+\lambda\mathbf{I}_{N_{\mathrm{train}}}]^{-1}\vec{Z}_i^{\mathrm{tr}}} \tag*{(31)}

where Ztr\mathbf{Z}^{\mathrm{tr}} is the matrix of features transformed into the basis of right singular vectors

Ztr:=ZVZ.(32)\mathbf{Z}^{\mathrm{tr}} := \mathbf{Z}\mathbf{V}_{Z}. \tag*{(32)}

With Ztr\mathbf{Z}^{\mathrm{tr}} and (Ztr)Tq⃗(\mathbf{Z}^{\mathrm{tr}})^{\mathsf{T}}\vec{q} pre-calculated with O(Ntrain2Nf)O(N_{\mathrm{train}}^{2}N_f) operations from Z\mathbf{Z}, calculating (A.6) for a given λ\lambda costs O(Ntrain2)O(N_{\mathrm{train}}^{2}); the same scaling holds for calculating matrices later used in quantity shift optimization. As a result, for each evaluation of Z\mathbf{Z} we optimize λ\lambda (or, to be more exact, ln⁡λ\ln\lambda to enforce λ\lambda’s positivity). Unlike the case of quantity shifts we cannot guarantee the optimization problem is convex and thus approach it in two steps. In the first we use Bayesian Optimization Structure Search (BOSS) package [98] to approximately find the minimum of 1/Lhyp1/L_{\mathrm{hyp}}, inversion done to give minimized function a well-defined limit should a value in Δq⃗\Delta\vec{q} go to infinity due to numerical instability.

The second step is to perform gradient optimization with ”Sequential Least-Squares Quadratic Programming” (SLSQP)4 algorithm [100] as implemented in SciPy.

We thus define L~hyp(σ⃗all,γ)\widetilde{L}_{\mathrm{hyp}}(\vec{\sigma}_{\mathrm{all}},\gamma) as minimum LhypL_{\mathrm{hyp}} obtained by optimizing quantity shifts and λ\lambda values for a set of σ\sigma hyperparameters [i.e. hyperparameters appearing in the definition of F\mathcal{F} (13) and E\mathcal{E} (22)] defined by σ⃗all\vec{\sigma}_{\mathrm{all}} and smoothness parameter γ\gamma. The general idea is to find global minimum of L~hyp\widetilde{L}_{\mathrm{hyp}} for γ→0\gamma\to0 with steepest descent, but setting γ=0\gamma= 0 from the beginning will likely make the optimization loop stuck in a local minimum. Hence we used several gradient descent steps and γ\gamma values to avoid the problem, as documented in Algorithm 1.

Algorithm 1 Gradient optimization of L~hyp\widetilde{L}_{\mathrm{hyp}}.

Require: Initial guess for hyperparameters σ⃗allinit\vec{\sigma}_{\mathrm{all}}^{\mathrm{init}};

for γ=γ1,…,γfinal\gamma= \gamma_1, \ldots, \gamma_{\mathrm{final}} do

for S=S1,…,SfinalS = S_1, \ldots, S_{\mathrm{final}} do

loop

G⃗←dL~hyp(σ⃗all,γ)/d(ln⁡σ⃗all)∣σ⃗all=σ⃗allinit\vec{G} \gets\left.\mathrm{d}\widetilde{L}_{\mathrm{hyp}}(\vec{\sigma}_{\mathrm{all}},\gamma)/\mathrm{d}(\ln\vec{\sigma}_{\mathrm{all}})\right|_{\vec{\sigma}_{\mathrm{all}}=\vec{\sigma}_{\mathrm{all}}^{\mathrm{init}}};

ln⁡σ⃗allnext←ln⁡σ⃗allinit−SG⃗/∣G⃗∣\ln\vec{\sigma}_{\mathrm{all}}^{\mathrm{next}} \gets\ln\vec{\sigma}_{\mathrm{all}}^{\mathrm{init}} - S\vec{G}/|\vec{G}|;

if L~hyp(σ⃗allnext,γ)>L~hyp(σ⃗allinit,γ)\widetilde{L}_{\mathrm{hyp}}(\vec{\sigma}_{\mathrm{all}}^{\mathrm{next}},\gamma) > \widetilde{L}_{\mathrm{hyp}}(\vec{\sigma}_{\mathrm{all}}^{\mathrm{init}},\gamma) then

break loop;

else

σ⃗allinit←σ⃗allnext\vec{\sigma}_{\mathrm{all}}^{\mathrm{init}} \gets\vec{\sigma}_{\mathrm{all}}^{\mathrm{next}};

end if

end loop

end for

end for

For initial guesses, we always set σmix\sigma_{\mathrm{mix}} from E\mathcal{E} (22) as unity. Initial guesses for σ\sigma values appearing in each F\mathcal{F} (13) were calculated as root mean square distance from training set vectors used as procedure’s input to their average. For example, for FJK features that meant that first σ⃗\vec{\sigma} values are guessed, then used to generate the corresponding WF\mathcal{WF} features (17), which are in turn used to estimate the initial guess for σglobal\sigma_{\mathrm{global}} used to generate the FNWF\mathcal{FNWF} features (19). Also, as mentioned in Subsec. 2.1, FJK applies four different values of σ\sigma to four different types of entries in the v⃗(ψ)\vec{v}(\psi) vectors; each of these σ\sigma values was calculated analogously to the case of single σ\sigma hyperparameter, but with only entries rescaled by the corresponding σ\sigma considered and the final result multiplied by 4=2\sqrt{4} = 2.

Lastly, in this work hyperparameter optimization always ran with consequitive γ\gamma values of 0.5, 0.25, and 0.125 and gradient step values of 0.5, 0.25, and 0.125. During L-BFGS and SLSQP algorithm calls we set gradient tolerance parameter as 10−910^{-9}. While performing BOSS optimization the search bounds for ln⁡λ\ln\lambda were −25.0-25.0 and 0.00.0 shifted by ln⁡[Tr⁡(D2)/Ntrain]\ln[\operatorname{Tr}(\mathbf{D}^{2})/N_{\mathrm{train}}], which is approximately the logarithm of average diagonal element of the training kernel matrix K\mathbf{K} in the SORF approximation. Each BOSS run started with putting 16 points on a uniform grid in the search bounds, then doing 16 more evaluations according to the Bayesian optimization algorithm.

Self-consistent versions of Huber and LogCosh loss functions

We start with a general statement about regression loss functions operating on error vectors of dimensionality NN.

Theorem 1. Suppose L(x⃗,β)L(\vec{x},\beta) is a function that

(i) for a fixed β>0\beta>0 is convex and first-order smooth w.r.t. x⃗\vec{x}, as well as strictly positive for any x⃗≠0⃗\vec{x}\ne\vec{0}, while for x⃗=0⃗\vec{x}=\vec{0} it equals zero and its gradient w.r.t. x⃗\vec{x} equals 0⃗\vec{0};

(ii) is positively homogenous of order 1 w.r.t its arguments, i.e. for all x⃗∈RN\vec{x}\in\mathbb{R}^{N} and C>0C>0

L(Cx⃗,βC)=CmL(x⃗,β)(33)L(C\vec{x},\beta C)=C^{m}L(\vec{x},\beta) \tag*{(33)}

with m=1m=1;

(iii) and satisfies the limiting behavior

lim⁡∣x⃗∣/β→∞L(x⃗,β)−MAE⁡(x⃗)β=−1.(34)\lim_{\lvert\vec{x}\rvert/\beta\to\infty}\frac{L(\vec{x},\beta)-\operatorname{MAE}(\vec{x})}{\beta}=-1. \tag*{(34)}

In this case we can define LSCL_{\mathrm{SC}} as the unique solution to the equation

L(x⃗,γLSC)−LSC=0,(35)L(\vec{x},\gamma L_{\mathrm{SC}})-L_{\mathrm{SC}}=0, \tag*{(35)}

where γ\gamma is a dimensionless parameter between 0 and 1. LSCL_{\mathrm{SC}} is a convex function of x⃗\vec{x} that is zero at x⃗=0⃗\vec{x}=\vec{0}, strictly positive everywhere else, and is related to MAE by

1≤MAE⁡(x⃗)LSC(x⃗)≤1+γ.(36)1\le\frac{\operatorname{MAE}(\vec{x})}{L_{\mathrm{SC}}(\vec{x})}\le1+\gamma. \tag*{(36)}

Proof. We first rewrite Euler’s theorem [101] for LL as

∂L(x⃗,β)∂β=1β[L(x⃗,β)−x⃗⋅∂L(x⃗,β)∂x⃗],(37)\frac{\partial L(\vec{x},\beta)}{\partial\beta}=\frac{1}{\beta}\left[L(\vec{x},\beta)-\vec{x}\cdot\frac{\partial L(\vec{x},\beta)}{\partial\vec{x}}\right], \tag*{(37)}

and use convexity of LL to write for β>0\beta>0

∂L(x⃗,β)∂β<1β[L(0⃗,β)+x⃗⋅∂L(x⃗,β)∂x⃗−x⃗⋅∂L(x⃗,β)∂x⃗]=0.(38)\frac{\partial L(\vec{x},\beta)}{\partial\beta}<\frac{1}{\beta}\left[L(\vec{0},\beta)+\vec{x}\cdot\frac{\partial L(\vec{x},\beta)}{\partial\vec{x}}-\vec{x}\cdot\frac{\partial L(\vec{x},\beta)}{\partial\vec{x}}\right]=0. \tag*{(38)}

We now define

FSC(x⃗,l):={L(x⃗,γl)−l,l>0MAE⁡(x⃗)−(γ+1)l,l≤0(39)F_{\mathrm{SC}}(\vec{x},l):=\begin{cases} L(\vec{x},\gamma l)-l, & l>0\\ \operatorname{MAE}(\vec{x})-(\gamma+1)l, & l\le0 \end{cases} \tag*{(39)}

and rewrite Equation (B.3) as

FSC(x⃗,LSC)=0.(40)F_{\mathrm{SC}}(\vec{x}, L_{\mathrm{SC}}) = 0. \tag*{(40)}

For a fixed x⃗\vec{x} FSC(x⃗,l)F_{\mathrm{SC}}(\vec{x}, l) is a monotonously decreasing function of ll [see Equation (B.6)] with a positive limit at l→0l \to0 [see Equation (B.2)], hence Equation (B.8) has a single positive solution, making LSCL_{\mathrm{SC}} well-defined. We also note that FSCF_{\mathrm{SC}} is homogeneous and convex w.r.t. x⃗\vec{x} for a fixed ll. By virtue of Equation (B.2) it is straightforward to check that at l→0l \to0 FSCF_{\mathrm{SC}} is continuous and its left and right derivatives w.r.t. ll both equal −(γ+1)-(\gamma+ 1). According to Ref. [102], these properties of FSCF_{\mathrm{SC}} guarantee that it is convex w.r.t. all arguments, and convex w.r.t. ll for fixed values of x⃗\vec{x}. To check convexity of LSCL_{\mathrm{SC}} we consider y⃗=a⃗+κb⃗\vec{y} = \vec{a} + \kappa\vec{b} and differentiate (B.8) w.r.t. κ\kappa twice to write

d2LSC(y⃗)dκ2=−[∂FSC(y⃗,l)∂l∣l=LSC]−1[dLSC(y⃗)dκ]2∂2FSC(y⃗,l)∂l2∣l=LSC≥0.(41)\frac{\mathrm{d}^{2}L_{\mathrm{SC}}(\vec{y})}{\mathrm{d}\kappa^{2}} = -\left[\left.\frac{\partial F_{\mathrm{SC}}(\vec{y},l)}{\partial l}\right|_{l=L_{\mathrm{SC}}}\right]^{-1}\left[\frac{\mathrm{d}L_{\mathrm{SC}}(\vec{y})}{\mathrm{d}\kappa}\right]^{2}\left.\frac{\partial^{2}F_{\mathrm{SC}}(\vec{y},l)}{\partial l^{2}}\right|_{l=L_{\mathrm{SC}}} \geq0. \tag*{(41)}

This completes the proof that LSCL_{\mathrm{SC}} is convex.

To prove Equation (B.4) recall convexity of LL w.r.t. x⃗\vec{x} for a given β\beta and Equation (B.2), which together imply

L(x⃗,β)≥MAE⁡(x⃗)−β,(42)L(\vec{x},\beta) \geq\operatorname{MAE}(\vec{x})-\beta, \tag*{(42)}

and when combined with ∂L(x⃗,β)/∂x⃗∣x⃗=0⃗=0\left.\partial L(\vec{x},\beta)/\partial\vec{x}\right|_{\vec{x}=\vec{0}}=0 yields

L(x⃗,β)<MAE⁡(x⃗).(43)L(\vec{x},\beta) < \operatorname{MAE}(\vec{x}). \tag*{(43)}

The inequalities (B.10) and (B.11) together with Equation (B.3) prove the inequalities in (B.4).

There are two closing remarks on Theorem 1. Firstly, since LSC(x⃗)L_{\mathrm{SC}}(\vec{x}) is order 1 positively homogeneous w.r.t. x⃗\vec{x}, its nn-th derivative is positively homogeneous of order 1−n1-n, implying that at x⃗=0⃗\vec{x}=\vec{0} the first and second derivatives of LSC(x⃗)L_{\mathrm{SC}}(\vec{x}) have a discontinuity and a singularity. This is the reason we instead use LSC2(x⃗)L_{\mathrm{SC}}^{2}(\vec{x}) (LSCL_{\mathrm{SC}} squared), as it is order 2 positively homogeneous, making its first derivative continous for all x⃗∈RN\vec{x}\in R^{N} and its second derivative bound, though discontinuous at x⃗=0⃗\vec{x}=\vec{0}. Secondly, Equation (B.4) means that using LSC(x⃗)L_{\mathrm{SC}}(\vec{x}) or LSC2(x⃗)L_{\mathrm{SC}}^{2}(\vec{x}) in hyperparameter optimization yields a minimum whose MAE does not differ from the global minimal MAE by more than γ\gamma in relative terms.

To derive dLSC(x⃗)/dx⃗\mathrm{d}L_{\mathrm{SC}}(\vec{x})/\mathrm{d}\vec{x} we differentiate Equation (B.3) w.r.t. x⃗\vec{x} and use Equation (B.5) to write

dLSC(x⃗)dx⃗=LSC(x⃗)∂L(x⃗,γLSC)/∂x⃗x⃗⋅∂L(x⃗,γLSC)/∂x⃗.(44)\frac{\mathrm{d}L_{\mathrm{SC}}(\vec{x})}{\mathrm{d}\vec{x}} = L_{\mathrm{SC}}(\vec{x})\frac{\partial L(\vec{x},\gamma L_{\mathrm{SC}})/\partial\vec{x}}{\vec{x}\cdot\partial L(\vec{x},\gamma L_{\mathrm{SC}})/\partial\vec{x}}. \tag*{(44)}

This expression was used in steepest descent optimization as described in Appendix § 9.

Though it is not, in general, possible to analytically solve equation (B.8), the latter’s solvability with Newton method [95] can be guaranteed by requiring second derivative of L(x⃗,β)L(\vec{x},\beta) w.r.t. β\beta to be continuous and non-zero for β>0\beta> 0 and fixed x⃗\vec{x}. Combined with convexity of FSC(x⃗,l)F_{\mathrm{SC}}(\vec{x},l) w.r.t. ll for fixed x⃗\vec{x}, this makes second derivative of FSC(x⃗,l)F_{\mathrm{SC}}(\vec{x},l) for fixed x⃗\vec{x} strictly positive and thus having positive lower and upper bounds for 0<l≤LSC0 < l \le L_{\mathrm{SC}}. Therefore, a Newton method run starting at LSC=0L_{\mathrm{SC}} = 0 will only consider points in 0<l≤LSC0 < l \le L_{\mathrm{SC}} where second derivative of the r.h.s. of (B.8) has positive lower and upper bounds, giving well-defined upper bound on the number of steps needed to converge the solution to a given accuracy [95]. The derivative required by the Newton method can be obtained by differentiating FSCF_{\mathrm{SC}} (B.7) and applying Euler’s theorem to the case of l>0l > 0

∂FSC(x⃗,l)∂l={[FSC(x⃗,l)−x⃗⋅∂L(x⃗,β)/∂x⃗]/ll>0−(γ+1)l≤0(45)\frac{\partial F_{\mathrm{SC}}(\vec{x},l)}{\partial l} = \begin{cases} \left[F_{\mathrm{SC}}(\vec{x},l)-\vec{x}\cdot\partial L(\vec{x},\beta)/\partial\vec{x}\right]/l & l>0 \\ -(\gamma+1) & l\le0 \end{cases} \tag*{(45)}

We now consider two loss functions satisfying the conditions of Theorem 1. The first one is Huber loss function [35], which we rescale to satisfy (B.2)

LH(xi,β):=1N∑i=1NfH(xi,β)(46)L_{\mathrm{H}}(x_i,\beta) := \frac{1}{N}\sum_{i=1}^{N}f_{\mathrm{H}}(x_i,\beta) \tag*{(46)}
fH(t):={t2/(4β)∣t∣<2β,∣t∣−β∣t∣≥2β.(47)f_{\mathrm{H}}(t) := \begin{cases} t^2/(4\beta) & |t|<2\beta, \\ |t|-\beta& |t|\ge2\beta. \end{cases} \tag*{(47)}

Substitution of Equation (B.14) into Equation (B.3) yields the equation for self consistent Huber loss function LSCHL_{\mathrm{SCH}}

(N+γN≥)LSCH2−LSCH∑≥∣xi∣−14γ∑<xi2=0,(48)(N+\gamma N_{\ge})L_{\mathrm{SCH}}^2-L_{\mathrm{SCH}}\sum_{\ge}|x_i|-\frac{1}{4\gamma}\sum_{<}x_i^2=0, \tag*{(48)}

where ∑≥\sum_{\ge} and ∑<\sum_{<} are shorthands for summation over ii such that ∣xi∣≥2γLSCH|x_i|\ge2\gamma L_{\mathrm{SCH}} and ∣xi∣<2γLSCH|x_i|<2\gamma L_{\mathrm{SCH}}, N≥N_{\ge} is the number of ∣xi∣|x_i| values such that ∣xi∣≥2γLSCH|x_i|\ge2\gamma L_{\mathrm{SCH}}. The sets of ii for both summation signs can be found by sorting ∣xi∣|x_i| values and finding the interval between two consecutive ∣xi∣|x_i| where r.h.s of (B.16) as a function of LSCHL_{\mathrm{SCH}} changes sign. Finding the only positive root of (B.16) yields the final expression

LSCH=12(N+N≥γ)[∑≥∣xi∣+(∑≥∣xi∣)2+(N+γN≥)γ∑<xi2].(49)L_{\mathrm{SCH}}=\frac{1}{2(N+N_{\ge}\gamma)}\left[\sum_{\ge}|x_i|+\sqrt{\left(\sum_{\ge}|x_i|\right)^2+\frac{(N+\gamma N_{\ge})}{\gamma}\sum_{<}x_i^2}\right]. \tag*{(49)}

An alternative way to find LSCHL_{\mathrm{SCH}} without sorting ∣xi∣|x_i| is derived by considering the FSCF_{\mathrm{SC}} arising in this case as a function defined as equaling F~SC(j)\widetilde{F}_{\mathrm{SC}}^{(j)} on each interval of the form (∣xjs∣/(2γ),∣xj+1s∣/(2γ))(|x_j^s|/(2\gamma),|x_{j+1}^s|/(2\gamma)), where x⃗s\vec{x}^s consists of values of x⃗\vec{x} sorted by increasing absolute value and augmented with x0s=0x_0^s=0, and F~SC(j)\widetilde{F}_{\mathrm{SC}}^{(j)} reads

F~SC(j)(x⃗,l)=MAE⁡(x⃗)−(γ+1)l+1N∑j′=1jmax⁡[0,(xj′s)24γl−∣xj′s∣+γl].(50)\widetilde{F}_{\mathrm{SC}}^{(j)}(\vec{x},l)=\operatorname{MAE}(\vec{x})-(\gamma+1)l+\frac{1}{N}\sum_{j'=1}^{j}\max\left[0,\frac{(x_{j'}^s)^2}{4\gamma l}-|x_{j'}^s|+\gamma l\right]. \tag*{(50)}

For fixed x⃗\vec{x} each F~SC(j)\widetilde{F}_{\mathrm{SC}}^{(j)} is a convex monotonously decreasing function, with F~SC(j+1)\widetilde{F}_{\mathrm{SC}}^{(j+1)} differing from F~SC(j)\widetilde{F}_{\mathrm{SC}}^{(j)} by a nonnegative function. This means each F~SC(j)\widetilde{F}_{\mathrm{SC}}^{(j)} has only one positive root w.r.t. ll, which is also smaller than the positive root of F~SC(j+1)\widetilde{F}_{\mathrm{SC}}^{(j+1)}. Uniqueness of LSCHL_{\mathrm{SCH}} also implies that of all the roots of F~SC(j)\widetilde{F}_{\mathrm{SC}}^{(j)}, only one actually lies in the corresponding interval (∣xjs∣/(2γ),∣xj+1s∣/(2γ))(|x_j^{s}|/(2\gamma), |x_{j+1}^{s}|/(2\gamma)). That means LSCHL_{\mathrm{SCH}} can be evaluated similarly to the Newton method, by starting with the initial guess LSCH=0L_{\mathrm{SCH}}=0 and calculating each successive guess with Equation (B.17) with ∑≥\sum_{\geq} and ∑≤\sum_{\leq} evaluated based on the previous guess. The procedure will yield increasing estimates of LSCHL_{\mathrm{SCH}} until the exact value is reached, as signified by ∑≥\sum_{\geq} and ∑≤\sum_{\leq} staying the same as in the previous iteration.

Secondly, we considered the LogCosh loss function [34] which, when rescaled to satisfy (B.2), reads

LLC(x⃗,β):=1N∑i=1Nβln⁡2ln⁡(cosh⁡xiln⁡2β).(51)L_{\mathrm{LC}}(\vec{x},\beta) := \frac{1}{N}\sum_{i=1}^{N}\frac{\beta}{\ln2}\ln\left(\cosh\frac{x_i\ln2}{\beta}\right). \tag*{(51)}

It is straightforward to check that for fixed x⃗\vec{x} the second derivative of LLCL_{\mathrm{LC}} w.r.t. β\beta is never zero, allowing to calculate the self-consistent version of LLCL_{\mathrm{LC}}, dubbed LSCLCL_{\mathrm{SCLC}}, numerically via the Newton method as discussed previously.

We implemented both LSCH2L_{\mathrm{SCH}}^{2} and LSCLC2L_{\mathrm{SCLC}}^{2} in the QML2 code. Though LSCLC2L_{\mathrm{SCLC}}^{2} is significantly more expensive to calculate than LSCH2L_{\mathrm{SCH}}^{2}, we chose the former for our work since its cost is small relative to the cost of finding the SVD of the ZZ matrix (A.5) and its derivatives are infinitely continuous for all x⃗≠0⃗\vec{x}\ne\vec{0} [as can be verified by differentiating (B.3)], with x⃗=0⃗\vec{x}=\vec{0} extremely unlikely to be encountered during hyperparameter optimization. We underline that Equation (B.3) implies that the γ\gamma operates similarly to the way β\beta operates in the definitions of LHL_{\mathrm{H}} (B.14) and LLCL_{\mathrm{LC}} (B.19). Since the β\beta parameter in this context is sometimes described as an absolute error value beyond which a point is considered an outlier, γ\gamma can be described as how large an error is compared to LSCL_{\mathrm{SC}} (i.e., to the “average smoothed error”) to be considered an outlier. As mentioned in Appendix § 9, we found γ\gamma to be a more convenient parameter to regulate in our applications.

As a closing remark, we note that LSCH2L_{\mathrm{SCH}}^{2} and LSCLC2L_{\mathrm{SCLC}}^{2} cannot be straightforwardly used with Stochastic Gradient Descent [103] algorithms since these loss functions are not sums of contributions from different elements of x⃗\vec{x}. While it should be possible to bypass the issue by using LHL_{\mathrm{H}} or LSCL_{\mathrm{SC}} and enforcing Equation (B.3) with a Lagrange multiplier, investigating feasibility and usefulness of such a procedure was beyond the scope of this work.

Supplementary Data: Kernel Ridge Regression for Boltzmann ensembles made easy with Structured Orthogonal Random Features

Konstantin Karandashev

Image not extracted

Figure 1 (PDF p. 30). Mean Absolute Errors (MAE) of predictions of oxidation potentials EoxE_{\mathrm{ox}} in acetonitrile from Ref. [1] obtained by combining MSORF with several molecular representations for different training set sizes NtrainN_{\mathrm{train}}. “FJK (s. det.)” indicates FJK was used with a single Slater determinant. Unlike Figure 1 of the main text, the results were obtained while shifting EoxE_{\mathrm{ox}} by a constant.

Additional results for Section 4 of the main text

Here we present results analogous to the ones in § 4 of the main text, but with LES oxidation potentials shifted by a constant and FreeSolv hydration energies shifted using qishift,totq_i^{\mathrm{shift,tot}} as defined in Equation (A.4) of the main text. The results for LES are presented in Figures 1 and 3 and Table 1 and 3, demonstrating that shifting the quantities did not affect the learning curves significantly. The results for FreeSolv are presented in Figures 2 and 3 and Tables 2 and 4; comparing the table values demonstrates that applying the stochiometry-dependent shift often increased MAE and RMSE for different representations, and the large statistical errors occasionally appearing on the learning curves correspond to cases when the procedure proved numerically unstable.

Image not extracted

Figure 2 (PDF p. 31). Mean Absolute Error (MAE) of predictions of EhydE_{\mathrm{hyd}} of the FreeSolv dataset obtained by combining MSORF with several molecular representations for different training set sizes NtrainN_{\mathrm{train}}. “S. det.” next to FJK entries indicates single Slater determinant was used, “m.-ext.” indicates using the “mixed-extensive” layer E\mathcal{E} [see Equation (22) in the main text]. Unlike Figure 2 of the main text, the results were obtained with shifting EhydE_{\mathrm{hyd}} by a stochiometry-dependent term.

methodmax. NtrainN_{\mathrm{train}} MAEmax. NtrainN_{\mathrm{train}} RMSEest. NtrainN_{\mathrm{train}} for target MAE 1.0 kcal/molest. NtrainN_{\mathrm{train}} for target MAE 0.6 kcal/mol
aSLATM0.61±0.020.61 \pm0.021.41±0.321.41 \pm0.32128-
aSLATM (m.-ext.)0.59±0.040.59 \pm0.041.30±0.311.30 \pm0.31153412
FCHL190.52±0.050.52 \pm0.050.91±0.140.91 \pm0.1493366
FCHL19 (m.-ext.)0.52±0.060.52 \pm0.060.91±0.160.91 \pm0.1698357
cMBDF0.63±0.040.63 \pm0.041.24±0.231.24 \pm0.23140-
cMBDF (m.-ext.)0.55±0.060.55 \pm0.060.91±0.160.91 \pm0.16156466
SOAP0.78±0.040.78 \pm0.041.23±0.111.23 \pm0.11262-
SOAP (m.-ext.)0.79±0.060.79 \pm0.061.28±0.131.28 \pm0.13287-
FJK1.01±0.071.01 \pm0.071.61±0.211.61 \pm0.21--
FJK (s. det.)0.90±0.120.90 \pm0.121.85±0.541.85 \pm0.54312-
FJK (m.-ext.)1.04±0.081.04 \pm0.081.68±0.061.68 \pm0.06--
FJK (m.-ext., s. det.)1.00±0.161.00 \pm0.161.85±0.481.85 \pm0.48347-
CM1.27±0.171.27 \pm0.171.98±0.441.98 \pm0.44--
SLATM0.75±0.230.75 \pm0.232.25±1.922.25 \pm1.92173-

Table 2 (PDF p. 32). Summary of results for machine learning EhydE_{\mathrm{hyd}} of the FreeSolv dataset labeled analogously to Table 1; note that for FreeSolv the maximum training set size was 514 molecules. Unlike Table 3 of the main text, the results were obtained with shifting EhydE_{\mathrm{hyd}} by a stoichiometry-dependent term.

methodmax. NtrainN_{\mathrm{train}} MAEmax. NtrainN_{\mathrm{train}} RMSEest. NtrainN_{\mathrm{train}} for target MAE 0.3 eVest. NtrainN_{\mathrm{train}} for target MAE 0.2 eV
CM0.36±0.010.36 \pm0.010.47±0.020.47 \pm0.02--
aSLATM0.19±0.010.19 \pm0.010.28±0.030.28 \pm0.0392438
FCHL190.17±0.020.17 \pm0.020.25±0.030.25 \pm0.0385314

Table 3 (PDF p. 32). Summary of results for machine learning EoxE_{\mathrm{ox}} from Ref. [1] while representing molecules with minimum energy conformers labeled analogously to Table 1. Unlike Table 4 of the main text, the results were obtained with shifting EoxE_{\mathrm{ox}} by a constant.

methodmax. NtrainN_{\mathrm{train}} MAEmax. NtrainN_{\mathrm{train}} RMSEest. NtrainN_{\mathrm{train}} for target MAE 1.0 kcal/molest. NtrainN_{\mathrm{train}} for target MAE 0.6 kcal/mol
CM1.27±0.171.27 \pm0.171.88±0.351.88 \pm0.35--
aSLATM0.61±0.040.61 \pm0.041.50±0.391.50 \pm0.39132-
FCHL190.52±0.050.52 \pm0.050.88±0.130.88 \pm0.13100372

Table 4 (PDF p. 32). Summary of results for machine learning EhydE_{\mathrm{hyd}} of the FreeSolv dataset while representing molecules with minimum energy conformers labeled analogously to Table 1. Unlike Table 5 of the main text, the results were obtained with shifting EhydE_{\mathrm{hyd}} by a stoichiometry-dependent term.

Image not extracted

Figure 3 (PDF p. 33). Comparison of MAEs obtained by representing molecules either with a conformer ensemble or the lowest energy conformer (“min. conf.”) for different representations and training set sizes NtrainN_{\mathrm{train}}. EoxE_{\mathrm{ox}} and EhydE_{\mathrm{hyd}} are obtained from Ref. [1] and FreeSolv dataset. Unlike Figure 3 of the main text, results for EoxE_{\mathrm{ox}} were obtained with shifting the quantities by a constant and results for EhydE_{\mathrm{hyd}} were obtained with shifting them by a stoichiometry-dependent term.

References

  1. [1]Lee, S, Heinen, S, Khan, D, & Anatole von Lilienfeld, O. (2024) Autonomous data extraction from peer reviewed literature for training machine learning models of oxidation potentials. Mach. Learn.: Sci. Technol. 5, 015052.

Paper details

vlnc:W_79gf6uj6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents