betaFeedback

WorksPhysics and Astronomy

Assessing the Performance of Nonlinear Regression based Machine Learning Models to Solve Coupled Cluster Theory

Abstract

The iteration dynamics of the coupled cluster equations exhibits a synergistic relationship among the cluster amplitudes. The iteration scheme may be viewed as a multivariate discrete-time propagation of nonlinearly coupled equations, which is dictated by only a few principal cluster amplitudes. These principal amplitudes usually correspond to only a few valence excitations, whereas all other cluster amplitudes are enslaved, and behave as auxiliary variables. Staring with a few trial iterations, we employ a supervised machine learning strategy to establish a mapping of the principal and auxiliary amplitudes. We introduce a machine learning-coupled cluster hybrid scheme where the coupled cluster equations are solved only to determine the principal amplitudes, which saves significant computation time. The auxiliary amplitudes, on the other hand, are determined via regression. Few different regression techniques have been introduced to express the auxiliary amplitudes as functions of the principal amplitudes. The scheme has been applied to several molecules in their equilibrium and stretched geometries, and our scheme, with both the regression models, shows a significant reduction in computation time without unduly sacrificing the accuracy.

the principal and auxiliary amplitudes. We introduce a coupled cluster - machine learning hybrid scheme where the coupled cluster equations are solved only to determine the principal amplitudes, which saves significant computation time. The auxiliary amplitudes, on the other hand, are determined via regression. Few different regression techniques have been introduced to express the auxiliary amplitudes as functions of the principal amplitudes. The scheme has been applied to several molecules in their equilibrium and stretched geometries, and our scheme, with all the regression models, shows a significant reduction in computation time over the canonical coupled cluster calculations without unduly sacrificing the accuracy.

Introduction

Coupled cluster theory[1, 2, 3, 4] is a well established method for solving the electronic Schrödinger wave equation for small to medium sized atoms and molecules. CC theory employs an exponential wave operator Ω\Omega, that brings in the effects of the excited Slater determinants on to the reference ground state wavefunction, which is usually taken to be the Hartree-Fock determinant. The wave operator Ω\Omega is chosen as Ω=eT\Omega= e^{T}, where TT is the sum of all possible many-body hole to particle excitation operators. In the most common cases, TT consists of one and two-body hole-particle excitation operators. The resultant theory, known as the CC with singles and doubles excitations (CCSD) predicts accurate quite energy for molecules with predominantly single reference character. The amplitudes corresponding to the excitation operators, which are the unknown quantities, are determined by projecting the similarity transformed effective Hamiltonian G=e−THeT\mathcal{G} = e^{-T} H e^{T} against the excited state determinants. The correlated ground state energy is calculated as the expectation value of the effective Hamiltonian with respect to the chosen reference function, Ecorr=⟨ϕHF∣Heff∣ϕHF⟩=⟨ϕHF∣e−THeT∣ϕHF⟩E_{\mathrm{corr}} = \langle\phi_{HF} | H_{\mathrm{eff}} | \phi_{HF} \rangle= \langle\phi_{HF} | e^{-T} H e^{T} | \phi_{HF} \rangle. Due to the exponential nature of the wave operator, the similarity transformed Hamiltonian, HeffH_{\mathrm{eff}} is highly non-linear in TT, and hence one employs iterative scheme to solve these equations. For CCSD, the most expensive computational step scales as no2nv4n_{o}^{2}n_{v}^{4} per iteration, where non_o is the number of the occupied orbitals, and nvn_v is the number of the virtual orbitals in the chosen reference determinant. This often makes the theory prohibitively time consuming for large systems. The theory since it’s inception has seen many developments to increase it’s accuracy with a reduced computational scaling. Due to the iterative nature of the solutions, there have been significant efforts to accelerate the convergence [5, 6, 7, 8] or scale down the steep computational scaling associated with the solution scheme.[9, 10, 11, 12] One may thus look for an iterative scheme where the expensive no2nv4n_o^2n_v^4 scaling may be bypassed, at least partially, which is likely to save a lot of computation time. This may be achieved by exploiting the nonlinearity of the iterative solution scheme. In line with this, some of the present authors imbibed ideas from nonlinear dynamics and synergetics [13, 14, 15] to demonstrate that not all the cluster amplitudes are equally important in the nonlinear iteration scheme. Based on the magnitude of the amplitudes, the authors classified the amplitudes into “unstable master amplitudes” (later to be referred to as the principal amplitudes) and “stable slave amplitudes” (later to be referred to as the auxiliary amplitudes). Borrowing the key concepts from nonlinear discrete time-series, [16, 17] coupled with principles imported from the areas of Synergetics, a Machine Learning (ML) [18] based hybrid numerical scheme was developed to establish a relationship between the two classes of amplitudes. This effectively reduces the independent degrees of freedom to accelerate the overall iteration process. In the resulting scheme, only a few initial iterations scale as no2nv4n_o^2n_v^4, while most of the other iterations scale significantly less. This saves a lot of computation time, as was demonstrated by the authors. Complementary to that, one may employ an adiabatic approximations based on the difference in the time scale of relaxation of various cluster amplitudes during the iteration, where one may formally reduce the scaling of the iterative scheme by at least one order of magnitude, without undue sacrifice of the accuracy.

In section II, we discuss the essential aspects of the newly developed iteration scheme, where we reiterate the existence of a circularly causal relationship among the principal and the auxiliary amplitudes. We will show how a machine learning strategy can be employed based on the circular causality to simplify coupled cluster calculations. In section III we discuss each of the machine learning models and their performance in terms of accuracy and time requirements. We will discuss four models, out of which, three are primarily regression based, and the last one is a classification based model. In section IV, we conclude our findings with overarching remarks on the performance of the models.

Nonlinearity in the CC iteration scheme from synergistic perspective, and overview of a new algorithm based on the circular causality relationship:

There exists a relationship among the set of most significant and less significant cluster amplitudes during the nonlinear iteration process. While the significant amplitudes dictate the iteration process in the macroscopic sense, other less significant amplitudes are simply enslaved, and their variation gets suppressed. These significant amplitudes, which are large in magnitude and mostly labelled by the active orbitals, are interchangeably termed as the principal or the driver amplitudes. From the viewpoint of the nonlinear discrete time dynamics, they are the unstable modes and are denoted by tLt^{L}. The space spanned by them are termed as the large amplitude subset (LAS), having dimension nLn_{L}. On the other hand, the amplitudes with lesser magnitudes than a pre-defined threshold have significantly less importance in the iteration dynamics, and they are the stable modes of the time-discrete iteration process. They are termed as the auxiliary amplitudes, denoted by tSt_{S}, which span the small amplitude subset (SAS) with dimension nSn_{S}. Note that nS≫nLn_{S} \gg n_{L}. The principal and the auxiliary amplitudes are known as master and slave variables in Synergetics. It was demonstrated that there exists a mapping FF, such that

F:tkL→tkS(1)F : t_{k}^{L} \to t_{k}^{S} \tag*{(1)}

where kk is the discrete time iteration step. We established that one may numerically exploit this mapping to reduce the computational time required for the coupled cluster calculations. In order to establish the mapping FF, one may modify the CC iteration scheme based on

the circular causality that exists between the principal and the auxiliary amplitudes. This is shown in Figure 1. However, the circular causality presumes that the map FF is known. In order to establish the mapping, one may use the routine CC framework to calculate both the sets of amplitudes, tLt^{L} and tSt^{S}. After a certain number of training iterations, the principal amplitudes are extracted and they are trained to map on to the auxiliary amplitudes for once. This mapping in our current scheme is done via various kernelized Regression based ML models where the principal amplitudes are taken as the independent input variables and the auxiliary amplitudes are the dependent variables that are predicted. For each individual molecular/atomic calculation, this training needs to be done right after the initial iterations only once, and there is no requirement of any previously computed data. Our model can be trained on the amplitudes on the fly during the iteration process. Depending on the desired level of accuracy, we will present four different regression models that have their own pros and cons, and we will discuss those in the sections to follow, along with their exhaustive numerical applications. Once the amplitudes are trained and the functional form of FF which maps tLt^{L} on to tSt^{S} is numerically established, the algorithm runs via the circular causality loops as shown in Figure 1. This consists of two steps: in step-I, all the auxiliary amplitudes are determined from the principal amplitudes of the same iteration time step via the ML regression model. In step-II, the principal amplitudes are updated via the feedback coupling of both sets of the amplitudes. This circular causality loop continues till the principal amplitudes (and hence the auxiliary amplitudes, due to the fixed functional dependence) converge. Note that in step-II, only the principal amplitudes are determined through the exact CC equations, although the auxiliary amplitudes that are plugged in are obtained via ML. The computational scaling of this step is nLnonv2n_{L}n_{o}n_{v}^{2} as demonstrated by Agarawal et. al.[19] Since nLn_{L} is only a very small fraction even compared to nonvn_{o}n_{v}, this step has a significant reduction in scaling from the usual no2nv4n_{o}^{2}n_{v}^{4} of the conventional scheme. Step-I involves only a single matrix multiplication, and hence it scales negligibly compared to step-II. We will henceforth refer to this algorithm as the hybrid CC-ML scheme.

Principal Amplitudes ($t_L$): Unstable Modes (Large Amplitude Subset); Auxiliary Amplitudes ($t_S$): Stable Modes (Small Amplitude Subset); Step I - ML Prediction; Step II - CC reduced

Figure 1. General structure of the circular Causality loop: the principal amplitudes are mapped on to the auxiliary amplitudes using a ML model (step-I), and their feedback coupling to obtain the updated set of the driver amplitudes via step-II. This loop is constructed following the initial iteration cycles and training of the relevant ML model.

The computational time requirement by the overall hybrid CC-ML scheme is governed by two major factors: (1) the number of conventional iterations (mm) used to train the model to determine FF, and (2) the dimension of the LAS, nLn_{L}. Simple calculations show that over 90% of the overall calculation time is consumed in generation of data through the initial iterations and training of model. Thus, there is a lot of scope to play around with different ML models that can predict the auxiliary amplitudes with fewer training iterations, and minimize the required dimension of the the LAS, nLn_{L}, without unduly sacrificing the accuracy.

In a previous publication, Agarawal et. al.[19] had studied the efficacy of KRR based ML model to represent FF. This was studied for a few pilot molecules in their equilibrium and away from equilibrium geometries, and it produced highly accurate energies (of the order of micro-Hartree (μH\mu H), compared to the canonical CC calculations). The authors also systematically studied the time requirements necessary for the overall process, and as expected, it was much less than the conventional CCSD scheme. In fact, in order to obtain a sub-μH\mu H accuracy in energy, the time requirement for the algorithm was similar to lower to that of a DIIS accelerated CC scheme. Thus, the pilot study showed extremely promising results, and it required further benchmarking. In this manuscript, we have thoroughly studied the performance of KRR model, both in terms of energy accuracy and time requirement, for about 60 molecular applications with various degrees of electronic complexity. We would also introduce a few more regression models, which would further greatly reduce the requirements of the numbers of initial training iterations, and the LAS dimension. All the models will be thoroughly benchmarked with numerous numerical applications.

Discussion on various regression based machine learning models and their benchmarking studies

In this section we will describe various regression based ML models that have been employed in this work. We have studied four different regression models to represent the forward mapping (step-I) of the circular causality loop. The models which we would be considering in this manuscript are:

  • Kernel Ridge Regression model (KRR)

  • Customized Kernel Regression model (CKR)

  • Polynomial-Kernel Ridge Regression model (PKR)

  • K-Nearest Neighbors model (KNN)

Following the initial iteration cycles, the model is trained only once for each molecule, after which the iteration runs through the circular causality loop. In all the ML models, the input parameters are the principal amplitudes, tLt^{L}, which are taken to be the independent variables, and the output is the set of auxiliary amplitudes tSt^{S} which are predicted. While CKR and PKR models superficially have the similar working philosophy to that of KRR, they differ from KRR in the detailed structure of the Kernel matrix. Hence, the underlying mathematical structure of all these three models will be discussed together in the section that deals with KRR. In the subsequent sections dealing with CKR and PKR models, we will only present the Kernelization techniques which distinguish them from KRR. In the following, we will briefly present the working principles of all the different models and would also present the benchmarking applications along with, in comparison to the canonical CCSD results. For each molecule, both the canonical CCSD and hybrid CC-ML calculations were done in the same machine for proper comparison of the relative computation timing.

The Kernel Ridge Regression Model:

Formal aspects and structure of the working equations:

In this subsection, we will discuss the main ideas behind the Kernel Ridge Regression model. As mentioned before, this model also forms the base for two other models under consideration, namely the Customized Kernel Regression and the Polynomial Kernel Ridge Regression. The KRR model is strictly based on the linear regression algorithm. A linear regressor fits the dependent auxiliary amplitudes as a linear function of the independent principal amplitudes. The training of model essentially translates to finding appropriate coefficients in a linear function of principal amplitudes to give auxiliary amplitudes. As one would expect, the accuracy of the function would be directly dependent on the number of training iterations mm.

As mentioned previously, during the training cycles, the CC equations are iterated in the full space spanned by {tL⊕tS}\{t^{L} \oplus t^{S}\} amplitudes and these exact amplitudes are used to train the model. During the training, the entire space of {tL⊕tS}\{t^{L} \oplus t^{S}\} is processed into two matrices of TST^{S} and TLT^{L}, where TLT^{L} is defined as

TL=(1t11Lt21L⋯tnL1L1t12Lt22L⋯tnL2L1t13Lt23L⋯tnL3L⋯⋯⋯⋯⋯1t1mLt2mL⋯tnLmL)(2)T^{L} = \begin{pmatrix} 1 & t^{L}_{11} & t^{L}_{21} & \cdots& t^{L}_{n_{L}1} \\ 1 & t^{L}_{12} & t^{L}_{22} & \cdots& t^{L}_{n_{L}2} \\ 1 & t^{L}_{13} & t^{L}_{23} & \cdots& t^{L}_{n_{L}3} \\ \cdots& \cdots& \cdots& \cdots& \cdots\\ 1 & t^{L}_{1m} & t^{L}_{2m} & \cdots& t^{L}_{n_{L}m} \end{pmatrix} \tag*{(2)}

. Here each row signifies the nLn_{L} independent principal cluster amplitudes for a given iteration. With mm number of training iterations performed to construct the TLT^{L} matrix, it is of the dimension m×(nL+1)m \times(n_{L} + 1). The extra column in TLT^{L} is added to take care of the intercept term. A generalized linear fit can be written in the form of TS=TLβT^{S} = T^{L}\beta, where β\beta is the coefficient matrix of the linear functions of tLt^{L}. It can be trivially defined as:

β=(β01β02β03⋯β0nSβ11β12β13⋯β1nSβ21β22β23⋯β2nS⋯⋯⋯⋯⋯βnL1βnL2βnL3⋯βnLnS)(3)\beta= \begin{pmatrix} \beta_{01} & \beta_{02} & \beta_{03} & \cdots& \beta_{0n_{S}} \\ \beta_{11} & \beta_{12} & \beta_{13} & \cdots& \beta_{1n_{S}} \\ \beta_{21} & \beta_{22} & \beta_{23} & \cdots& \beta_{2n_{S}} \\ \cdots& \cdots& \cdots& \cdots& \cdots\\ \beta_{n_{L}1} & \beta_{n_{L}2} & \beta_{n_{L}3} & \cdots& \beta_{n_{L}n_{S}} \end{pmatrix} \tag*{(3)}

where nSn_{S}, as mentioned before, is the number of the elements in the dependent SAS amplitudes for a given iteration. Thus the matrix TST^{S} is of the dimension m×nSm \times n_{S}, defined as:

TS=(t11St21S⋯tnS1St12St22S⋯tnS2St13St23S⋯tnS3S⋯⋯⋯⋯t1mSt2mS⋯tnSmS)(4)T^{S} = \begin{pmatrix} t^{S}_{11} & t^{S}_{21} & \cdots& t^{S}_{n_{S}1} \\ t^{S}_{12} & t^{S}_{22} & \cdots& t^{S}_{n_{S}2} \\ t^{S}_{13} & t^{S}_{23} & \cdots& t^{S}_{n_{S}3} \\ \cdots& \cdots& \cdots& \cdots\\ t^{S}_{1m} & t^{S}_{2m} & \cdots& t^{S}_{n_{S}m} \end{pmatrix} \tag*{(4)}

Starting from a guess coefficient matrix, β^\hat{\beta}, the optimized coefficients may be obtained by minimizing the loss function ηTη\eta^{T}\eta. Here η\eta is the error function defined by: η=TS−T^S\eta= T^{S} - \hat{T}^{S}, where T^S=TL⋅β^\hat{T}^{S} = T^{L} \cdot\hat{\beta} is the predicted auxiliary amplitude matrix obtained during the training cycles. With mm training iterations, β^\hat{\beta} can be optimized to give β\beta, which will be used for prediction in the subsequent iterations.

One may further increase the power of the independent principal amplitudes to quadratic, cubic etc. to capture the nonlinear features of the trajectory. This leads to increase the dimension on the independent variable space, and hence there’s a better chance of getting a better fit. One thus increases the independent variable space from a dimension of nLn_{L} to a higher dimensional space by including polynomial forms of tLt^{L} and treat each terms as independent variable. Let the higher dimension space with non linear terms be connected to the feature vectors by a map ϕ\phi, which can be defined as:

ϕ:{tμL,μ∈nL}→{tμL,tμLtνL,…,(tμL)d,μ,ν∈nL}(5)\phi: \{t^{L}_{\mu}, \mu\in n_{L}\} \to\{t^{L}_{\mu}, t^{L}_{\mu}t^{L}_{\nu}, \ldots, (t^{L}_{\mu})^{d}, \mu,\nu\in n_{L}\} \tag*{(5)}

where dd is the degree of the polynomial. The loss function now becomes β^μ=ϕT(ϕϕT)−1(TμS)\hat{\beta}_{\mu} = \phi^{T}(\phi\phi^{T})^{-1}(T^{S}_{\mu}). The whole operation, if done directly, is expensive and one usually makes use of the Kernelization technique instead, which allows evaluation the expression without explicit knowledge of the function ϕ\phi.

According to the Mercer theorem, [20] one may define the Kernel Function K=ϕϕTK = \phi\phi^{T} for every symmetric positive definite matrix. Thus, the loss function now becomes β^μ=ϕT(K)−1(TμS)\hat{\beta}_{\mu} = \phi^{T}(K)^{-1}(T^{S}_{\mu}). After sufficient training of the model, the auxiliary SAS amplitudes for the ii-th iteration (i>mi > m) (tiSt^{S}_{i}) are predicted using only the optimised coefficient matrix and the LAS amplitude vector of the same iteration (tiLt^{L}_{i}).

tμ,iS(predicted)=ϕ({tiL})β^μ(6)t^{S}_{\mu,i}(predicted) = \phi(\{t^{L}_{i}\})\hat{\beta}_{\mu} \tag*{(6)}

Note that the effect of tSt^{S} or previous iterations is included in the β^\hat{\beta} matrix.

To bypass the usage of ϕ\phi, a new kernel function KLK_{L} is defined, which takes all the previous training amplitudes, and the tiLt_i^L to predict the tiSt_i^S. Thus, KL=ϕ.(ϕ(tiL))TK_L=\phi.(\phi(t_i^L))^T, where the right most ϕ\phi appearing in the above equation is a function of tiLt_i^L.

tpredS=(KL)TK−1TS(7)t_{\mathrm{pred}}^S=(K_L)^T K^{-1}T^S \tag*{(7)}

Here the subscript indicates that these are the predicted SAS amplitudes.

To avoid unphysical underfitting or overfitting due to erroneous weight in the training data set, a regularization parameter is introduced which penalizes the model each time a certain term gets unphysical weight. Thus one may modify Eq (7) by adding a regularization term as

tpredS=(KL)T(K+λI)−1TS(8)t_{\mathrm{pred}}^S=(K_L)^T(K+\lambda I)^{-1}T^S \tag*{(8)}

Following the convention, we denote the regularization parameter with α\alpha. Here λ=α/2\lambda=\alpha/2. The value of λ(α)\lambda(\alpha) manages overfitting at the cost of rate of learning. A small value of α\alpha trains a model very quickly and often overfits the data points, which could result in slight inaccuracies with few training data sets. On the other hand, a large α\alpha slows down the learning process, and the model takes a larger number of training data sets to produce accurate results. Thus the value of α\alpha requires tuning.

In Eq. (8), we note that the quantity (K+λI)−1TS(K+\lambda I)^{-1}T^S can be computed only once for all after the training data set is produced. Thus all the tiSt_i^S amplitudes may be computed each iteration via a single matrix multiplication of (KL)T(K_L)^T and (K+λI)−1TS(K+\lambda I)^{-1}T^S instead of algebraically or diagrammatically solving the coupled cluster equations, which results in savings in the overall computational time. We must also note that all these processes are already standardized in scikit, [21] and one has to only supply TLT^L and TST^S for training.

In an earlier paper, we had presented a few pilot molecular applications to show the potential of the hybrid CC-ML(KRR) method. However, a more thorough analysis of the model on a large number of molecules with varying electronic complexity is necessary to show the efficacy of the method with statistical significance. We have employed the KRR model to study the ground state energetics of about 60 cases with different molecules/geometries/basis sets. In the next subsection, we will study the efficacy of the KRR model in terms of accuracy in predicted energy compared to CCSD, and the associated time requirements.

Assessment of the performance of KRR model

The accuracy of any regression method largely depends on the number of iterations used to train the model. As expected, larger the training iterations employed, better would be the accuracy. In the first row of Figure 2, we have plotted the difference in the energy obtained via canonical CCSD and our hybrid CC-ML(KRR) model for 6, 7, and 8 training iterations for 60 different combinations of molecules, basis sets and geometries. With 6 training iterations, for most of the molecules in their respective equilibrium geometries, the hybrid CC-ML(KRR) predicts energy which is accurate up to 2 μH\mu H to the exact CCSD energy. However, there are quite a few cases, particularly for molecules in distorted geometries, for which 6 training iterations are not sufficient to accurately determine FF, and they provide results which are off by a large margin. They are not included in the scale of the plot. However, with higher number of training iterations, the difference between the energy obtained via canonical CCSD and hybrid CC-ML(KRR) tend to converge more towards the middle. This is particularly true for molecules in away from their equilibrium geometries, which are shown by the red dots. The mean absolute deviation (MAD) for molecules in away from equilibrium regions converge very fast with the number of training iterations with MAD=29.1 μH\mu H with m=6m = 6 to MAD=6.4 μH\mu H with m=7m = 7 to MAD=2.6 μH\mu H with m=8m = 8. For the molecules in their equilibrium geometries, even with 6 training iterations, MAD of 1.2 μH\mu H is observed, which further systematically improves as we increase the number of training iterations. Overall, irrespective of the molecular geometry, with m=7m = 7, the CC-ML(KRR) predicts energy within ±2 μH\pm2\ \mu H, which are shown by the yellow band.

Performance of hybrid CC-ML(KRR) model

Figure 2. Performance of hybrid CC-ML(KRR) model: the x-axis on all sub-plots are arranged in the increasing number of the non-zero cluster amplitudes, which is taken as a measure of the system size. The blue dots in all sub-plots are for the molecules in their respective equilibrium geometries, and the red dots are for the molecules in stretched geometries. The first row shows the difference in energy between the CC-ML(KRR) and exact CCSD. The black horizontal line is the reference, and the green horizontal bar signifies the convergence accuracy of ±1μH\pm1\mu\mathrm{H}. The second row shows the relative time taken by CC-ML(KRR) with respect to the exact CCSD (without DIIS acceleration). The red and blue lines show the mean fraction of the computational timing. The third row shows the time taken by CC-ML(KRR) with respect to DIIS accelerated CCSD scheme. “Eq” and “Away” in the sub-figure captions refer to the molecules in their equilibrium and away from equilibrium geometries, respectively. The three columns denote the number of training iterations employed.

In the second row of Figure 2, we have plotted the fraction of the time taken by the hybrid CCSD-ML(KRR) scheme compared to the conventional CCSD calculations. With m=6m = 6, the hybrid CC-ML(KRR) scheme takes at around 61% time compared to the conventional CCSD calculations for molecules in the equilibrium geometries. For the molecules with away from equilibrium geometries, the hybrid CC-ML(KRR) on an average takes around 53% of the total computational time to that required for the an CCSD. As one includes more training iteration, the time requirement increases, and the scattered points shift upwards. One may also note that the blue points (which are assigned for molecules with equilibrium geometries) appear higher than the red points (for molecules with stretched geometries). This is due to the fact that the molecules in equilibrium geometries take much lesser number of CCSD iterations to converge, and hence the training cycles take a large fraction of the overall calculation. In stretched geometries, the CCSD calculations take longer time to converge, which means that for the hybrid CC-ML scheme, the cost of initial iterations and training of the model is small relative to the total computation time, and hence the red dots appear consistently below the blue points. Note that with 8 training iterations, the computation time ratio is only 0.72 and 0.57 for equilibrium and away from equilibrium geometries respectively, relative to the canonical CCSD calculations. In the case where the CCSD scheme is accelerated via DIIS, the ratio is slightly higher, as expected and observed from the third row of Figure 2. However, the hybrid CC-ML(KRR) still outperforms DIIS accelerated CCSD in terms of the required computational time for most case irrespective of the molecular geometry.

Customized Kernel Regression Model:

While the hybrid CC-ML(KRR) model is very stable and accurate, it requires a larger number of training set. Moreover, one needs to start with an unknown polynomial form of the coupling map. Moreover, in the CC-ML(KRR) model, even starting from the first order guess amplitudes, one needs to discard a couple of initial iterations as the iteration pattern does not get stabilised to be accurately predicted by the model. This makes the KRR model quite data intensive and takes high number of initial iterations to train the data for sufficiently accurate result. Moving towards a map that is more consistent with the CC iteration scheme is expected to reduce the number of training iterations and the required dimension of the independent amplitude space. We have generated a new kernelization technique that is more consistent with the CC iteration scheme. As regularization often slows down the learning process, one may remove the Ridge regularization from the Eq (8). In order to speed up the calculations and get as much features as possible from very less amount of input parameters, we go back to Eq. (7), and define our own custom made Kernel function. Note that KRR uses a ready-made polynomial kernel available in the standard library; however, CKR is based on our own kernel matrix, which we integrated with the library of the Scikit.

Following a similar philosophy as KRR, we define the input training matrix TLT^{L} and TST^{S} exactly the same way as defined in Eqs (2) and (4). To extract the input features from the TLT^{L} and TST^{S} matrices, we define two different feature matrices

  • The Squared Root Euclidean Distance Matrix (EDM), or what has simply been defined as distance matrix DD in our earlier works. [19, 22] The elements of the distance matrix DD is defined as

dij=∥x⃗i−x⃗j∥,(9)d_{ij} = \lVert\vec{x}_{i} - \vec{x}_{j} \rVert, \tag*{(9)}

where x⃗i\vec{x}_{i} is defined as the ii − th row of TLT^{L}.

  • Linear Kernel (QQ, elements denoted as qijq_{ij}) is defined as

Q=TL(TL)T(10)Q = T^{L}(T^{L})^{T} \tag*{(10)}

Both of the matrices are symmetric and of the size m×mm \times m, where mm is, as usual, the number of the training data sets. The EDM and the linear kernel matrix are the input to the kernel, and they work as feature extractors from the training data.

With these feature matrices, a new kernel is constructed with the combination of an exponential and sinusoidal parts. The the kernel KK with elements kijk_{ij}, i,j=={1,2,...p}i, j == \{1, 2, ...p\} can be written as

kij=exp(−sin(gij+lij))(11)k_{ij} = exp(-sin(g_{ij} + l_{ij})) \tag*{(11)}

where,

gij=exp(−γ∗dij),lij=log(qij)(12)g_{ij} = exp(-\gamma* d_{ij}), l_{ij} = log(q_{ij}) \tag*{(12)}

where γ\gamma is a hyper-parameter, which can be optimized.

Once the model is trained through the initial cycles, we first calculate the new kernel matrix, KLK_L with the new input values of the principal amplitudes through the new vector TnewLT^L_{new}. In order to do so, we first calculate the feature extraction matrices, where the new squared root EDM DLD_L is defined as:

DLi=∑k=1n(t(new)ikL−tikL)2, i={1,2,…,m}(13)D_{L_i} = \sqrt{\sum_{k=1}^{n} \left(t^L_{(new)ik} - t^L_{ik}\right)^2},\ i = \{1,2,\ldots,m\} \tag*{(13)}

and the linear kernel matrix QLQ_L as:

QL=TL(TnewL)T(14)Q_L = T^L\left(T^L_{new}\right)^T \tag*{(14)}

In Eqs. (9) and (10), we now replace the elements of DD and QQ matrices by the elements of the new feature matrices DLD_L and QLQ_L from Eqs. (13) and (14). Finally, using Eq. (11), we get the predicted auxiliary amplitudes as:

tpredS=(KL)TK−1TS(15)t^S_{pred} = \left(K_L\right)^T K^{-1} T^S \tag*{(15)}

Although the hybrid CC-ML(CKR) model performs quite well in most of the cases, it is observed that due to the sinusoidal nature of the kernel matrix, sometimes for some highly fluctuating larger amplitudes, combined with a bad choice of γ\gamma, may make (gij+lij)(g_{ij} + l_{ij}) inside the exponential lose its monotonic nature. In that case, KK may contain very small diagonal value, and taking its inverse via Eq. (15) makes the whole model to explode with a very large value of the predicted auxiliary amplitudes. Forceful regularization procedure on the diagonal values does not work well, as it changes the overall nature of the kernel matrix, which later gives poor accuracy. One of possible way is to remove more fluctuating larger amplitudes from the training matrix itself before training. However, the consequence of the removal of the highly fluctuating large amplitudes results in poorer accuracy of the overall model. Furthermore, extracting the highly oscillating large amplitudes beforehand is a highly nontrivial exercise in ML. A more stable model in this regard will be a subject of our forthcoming publication. In our model, to prevent this from happening, we integrated a separate algorithm which keeps track of the KLK_L matrix, and prevents it from changing any of its element value by more than 0.02 by pushing it backwards. This whole process makes the learning process slow and does not give the desired accuracy. Nonetheless, this procedure prevents the whole code from catastrophic breakdown. With this caveat, the results for the CKR model are discussed below.

Assessment of the performance of CKR model:

It is evident from the previous discussion that the CKR method is a high-risk/high-profit model where for almost all the cases under consideration, one gets highly accurate result with less computation time compared to KRR. Like all other models considered in this manuscript, the accuracy largely depends on the number of training iterations. However, unlike KRR, the customized kernelization predicts the trajectory with less number of training iterations and discarding the couple of initial iterations is not needed. This saves a significant computation time due to less training iterations. Moreover, The CKR model requires a much smaller size of the independent variables compared to KRR, (nLKRR≫nLCKRn_{L_{KRR}} \gg n_{L_{CKR}}). This means the major computational bottleneck of nLno2nv2n_{L}n_{o}^{2}n_{v}^{2} scaling in step-II is significantly reduced. This thus allows a faster loop over the LAS elements, reducing the overall computation time significantly over KRR. Note that with a higher value of nLn_{L}, although the computation time increases slightly due to the nLno2nv2n_{L}n_{o}^{2}n_{v}^{2} scaling in step-II, the results become significantly more accurate.

The performance of hybrid CC-ML(CKR) model is shown in Figure 3 for various numbers of the training iterations for about 52 different molecules/basis/geometries. With m=6m = 6, the model is significantly less accurate than KRR with MAD of 46.5 μH\mu H and 92.1 μH\mu H for molecules in equilibrium and away from equilibrium geometries respectively, from the canonical CCSD calculations. However, the model has a sharper convergence with respect to the number of the training cycles compared to KRR. With m=8m = 8, the MAD observed with the CKR model is only 2.8 μH\mu H for the molecules in equilibrium geometry, while for the molecules with away from equilibrium geometry, the MAD is about 4.2 μH\mu H. As we mentioned before, unlike KRR, one does not need to discard the initial iterations to train the model, and hence the training can be performed with less number of iterations effectively. This amounts to significant reduction in the computation time than KRR. This is reflected in the second and third row panels of Figure 3 where we have plotted the relative time requirement compared to the conventional CCSD. Note that for m=8m = 8, the average time requirement to obtain energy of μH\mu H precision is only 56% of that of the canonical CCSD calculation for molecules in equilibrium. To achieve a similar accuracy for the molecules with distorted geometry, the average time requirement is about 45% compared to canonical CCSD. If the canonical CCSD calculations are accelerated by DIIS as is done in most of the cases, the time requirement for the hybrid CC-ML(CKR) slightly goes up to 60% on an average to the DIIS accelerated CCSD calculation. However, one should monitor the predicted energy from CKR as in few rare cases, the model may get unstable as mentioned before producing somewhat inaccurate energy.

Performance of CC-ML(CKR)

Figure 3. Performance of CC-ML(CKR). The first row shows the difference in energy between the CC-ML(CKR) and the exact CCSD with the green bar showing an energy accuracy of ±10μH\pm10\mu\mathrm{H}. The second and third rows show the relative computational time of CC-ML(CKR) with respect to CCSD (without DIIS acceleration) and DIIS accelerated CCSD schemes, respectively. The three columns denote the number of training iterations employed for training. All other details are same as Figure 2.

Polynomial Kernel Ridge Regression Model:

The Polynomial Kernel Ridge Regression model has a similar structure to that of the CKR model presented in the previous section, and hence again, we only present the essential aspects which distinguish PKR from KRR. While the KRR model is linear by default, there is supposed to have more variational flexibility by introducing the nonlinear terms in the kernel function. To proceed along this direction, we define the input TLT^{L} and TST^{S} matrices the same way as Eq. (2) and Eq. (4). Furthermore, we define the linear kernel, QQ as Eq. (10). We then introduce our polynomial kernel KK as

K=(γQ+C)o(16)K = (\gamma Q + C)^{o} \tag*{(16)}

where C=1C = 1 is a constant, and o=3o = 3 is the order of the polynomial. These values of CC and oo are taken by default in the ML library. [21] Like the previous two models, this kernel is now used in training of the amplitudes. The prediction is done by a similar technique used in the KRR model using Eq. (8). The most fundamental difference between the KRR and PKR is in the choice of the kernel function: while the former uses a linear regression type fitting, the latter uses polynomial function, in this case of degree 3, for a better accuracy.

Assessment of the performance of PKR model:

The PKR method is a variant of the parent KRR model. This model, as mentioned previously, increases the dimension of the feature matrix, which allows us a better fit of the auxiliary amplitudes in terms of the principal amplitudes. PKR model, as a result, is highly accurate. Howeverm, like CKR, this too does not discard any initial iterations, and hence has a similar requirement of the number of training cycles as that of CKR. Since the number of independent degrees of freedom increases due to the polynomial structure of the kernel via Eq. (5), a small dimension of the LAS, nLn_{L}, is sufficient to achieve high accuracy. The performance of the hybrid CC-ML(PKR) model is shown in Figure 4 as a function of different number of the training iterations for about 60 different molecules/basis/geometries. With m=6m=6, the model is less accurate than KRR with MAD of 16.1 μH\mu H and 36.5 μH\mu H for molecules in equilibrium and away from equilibrium geometries respectively, from the canonical CCSD calculations. This is substantially better than CKR model with equal number of training iterations. Even though the model has a sharper convergence of the estimated energy than KRR, it is not as sharp as CKR. This is presumably due to the fact that the PKR model, by construction, starts with an unknown polynomial mapping, and also it has a ridge regularization term. With m=8m=8, the MAD for the PKR model goes down to 3.4 μH\mu H for the molecules

Performance of CC-ML(PKR)

Figure 4. Performance of CC-ML(PKR). The first row shows the difference in energy between the CC-ML(PKR) and the exact CCSD with the green bar showing an energy accuracy of ±12.5μH\pm12.5\mu H. The second and third rows show the relative computational time of the model with respect to un-accelerated and DIIS accelerated CCSD schemes, respectively. The three columns are for three different training iterations (m=6m=6, 7, 8). All other details are same as Figure 2.

in equilibrium geometry, and 6.9μH6.9\mu\mathrm{H} for those in the away from equilibrium geometry. PKR, due to its inherent polynomial structure, does not require to discard any initial iterations to construct the kernel, and hence there is a significant reduction in the computation time than KRR. Moreover, PKR requires a very small dimension of the principal amplitude space, nLn_L, like CKR. This leads to the time requirement of PKR very similar to CKR. Following discussions from previous models, we note that for m=8m = 8, the average time requirement is only 55% of that of the canonical CCSD calculation for molecules in equilibrium. To achieve a similar accuracy for the molecules with distorted geometry, the average time requirement is about 44% compared to canonical CCSD. If the canonical CCSD calculations are accelerated by DIIS, the time requirement for the hybrid CC-ML(PKR) slightly goes up to 60% on an average to the DIIS accelerated CCSD calculation with m=8m = 8. The model is, however, much more stable than CKR due to inclusion of regularization terms via Eq. (8).

K-Nearest Neighbours Regression Model:

The K-nearest neighbor model, strictly speaking, is a classification based method and not a regression method. However, it can be modified slightly to turn it into an approximate regression model, as we briefly discuss below. Traditionally, the KNN is used in classify a quantity into groups looking at nearest points through features such as Euclidean or Manhattan Distance. Here, we have used the Euclidean Distance as the measure of the closeness.

The neighbours are defined as the individual training iterations. Therefore, with mm training iterations, we get mm neighbours, each neighbour having an input principal amplitude feature matrix defined as TjL={tijL,i∈(1,nL)}T_j^L = \{t_{ij}^L, i \in(1,n_L)\}, and output as the remaining auxiliary amplitudes TjS={tijS,i∈(1,nS)}T_j^S = \{t_{ij}^S, i \in(1,n_S)\}, where jj is the number of the training iteration. At the end of the training iterations, we have mm feature vectors and mm output vectors. After starting the reduced iteration, we calculate the tnewLt_{\mathrm{new}}^L using CC via step-II of the circular causality loop, and the tnewSt_{\mathrm{new}}^S is predicted as a average of the kk nearest neighbors, and it’s equation is given as

Performance of CC-ML(KNN)

Figure 5. Performance of CC-ML(KNN): the first row shows the difference in energy between the CC-ML(KNN) and exact CCSD with the green bar showing an energy accuracy of ±50μH\pm50\mu H. The second and third rows show the relative computational time of the model with respect to un-accelerated and DIIS accelerated CCSD schemes, respectively. The three columns are for three different training iterations (m=6,7,8m=6, 7, 8). All other details are same as Figure 2.

tnewS=1k∑ikTiS(17)t^{S}_{new} = \frac{1}{k}\sum_{i}^{k} T^{S}_{i} \tag*{(17)}

where the nearness is defined as the Euclidean distance between tnewLt^{L}_{new} and tLt^{L} from earlier iterations. As is clear, the ’nearness’ of a point is completely arbitrary and, we noticed that at just one neighbor, the accuracy nearly saturated and did not increase with the increase in the number of neighbors. It is not always necessary that the accuracy would increase with increase in neighbors as it may lead to overfitting. With these ideas in mind, and through a well established ML library[21] we converted our ML model to be of KNN-Regression form. In the subsequent paragraph, we demonstrate the performance of the model over several test cases.

Assessment of the performance of KNN model:

In Fig 5, we have presented the performance of KNN for several molecular systems in their equilibrium and away from equilibrium geometries. Clearly, the model doesn’t perform nearly well as compared to the other robust regression models discussed previously. While the average deviation improves sharply by taking high number of training iterations, with fewer training iterations, the model performs poorly. This contradicts the basic idea of supervised machine learning where we target to model the CC iteration convergence with only a few number of training cycles. This is primarily due to it being a classification model, rather than a regression based model. While the process of training remaining the same, which guarantees that it takes approximately similar computational time to CKR and PKR, the accuracy is not nearly as good as the previously discussed models. We must also note that being a rudimentary model, it provides us with a base case, and any sophisticated machine learning model would be an improvement over it. Thus the use of such a classification based model is not recommended to simulate the CC iteration series.

Discussion and summary:

Through this work, we have thoroughly benchmarked some of the common supervised machine learning models based on the regression technique to solve CC equations1. The average accuracy and computation time requirements for various models under consideration are summarized in Table 1. The CC-ML(KRR) seems to be the most stable and robust model with guaranteed convergence beyond μH\mu H at the cost of large number of training iterations. The model saves around 30-40% computation time over the unaccelerated canonical CCSD calculations, and 20-30% computation time over the DIIS accelerated CCSD calculations. The bottleneck for the CC-ML(KRR) model is the high number of iterations required for the training on top of a few discarded initial iterations. The CC-ML(CKR) and CC-ML(PKR) perform much better in terms of the time requirement since it can be trained with fewer number of iterations without discarding the initial steps. For both the models, the accuracy is marginally poorer than CC-ML(KRR) model, although the average error in both the cases is only a few μH\mu H irrespective of the molecular correlation complexity. Apart from that, a relatively ’loose’ classification based KNN machine learning model gives a respectable accuracy of 9 μH\mu H for molecules in equilibrium geometries and 40μH\mu H in geometries away from the equilibrium, with the time saving similar to the previous two models. We thus conclude that the hybrid CC-ML technique is statistically stable and could be used as a standardized method of calculation. The method works well with several supervised machine learning models, and is highly tunable as per the requirement of accuracy and cost affordability. This is also a demonstration of the robustness of the synergistic interdependence of the cluster amplitudes and the resulting hybrid CC-ML models under various electronic complexity. In our implementation, the construction of the diagrams for only the selected excitations

Parametersntrainn_{train}KRR Eq.KRR AwayCRM Eq.CRM AwayPoly-KR Eq.Poly-KR AwayKNN Eq.KNN Away
Accuracy(μH\mu H)61.229.146.592.116.136.542.9133.4
Accuracy(μH\mu H)70.86.415.617.17.025.424.9106.0
Accuracy(μH\mu H)80.72.62.84.23.46.98.940.1
Time/CCSD(UA)60.610.530.440.380.440.370.420.37
Time/CCSD(UA)70.670.550.50.410.50.40.480.43
Time/CCSD(UA)80.720.570.560.450.550.440.540.48
Time/CCSD(DIIS)60.660.730.470.470.480.490.460.44
Time/CCSD(DIIS)70.720.790.530.540.540.540.520.53
Time/CCSD(DIIS)80.780.790.60.60.60.60.590.59

Table 1. Performance of the different Machine Learning models. The accuracy is defined as the absolute difference of the energy obtained by the hybrid CCSD-ML model and the canonical CCSD. The Time/CCSD(UA) denotes the fraction of time taken by the CC-ML model with respect to the unaccelerated CCSD scheme. The Time/CCSD(DIIS) denotes the fraction of time taken by the CC-ML model with respect to an DIIS based CCSD method. All the models, particularly those based on regression, provide excellent accuracy with substantial savings in computational time.

belonging to the LAS, as shown in step-II, Figure 1, is far from being optimal, and there is plenty of room to further improve upon. This would further reduce the computation time significantly. We note that the hybrid CC-ML scheme does not require any previously computed data; rather they can be trained on the fly based on the various cluster amplitudes determined at the initial steps during the optimization process for individual molecules.

Future Directions

This work reinforces the synergistic interrelation of the cluster amplitudes during the CC iteration scheme, and demonstrates the effectiveness of the hybrid CC-ML methodology. However, being one of the first instances of the numerical inclusion of Synergetics via ML in CC, the possibilities are endless. Since the field of ML is still new, development of better models that resonate with the exact analytical structure of the CC iteration scheme would be an exciting avenue to explore. The development of an analytical mapping is a highly non-trivial challenge. While the adiabatic decoupling scheme to map the auxiliary amplitudes in terms of the principal amplitudes is available[23] in literature, conversion of the technique to machine learning based methods for numerical efficiency is a challenge. Other areas of work would be the inclusion of DIIS in the CC-ML technique for even faster calculations. Inclusion of higher order terms like triples and quadruples would be an interesting avenue to venture. This would make high order calculations with large basis sets faster in near future. An extension of this model to treat molecular excited states would also be a subject to a forthcoming publication.

Data Availability

The data generated in this study is available upon reasonable request to the corresponding author.

Acknowledgement

The authors thank Mr. Anish Chakraborty for many stimulating discussions about the structure of the program.

Conflict of Interest

The authors declare no competing financial interest.

Funding

The authors thank IRCC, Indian Institute of Technology Bombay for the research seed grant, and SERB, Department of Science and Technology, Government of India for their financial support.

References

References

  1. [1]Čížek, J. On the correlation problem in atomic and molecular systems. Calculation of wavefunction components in Ursell-Type expansion Using Quantum-Field theoretical methods. J. Chem. Phys. 1966, 45, 4256–4266.DOI
  2. [2]Čížek, J. On the use of the cluster expansion and the technique of diagrams in calculations of correlation effects in atoms and molecules. Adv. Chem. Phys. 1969, 14, 35–89.DOI
  3. [3]Čížek, J.; Paldus, J. Correlation problems in atomic and molecular systems III. Re-derivation of the coupled-pair many-electron theory using the traditional quantum chemical methods. Int. J. Quantum Chem. 1971, 5, 359–379.
  4. [4]Bartlett, R. J.; Musial, M. Coupled-cluster theory in quantum chemistry. Reviews of Modern Physics 2007, 79, 291.DOI
  5. [5]Pulay, P. Convergence acceleration of iterative sequences. The case of SCF iteration. Chemical Physics Letters 1980, 73, 393–398.DOI
  6. [6]Piecuch, P.; Adamowicz, L. Solving the single-reference coupled-cluster equations involving highly excited clusters in quasidegenerate situations. The Journal of chemical physics 1994, 100, 5857–5869.
  7. [7]Kjønstad, E. F.; Folkestad, S. D.; Koch, H. Accelerated multimodel Newton-type algorithms for faster convergence of ground and excited state coupled cluster equations. The Journal of Chemical Physics 2020, 153, 014104.DOI
  8. [8]Yang, C.; Brabec, J.; Veis, L.; Williams-Young, D. B.; Kowalski, K. Solving Coupled Cluster Equations by the Newton Krylov Method. Frontiers in chemistry 2020, 8.DOI
  9. [9]Parrish, R. M.; Zhao, Y.; Hohenstein, E. G.; Martínez, T. J. Rank reduced coupled cluster theory. I. Ground state energies and wavefunctions. The Journal of chemical physics 2019, 150, 164118.DOI
  10. [10]Schutski, R.; Zhao, J.; Henderson, T. M.; Scuseria, G. E. Tensor-structured coupled cluster theory. The Journal of chemical physics 2017, 147, 184113.DOI
  11. [11]DePrince III, A. E.; Sherrill, C. D. Accuracy and efficiency of coupled-cluster theory using density fitting/cholesky decomposition, frozen natural orbitals, and at 1-transformed hamiltonian. Journal of chemical theory and computation 2013, 9, 2687–2696.DOI
  12. [12]Schütz, M.; Manby, F. R. Linear scaling local coupled cluster theory with density fitting. Part I: 4-external integrals. Physical Chemistry Chemical Physics 2003, 5, 3349–3358.DOI
  13. [13]Haken, H.; Wunderlin, A. Slaving principle for stochastic differential equations with additive and multiplicative noise and for discrete noisy maps. Z. Phys. B 1982, 47, 179–187.DOI
  14. [14]Haken, H. Advanced Synergetics: Instability Hierarchies of Self-Organizing Systems and Devices; Springer Berlin Heidelberg: Berlin, Heidelberg, 1983; pp 187–221.DOI
  15. [15]Haken, H. Synergetics: an overview. Rep. Prog. Phys. 1989, 52, 515–553.
  16. [16]Marwan, N.; Romano, M. C.; Thiel, M.; Kurths, J. Recurrence plots for the analysis of complex systems. Phys. Rep. 2007, 438, 237–329.DOI
  17. [17]Marwan, N.; Romano, M. C.; Thiel, M.; Kurths, J. Recurrence Plots. accessed May 19, 2020; www.recurrence-plot.tk.
  18. [18]Murphy, K. P. Machine learning: a probabilistic perspective; MIT press, 2012.
  19. [19]Agarawal, V.; Roy, S.; Chakraborty, A.; Maitra, R. Accelerating coupled cluster calculations with nonlinear dynamics and supervised machine learning. The Journal of Chemical Physics 2021, 154, 044110.DOI
  20. [20]Minh, H. Q.; Niyogi, P.; Yao, Y. Mercers theorem, feature maps, and smoothing. International Conference on Computational Learning Theory. 2006; pp 154–168.
  21. [21]Pedregosa, F. et al. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 2011, 12, 2825–2830.arxiv.org/abs/1201.0490
  22. [22]Agarawal, V.; Chakraborty, A.; Maitra, R. Stability analysis of a double similarity transformed coupled cluster theory. The Journal of Chemical Physics 2020, 153, 084113.arxiv.org/abs/2006.13691
  23. [23]Agarawal, V.; Patra, C.; Maitra, R. An Approximate Coupled Cluster Theory via Nonlinear Dynamics and Synergetics: the Adiabatic Decoupling Conditions. 2021; https://arxiv.org/abs/2106.05069.

Paper details

vlnc:W_gt8wfydm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents