betaFeedback

WorksPhysics and Astronomy

Neural network backflow for ab-initio quantum chemistry

Abstract

The ground state of second-quantized quantum chemistry Hamiltonians provides access to an important set of chemical properties. Wavefunctions based on ML architectures have shown promise in approximating these ground states in a variety of physical systems. In this work, we show how to achieve state-of-the-art energies for molecular Hamiltonians using the the neural network backflow wave-function. To accomplish this, we optimize this ansatz with a variant of the deterministic optimization scheme based on SCI introduced by [Li, et. al JCTC (2023)] which we find works better than standard MCMC sampling. For the molecules we studied, NNBF gives lower energy states than both CCSD and other neural network quantum states. We systematically explore the role of network size as well as optimization parameters in improving the energy. We find that while the number of hidden layers and determinants play a minor role in improving the energy, there is significant improvements in the energy from increasing the number of hidden units as well as the batch size used in optimization with the batch size playing a more important role.

Introduction

The solution to the electronic Schrödinger equation for a given molecular system provides comprehensive access to its chemical properties from first principles. However, analytically solving the Schrödinger equation is feasible only for hydrogen-like atoms in real space. Exact approaches in second quantization, such as the full configuration interaction (FCI) method, are intractable for large systems due to the exponential growth of the Hilbert space with the system size. The many-body electronic problem is fundamentally NP hard [1, 2], motivating efforts to develop numerical methods for approximating solutions in ab-initio quantum chemistry (QC).

Configuration interaction (CI) methods [3] consider excitations above the Hartree-Fock (HF) reference state up to a fixed order, while coupled cluster (CC) approaches [4] use exponential excitation operators up to a certain order to access all excited states. Although CC techniques are often considered the “gold standard” in ab-initio QC, they may fail in regimes of strong static correlations [5].

Alternatively, variational wave-functions such as the Slater-Jastrow (SJ) [6] and matrix-product states (MPS) [7, 8], can be optimized to approximate the ground state. Various approaches build on top of the SJ approach such as multi-determinant Jastrow wave-functions [9] or alternatively backflow transformation [10], which renders orbitals dependent on the coordinates of all electrons [11]. Projector methods like diffusion Monte Carlo (DMC) [12] systematically refine the variational starting point. However, the effectiveness of these approaches is often bounded by the expressiveness of the wave function and the efficiency of the optimization. For instance, MPS are inadequate for highly entangled systems and face challenges in dimensions greater than one. Slater-Jastrow-Backflow (SJB) are often hard to systematically improve.

Machine learning architectures presents significant promise as concise wave-functions to address certain constraints of existing variational representations. Ref. 13 initially demonstrated that restricted Boltzmann machines (RBM), trained via variational Monte Carlo (VMC), could effectively capture the intricate correlations in certain interacting spin models and numerous advances [14, 15, 16, 17] since then have improved neural network quantum states (NNQS) for spin models. More recently, there has been significant work using machine learning architectures to develop wave-functions for Fermion systems [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33].

The advent of wave-functions for Fermions has motivated an interest in addressing molecular and QC Hamiltonians. The FermiNet [25] and PauliNet [26] wave-functions, which are real-space generalizations of the Neural Net Backflow (NNBF) [23], have shown to be competitive with other ab-initio QC methods in continuous space. Using a second quantized Hamiltonian in a finite basis, ref. 27 mapped the molecular system onto an equivalent spin problem and demonstrated that RBM could achieve higher accuracy than CCSD(T). However, they noted that using ordinary Markov chain Monte Carlo (MCMC) methods for sampling was inefficient due to sharp peaks in the underlying probability distribution around the HF state and neighboring excited states. To address this sampling inefficiency as well as gain more accuracy than the RBM, subsequent studies [28, 29, 31, 32] introduced autoregressive neural networks (ARN) for exact sampling. Further improvements in optimization were introduced by ref. 30 which proposed a non-stochastic optimization scheme adapted from the selected configuration interaction (SCI) method to deterministically select samples on-the-fly leading to competitive molecular energies using RBMs. Interestingly, despite NNBF style ansatz demonstrating high accuracy in the continuum for QC, the NNBF approach has not yet been applied to second quantized QC Hamiltonians.

In this work, we address this omission using the NNBF [23] to approximate the ground state of molecular systems in second quantization. To optimize our wave-function, we apply and demonstrate the superiority of a deterministic fixed-size selected configuration (FSSC) iteration scheme, adapted from ref. 30. Our experiments on various molecules indicate that the NNBF ansatz not only surpasses traditional CCSD benchmarks but also finds lower energies than all other existing NNQS methods on large molecules (and equally good energies on smaller molecules). We also evaluate the dissociation curve for diatomic Nitrogen demonstrating NNBF’s ability to capture both strong and weak quantum correlation. Additionally, we conduct a systematic investigation into the impact of network architecture and batch size, revealing that increasing the batch size is a more efficient and effective approach to improving NNBF performance.

Methods

In this section, we provide a detailed description of the structure of NNBF and the workflow of FSSC.

Neural Network BackFlow Architecture

For a molecular system containing NeN_e electrons, a basis set composed of No(>Ne)N_o(>N_e) spin-orbitals B={∣ϕi⟩}i=1No\mathcal{B}=\{|\phi_i\rangle\}_{i=1}^{N_o} is given to define a many electron wave-function in the form of ∣ψ⟩=∑i=1Noψ(xi)∣xi⟩|\psi\rangle=\sum_{i=1}^{N_o}\psi(\boldsymbol{x}_i)|\boldsymbol{x}_i\rangle where ∣xi⟩=∣xi1,…,xiNo⟩|\boldsymbol{x}_i\rangle=|x_i^1,\ldots,x_i^{N_o}\rangle is the ii-th computational basis vector in second quantization. Here, xij∈{0,1}x_i^j\in\{0,1\} denotes whether the jj-th spin-orbital is occupied in the ii-th computational basis vector.

The NNBF consists of a multilayer perceptron (MLP) which generates the ”configuration-dependent” single particle orbitals used in evaluating the Slater determinant (see Figure 1). The MLP directly takes the configuration strings as input feature vector and feeds it through a series of intermediate layers consisting of a fully-connected feed-forward neural network (FNN) combined with a ReLU nonlinear activation function i.e., hl+1=ReLU⁡(Wlhl+bl)h^{l+1}=\operatorname{ReLU}(W^lh^l+b^l). Instead of directly returning a scalar wave-function amplitude, the MLP outputs a configuration-dependent additive change to the set of single-particle orbitals for each determinant. In practice, the final output vector of the network, y=WLhL+bLy=W^Lh^L+b^L, will be reshaped into an array of DD matrices, where each matrix ϕijk\phi_{ij}^k is of shape (No,Ne)(N_o,N_e). The ii-th row of the kk-th matrix has a physical interpretation that it is the evaluation of the kk-th set of configuration-dependent orbitals if the ii-th spin-orbital is occupied. A square matrix can be generated by extracting the rows whose indices correspond to the locations of the electrons in the input configuration, and the amplitude of this configuration is defined as ψθ(x)=∑k=1Ddet⁡[ϕi={l∣xl=1},jk]\psi_\theta(\boldsymbol{x})=\sum_{k=1}^{D}\det[\phi_{i=\{l\mid x_l=1\},j}^k] where θ\theta is the model parameters. Consequently, the full NNBF wave-function is described as ∣ψθ⟩=∑i=1Noψθ(xi)∣xi⟩|\psi_\theta\rangle=\sum_{i=1}^{N_o}\psi_\theta(\boldsymbol{x}_i)|\boldsymbol{x}_i\rangle.

Illustration of the NNBF architecture

Figure 1. Illustration of the NNBF architecture with an example input configuration: 2 spin-orbitals with 1 spin-up electron occupying the first spin-orbital and 1 spin-down electron occupying the second spin-orbital. The neural network takes the configuration string as input and outputs a set of DD configuration-dependent single-particle orbitals of shape (No,Ne)(N_o,N_e). Rows of these orbitals are then selected based on the electron locations to form square matrices, from which determinants are computed. For this example, the first and last rows (the gray orbitals) are selected to compute the determinant. The sum of these determinants yields the amplitude for the input configuration.

Wavefunction Optimization

Given a quantum chemical Hamiltonian,

H^=∑i,j,σtijci,σ†cj,σ+12∑i,j,k,l,σ,σ′Vijklci,σ†cj,σ′†cl,σ′ck,σ.(1)\hat{H}=\sum_{i,j,\sigma}t_{ij}c_{i,\sigma}^{\dagger}c_{j,\sigma}+\frac{1}{2}\sum_{i,j,k,l,\sigma,\sigma'}V_{ijkl}c_{i,\sigma}^{\dagger}c_{j,\sigma'}^{\dagger}c_{l,\sigma'}c_{k,\sigma}. \tag*{(1)}

our target is to minimize the variational energy of our wave-function,

Eθ=⟨ψθ∣H^∣ψθ⟩⟨ψθ∣ψθ⟩=Epθ(x)[Eloc(x)](2)E_\theta=\frac{\langle\psi_\theta|\hat{H}|\psi_\theta\rangle}{\langle\psi_\theta|\psi_\theta\rangle}=\mathbb{E}_{p_\theta(\boldsymbol{x})}[E_{\mathrm{loc}}(\boldsymbol{x})] \tag*{(2)}

which can be evaluated by MCMC by sampling from the probability distribution pθ(x)∝∣Ψθ(x)∣2p_\theta(\boldsymbol{x})\propto|\Psi_\theta(\boldsymbol{x})|^2 where the local energy Eloc(x)=⟨ψθ∣H^∣x⟩⟨ψθ∣x⟩E_{\mathrm{loc}}(\boldsymbol{x})=\frac{\langle\psi_\theta|\hat{H}|\boldsymbol{x}\rangle}{\langle\psi_\theta|\boldsymbol{x}\rangle}. To optimize this energy, we will estimate the gradient of it, ∇θEθ\nabla_\theta E_\theta, and update θ\theta using ADAM [34]. We perform a CISD pretraining before optimizing with expected energy; the implementation details of this pretraining are provided in Appendix B.

One approach for computing the gradient

∇θEθ=2Re⁡{Epθ(x)[[Eloc(x)−Eθ]∇θln⁡∣ψθ(x)∣]}(3)\nabla_\theta E_\theta=2\operatorname{Re}\left\{\mathbb{E}_{p_\theta(\boldsymbol{x})}\left[\left[E_{\mathrm{loc}}(\boldsymbol{x})-E_\theta\right]\nabla_\theta\ln|\psi_\theta(\boldsymbol{x})|\right]\right\} \tag*{(3)}

is through an MCMC sampling of the energy. For optimizing molecular Hamiltonians, we find that this is a suboptimal way to update the variational parameters. A similar observation was made by ref. 27 which found that typical MCMC sampling technique is inefficient for the ground state of molecular system due to the peaks around the HF state and neighboring excited states [35, 36] which results in drawing many identical samples from the dominant states when sampling. Various techniques have been developed to overcome these issues. For autoregressive variational wave-functions, the probability of a configuration xx can be modeled using its conditional probabilities, expressed as pθ(x)=∏ipθ(xi∣x1,x2,…,xi−1)p_\theta(x)=\prod_i p_\theta(x_i|x_1,x_2,\ldots,x_{i-1}). This autoregressive property, combined with more efficient techniques, enables sampling the numbers of occurrences for each unique string, thus avoiding duplicate sampling efforts [28, 29, 31, 33]. Recently, Li et al. [30] proposed an algorithm based on SCI to select a set of states and approximate their frequency of occurrences instead of statistically sampling them. This set of states is a deterministic selection of a set of important configurations identified through the modulus of the wave-function ansatz during energy evaluation.

Deterministically Select Sample Space

In this work, we will use a variant of the scheme of ref. [30] which we call the FSSC scheme. For completeness, we describe the entire FSSC approach below while a diagrammatic description of it is also provided in Figure 2.

Diagrammatic description of the workflow of FSSC

Figure 2. A diagrammatic description of the workflow of FSSC. Each circle represents one configuration. Initially, the algorithm initializes a parametrized wave-function and a core space V0\mathcal{V}^0 of size Nu=10N_u=10. After each iteration, the amplitude is computed for all configurations in Vn−1∪Cn−1\mathcal{V}^{n-1}\cup\mathcal{C}^{n-1} and the 10 largest unique ones (denoted by red configurations) are selected to form the new core space Vn\mathcal{V}^n. The energy (depicted as the loss function in the orange box) and its gradient are estimated by constraining the relative sum to only consider Vn\mathcal{V}^n, and the latter is used to update the model parameters.

The full configuration space is divided into three parts as in a typical SCI method: the core space V\mathcal{V} consisted of the dynamically selected important configurations, the connected space C\mathcal{C} containing all other configurations connected to the current core space through nonzero Hamiltonian matrix elements, and the remaining part of the configuration space. The HF state and some neighboring excited states are deterministically pre-selected to form the initial core space V0\mathcal{V}^0 of size NuN_u. After each iteration of optimization, we find the connected space Cn−1\mathcal{C}^{n-1} induced from the core space Vn−1\mathcal{V}^{n-1} from the last iteration. The new core space Vn\mathcal{V}^n will be deterministically chosen by selecting the NuN_u configurations from the union of Cn−1\mathcal{C}^{n-1} and Vn−1\mathcal{V}^{n-1} which have the largest unique amplitudes with respect to the new state with updated parameters. This Vn\mathcal{V}^n will then be used to evaluate an approximation to the gradient and update the model parameters:

∇θEθ=2Re⁡{∑∣xi⟩∈Vnpθ(xi)[[Eloc(xi)−Eθ]∇θln⁡∣ψθ(xi)∣]}(4)\nabla_\theta E_\theta=2\operatorname{Re}\left\{\sum_{|x_i\rangle\in\mathcal{V}^n}p_\theta(x_i)\left[\left[E_{\mathrm{loc}}(x_i)-E_\theta\right]\nabla_\theta\ln\left|\psi_\theta(x_i)\right|\right]\right\} \tag*{(4)}

where

Eθ=∑∣xi⟩∈Vnpθ(xi)Eloc(xi).(5)E_\theta=\sum_{|x_i\rangle\in\mathcal{V}^n}p_\theta(x_i)E_{\mathrm{loc}}(x_i). \tag*{(5)}

and

pθ(xi)=∣⟨ψθ∣xi⟩∣2∑∣xj⟩∈Vn∣⟨ψθ∣xj⟩∣2(6)p_\theta(x_i)=\frac{\left|\langle\psi_\theta|x_i\rangle\right|^2}{\sum_{|x_j\rangle\in\mathcal{V}^n}\left|\langle\psi_\theta|x_j\rangle\right|^2} \tag*{(6)}

where the relevant sums are over the core space. Approximating the various quantities above by summing over only the core space performs well when the ground state is predominantly influenced by a small number of significant configurations, which is a typical characteristic of

molecular systems in the canonical HF basis. We note that all the energies reported in this work (not including traces over optimization time) are computed exactly by Monte Carlo (MC) sampling. See Appendix A for details.

The major distinction between our approach and ref. [30] lies in fixing the size of the core space, as opposed to selecting configurations whose modulus relative to the largest one is greater than a selection cutoff, ϵ\epsilon. This allows us to leverage the advantages of just-in-time (JIT) compilation with JAX, as the shape of the input array to the jitted functions remains constant throughout iterations. Additionally, a fixed number of unique configurations determines the computational cost, which is directly proportional to NuN_u, whereas in most other NNQS works [27, 28, 30, 31], only the total number of samples could be specified. Moreover, a fixed batch size prevents the possibility of an unexpectedly large number of configurations overwhelming the limited computational resources, whereas the core space could potentially encompass the entire configuration space at some point in the ref. [30]. At one point, we have compared a fixed batch size to a dynamically adjusted batch size. The optimized energy using a selection cutoff is essentially the same as the optimized energy with a fixed batch size which is equal to the average batch size of the selection cutoff (Figure 7a).

Three-panel plots showing batch-size experiments, batch-size scheduling, and dependence on hidden units

Figure 7. (a) Experiments exploring the impact of batch size NuN_u on energy improvements with (L,h,D)=(2,64,1)(L,h,D)=(2,64,1) and (L,h,D)=(2,256,1)(L,h,D)=(2,256,1) on lithium oxide. The energy error decreases approximately following O(Nu−1.137885)\mathcal{O}(N_u^{-1.137885}) with an R2R^2 value of 0.900605 for h=64h=64 and O(Nu−1.288627)\mathcal{O}(N_u^{-1.288627}) with R2=0.964273R^2=0.964273 for h=256h=256. Some data points from Figure 6b are included for comparison in the inset. The black star represents a trial trained with (L,h,D)=(2,64,1)(L,h,D)=(2,64,1) using the sampling scheme from ref. [30]. Its x-position is determined by the average batch size at convergence. The energy closely aligns with the fitted line for the FSSC scheme, indicating that using a dynamic batch size does not offer a noticeable improvement in training. (b) Demonstration of the effectiveness of the batch size scheduling (BSS) strategy, with a moving average window of 100 applied for improved readability. (c) Experiments examining the dependence of energy improvements on batch size NuN_u with various hh values on methane. Each data point represents the average and the standard deviation of the exact NNBF energy across 3 seeds.

Results

Here we showcase the effectiveness of the FSSC scheme by comparing it to the typical MCMC scheme, evaluate the performance of the NNBF on various molecules, train the NNBF model to emulate the dissociation curve of nitrogen molecule, and investigate the impact of network architecture and batch size on NNBF. Further details on the default network architectures, hyperparameters, training routine, and the post-training MCMC inference routine can be found in Appendix A.

FSSC vs. Traditional MCMC Optimization

To demonstrate the efficacy of the FSSC scheme in addressing the inefficiencies of MC sampling, we conduct a comparison on lithium oxide using both the FSSC scheme and the standard MCMC scheme. For a fair comparison, we set the number of unique configurations in the FSSC scheme equal to the number of walkers in the MCMC scheme, i.e. Nu=Nw=8192N_u = N_w = 8192.

In Figure 3 we see both faster convergence as well as a lower converged energy for the FSSC scheme compared to the conventional MCMC scheme, which happens to only converge to the CCSD baseline. Additionally, we observe that there are only 111 unique configurations in the 8192 MCMC samples at convergence, with the most frequent sample being the HF configuration appearing in 7686 walkers. This observation validates the inefficient MC sampling discovered by ref. 27 and highlights the FSSC scheme’s ability to effectively capture important configurations in large configuration space, akin to conducting MC sampling with a sample size Nw≫NuN_w \gg N_u. Additionally, the FSSC method shows a smoother, more monotonic trend in energy values during the optimization steps, especially compared to the MCMC scheme, which, although it displays an average reduction in energy, does so with noticeable oscillations. This demonstration aligns with the results presented by ref. 30.

Comparison between FSSC and MCMC schemes on lithium oxide

Figure 3. Comparison between FSSC and MCMC schemes on lithium oxide (NO=30N_O = 30, Ne=14N_e = 14, Nu=Nw=8192N_u = N_w = 8192). The red band represents the post-training MCMC inference energy for the FSSC scheme with a width of σ\sigma in each direction around the mean. A moving average window of 100 is applied to improve readability.

Benchmark With Other QC Methods

We assess the performance of our algorithm by comparing molecular ground state energies with CCSD baselines and other NNQS results, utilizing a list of molecules with geometries sourced from PubChem [37]. Table I demonstrates that NNBF not only surpasses conventional CCSD methods but also is equal or lower in energy than all other existing NNQS approaches, particularly showing superiority in larger molecular systems.

Additionally, we explore the NNBF’s ability to capture quantum correlation by computing the dissociation curve of the diatomic nitrogen molecule. As illustrated in Figure 4, the NNBF precisely matches the FCI energy and outperforms all other conventional QC approaches in scenarios involving both near-equilibrium geometry and bond breaking, where other methods falter due to the presence of strong quantum correlations at large bond separation. This experiment highlights NNBF’s capability to accurately capture both strong and weak correlations.

Dissociation curve for N₂ obtained with NNBF, HF, CCSD, and CCSD(T) methods

Figure 4. Dissociation curve for N₂ obtained with NNBF, HF, CCSD, and CCSD(T) methods. The FCI energy is used as the ground-truth energy. The NNBF energy is trained using the FSSC scheme with Nu=4096<NtN_u = 4096 < N_t, and the reported energy here is computed exactly, as it remains feasible to compute.

The Effect of Network Architecture With Full Hilbert Space

One of the critical factors influencing the performance of NNBF is the expressiveness of the ansatz, which is directly linked to the network architecture, including the number of hidden units hh, the number of backflow determinants DD, and the number of hidden layers LL. We first explore the impact of the size and shape of the network by comparing NNBF models with different values of hh, LL, and DD on methane. It’s worth noting that we employ the entire Hilbert space (Nu=NtN_u = N_t) during optimization to eliminate the approximation error in equation (4) and (5) for small systems like methane. While this approach inherently encompasses the combinatorial complexity in the summation, making it non-scalable, it still provides insight into how the network architecture influences the representability of the ansatz and its performance.

MoleculeNtN_tNuN_uCCSDFCIBest NNQSNNBF
N2_2144004096-107.656080-107.660206-107.6602a-107.660198(5)
CH4_4158764096-39.806022-39.806259-39.8062a-39.806255(3)*
LiF441004096-105.159235-105.166172-105.1661a-105.166178(9)
LiCl10020018192-460.847579-460.849618-460.8496b-460.849615(17)
Li2_2O4140922565536-87.885514-87.892693-87.8922a-87.892645(9)
C2_2H4_4O2538950544131072-151.120474--151.120486c-151.121703(68)

Table 1. Ground state energies obtained by NNBF. The conventional CCSD and FCI results are listed for comparison. NtN_t indicates the total number of physically valid configurations conserving the particle number and the total charge. Best NNQS column depicts best published NNQS energies, and footnotes mark the methods: a: QiankunNet [31], b: NAQS [28], c: MADE [29]. The NNBF energy column displays the post-training MCMC inference energy. The energy marked by * is obtained by setting D=3D = 3.

As shown in Figure 5, while there is an overall improvement in accuracy with an increase in the number of layers, the transition from 2 to 3 layers does not exhibit a significant enhancement; in fact, the energy increases slightly which suggests it gets caught in a slightly higher local minima. Additionally, increasing the number of hidden units results in a uniform improvement in accuracy before reaching saturation. According to a linear regression analysis of the log-errors, the error scales with the number of hidden units as O(h−2.253083)O(h^{-2.253083}). Moreover, augmenting the number of determinants reduces the energy error while keeping LL and hh fixed, although the benefits seem to plateau after reaching 4 determinants. This suggests that extensive linear combinations of determinants may not be essential as in multi-determinant SJ, consistent with findings reported in FermiNet [25].

Effects of network architecture on the NNBF performance on CH4

Figure 5. Effects of network architecture on the NNBF performance on CH4_4 (No=18N_o = 18, Ne=10N_e = 10, Nu=NtN_u = N_t, L=1L = 1). Each point is one run of the same model. Double precision is employed for calculating the exact inference energy, as higher precision is necessary when the NNBF state closely approaches the true ground state. (a) Effect of network depth. The increase in performance levels off after the addition of 2 hidden layers to NNBF states. (b) Effect of number of hidden units. A wider hidden layer consistently improves accuracy, with the energy error decreasing at a rate of O(h−2.256228)O(h^{-2.256228}) with R2=0.968270R^2 = 0.968270 until h=64h = 64. (c) Effect of number of determinants (h=30h = 30). Expanding the number of determinants does reduce the energy error and begins to plateau after 4 determinants.

It’s noteworthy that all curves in Figure 5 converge to an energy level close to the FCI energy within single precision. This indicates that these saturation points likely represent the global minima, as they have reached machine precision.

The Effect of Network Architecture With Limited Batch Size

It’s intriguing to explore whether the impact of the network architecture on NNBF remains consistent when using the entire Hilbert space is not feasible in large systems, which is the typical regime of interest. Therefore, we conducted the same procedure as described in Section IIIC on Li2_2O with Nu=8192N_u = 8192. The results are presented in Figure 6.

Effects of network architecture on NNBF performance on the Li2O

Figure 6. Effects of network architecture on NNBF performance on the Li2_2O with Nu=8192N_u = 8192. (a) Effect of network depth. The improvement with more layers quickly saturates after 2 layers are added to NNBF states. Baselines from other NNQS works have been provided for comparison: QiankunNet [31], NAQS [28], MADE [29], QiankunNet* [32], RBM-SC [30]. The dark orange star denotes the best NNBF energy we have obtained with a greater NuN_u. (b) Effect of number of hidden units. Wider hidden layer continuously improves the accuracy with the absolute error drops at a rate of O(h−0.671855)O(h^{-0.671855}) with R2=0.952529R^2 = 0.952529 until h=256h = 256. (c) Effect of number of determinants (h=64h = 64). Increasing the number of determinants reduces the energy error, but the improvement begins to plateau after reaching 4 determinants.

As the number of hidden units increases while maintaining 2 layers and 1 backflow determinant, accuracy uniformly improves and saturates after reaching 256 units. Additionally, adding more layers also increases accuracy, but the improvement from 2 to 3 layers is not significant, suggesting that additional layers may yield only minor gains. Expanding the number of determinants does reduce the energy error, with the improvement beginning to plateau after 4 determinants. The pattern shown in Figure 6 is essentially consistent with that shown in Figure 5.

A significant observation from Figure 6a is that our NNBF state with zero hidden layer, i.e. the configuration-dependent single particle orbitals are equal to Winput+bW_{\mathrm{input}} + b, yields a respectable energy −87.892004(67) Ha. Remarkably, this performance surpasses not only the CCSD baseline but also all existing NNQS results except QiankunNet [31].

Interplay Between Batch Size And Network Architecture

In the previous subsections, we considered the role of increasing the size of the neural network to improve the representability of the wave-function and hence its energy. Somewhat surprisingly, we find that the final energy depends not only on the network architecture but also the batch size used in optimization. In particular, in Figure 6a, we see that by using a larger batch size, we get a non-trivial improvement in the energy for Li2_2O. In this section, we more systematically study the role of changing the batch size of the optimization for a fixed architecture.

The primary role of the batch size NuN_u is to determine the accuracy of the energy and gradients; notice that because we are using FSSС, larger batch sizes involve more unique independent configurations and therefore involve configurations that might not otherwise be seen.

To investigate NuN_u, we trained our model with batches of varying sizes for Li2_2O, employing two sets of network architectures: an optimal one (L,h,D)=(2,256,1)(L,h,D) = (2,256,1) and a sub-optimal one (L,h,D)=(2,64,1)(L,h,D) = (2,64,1) identified in

Section III D. Figure 7a illustrates that increasing the batch size uniformly enhances the performance of the NNBF state for both network architectures. Remarkably, from the inset in Figure 7a, we see that inferior architectures with larger batch size can reach lower energies than better architectures with smaller batch sizes. Both the improvement in energy from increasing the batch size and hidden units appears to scale polynomially (i.e. roughly linear on a log-log plot) with other hyperparameters fixed exhibiting convergence rates of O(Nu−1.137885)O(N_u^{-1.137885}) and O(h−0.671855)O(h^{-0.671855}) respectively until a point of saturation. Surprisingly increasing batch size converges faster; note that the computational complexity to increasing NuN_u scales linearly while increasing hh scales quadratically

(since L>1L > 1; details are provided in Appendix C). Increasing NuN_u also offers another inherent advantage: we can conveniently initiate our optimization at a smaller NuN_u, and once it has reached convergence, we can employ the trained NNBF state as the initial state for training with a larger batch size, as the network architecture remains unchanged. This enables a non-trivial portion of our optimization iterations to be conducted at smaller NuN_u, thereby saving computational time. In Figure 7b, we demonstrate both ramping up NuN_u from a smaller NuN_u as well as running the full optimization at a larger NuN_u. We find that ramping up NuN_u reaches very similar energies with the same number of iterations and has the additional benefit that, in the energy trace, there is less spikiness caused by selection of configurations from regions that haven’t been well-trained. These spikes can significantly impair the initial training process, particularly in large systems, and potentially prevent the energy from descending. Initiating with a smaller NuN_u can help mitigate this issue by exposing fewer new configurations to the algorithm at each iteration. This reduces the likelihood of encountering the not-well-trained regions before the NNBF state has generalized to unseen configurations. Notice that ramping hh is significantly more difficult as it requires some form of transfer learning to “move” the trained wave-function from a smaller to larger network.

We further wish to understand if there is a relationship between the complexity of the network and the batch size required to saturate the optimization of that network. Another notable observation from Figure 7a is that the energy for the network with h=64h = 64 begins to saturate within the range of batch sizes tested, while the network with h=256h = 256 shows ongoing improvements in energy. To better understand this, we focus on a smaller molecule, CH4_4, where we can thoroughly investigate this correlation. Figure 7c consistently shows that as the number of hidden units increases, the saturating batch size also grows larger. This further validates that a larger NuN_u is required for a more expressive NNBF wave-function to reach the global minimum, mirroring a common relationship observed in traditional machine learning between neural network size and dataset size.

Conclusions

In this work, we show how to use NNBF with a deterministic FSSC optimization scheme to reach state-of-the-art energies on ab-initio second quantized molecular Hamiltonians. On the large molecules, these energies are better than any other variational NNQS approaches as well as the CCSD method. This gives strong evidence that NNBF is a powerful ansatz for second-quantized ab-initio QC. We additionally show that the FSSC scheme effectively captures crucial configurations, leading to faster, lower, and smoother energy convergence compared to conventional VMC with equivalent batch sizes.

We systematically investigated the impact of network architecture on NNBF performance. Increasing the number of hidden layers and backflow determinants reduces energy error but reaches saturation quickly. The aspect of the network architecture which makes the largest difference in improving the energy is increasing the number of hidden units which seems to decrease the energy polynomially until saturation. These trends remain consistent across experiments both on small molecules optimized with batch sizes equal to the full Hilbert space as well as on larger molecules optimized with batch sizes much smaller than the full Hilbert space.

Our analysis of batch size suggests that increasing it is crucial to improving the final optimized energy and, in fact, is a more efficient and effective strategy for improving NNBF energies compared to expanding hidden units in the regime where neither of these parameters is saturated. Increasing the batch size is also less costly computationally and easier to ramp up as a function of optimization iteration.

Further technical advancements, as explored in prior works [29, 32, 33], hold the potential to further enhance the performance and efficiency of NNBF. Given the more favorable computational costs of our NNBF ansatz compared to RBMs and ARNs in the large basis set limit (see Appendix C), future research could focus on investigating how easily the NNBF model extrapolates to the complete basis set limit allowing closer comparison with continuum methods based on first quantization [25, 26].

Acknowledgments

We would like to express our gratitude for the insightful discussions with Zejun Liu and Kieran Loehr. This work utilized the Illinois Campus Cluster, a computing resource operated by the Illinois Campus Cluster Program in collaboration with the National Center for Supercomputing Applications and supported by funds from the University of Illinois at Urbana-Champaign. We also acknowledge support from the NSF Quantum Leap Challenge Institute for Hybrid Quantum Architectures and Networks (NSF Award 2016136).

Experimental Setup

In this work, we utilize the Adam optimizer to minimize the energy expectation value of our NNBF wave-function to approximate the ground state of various molecules with the STO-3G basis set. The default hyperparameters are shown in Table II, and the energies for HF, CCSD, CCSD(T), and FCI baselines are calculated using the PyP_ySCF software package [38]. Following the training process with FSSC scheme whose detailed implementation is provided in Sec II C, we conduct a separate MCMC inference procedure to obtain the stochastic estimation of the energy of the trained NNBF state and report these numbers in all the experiments, except for the demonstration of optimization progress with FSSC scheme.

Sampling from the current unnormalized probability distribution pˉθ(x)=ψθ(x)2\bar{p}_{\theta}(x) = \psi_{\theta}(x)^2, each walker generates a Markov chain {∣x1⟩→∣x2⟩→∣x3⟩→⋯ }\lbrace|x_1\rangle\rightarrow|x_2\rangle\rightarrow|x_3\rangle\rightarrow\cdots\rbrace using the Metropolis–Hastings algorithm, proposing a swap between an occupied site and an unoccupied one at each iteration. Subsequently, samples are downsampled from the Markov chain at an interval K1K_1, forming {∣x1⟩,∣xK1⟩,∣x2K1⟩,⋯ ,∣xMK1⟩}\lbrace|x_1\rangle, |x_{K_1}\rangle, |x_{2K_1}\rangle, \cdots, |x_{MK_1}\rangle\rbrace. In this work, we typically set K1=10NeK_1 = 10N_e and M=1000M = 1000, with Nw=1024N_w = 1024 walkers operating concurrently. The MM samples will be averaged within each walker to generate NwN_w energies.

Subsequently, the energy expectation value is computed by averaging over these energies, while the uncertainties are derived from the variance among these NwN_w energies to mitigate the influence of auto-correlation time. The NNBF energy and error bars presented in this report are obtained through this procedure. Note that we followed the approach outlined in ref. [39] for initializing walkers. Specifically, we selected the M=8M=8 most dominant configurations from the last core space utilized in the FSSC scheme. These configurations were used to set the initial positions of the NwN_w walkers based on their relative probabilities, and the walkers underwent K2=200K_2=200 burn-in steps before starting to collect samples. This approach helps prevent the walkers from becoming trapped in low-probability modes of a multi-modal probability landscape, facilitating faster convergence to equilibrium.

The same sampling routine can be employed to draw MCMC samples during the training phase with MCMC scheme, with the distinction that the network is updated at an interval KK, and a larger NwN_w may be utilized to ensure an accurate stochastic estimate of energy and its gradient.

HyperparameterValue
Energy unitHartree
Atomic orbitalSTO-3G
Molecular orbitalcanonical Hartree-Fock
FrameworkJAX
Precisionfloat32
OptimizerAdam
Adam’s β1\beta_10.9
Adam’s β2\beta_20.999
Adam’s ϵ\epsilon1×10−81 \times10^{-8}
Learning rate10−3×(1+10−4t)−110^{-3} \times(1+10^{-4}t)^{-1}
Training iterations50000
Number of pretraining walkers8192
Pretraining iterations500
Number of dominant configurations for walker initialization8
MCMC burn-in steps right after initialization200
Number of MCMC walkers1024
MCMC downsample interval10Ne10N_e
MCMC iterations1000
Number of hidden layers2
Hidden units256
Number of determinants1

Table 2. TABLE II: Default hyperparameters used for all experiments in the paper, unless explicitly stated otherwise

CISD Pre-training

Machine learning scientists understand that initializing the model close to the global minimum significantly aids training. This insight prompted us to pretrain the network to approximate the configuration interaction with single and double excitations (CISD) state, computed using PySCF [38], before optimizing for the expected energy. The pretraining loss function we employed is the negative logarithm of the fidelity between the current

symboldescription
NeN_enumber of electrons
NoN_onumber of spin-orbitals
NtN_tnumber of physically valid configurations
NuN_unumber of unique configurations
LLnumber of hidden layers
hhhidden units
DDnumber of determinants
NwN_wnumber of MCMC walkers

caption: TABLE III: Table of notations

:::

NNBF ansatz and the CISD wave-function:

F(θ)=−ln⁡∣⟨Ψθ∣ΨT⟩∣2⟨Ψθ∣Ψθ⟩⟨ΨT∣ΨT⟩(7)\mathcal{F}(\theta)=-\ln\frac{\lvert\langle\Psi_\theta\mid\Psi_T\rangle\rvert^2}{\langle\Psi_\theta\mid\Psi_\theta\rangle\langle\Psi_T\mid\Psi_T\rangle} \tag*{(7)}

where each component can be formulated as expectation values such as:

⟨Ψθ∣ΨT⟩=Epθ(x)[⟨x∣ΨT⟩⟨x∣Ψθ⟩](8)\langle\Psi_\theta\mid\Psi_T\rangle=\mathbb{E}_{p_\theta(x)}\left[\frac{\langle x\mid\Psi_T\rangle}{\langle x\mid\Psi_\theta\rangle}\right] \tag*{(8)}

Also, the gradient of F(θ)\mathcal{F}(\theta) can be expressed as:

∂F∂θ=−2Re⁡{Epθ(x)[(1⟨Ψθ∣ΨT⟩⟨x∣ΨT⟩⟨x∣Ψθ⟩−1⟨Ψθ∣Ψθ⟩)∇θln⁡⟨Ψθ∣x⟩]}.(9)\frac{\partial\mathcal{F}}{\partial\theta}=-2\operatorname{Re}\left\{\mathbb{E}_{p_\theta(x)}\left[\left(\frac{1}{\langle\Psi_\theta\mid\Psi_T\rangle}\frac{\langle x\mid\Psi_T\rangle}{\langle x\mid\Psi_\theta\rangle}-\frac{1}{\langle\Psi_\theta\mid\Psi_\theta\rangle}\right)\nabla_\theta\ln\langle\Psi_\theta\mid x\rangle\right]\right\}. \tag*{(9)}

Here, (B1) and (B3) are estimated using the standard MCMC procedure, where the samples are drawn from the unnormalized probability pˉθ(x)=ψθ(x)2\bar{p}_\theta(x)=\psi_\theta(x)^2.

The comparison between the trials with and without pretraining is depicted in Figure 8. It’s evident that the objective energy converges faster with CISD pretraining, as it avoids optimizing through a region known to be physically uninteresting. Using MCMC instead of FSSC to draw samples could prevent the time-consuming local energy calculation. Hence, prepending the pretraining to our training routine seems to be a good practice, although we did observe that pretraining stranded the neural network in a poor local optimum in rare cases.

Optimization progress for CH4, with and without CISD pretraining

Figure 8. Optimization progress for CH4_4, with and without CISD pretraining (L=1L=1, h=512h=512, Nu=NtN_u=N_t). The training iteration comprises 20000 steps, and a moving average window of 1000 is applied for better readability.

Computational Complexity Analysis

Using the notations in Table III, we can express the computational cost of NNBF as follows: evaluating the MLP scales as O(Noh+(L−1)h2+hDNoNe)\mathcal{O}(N_oh+(L-1)h^2+hDN_oN_e), while evaluating the determinants scales as O(DNe3)\mathcal{O}(DN_e^3). This yields a total computational cost of O(Noh+(L−1)h2+hDNoNe+DNe3)\mathcal{O}(N_oh+(L-1)h^2+hDN_oN_e+DN_e^3) for calculating the amplitude for one configuration and O(Nu[Noh+(L−1)h2+hDNoNe+DNe3])\mathcal{O}(N_u[N_oh+(L-1)h^2+hDN_oN_e+DN_e^3]) for a batch of NuN_u configurations.

In the large basis limit, where NoN_o increases while NeN_e is fixed, the time complexity (without optimization) of

NNBF would be O(No)\mathcal{O}(N_o) if we also fix the network architecture (LL, hh, DD). In comparison, for a simple shallow RBM, its computational cost is O(No+αNo+αNo2)=O(No2)\mathcal{O}(N_o+\alpha N_o+\alpha N_o^2)=\mathcal{O}(N_o^2), where α\alpha is the hidden layer density. Similarly, for an ARN, the computational cost for calculating the conditional probability distribution pθ(xi∣x1,x2,…,xi−1)p_\theta(x_i\mid x_1,x_2,\ldots,x_{i-1}) is at least Ω(i)\Omega(i) (transformer-based model has even worse complexity, O(i2)\mathcal{O}(i^2)), resulting in a cost of Ω(No2)\Omega(N_o^2) for exact sampling of one unique configuration, pθ(x)=∏i=2Nopθ(xi∣x1,x2,…,xi−1)p_\theta(x)=\prod_{i=2}^{N_o}p_\theta(x_i\mid x_1,x_2,\ldots,x_{i-1}). Both architectures exhibit a computational cost quadratic in NoN_o, not NeN_e. This distinction renders our NNBF model computationally more favorable when NoN_o is large with NeN_e fixed.

References

References

  1. [1]J. D. Whitfield, P. J. Love, and A. Aspuru-Guzik, Computational complexity in electronic structure, Phys. Chem. Chem. Phys. 15, 397–411 (2013).
  2. [2]B. O’Gorman, S. Irani, J. Whitfield, and B. Fefferman, Intratability of electronic structure in a fixed basis, PRX Quantum 3, 020322 (2022).
  3. [3]C. David Sherrill and H. F. Schaefer, The configuration interaction method: Advances in highly correlated approaches, in Advances in Quantum Chemistry (Elsevier, 1999) p. 143–269.
  4. [4]F. Coester and H. Kümmel, Short-range correlations in nuclear wave functions, Nuclear Physics 17, 477–485 (1960).
  5. [5]I. W. Bulik, T. M. Henderson, and G. E. Scuseria, Can single-reference coupled cluster theory describe static correlation?, Journal of Chemical Theory and Computation 11, 3171–3179 (2015).
  6. [6]W. M. C. Foulkes, L. Mitas, R. J. Needs, and G. Rajagopal, Quantum monte carlo simulations of solids, Rev. Mod. Phys. 73, 33 (2001).
  7. [7]S. R. White, Density matrix formulation for quantum renormalization groups, Phys. Rev. Lett. 69, 2863 (1992).
  8. [8]S. R. White and R. L. Martin, Ab initio quantum chemistry using the density matrix renormalization group, The Journal of Chemical Physics 110, 4127–4130 (1999).
  9. [9]B. K. Clark, M. A. Morales, J. McMinis, J. Kim, and G. E. Scuseria, Computing the energy of a water molecule using multideterminants: A simple, efficient algorithm, The Journal of Chemical Physics 135, 10.1063/1.3665391 (2011).
  10. [10]R. P. Feynman and M. Cohen, Energy spectrum of the excitations in liquid helium, Phys. Rev. 102, 1189 (1956).
  11. [11]Y. Kwon, D. M. Ceperley, and R. M. Martin, Effects of three-body and backflow correlations in the two-dimensional electron gas, Phys. Rev. B 48, 12037 (1993).
  12. [12]J. B. Anderson, A random-walk simulation of the schrödinger equation: H+3, The Journal of Chemical Physics 63, 1499–1503 (1975).
  13. [14]K. Choo, T. Neupert, and G. Carleo, Two-dimensional frustrated J1–J2 model studied with neural network quantum states, Phys. Rev. B 100, 125124 (2019).
  14. [15]K. Choo, G. Carleo, N. Regnault, and T. Neupert, Symmetries and many-body excitations with neural-network quantum states, Phys. Rev. Lett. 121, 167204 (2018).
  15. [16]A. Nagy and V. Savona, Variational quantum monte carlo method with a neural-network ansatz for open quantum systems, Phys. Rev. Lett. 122, 250501 (2019).
  16. [17]O. Sharir, Y. Levine, N. Wies, G. Carleo, and A. Shashua, Deep autoregressive models for the efficient variational simulation of many-body quantum systems, Phys. Rev. Lett. 124, 020503 (2020).
  17. [18]Y. Nomura, A. S. Darmawan, Y. Yamaji, and M. Imada, Restricted boltzmann machine learning for solving strongly correlated quantum systems, Phys. Rev. B 96, 205152 (2017).
  18. [19]F. Ferrari, F. Becca, and J. Carrasquilla, Neural gutzwiller-projected variational wave functions, Phys. Rev. B 100, 125131 (2019).
  19. [20]J. Stokes, J. R. Moreno, E. A. Pnevmatikakis, and G. Carleo, Phases of two-dimensional spinless lattice fermions with first-quantized deep neural-network quantum states, Phys. Rev. B 102, 205122 (2020).
  20. [21]J. Lin, G. Goldshlager, and L. Lin, Explicitly antisymmetrized neural network layers for variational monte carlo simulation, Journal of Computational Physics 474, 111765 (2023).
  21. [22]Z. Chen, D. Luo, K. Hu, and B. K. Clark, Simulating 2+1d lattice quantum electrodynamics at finite density with neural low wave functions (2022).
  22. [23]D. Luo and B. K. Clark, Backflow transformations via neural networks for quantum many-body wave functions, Phys. Rev. Lett. 122, 226401 (2019).
  23. [24]Z. Liu and B. K. Clark, Unifying view of fermionic neural network quantum states: From neural network backflow to hidden fermion determinant states, Phys. Rev. B 110, 115124 (2024).
  24. [25]D. Pfau, J. S. Spencer, A. G. D. G. Matthews, and W. M. Foulkes, Ab initio solution of the many-electron schrödinger equation with deep neural networks, Physical Review Research 2, 10.1103/physrevresearch.2.033429 (2020).
  25. [26]J. Hermann, Z. Schätzle, and F. Noé, Deep-neural-network solution of the electronic schrödinger equation, Nature Chemistry 12, 891 (2020).
  26. [27]K. Choo, A. Mezzacapo, and G. Carleo, Fermionic neural-network states for ab-initio electronic structure, Nature Communications 11, 2368 (2020).
  27. [28]T. D. Barrett, A. Malyshev, and A. I. Lvovsky, Autoregressive neural-network wavefunctions for ab initio quantum chemistry, Nature Machine Intelligence 4, 351 (2022).
  28. [29]T. Zhao, J. Stokes, and S. Veerapaneni, Scalable neural quantum states architecture for quantum chemistry, Machine Learning: Science and Technology 4, 025034 (2023).
  29. [30]X. Li, J.-C. Huang, G.-Z. Zhang, H.-E. Li, C.-S. Cao, D. Lv, and H.-S. Hu, A nonstochastic optimization algorithm for neural-network quantum states, Journal of Chemical Theory and Computation 19, 8156 (2023).
  30. [31]H. Shang, C. Guo, Y. Wu, Z. Li, and J. Yang, Solving schrödinger equation with a language model (2023).
  31. [32]Y. Wu, C. Guo, Y. Fan, P. Zhou, and H. Shang, Nnqs-transformer: an efficient and scalable neural network quantum states approach for ab initio quantum chemistry, in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’23 (Association for Computing Machinery, New York, NY, USA, 2023).
  32. [33]A. Malyshev, J. M. Arrazola, and A. I. Lvovsky, Autoregressive neural quantum states with quantum number symmetries (2023).
  33. [34]D. P. Kingma and J. Ba, Adam: A method for stochastic optimization (2014).
  34. [35]L. Bytautas and K. Ruedenberg, A priori identification of configurational deadwood, Chemical Physics 356, 64 (2009), moving Frontiers in Quantum Chemistry:.
  35. [36]J. S. Anderson, F. H. Deiner-Zadeh, and P. W. Ayers, Breaking the curse of dimension for the electronic schrödinger equation with functional analysis, Compu- tational and Theoretical Chemistry 1142, 66 (2018).
  36. [37]Y. Wang, J. Xiao, T. O. Suzek, J. Zhang, J. Wang, and S. H. Bryant, Pubchem: a public information system for analyzing bioactivities of small molecules, Nucleic Acids Research 37, W623–W633 (2009).
  37. [38]Q. Sun, T. C. Berkelbach, N. S. Blunt, G. H. Booth, S. Guo, Z. Li, J. Liu, J. D. McClain, E. R. Sayfutyarova, S. Sharma, S. Wouters, and G. K. Chan, PySCF: the python-based simulations of chemistry framework, WIREs Computational Molecular Science 8, 10.1002/wcms.1340 (2017).
  38. [39]D. Foreman-Mackey, D. W. Hogg, D. Lang, and J. Goodman, emcee: The mcmc hammer, Publications of the Astronomical Society of the Pacific 125, 306–312 (2013).

Paper details

vlnc:W_z8jxqgfb
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents