WorksPhysics and Astronomy
Neural network backflow for ab-initio quantum chemistry
Abstract
The ground state of second-quantized quantum chemistry Hamiltonians provides access to an important set of chemical properties. Wavefunctions based on ML architectures have shown promise in approximating these ground states in a variety of physical systems. In this work, we show how to achieve state-of-the-art energies for molecular Hamiltonians using the the neural network backflow wave-function. To accomplish this, we optimize this ansatz with a variant of the deterministic optimization scheme based on SCI introduced by [Li, et. al JCTC (2023)] which we find works better than standard MCMC sampling. For the molecules we studied, NNBF gives lower energy states than both CCSD and other neural network quantum states. We systematically explore the role of network size as well as optimization parameters in improving the energy. We find that while the number of hidden layers and determinants play a minor role in improving the energy, there is significant improvements in the energy from increasing the number of hidden units as well as the batch size used in optimization with the batch size playing a more important role.
Introduction
The solution to the electronic Schrödinger equation for a given molecular system provides comprehensive access to its chemical properties from first principles. However, analytically solving the Schrödinger equation is feasible only for hydrogen-like atoms in real space. Exact approaches in second quantization, such as the full configuration interaction (FCI) method, are intractable for large systems due to the exponential growth of the Hilbert space with the system size. The many-body electronic problem is fundamentally NP hard [1, 2], motivating efforts to develop numerical methods for approximating solutions in ab-initio quantum chemistry (QC).
Configuration interaction (CI) methods [3] consider excitations above the Hartree-Fock (HF) reference state up to a fixed order, while coupled cluster (CC) approaches [4] use exponential excitation operators up to a certain order to access all excited states. Although CC techniques are often considered the “gold standard” in ab-initio QC, they may fail in regimes of strong static correlations [5].
Alternatively, variational wave-functions such as the Slater-Jastrow (SJ) [6] and matrix-product states (MPS) [7, 8], can be optimized to approximate the ground state. Various approaches build on top of the SJ approach such as multi-determinant Jastrow wave-functions [9] or alternatively backflow transformation [10], which renders orbitals dependent on the coordinates of all electrons [11]. Projector methods like diffusion Monte Carlo (DMC) [12] systematically refine the variational starting point. However, the effectiveness of these approaches is often bounded by the expressiveness of the wave function and the efficiency of the optimization. For instance, MPS are inadequate for highly entangled systems and face challenges in dimensions greater than one. Slater-Jastrow-Backflow (SJB) are often hard to systematically improve.
Machine learning architectures presents significant promise as concise wave-functions to address certain constraints of existing variational representations. Ref. 13 initially demonstrated that restricted Boltzmann machines (RBM), trained via variational Monte Carlo (VMC), could effectively capture the intricate correlations in certain interacting spin models and numerous advances [14, 15, 16, 17] since then have improved neural network quantum states (NNQS) for spin models. More recently, there has been significant work using machine learning architectures to develop wave-functions for Fermion systems [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33].
The advent of wave-functions for Fermions has motivated an interest in addressing molecular and QC Hamiltonians. The FermiNet [25] and PauliNet [26] wave-functions, which are real-space generalizations of the Neural Net Backflow (NNBF) [23], have shown to be competitive with other ab-initio QC methods in continuous space. Using a second quantized Hamiltonian in a finite basis, ref. 27 mapped the molecular system onto an equivalent spin problem and demonstrated that RBM could achieve higher accuracy than CCSD(T). However, they noted that using ordinary Markov chain Monte Carlo (MCMC) methods for sampling was inefficient due to sharp peaks in the underlying probability distribution around the HF state and neighboring excited states. To address this sampling inefficiency as well as gain more accuracy than the RBM, subsequent studies [28, 29, 31, 32] introduced autoregressive neural networks (ARN) for exact sampling. Further improvements in optimization were introduced by ref. 30 which proposed a non-stochastic optimization scheme adapted from the selected configuration interaction (SCI) method to deterministically select samples on-the-fly leading to competitive molecular energies using RBMs. Interestingly, despite NNBF style ansatz demonstrating high accuracy in the continuum for QC, the NNBF approach has not yet been applied to second quantized QC Hamiltonians.
In this work, we address this omission using the NNBF [23] to approximate the ground state of molecular systems in second quantization. To optimize our wave-function, we apply and demonstrate the superiority of a deterministic fixed-size selected configuration (FSSC) iteration scheme, adapted from ref. 30. Our experiments on various molecules indicate that the NNBF ansatz not only surpasses traditional CCSD benchmarks but also finds lower energies than all other existing NNQS methods on large molecules (and equally good energies on smaller molecules). We also evaluate the dissociation curve for diatomic Nitrogen demonstrating NNBF’s ability to capture both strong and weak quantum correlation. Additionally, we conduct a systematic investigation into the impact of network architecture and batch size, revealing that increasing the batch size is a more efficient and effective approach to improving NNBF performance.
Methods
In this section, we provide a detailed description of the structure of NNBF and the workflow of FSSC.
Neural Network BackFlow Architecture
For a molecular system containing electrons, a basis set composed of spin-orbitals is given to define a many electron wave-function in the form of where is the -th computational basis vector in second quantization. Here, denotes whether the -th spin-orbital is occupied in the -th computational basis vector.
The NNBF consists of a multilayer perceptron (MLP) which generates the ”configuration-dependent” single particle orbitals used in evaluating the Slater determinant (see Figure 1). The MLP directly takes the configuration strings as input feature vector and feeds it through a series of intermediate layers consisting of a fully-connected feed-forward neural network (FNN) combined with a ReLU nonlinear activation function i.e., . Instead of directly returning a scalar wave-function amplitude, the MLP outputs a configuration-dependent additive change to the set of single-particle orbitals for each determinant. In practice, the final output vector of the network, , will be reshaped into an array of matrices, where each matrix is of shape . The -th row of the -th matrix has a physical interpretation that it is the evaluation of the -th set of configuration-dependent orbitals if the -th spin-orbital is occupied. A square matrix can be generated by extracting the rows whose indices correspond to the locations of the electrons in the input configuration, and the amplitude of this configuration is defined as where is the model parameters. Consequently, the full NNBF wave-function is described as .

Figure 1. Illustration of the NNBF architecture with an example input configuration: 2 spin-orbitals with 1 spin-up electron occupying the first spin-orbital and 1 spin-down electron occupying the second spin-orbital. The neural network takes the configuration string as input and outputs a set of configuration-dependent single-particle orbitals of shape . Rows of these orbitals are then selected based on the electron locations to form square matrices, from which determinants are computed. For this example, the first and last rows (the gray orbitals) are selected to compute the determinant. The sum of these determinants yields the amplitude for the input configuration.
Wavefunction Optimization
Given a quantum chemical Hamiltonian,
our target is to minimize the variational energy of our wave-function,
which can be evaluated by MCMC by sampling from the probability distribution where the local energy . To optimize this energy, we will estimate the gradient of it, , and update using ADAM [34]. We perform a CISD pretraining before optimizing with expected energy; the implementation details of this pretraining are provided in Appendix B.
One approach for computing the gradient
is through an MCMC sampling of the energy. For optimizing molecular Hamiltonians, we find that this is a suboptimal way to update the variational parameters. A similar observation was made by ref. 27 which found that typical MCMC sampling technique is inefficient for the ground state of molecular system due to the peaks around the HF state and neighboring excited states [35, 36] which results in drawing many identical samples from the dominant states when sampling. Various techniques have been developed to overcome these issues. For autoregressive variational wave-functions, the probability of a configuration can be modeled using its conditional probabilities, expressed as . This autoregressive property, combined with more efficient techniques, enables sampling the numbers of occurrences for each unique string, thus avoiding duplicate sampling efforts [28, 29, 31, 33]. Recently, Li et al. [30] proposed an algorithm based on SCI to select a set of states and approximate their frequency of occurrences instead of statistically sampling them. This set of states is a deterministic selection of a set of important configurations identified through the modulus of the wave-function ansatz during energy evaluation.
Deterministically Select Sample Space
In this work, we will use a variant of the scheme of ref. [30] which we call the FSSC scheme. For completeness, we describe the entire FSSC approach below while a diagrammatic description of it is also provided in Figure 2.

Figure 2. A diagrammatic description of the workflow of FSSC. Each circle represents one configuration. Initially, the algorithm initializes a parametrized wave-function and a core space of size . After each iteration, the amplitude is computed for all configurations in and the 10 largest unique ones (denoted by red configurations) are selected to form the new core space . The energy (depicted as the loss function in the orange box) and its gradient are estimated by constraining the relative sum to only consider , and the latter is used to update the model parameters.
The full configuration space is divided into three parts as in a typical SCI method: the core space consisted of the dynamically selected important configurations, the connected space containing all other configurations connected to the current core space through nonzero Hamiltonian matrix elements, and the remaining part of the configuration space. The HF state and some neighboring excited states are deterministically pre-selected to form the initial core space of size . After each iteration of optimization, we find the connected space induced from the core space from the last iteration. The new core space will be deterministically chosen by selecting the configurations from the union of and which have the largest unique amplitudes with respect to the new state with updated parameters. This will then be used to evaluate an approximation to the gradient and update the model parameters:
where
and
where the relevant sums are over the core space. Approximating the various quantities above by summing over only the core space performs well when the ground state is predominantly influenced by a small number of significant configurations, which is a typical characteristic of
molecular systems in the canonical HF basis. We note that all the energies reported in this work (not including traces over optimization time) are computed exactly by Monte Carlo (MC) sampling. See Appendix A for details.
The major distinction between our approach and ref. [30] lies in fixing the size of the core space, as opposed to selecting configurations whose modulus relative to the largest one is greater than a selection cutoff, . This allows us to leverage the advantages of just-in-time (JIT) compilation with JAX, as the shape of the input array to the jitted functions remains constant throughout iterations. Additionally, a fixed number of unique configurations determines the computational cost, which is directly proportional to , whereas in most other NNQS works [27, 28, 30, 31], only the total number of samples could be specified. Moreover, a fixed batch size prevents the possibility of an unexpectedly large number of configurations overwhelming the limited computational resources, whereas the core space could potentially encompass the entire configuration space at some point in the ref. [30]. At one point, we have compared a fixed batch size to a dynamically adjusted batch size. The optimized energy using a selection cutoff is essentially the same as the optimized energy with a fixed batch size which is equal to the average batch size of the selection cutoff (Figure 7a).

Figure 7. (a) Experiments exploring the impact of batch size on energy improvements with and on lithium oxide. The energy error decreases approximately following with an value of 0.900605 for and with for . Some data points from Figure 6b are included for comparison in the inset. The black star represents a trial trained with using the sampling scheme from ref. [30]. Its x-position is determined by the average batch size at convergence. The energy closely aligns with the fitted line for the FSSC scheme, indicating that using a dynamic batch size does not offer a noticeable improvement in training. (b) Demonstration of the effectiveness of the batch size scheduling (BSS) strategy, with a moving average window of 100 applied for improved readability. (c) Experiments examining the dependence of energy improvements on batch size with various values on methane. Each data point represents the average and the standard deviation of the exact NNBF energy across 3 seeds.
Results
Here we showcase the effectiveness of the FSSC scheme by comparing it to the typical MCMC scheme, evaluate the performance of the NNBF on various molecules, train the NNBF model to emulate the dissociation curve of nitrogen molecule, and investigate the impact of network architecture and batch size on NNBF. Further details on the default network architectures, hyperparameters, training routine, and the post-training MCMC inference routine can be found in Appendix A.
FSSC vs. Traditional MCMC Optimization
To demonstrate the efficacy of the FSSC scheme in addressing the inefficiencies of MC sampling, we conduct a comparison on lithium oxide using both the FSSC scheme and the standard MCMC scheme. For a fair comparison, we set the number of unique configurations in the FSSC scheme equal to the number of walkers in the MCMC scheme, i.e. .
In Figure 3 we see both faster convergence as well as a lower converged energy for the FSSC scheme compared to the conventional MCMC scheme, which happens to only converge to the CCSD baseline. Additionally, we observe that there are only 111 unique configurations in the 8192 MCMC samples at convergence, with the most frequent sample being the HF configuration appearing in 7686 walkers. This observation validates the inefficient MC sampling discovered by ref. 27 and highlights the FSSC scheme’s ability to effectively capture important configurations in large configuration space, akin to conducting MC sampling with a sample size . Additionally, the FSSC method shows a smoother, more monotonic trend in energy values during the optimization steps, especially compared to the MCMC scheme, which, although it displays an average reduction in energy, does so with noticeable oscillations. This demonstration aligns with the results presented by ref. 30.

Figure 3. Comparison between FSSC and MCMC schemes on lithium oxide (, , ). The red band represents the post-training MCMC inference energy for the FSSC scheme with a width of in each direction around the mean. A moving average window of 100 is applied to improve readability.
Benchmark With Other QC Methods
We assess the performance of our algorithm by comparing molecular ground state energies with CCSD baselines and other NNQS results, utilizing a list of molecules with geometries sourced from PubChem [37]. Table I demonstrates that NNBF not only surpasses conventional CCSD methods but also is equal or lower in energy than all other existing NNQS approaches, particularly showing superiority in larger molecular systems.
Additionally, we explore the NNBF’s ability to capture quantum correlation by computing the dissociation curve of the diatomic nitrogen molecule. As illustrated in Figure 4, the NNBF precisely matches the FCI energy and outperforms all other conventional QC approaches in scenarios involving both near-equilibrium geometry and bond breaking, where other methods falter due to the presence of strong quantum correlations at large bond separation. This experiment highlights NNBF’s capability to accurately capture both strong and weak correlations.

Figure 4. Dissociation curve for N₂ obtained with NNBF, HF, CCSD, and CCSD(T) methods. The FCI energy is used as the ground-truth energy. The NNBF energy is trained using the FSSC scheme with , and the reported energy here is computed exactly, as it remains feasible to compute.
The Effect of Network Architecture With Full Hilbert Space
One of the critical factors influencing the performance of NNBF is the expressiveness of the ansatz, which is directly linked to the network architecture, including the number of hidden units , the number of backflow determinants , and the number of hidden layers . We first explore the impact of the size and shape of the network by comparing NNBF models with different values of , , and on methane. It’s worth noting that we employ the entire Hilbert space () during optimization to eliminate the approximation error in equation (4) and (5) for small systems like methane. While this approach inherently encompasses the combinatorial complexity in the summation, making it non-scalable, it still provides insight into how the network architecture influences the representability of the ansatz and its performance.
| Molecule | CCSD | FCI | Best NNQS | NNBF | ||
| N | 14400 | 4096 | -107.656080 | -107.660206 | -107.6602a | -107.660198(5) |
| CH | 15876 | 4096 | -39.806022 | -39.806259 | -39.8062a | -39.806255(3)* |
| LiF | 44100 | 4096 | -105.159235 | -105.166172 | -105.1661a | -105.166178(9) |
| LiCl | 1002001 | 8192 | -460.847579 | -460.849618 | -460.8496b | -460.849615(17) |
| LiO | 41409225 | 65536 | -87.885514 | -87.892693 | -87.8922a | -87.892645(9) |
| CHO | 2538950544 | 131072 | -151.120474 | - | -151.120486c | -151.121703(68) |
Table 1. Ground state energies obtained by NNBF. The conventional CCSD and FCI results are listed for comparison. indicates the total number of physically valid configurations conserving the particle number and the total charge. Best NNQS column depicts best published NNQS energies, and footnotes mark the methods: a: QiankunNet [31], b: NAQS [28], c: MADE [29]. The NNBF energy column displays the post-training MCMC inference energy. The energy marked by * is obtained by setting .
As shown in Figure 5, while there is an overall improvement in accuracy with an increase in the number of layers, the transition from 2 to 3 layers does not exhibit a significant enhancement; in fact, the energy increases slightly which suggests it gets caught in a slightly higher local minima. Additionally, increasing the number of hidden units results in a uniform improvement in accuracy before reaching saturation. According to a linear regression analysis of the log-errors, the error scales with the number of hidden units as . Moreover, augmenting the number of determinants reduces the energy error while keeping and fixed, although the benefits seem to plateau after reaching 4 determinants. This suggests that extensive linear combinations of determinants may not be essential as in multi-determinant SJ, consistent with findings reported in FermiNet [25].

Figure 5. Effects of network architecture on the NNBF performance on CH (, , , ). Each point is one run of the same model. Double precision is employed for calculating the exact inference energy, as higher precision is necessary when the NNBF state closely approaches the true ground state. (a) Effect of network depth. The increase in performance levels off after the addition of 2 hidden layers to NNBF states. (b) Effect of number of hidden units. A wider hidden layer consistently improves accuracy, with the energy error decreasing at a rate of with until . (c) Effect of number of determinants (). Expanding the number of determinants does reduce the energy error and begins to plateau after 4 determinants.
It’s noteworthy that all curves in Figure 5 converge to an energy level close to the FCI energy within single precision. This indicates that these saturation points likely represent the global minima, as they have reached machine precision.
The Effect of Network Architecture With Limited Batch Size
It’s intriguing to explore whether the impact of the network architecture on NNBF remains consistent when using the entire Hilbert space is not feasible in large systems, which is the typical regime of interest. Therefore, we conducted the same procedure as described in Section IIIC on LiO with . The results are presented in Figure 6.

Figure 6. Effects of network architecture on NNBF performance on the LiO with . (a) Effect of network depth. The improvement with more layers quickly saturates after 2 layers are added to NNBF states. Baselines from other NNQS works have been provided for comparison: QiankunNet [31], NAQS [28], MADE [29], QiankunNet* [32], RBM-SC [30]. The dark orange star denotes the best NNBF energy we have obtained with a greater . (b) Effect of number of hidden units. Wider hidden layer continuously improves the accuracy with the absolute error drops at a rate of with until . (c) Effect of number of determinants (). Increasing the number of determinants reduces the energy error, but the improvement begins to plateau after reaching 4 determinants.
As the number of hidden units increases while maintaining 2 layers and 1 backflow determinant, accuracy uniformly improves and saturates after reaching 256 units. Additionally, adding more layers also increases accuracy, but the improvement from 2 to 3 layers is not significant, suggesting that additional layers may yield only minor gains. Expanding the number of determinants does reduce the energy error, with the improvement beginning to plateau after 4 determinants. The pattern shown in Figure 6 is essentially consistent with that shown in Figure 5.
A significant observation from Figure 6a is that our NNBF state with zero hidden layer, i.e. the configuration-dependent single particle orbitals are equal to , yields a respectable energy −87.892004(67) Ha. Remarkably, this performance surpasses not only the CCSD baseline but also all existing NNQS results except QiankunNet [31].
Interplay Between Batch Size And Network Architecture
In the previous subsections, we considered the role of increasing the size of the neural network to improve the representability of the wave-function and hence its energy. Somewhat surprisingly, we find that the final energy depends not only on the network architecture but also the batch size used in optimization. In particular, in Figure 6a, we see that by using a larger batch size, we get a non-trivial improvement in the energy for LiO. In this section, we more systematically study the role of changing the batch size of the optimization for a fixed architecture.
The primary role of the batch size is to determine the accuracy of the energy and gradients; notice that because we are using FSSС, larger batch sizes involve more unique independent configurations and therefore involve configurations that might not otherwise be seen.
To investigate , we trained our model with batches of varying sizes for LiO, employing two sets of network architectures: an optimal one and a sub-optimal one identified in
Section III D. Figure 7a illustrates that increasing the batch size uniformly enhances the performance of the NNBF state for both network architectures. Remarkably, from the inset in Figure 7a, we see that inferior architectures with larger batch size can reach lower energies than better architectures with smaller batch sizes. Both the improvement in energy from increasing the batch size and hidden units appears to scale polynomially (i.e. roughly linear on a log-log plot) with other hyperparameters fixed exhibiting convergence rates of and respectively until a point of saturation. Surprisingly increasing batch size converges faster; note that the computational complexity to increasing scales linearly while increasing scales quadratically
(since ; details are provided in Appendix C). Increasing also offers another inherent advantage: we can conveniently initiate our optimization at a smaller , and once it has reached convergence, we can employ the trained NNBF state as the initial state for training with a larger batch size, as the network architecture remains unchanged. This enables a non-trivial portion of our optimization iterations to be conducted at smaller , thereby saving computational time. In Figure 7b, we demonstrate both ramping up from a smaller as well as running the full optimization at a larger . We find that ramping up reaches very similar energies with the same number of iterations and has the additional benefit that, in the energy trace, there is less spikiness caused by selection of configurations from regions that haven’t been well-trained. These spikes can significantly impair the initial training process, particularly in large systems, and potentially prevent the energy from descending. Initiating with a smaller can help mitigate this issue by exposing fewer new configurations to the algorithm at each iteration. This reduces the likelihood of encountering the not-well-trained regions before the NNBF state has generalized to unseen configurations. Notice that ramping is significantly more difficult as it requires some form of transfer learning to “move” the trained wave-function from a smaller to larger network.
We further wish to understand if there is a relationship between the complexity of the network and the batch size required to saturate the optimization of that network. Another notable observation from Figure 7a is that the energy for the network with begins to saturate within the range of batch sizes tested, while the network with shows ongoing improvements in energy. To better understand this, we focus on a smaller molecule, CH, where we can thoroughly investigate this correlation. Figure 7c consistently shows that as the number of hidden units increases, the saturating batch size also grows larger. This further validates that a larger is required for a more expressive NNBF wave-function to reach the global minimum, mirroring a common relationship observed in traditional machine learning between neural network size and dataset size.
Conclusions
In this work, we show how to use NNBF with a deterministic FSSC optimization scheme to reach state-of-the-art energies on ab-initio second quantized molecular Hamiltonians. On the large molecules, these energies are better than any other variational NNQS approaches as well as the CCSD method. This gives strong evidence that NNBF is a powerful ansatz for second-quantized ab-initio QC. We additionally show that the FSSC scheme effectively captures crucial configurations, leading to faster, lower, and smoother energy convergence compared to conventional VMC with equivalent batch sizes.
We systematically investigated the impact of network architecture on NNBF performance. Increasing the number of hidden layers and backflow determinants reduces energy error but reaches saturation quickly. The aspect of the network architecture which makes the largest difference in improving the energy is increasing the number of hidden units which seems to decrease the energy polynomially until saturation. These trends remain consistent across experiments both on small molecules optimized with batch sizes equal to the full Hilbert space as well as on larger molecules optimized with batch sizes much smaller than the full Hilbert space.
Our analysis of batch size suggests that increasing it is crucial to improving the final optimized energy and, in fact, is a more efficient and effective strategy for improving NNBF energies compared to expanding hidden units in the regime where neither of these parameters is saturated. Increasing the batch size is also less costly computationally and easier to ramp up as a function of optimization iteration.
Further technical advancements, as explored in prior works [29, 32, 33], hold the potential to further enhance the performance and efficiency of NNBF. Given the more favorable computational costs of our NNBF ansatz compared to RBMs and ARNs in the large basis set limit (see Appendix C), future research could focus on investigating how easily the NNBF model extrapolates to the complete basis set limit allowing closer comparison with continuum methods based on first quantization [25, 26].
Acknowledgments
We would like to express our gratitude for the insightful discussions with Zejun Liu and Kieran Loehr. This work utilized the Illinois Campus Cluster, a computing resource operated by the Illinois Campus Cluster Program in collaboration with the National Center for Supercomputing Applications and supported by funds from the University of Illinois at Urbana-Champaign. We also acknowledge support from the NSF Quantum Leap Challenge Institute for Hybrid Quantum Architectures and Networks (NSF Award 2016136).
Experimental Setup
In this work, we utilize the Adam optimizer to minimize the energy expectation value of our NNBF wave-function to approximate the ground state of various molecules with the STO-3G basis set. The default hyperparameters are shown in Table II, and the energies for HF, CCSD, CCSD(T), and FCI baselines are calculated using the SCF software package [38]. Following the training process with FSSC scheme whose detailed implementation is provided in Sec II C, we conduct a separate MCMC inference procedure to obtain the stochastic estimation of the energy of the trained NNBF state and report these numbers in all the experiments, except for the demonstration of optimization progress with FSSC scheme.
Sampling from the current unnormalized probability distribution , each walker generates a Markov chain using the Metropolis–Hastings algorithm, proposing a swap between an occupied site and an unoccupied one at each iteration. Subsequently, samples are downsampled from the Markov chain at an interval , forming . In this work, we typically set and , with walkers operating concurrently. The samples will be averaged within each walker to generate energies.
Subsequently, the energy expectation value is computed by averaging over these energies, while the uncertainties are derived from the variance among these energies to mitigate the influence of auto-correlation time. The NNBF energy and error bars presented in this report are obtained through this procedure. Note that we followed the approach outlined in ref. [39] for initializing walkers. Specifically, we selected the most dominant configurations from the last core space utilized in the FSSC scheme. These configurations were used to set the initial positions of the walkers based on their relative probabilities, and the walkers underwent burn-in steps before starting to collect samples. This approach helps prevent the walkers from becoming trapped in low-probability modes of a multi-modal probability landscape, facilitating faster convergence to equilibrium.
The same sampling routine can be employed to draw MCMC samples during the training phase with MCMC scheme, with the distinction that the network is updated at an interval , and a larger may be utilized to ensure an accurate stochastic estimate of energy and its gradient.
| Hyperparameter | Value |
| Energy unit | Hartree |
| Atomic orbital | STO-3G |
| Molecular orbital | canonical Hartree-Fock |
| Framework | JAX |
| Precision | float32 |
| Optimizer | Adam |
| Adam’s | 0.9 |
| Adam’s | 0.999 |
| Adam’s | |
| Learning rate | |
| Training iterations | 50000 |
| Number of pretraining walkers | 8192 |
| Pretraining iterations | 500 |
| Number of dominant configurations for walker initialization | 8 |
| MCMC burn-in steps right after initialization | 200 |
| Number of MCMC walkers | 1024 |
| MCMC downsample interval | |
| MCMC iterations | 1000 |
| Number of hidden layers | 2 |
| Hidden units | 256 |
| Number of determinants | 1 |
Table 2. TABLE II: Default hyperparameters used for all experiments in the paper, unless explicitly stated otherwise
CISD Pre-training
Machine learning scientists understand that initializing the model close to the global minimum significantly aids training. This insight prompted us to pretrain the network to approximate the configuration interaction with single and double excitations (CISD) state, computed using PySCF [38], before optimizing for the expected energy. The pretraining loss function we employed is the negative logarithm of the fidelity between the current
| symbol | description |
| number of electrons | |
| number of spin-orbitals | |
| number of physically valid configurations | |
| number of unique configurations | |
| number of hidden layers | |
| hidden units | |
| number of determinants | |
| number of MCMC walkers |
caption: TABLE III: Table of notations
:::
NNBF ansatz and the CISD wave-function:
where each component can be formulated as expectation values such as:
Also, the gradient of can be expressed as:
Here, (B1) and (B3) are estimated using the standard MCMC procedure, where the samples are drawn from the unnormalized probability .
The comparison between the trials with and without pretraining is depicted in Figure 8. It’s evident that the objective energy converges faster with CISD pretraining, as it avoids optimizing through a region known to be physically uninteresting. Using MCMC instead of FSSC to draw samples could prevent the time-consuming local energy calculation. Hence, prepending the pretraining to our training routine seems to be a good practice, although we did observe that pretraining stranded the neural network in a poor local optimum in rare cases.

Figure 8. Optimization progress for CH, with and without CISD pretraining (, , ). The training iteration comprises 20000 steps, and a moving average window of 1000 is applied for better readability.
Computational Complexity Analysis
Using the notations in Table III, we can express the computational cost of NNBF as follows: evaluating the MLP scales as , while evaluating the determinants scales as . This yields a total computational cost of for calculating the amplitude for one configuration and for a batch of configurations.
In the large basis limit, where increases while is fixed, the time complexity (without optimization) of
NNBF would be if we also fix the network architecture (, , ). In comparison, for a simple shallow RBM, its computational cost is , where is the hidden layer density. Similarly, for an ARN, the computational cost for calculating the conditional probability distribution is at least (transformer-based model has even worse complexity, ), resulting in a cost of for exact sampling of one unique configuration, . Both architectures exhibit a computational cost quadratic in , not . This distinction renders our NNBF model computationally more favorable when is large with fixed.
References
References
- [1]J. D. Whitfield, P. J. Love, and A. Aspuru-Guzik, Computational complexity in electronic structure, Phys. Chem. Chem. Phys. 15, 397–411 (2013).
- [2]B. O’Gorman, S. Irani, J. Whitfield, and B. Fefferman, Intratability of electronic structure in a fixed basis, PRX Quantum 3, 020322 (2022).
- [3]C. David Sherrill and H. F. Schaefer, The configuration interaction method: Advances in highly correlated approaches, in Advances in Quantum Chemistry (Elsevier, 1999) p. 143–269.
- [4]F. Coester and H. Kümmel, Short-range correlations in nuclear wave functions, Nuclear Physics 17, 477–485 (1960).
- [5]I. W. Bulik, T. M. Henderson, and G. E. Scuseria, Can single-reference coupled cluster theory describe static correlation?, Journal of Chemical Theory and Computation 11, 3171–3179 (2015).
- [6]W. M. C. Foulkes, L. Mitas, R. J. Needs, and G. Rajagopal, Quantum monte carlo simulations of solids, Rev. Mod. Phys. 73, 33 (2001).
- [7]S. R. White, Density matrix formulation for quantum renormalization groups, Phys. Rev. Lett. 69, 2863 (1992).
- [8]S. R. White and R. L. Martin, Ab initio quantum chemistry using the density matrix renormalization group, The Journal of Chemical Physics 110, 4127–4130 (1999).
- [9]B. K. Clark, M. A. Morales, J. McMinis, J. Kim, and G. E. Scuseria, Computing the energy of a water molecule using multideterminants: A simple, efficient algorithm, The Journal of Chemical Physics 135, 10.1063/1.3665391 (2011).
- [10]R. P. Feynman and M. Cohen, Energy spectrum of the excitations in liquid helium, Phys. Rev. 102, 1189 (1956).
- [11]Y. Kwon, D. M. Ceperley, and R. M. Martin, Effects of three-body and backflow correlations in the two-dimensional electron gas, Phys. Rev. B 48, 12037 (1993).
- [12]J. B. Anderson, A random-walk simulation of the schrödinger equation: H+3, The Journal of Chemical Physics 63, 1499–1503 (1975).
- [14]K. Choo, T. Neupert, and G. Carleo, Two-dimensional frustrated J1–J2 model studied with neural network quantum states, Phys. Rev. B 100, 125124 (2019).
- [15]K. Choo, G. Carleo, N. Regnault, and T. Neupert, Symmetries and many-body excitations with neural-network quantum states, Phys. Rev. Lett. 121, 167204 (2018).
- [16]A. Nagy and V. Savona, Variational quantum monte carlo method with a neural-network ansatz for open quantum systems, Phys. Rev. Lett. 122, 250501 (2019).
- [17]O. Sharir, Y. Levine, N. Wies, G. Carleo, and A. Shashua, Deep autoregressive models for the efficient variational simulation of many-body quantum systems, Phys. Rev. Lett. 124, 020503 (2020).
- [18]Y. Nomura, A. S. Darmawan, Y. Yamaji, and M. Imada, Restricted boltzmann machine learning for solving strongly correlated quantum systems, Phys. Rev. B 96, 205152 (2017).
- [19]F. Ferrari, F. Becca, and J. Carrasquilla, Neural gutzwiller-projected variational wave functions, Phys. Rev. B 100, 125131 (2019).
- [20]J. Stokes, J. R. Moreno, E. A. Pnevmatikakis, and G. Carleo, Phases of two-dimensional spinless lattice fermions with first-quantized deep neural-network quantum states, Phys. Rev. B 102, 205122 (2020).
- [21]J. Lin, G. Goldshlager, and L. Lin, Explicitly antisymmetrized neural network layers for variational monte carlo simulation, Journal of Computational Physics 474, 111765 (2023).
- [22]Z. Chen, D. Luo, K. Hu, and B. K. Clark, Simulating 2+1d lattice quantum electrodynamics at finite density with neural low wave functions (2022).
- [23]D. Luo and B. K. Clark, Backflow transformations via neural networks for quantum many-body wave functions, Phys. Rev. Lett. 122, 226401 (2019).
- [24]Z. Liu and B. K. Clark, Unifying view of fermionic neural network quantum states: From neural network backflow to hidden fermion determinant states, Phys. Rev. B 110, 115124 (2024).
- [25]D. Pfau, J. S. Spencer, A. G. D. G. Matthews, and W. M. Foulkes, Ab initio solution of the many-electron schrödinger equation with deep neural networks, Physical Review Research 2, 10.1103/physrevresearch.2.033429 (2020).
- [26]J. Hermann, Z. Schätzle, and F. Noé, Deep-neural-network solution of the electronic schrödinger equation, Nature Chemistry 12, 891 (2020).
- [27]K. Choo, A. Mezzacapo, and G. Carleo, Fermionic neural-network states for ab-initio electronic structure, Nature Communications 11, 2368 (2020).
- [28]T. D. Barrett, A. Malyshev, and A. I. Lvovsky, Autoregressive neural-network wavefunctions for ab initio quantum chemistry, Nature Machine Intelligence 4, 351 (2022).
- [29]T. Zhao, J. Stokes, and S. Veerapaneni, Scalable neural quantum states architecture for quantum chemistry, Machine Learning: Science and Technology 4, 025034 (2023).
- [30]X. Li, J.-C. Huang, G.-Z. Zhang, H.-E. Li, C.-S. Cao, D. Lv, and H.-S. Hu, A nonstochastic optimization algorithm for neural-network quantum states, Journal of Chemical Theory and Computation 19, 8156 (2023).
- [31]H. Shang, C. Guo, Y. Wu, Z. Li, and J. Yang, Solving schrödinger equation with a language model (2023).
- [32]Y. Wu, C. Guo, Y. Fan, P. Zhou, and H. Shang, Nnqs-transformer: an efficient and scalable neural network quantum states approach for ab initio quantum chemistry, in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’23 (Association for Computing Machinery, New York, NY, USA, 2023).
- [33]A. Malyshev, J. M. Arrazola, and A. I. Lvovsky, Autoregressive neural quantum states with quantum number symmetries (2023).
- [34]D. P. Kingma and J. Ba, Adam: A method for stochastic optimization (2014).
- [35]L. Bytautas and K. Ruedenberg, A priori identification of configurational deadwood, Chemical Physics 356, 64 (2009), moving Frontiers in Quantum Chemistry:.
- [36]J. S. Anderson, F. H. Deiner-Zadeh, and P. W. Ayers, Breaking the curse of dimension for the electronic schrödinger equation with functional analysis, Compu- tational and Theoretical Chemistry 1142, 66 (2018).
- [37]Y. Wang, J. Xiao, T. O. Suzek, J. Zhang, J. Wang, and S. H. Bryant, Pubchem: a public information system for analyzing bioactivities of small molecules, Nucleic Acids Research 37, W623–W633 (2009).
- [38]Q. Sun, T. C. Berkelbach, N. S. Blunt, G. H. Booth, S. Guo, Z. Li, J. Liu, J. D. McClain, E. R. Sayfutyarova, S. Sharma, S. Wouters, and G. K. Chan, PySCF: the python-based simulations of chemistry framework, WIREs Computational Molecular Science 8, 10.1002/wcms.1340 (2017).
- [39]D. Foreman-Mackey, D. W. Hogg, D. Lang, and J. Goodman, emcee: The mcmc hammer, Publications of the Astronomical Society of the Pacific 125, 306–312 (2013).