Introduction
We study the effect of symmetrizing neural network solutions to the Schrödinger equation. Solving the many-body Schrödinger equation is of fundamental importance in science, because it provides the key to understanding and predicting the behavior of quantum systems and thereby many physical phenomena. Ab initio computational methods seek to solve the non-relativistic electronic Schrödinger equation from first principles. There, the computation is performed directly from physical constraints and without relying on empirical approximations or training data, with the promise of producing high-accuracy electronic wavefunctions. However, the strict requirement on physical constraints makes it challenging to incorporate neural networks into these methods. Carleo & Troyer [9 ] , Hermann et al. [22 ] and FermiNet Pfau et al., [40 ] are some of the first successful ab initio neural network methods, which learn the ground state wavefunctions in atoms and molecules via a variational Monte Carlo (VMC) approach. Many neural network methods have emerged since then [10 ] (Li et al. 2022; von Glehn et al. 2023), each seeking to improve how physical constraints are modelled in different systems. While these approaches have produced state-of-the-art results on ground state energy and other physical properties, one notable drawback is their exorbitant training cost compared to classical VMC methods. This issue becomes dire for modelling large systems, as the Hilbert space of wavefunctions grows exponentially with the number of electrons.
In other AI-for-science approaches, symmetrization has proved to be successful both for modelling physical constraints and for improving neural network performance [6 , 16 ] (Batatıa et al. 2022; Du et al. 2022). One VMC example is DeepSolid Li et al., 2022, a FermiNet-type wavefunction that employs translationally invariant features to model periodic solids. However, recent findings in protein structures Abramson et al., [1 ] , atomic potential Qu & Krishnapriyan, 2024 and machine learning theory Huang et al., [24 ] ; Balestriero et al. [4 ] demonstrate that symmetries may be unnecessary and could sometimes be harmful for performance. This raises the question how much the previously observed improvements can be attributed to symmetries versus other factors such as hyperparameter and architecture choices.
Motivated by these questions, we examine the effectiveness of symmetrizing ab initio neural network solvers under the natural symmetry groups of a many-body problem, which are the diagonal groups of isometries. These groups, roughly speaking, describe the invariance of the system under a simultaneous movement of multiple objects; see § 2 for a detailed review. Common symmetrization approaches in machine learning (ML) roughly fall under three categories:
Randomly transforming data by symmetry operations (data augmentation );
Averaging over group operations (group averaging );
Invariant maps (invariant features and canonicalization ).
We compare these approaches on DeepSolid (Li et al., 2022), the state-of-the-arts architecture for VMC on solids, for different solid systems. The emphasis is on providing an apples-to-apples comparison by fixing the architecture and hyperparameters, while varying the symmetry parameters. We find that, perhaps surprisingly, the effects of diagonal symmetrization for VMC problems are mixed and nuanced. This arises due to the unique combination of challenges posed by VMC and diagonal invariance, as well as the joint consideration of computational cost, statistical behaviors and physical constraints. In particular, our analyses indicate:
In-training symmetrization can hurt. VMC training operates
in an “infinite-data” regime, and every symmetry operation comes at the cost of forgoing one new data point. Holding the computational budget constant, symmetrization can destabilize training and lead to worse performance. This is demonstrated by theoretical and numerical results in § 4 .
Post hoc symmetrization helps. At inference time, VMC solvers are less sensitive to computational costs, and allow for averaging over a moderate number of group operations (§ 5 ). We show that post hoc averaging (PA) leads to improved energy, variance and symmetry properties of the learned wavefunction (Figure 1 , Table 2 ). In one case, post hoc averaged DeepSolid achieves performance close to DeepSolid trained with 10 × 10\times 10 × more computational budget (§ 7 ).
Figure 1. Visualizations of the (partial) diagonal invariance of an unsymmetrized wavefunction versus a symmetrized wavefunction in a graphene 1 × 1 1 \times1 1 × 1 system. (a) and (b) are generated by evaluating log ∣ ψ ( x 1 + t , … , x 12 + t ) ∣ 2 \log\lvert\psi(x_1+t,\ldots,x_{12}+t)\rvert^2 log ∣ ψ ( x 1 + t , … , x 12 + t ) ∣ 2 under a simultaneous 2d translation t t t of the configuration ( x 1 , … , x 12 ) (x_1,\ldots,x_{12}) ( x 1 , … , x 12 ) given by the 12 blue spheres in (e). The red overlay indicates the unit cell in (e) such that the ref. line is exactly at the origin when t = 0 t=0 t = 0 . (d) shows 2 atoms (orange spheres) in an 1 × 1 1 \times1 1 × 1 planar supercell, with G G G illustrated by the objects. (e) and (f) are shifted copies of the same configuration, with their positions marked in (b). This method only visualizes the partial P3m1 symmetry; see Appendix § 13 for a method that shows the full P6mm symmetry of ψ θ ( P A ; G ) \psi_\theta^{(\mathrm{PA;G})} ψ θ ( PA ; G ) . Details setups are in § 6 , 7 and Appendix § 12 .
Table 2. Performance of symmetrization methods with similar computational budgets. Energy and variance are both reported at the per unit cell level. *See Appendix § 12 .§ 3 for specifications of the GPUs used for training.
The remainder of the paper provides mathematical discussions and computational tools for understanding diagonal symmetries in VMC. § 2 reviews the concept of diagonal invariance and discusses why, except for simple cases e.g., translations or E ( 3 ) E(3) E ( 3 ) , finding a natural smooth invariant map is difficult. This is further corroborated by mathematical and empirical results in the special case of a smoothed canonicalization (§ 4.3 and Appendix § 15 ). § 3 briefly reviews the VMC setup and how different computational costs arise. § 4 and 5 respectively examine in-training and post hoc symmetrization. § 6 discusses our evaluation metrics and develops a method for visualizing diagonal symmetries (Figure 1 ). § 7 discusses experiment details. Additional results and proofs are included in the appendix.
Many-body Schrödinger equation
Throughout, we shall consider finding the n n n -electron ground state wavefunction ψ : R 3 n → C \psi: \mathbb{R}^{3n} \to\mathbb{C} ψ : R 3 n → C and the corresponding minimal energy E ∈ R E \in\mathbb{R} E ∈ R of the Schrödinger equation
H ψ ( x ) = E ψ ( x ) , x : = ( x 1 , … , x n ) ∈ R 3 n . (1) H\psi(\mathbf{x}) = E\psi(\mathbf{x}), \quad\mathbf{x} := (x_1,\ldots,x_n) \in\mathbb{R}^{3n}. \tag*{(1)} H ψ ( x ) = E ψ ( x ) , x := ( x 1 , … , x n ) ∈ R 3 n . (1)
The Hamiltonian H H H is given by H ψ ( x ) : = − 1 2 Δ ψ ( x ) + V ( x ) ψ ( x ) H\psi(\mathbf{x}) := -\frac{1}{2}\Delta\psi(\mathbf{x}) + V(\mathbf{x})\psi(\mathbf{x}) H ψ ( x ) := − 2 1 Δ ψ ( x ) + V ( x ) ψ ( x ) , Δ \Delta Δ is the Laplacian representing the kinetic energy and V : R 3 n → R V : \mathbb{R}^{3n} \to\mathbb{R} V : R 3 n → R is the potential energy of the physical system. We also denote neural network ansätze by ψ θ \psi_\theta ψ θ , parametrized by some network weights θ ∈ R q \theta\in\mathbb{R}^q θ ∈ R q . Note that in general, the wavefunction depends on each electron via ( x i , σ i ) (x_i,\sigma_i) ( x i , σ i ) , where σ i ∈ { ↑ , ↓ } \sigma_i \in\{\uparrow,\downarrow\} σ i ∈ { ↑ , ↓ } is the spin, and ψ \psi ψ is required to be anti-symmetric with respect to permutations of ( x i , σ i ) (x_i,\sigma_i) ( x i , σ i ) . We focus on the case with fixed spins for simplicity, as is done in FermiNet and DeepSolid.
Diagonal invariance
Many physical systems of interest exhibit symmetry under some group G \mathbb{G} G of isometries. G \mathbb{G} G consists of maps of the form
x ↦ A x + b , x ∈ R 3 , (2) x \mapsto Ax+b,\qquad x\in\mathbb{R}^3, \tag*{(2)} x ↦ A x + b , x ∈ R 3 , (2)
for some orthogonal A ∈ R 3 × 3 A\in\mathbb{R}^{3\times3} A ∈ R 3 × 3 and some translation b ∈ R 3 b\in\mathbb{R}^3 b ∈ R 3 . We focus on groups that are countable. For systems with n > 1 n>1 n > 1 electrons, G \mathbb{G} G typically causes the potential V V V in (1) to be invariant under a diagonal group G d i a g \mathbb{G}_{\mathrm{diag}} G diag acting on R 3 n \mathbb{R}^{3n} R 3 n :
G d i a g : = { ( g , … , g ) ∣ g ∈ G } . \mathbb{G}_{\mathrm{diag}} := \{(g,\ldots,g)\mid g\in\mathbb{G}\}. G diag := {( g , … , g ) ∣ g ∈ G } .
To see how G d i a g \mathbb{G}_{\mathrm{diag}} G diag may arise, consider the Coulomb potential
V C o u l ( x ) = ∑ i < j 1 ∥ x i − x j ∥ + ∑ i , I 1 ∥ x i − r I ∥ + ⋯ . V_{\mathrm{Coul}}(\mathbf{x})=\sum_{i<j}\frac{1}{\lVert x_i-x_j\rVert}+\sum_{i,I}\frac{1}{\lVert x_i-r_I\rVert}+\cdots. V Coul ( x ) = i < j ∑ ∥ x i − x j ∥ 1 + i , I ∑ ∥ x i − r I ∥ 1 + ⋯ .
Each r I r_I r I is the fixed, known position of the I I I -th atom (under the Born-Oppenheimer approximation), ∥ ∙ ∥ \lVert\bullet\rVert ∥ ∙ ∥ is the Euclidean norm, and … \ldots … are the omitted electron-independent terms.
terms. If the set of atom positions { r I } \{r_I\} { r I } is invariant under some G \mathbb{G} G , V C o u l V_{\mathrm{Coul}} V Coul is invariant under a simultaneous transformation of all x i x_i x i ’s by the same g ∈ G g\in\mathbb{G} g ∈ G . Note however that V C o u l V_{\mathrm{Coul}} V Coul does not satisfy separate invariance , i.e. invariance does not hold if x 1 x_1 x 1 and x 2 x_2 x 2 are transformed by different group elements, since
1 ∥ g 1 ( x 1 ) − g 2 ( x 2 ) ∥ ≠ 1 ∥ x 1 − x 2 ∥ \frac{1}{\lVert g_1(x_1)-g_2(x_2)\rVert}\ne\frac{1}{\lVert x_1-x_2\rVert} ∥ g 1 ( x 1 ) − g 2 ( x 2 )∥ 1 = ∥ x 1 − x 2 ∥ 1
in general. Mathematically, separate invariance can be modelled by the product group G n \mathbb{G}^n G n acting on R 3 n \mathbb{R}^{3n} R 3 n , and diagonal invariance arises as a specific subgroup of G n \mathbb{G}^n G n , i.e. G d i a g ⊂ G n \mathbb{G}_{\mathrm{diag}}\subset\mathbb{G}^n G diag ⊂ G n . Figure 2 illustrates the difference between G d i a g \mathbb{G}_{\mathrm{diag}} G diag and G n \mathbb{G}^n G n in a 2-electron system: Under the given symmetry, a potential function is left unchanged when both electrons are reflected, but not when one is reflected and the other is rotated.
Figure 2. Different invariances for two electrons in a graphene system. Left. Separate invariance under a reflection and a rotation. Right. Diagonal invariance under a simultaneous reflection.
G d i a g \mathbb{G}_{\mathrm{diag}} G diag -invariant wavefunction. Throughout this paper, we use the shorthand g ( x ) = ( g ( x 1 ) , … , g ( x n ) ) g(\mathbf{x})=(g(x_1),\ldots,g(x_n)) g ( x ) = ( g ( x 1 ) , … , g ( x n )) for g ∈ G g\in\mathbb{G} g ∈ G , and focus on G d i a g \mathbb{G}_{\mathrm{diag}} G diag -invariant potentials V V V , i.e.,
V ( x ) = V ( g ( x ) ) for all g ∈ G . (3) V(\mathbf{x})=V(g(\mathbf{x}))\qquad\text{for all }g\in\mathbb{G}. \tag*{(3)} V ( x ) = V ( g ( x )) for all g ∈ G . (3)
Invariance of V V V does not imply the invariance of ψ \psi ψ : For a translation-invariant V V V with one electron, Bloch [7 ] proves that ψ \psi ψ is only invariant up to a phase factor, and non-invariant solutions can occur when the ground state is degenerate (Tinkham, [45 ] ). Nonetheless, an invariant solution can always be constructed from a linear combination of these states. The next result confirms this for the general case of n ≥ 1 n\geq1 n ≥ 1 electrons and a diagonal group of isometries.
Fact 2.1. Suppose ( ψ , E ) (\psi,E) ( ψ , E ) solves (1) and V V V is invariant under some G d i a g \mathbb{G}_{\mathrm{diag}} G diag induced by a group G \mathbb{G} G of isometries. Then for any finite subgroup G \mathcal{G} G of G \mathbb{G} G ,
ψ G ( x ) : = 1 ∣ G ∣ ∑ g ∈ G ψ ( g ( x ) ) (4) \psi^{\mathcal{G}}(\mathbf{x}):=\frac{1}{|\mathcal{G}|}\sum_{g\in\mathcal{G}}\psi(g(\mathbf{x})) \tag*{(4)} ψ G ( x ) := ∣ G ∣ 1 g ∈ G ∑ ψ ( g ( x )) (4) also solves (1) with respect to same energy E E E . In particular, if G \mathcal{G} G is a subgroup, ψ G \psi^{\mathcal{G}} ψ G is invariant under G \mathcal{G} G .
Fact 2.1 motivates us to seek wavefunctions that respect the G d i a g \mathbb{G}_{\mathrm{diag}} G diag -invariance of the system. Figure 1 (a) also shows that an unsymmetrized, well-trained wavefunction already attempts to achieve some extent of approximate invariance.
For more references on how G d i a g \mathbb{G}_{\mathrm{diag}} G diag -invariance arises and plays an important role in modelling ψ \psi ψ , see Rajagopal et al. [42 ] ; Zicovich-Wilson & Dovesi [51 ] .
Challenges in modelling G d i a g \mathbb{G}_{\mathrm{diag}} G diag -invariance. Despite successes in modelling simple n n n -electron symmetries such as the Euclidean group E ( 3 ) E(3) E ( 3 ) (Batzner et al., [6 ] ) and translations (Whitehead et al., [46 ] ), generalizing the approaches in those settings to general isometries can be challenging:
Translation groups possess simple and well-understood symmetries. Common symmetrizations include periodic Fourier bases (Rajagopal et al., [42 ] ) and projection via taking a modulus (Dym et al., [17 ] ). The former trades off representation power against computational cost as the
number of bases used varies, whereas the latter suffers from discontinuity and requires smoothing (§ 4.3 ). Neither has been extended to a general G d i a g G_{\mathrm{diag}} G diag .
E ( 3 ) E(3) E ( 3 ) consists of all isometries in R 3 \mathbb{R}^{3} R 3 , and also admits simple invariant features ([6 ] ). However, the additional symmetries under E ( 3 ) E(3) E ( 3 ) are undesirable when G G G is only a subgroup of E ( 3 ) E(3) E ( 3 ) , since the missing asymmetries represent a loss of information and therefore limit the representative power of the feature. Indeed, building an invariant map without losing information necessitates the availability of a maximal invariant ([30 ] ), whereas building a maximal invariant for G d i a g G_{\mathrm{diag}} G diag while respecting continuity constraints requires extending the mathematical theory of orbifolds ([43 , 2 ] ) to G d i a g G_{\mathrm{diag}} G diag of isometries. Such extensions are not known to the best of our knowledge.
In short, invariant maps are well-understood for groups with simple isometries and with many isometries, and the difficult cases are often found in the ones with restricted symmetries. This problem is particularly pronounced in the case of a diagonal group. Mathematically, the symmetries of G d i a g G_{\mathrm{diag}} G diag are described by a fixed group G G G in R 3 \mathbb{R}^{3} R 3 , but act on a larger and larger space R 3 n \mathbb{R}^{3n} R 3 n as the number of electrons n n n grows. This means that G d i a g G_{\mathrm{diag}} G diag admits substantially less structure than that of e.g. the product group G n G^{n} G n .
Space group G s p G_{\mathrm{sp}} G sp for a crystal system. A particular difficult case of G d i a g G_{\mathrm{diag}} G diag -invariance, in view of the above discussion, is the one induced by a space group G s p G_{\mathrm{sp}} G sp in a crystal lattice. These groups are described by (2) with a finite number of orthogonal matrices A A A and a countable number of translations b b b , which tile the R 3 \mathbb{R}^{3} R 3 space with a finite unit volume called the fundamental region . Figure 1 (d) illustrates the P 6 m m P6mm P 6 mm point group in a graphene system, where the space is tiled by a triangular fundamental region. The study of space group is a fundamental subject of solid state physics, and we refer interested readers to [3 ] .
Since G s p G_{\mathrm{sp}} G sp is infinite, Fact 2.1 does not apply directly to G = G s p G=G_{\mathrm{sp}} G = G sp . In practice however, to avoid computing an infinite crystal lattice, a common VMC practice is to restrict the system to a finite volume called the supercell ([18 , 29 ] ). This effectively reduces the set of translations b b b in (2) to a finite set and hence G s p G_{\mathrm{sp}} G sp to a finite group, which allows Fact 2.1 to apply.
While our theoretical results apply to all finite groups of isometries, our numerical experiments focus on G s p G_{\mathrm{sp}} G sp . Each G s p G_{\mathrm{sp}} G sp is denoted by standard shorthand e.g., P 6 m m P6mm P 6 mm , and an exhaustive list of G s p G_{\mathrm{sp}} G sp can be found in Brock et al. [8 ] . We emphasize that our focus on the diagonal action of G s p G_{\mathrm{sp}} G sp , a difficult class of restricted symmetries found in crystals, sets our work apart from existing VMC works that have investigated translations, simple point symmetries in molecules, as well as the continuous symmetries in S O ( 2 ) SO(2) S O ( 2 ) , S U ( 2 ) SU(2) S U ( 2 ) and E ( 3 ) E(3) E ( 3 ) . A non-exhaustive list of those works can be found in [36 , 33 , 34 , 50 ] and further comparisons are discussed in § 4.3 .
Symmetrization of many-body wavefunctions. We shall focus on generic symmetrization techniques from ML that can be applied directly to many-electron wavefunctions. This is to be distinguished with classical techniques, e.g., [51 ] : There, symmetrization is performed by building invariance into single-electron features, which does not generalize well to state-of-the-art neural network solvers that are many-body by design.
Neural network VMC solver
We briefly review the VMC approach for training our ab initio neural network solvers. VMC seeks to solve the minimum eigenvalue problem of (1) via the optimization
argmin θ ∈ R q ⟨ ψ θ , H ψ θ ⟩ ⟨ ψ θ , ψ θ ⟩ = argmin θ ∈ R q E [ E l o c a l ; ψ θ ( X ) ] , (5) \underset{\theta\in\mathbb{R}^{q}}{\operatorname{argmin}}\frac{\langle\psi_{\theta},H\psi_{\theta}\rangle}{\langle\psi_{\theta},\psi_{\theta}\rangle}
=
\underset{\theta\in\mathbb{R}^{q}}{\operatorname{argmin}}\mathbb{E}[E_{\mathrm{local};\psi_{\theta}}(\mathbf{X})], \tag*{(5)} θ ∈ R q argmin ⟨ ψ θ , ψ θ ⟩ ⟨ ψ θ , H ψ θ ⟩ = θ ∈ R q argmin E [ E local ; ψ θ ( X )] , (5)
where X ∼ p ψ θ \mathbf{X}\sim p_{\psi_{\theta}} X ∼ p ψ θ and E l o c a l ; ψ θ ( x ) : = H ψ θ ( x ) / ψ θ ( x ) E_{\mathrm{local};\psi_{\theta}}(\mathbf{x}) := H\psi_{\theta}(\mathbf{x})/\psi_{\theta}(\mathbf{x}) E local ; ψ θ ( x ) := H ψ θ ( x ) / ψ θ ( x ) is the local energy. ⟨ f , g ⟩ : = ∫ f ( x ) ∗ g ( x ) d x \langle f,g\rangle:= \int f(\mathbf{x})^{*}g(\mathbf{x})\,\mathrm{d}x ⟨ f , g ⟩ := ∫ f ( x ) ∗ g ( x ) d x is the complex inner product, and p ψ ( x ) : = ∣ ψ ( x ) ∣ 2 ⟨ ψ , ψ ⟩ p_{\psi}(\mathbf{x}) := \frac{|\psi(\mathbf{x})|^{2}}{\langle\psi,\psi\rangle} p ψ ( x ) := ⟨ ψ , ψ ⟩ ∣ ψ ( x ) ∣ 2 is the probability distribution obtained by normalizing a wavefunction ψ \psi ψ . The optimization may be performed by first or second-order methods, which can be represented by the generic functions F X ; ψ θ = F ( ψ θ ( x ) , Δ ψ θ ( x ) ) ∈ R q F_{X;\psi_{\theta}}=F(\psi_{\theta}(\mathbf{x}),\Delta\psi_{\theta}(\mathbf{x}))\in\mathbb{R}^{q} F X ; ψ θ = F ( ψ θ ( x ) , Δ ψ θ ( x )) ∈ R q and Q X ; ψ θ = Q ( ψ θ ( x ) , Δ ψ θ ( x ) ) ∈ R q × q Q_{X;\psi_{\theta}}=Q(\psi_{\theta}(\mathbf{x}),\Delta\psi_{\theta}(\mathbf{x}))\in\mathbb{R}^{q\times q} Q X ; ψ θ = Q ( ψ θ ( x ) , Δ ψ θ ( x )) ∈ R q × q and
θ ↦ θ − E [ F X ; ψ θ ] , θ ↦ θ − E [ Q X ; ψ θ ] − 1 E [ F X ; ψ θ ] , \theta\mapsto\theta-\mathbb{E}[F_{X;\psi_{\theta}}],\qquad
\theta\mapsto\theta-\mathbb{E}[Q_{X;\psi_{\theta}}]^{-1}\mathbb{E}[F_{X;\psi_{\theta}}], θ ↦ θ − E [ F X ; ψ θ ] , θ ↦ θ − E [ Q X ; ψ θ ] − 1 E [ F X ; ψ θ ] ,
Examples of F X ; ψ θ F_{X;\psi_{\theta}} F X ; ψ θ and Q X ; ψ θ Q_{X;\psi_{\theta}} Q X ; ψ θ can be found in Pfau et al. [40 ] . Notably, the expectation formulation above converts the expensive integral over the entire space into an expectation, which can then be approximated by Monte Carlo averages computed on finitely many samples from p ψ θ p_{\psi_{\theta}} p ψ θ .
Training (optimization) phase. Every training step consists of two sub-steps: (i) Sampling . Samples are obtained from running N N N independent MCMC chains with p ψ θ p_{\psi_{\theta}} p ψ θ as the target distribution; (ii) Gradient computation . F X ; ψ θ F_{X;\psi_{\theta}} F X ; ψ θ (or Q X ; ψ θ Q_{X;\psi_{\theta}} Q X ; ψ θ ) is computed on the N N N samples. Since (i) typically requires computing only the derivative ∂ x ψ θ \partial_{\mathbf{x}}\psi_{\theta} ∂ x ψ θ , whereas (ii) involves at least ∂ θ ψ θ \partial_{\theta}\psi_{\theta} ∂ θ ψ θ and ∂ θ 2 ψ θ \partial_{\theta}^{2}\psi_{\theta} ∂ θ 2 ψ θ , the one-step computational costs of (i) and (ii) typically compare as C s a m p ≪ C g r a d C_{\mathrm{samp}}\ll C_{\mathrm{grad}} C samp ≪ C grad . This is particularly true for neural network solvers, where short chains (20–100 steps) are typically used due to the expensive gradient evaluation and that any small increase in per-step cost is amplified by the large number of training steps. We verify this cost comparison in Table 1 and discuss the alternative case with C s a m p ≥ C g r a d C_{\mathrm{samp}}\geq C_{\mathrm{grad}} C samp ≥ C grad in Appendix § 19 .
Table 1. Computational cost per training step. Details in § 7 .
Inference phase. Having obtained a trained wavefunction ψ θ ^ \psi_{\hat{\theta}} ψ θ ^ parametrized by θ ^ \hat{\theta} θ ^ , we draw samples from long chains that target p ψ θ ^ p_{\psi_{\hat{\theta}}} p ψ θ ^ . These samples are used to compute various physical properties of ψ θ ^ \psi_{\hat{\theta}} ψ θ ^ that can be expressed as expectations of p ψ θ ^ p_{\psi_{\hat{\theta}}} p ψ θ ^ ; see § 6 for details. Note that the only computational cost occurred here is in terms of C s a m p C_{\mathrm{samp}} C samp .
Symmetrization can be performed during training, inference or both. We analyze symmetrization techniques in the two phases separately in § 4 and § 5 , as they involve different computational tradeoffs. Our theoretical analysis considers first-order methods for simplicity. Our numerical results focus on the symmetrization of DeepSolid (Li et al., 2022), a state-of-the-arts neural network solver for solid systems with G s p \mathcal{G}_{\mathrm{sp}} G sp -symmetry, trained with the second-order method KFAC (Martens & Grosse, [37 ] ); see Appendix § 12 .§ 1 .
Symmetrization during training
We first present a theoretical analysis of the behavior of gradient updates under different in-training symmetrization techniques. Let X 1 , … , X N \mathbf{X}_1,\ldots,\mathbf{X}_N X 1 , … , X N be i.i.d. samples from p ψ θ ^ ( m ) p_{\psi_{\hat{\theta}}}^{(m)} p ψ θ ^ ( m ) , the distribution of an m m m -th step MCMC chain with p ψ θ ^ p_{\psi_{\hat{\theta}}} p ψ θ ^ as the target distribution. Our benchmark for comparison is the update rule with the original (OG) unsymmetrized ψ θ \psi_{\theta} ψ θ ,
θ ↦ θ − δ θ ( O G ) , δ θ ( O G ) : = 1 N ∑ i ≤ N F X i ; ψ θ . (6) \theta\mapsto\theta-\delta\theta^{(\mathrm{OG})},\qquad\delta\theta^{(\mathrm{OG})}:=\frac{1}{N}\sum_{i\leq N}F_{\mathbf{X}_i;\psi_{\theta}}. \tag*{(6)} θ ↦ θ − δ θ ( OG ) , δ θ ( OG ) := N 1 i ≤ N ∑ F X i ; ψ θ . (6)
Sample size bottlenecked by computational cost. A key element of our analysis is that the VMC methods are in an infinite data regime . More precisely, unlike setups where sample size is constrained by the number of data points—commonly found in many theoretical analyses of symmetrization techniques (Chen et al., 2020; Lyle et al., 2020; Huang et al., 2022})—we are theoretically allowed to draw infinitely many samples from the sampling step during training. The practical limitation comes from C s a m p C_{\mathrm{samp}} C samp and C g r a d C_{\mathrm{grad}} C grad , both of which are affected by the batch size N N N as well as the symmetrization techniques used. By taking the computational effects into account, we shall see that symmetrization techniques exhibit substantially different statistical behaviors from those observed in the existing literature.
Pitfalls of data augmentation (DA)
Since “training data” correspond to samples drawn from p ψ θ ^ p_{\psi_{\hat{\theta}}} p ψ θ ^ , a k k k -fold data augmentation is performed as follows:
Sample N / k N/k N / k X 1 , … , X N / k \mathbf{X}_1,\ldots,\mathbf{X}_{N/k} X 1 , … , X N / k i.i.d. ∼ p ψ θ ^ ( m ) \sim p_{\psi_{\hat{\theta}}}^{(m)} ∼ p ψ θ ^ ( m ) ;
Sample g 1 , 1 , … , g N / k , k \mathbf{g}_{1,1},\ldots,\mathbf{g}_{N/k,k} g 1 , 1 , … , g N / k , k i.i.d. from some distribution on G \mathcal{G} G ;
Compute the DA update as θ ↦ θ − δ θ ( D A ) \theta\mapsto\theta-\delta\theta^{(\mathrm{DA})} θ ↦ θ − δ θ ( DA ) , where
δ θ ( D A ) : = 1 N ∑ i ≤ N / k ∑ j ≤ k F g i , j ( X i ) ; ψ θ . \delta\theta^{(\mathrm{DA})}:=\frac{1}{N}\sum_{i\leq N/k}\sum_{j\leq k}F_{\mathbf{g}_{i,j}(\mathbf{X}_i);\psi_{\theta}}. δ θ ( DA ) := N 1 i ≤ N / k ∑ j ≤ k ∑ F g i , j ( X i ) ; ψ θ .
Notice that the sample size in (i) is reduced to N / k N/k N / k , since a set of k k k -times augmented N / k N/k N / k samples and a set of unaugmented N N N samples incur the same gradient evaluation cost C g r a d C_{\mathrm{grad}} C grad , which dominates the overall computational cost.
While the sampling cost, C s a m p / k C_{\mathrm{samp}}/k C samp / k , enjoys a minor speed-up, the next result shows that this comes at the cost of increased instability of the gradient estimate. Below, we denote the distribution of X 1 g : = g 1 , 1 ( X 1 ) \mathbf{X}_{1}^{\mathbf{g}}:=\mathbf{g}_{1,1}(\mathbf{X}_1) X 1 g := g 1 , 1 ( X 1 ) by p ψ θ ^ ; D A ( m ) p_{\psi_{\hat{\theta}};\mathrm{DA}}^{(m)} p ψ θ ^ ; DA ( m ) , and define a distance between two distributions p , p ′ p,p' p , p ′ on R 3 n \mathbb{R}^{3n} R 3 n as
d F ( p , p ′ ) = sup f ∈ F ∥ E p [ f ( X ) ] − E p ′ [ f ( X ) ] ∥ . d_{\mathcal{F}}(p,p')=\sup_{f\in\mathcal{F}}\left\lVert\mathbb{E}_{p}[f(\mathbf{X})]-\mathbb{E}_{p'}[f(\mathbf{X})]\right\rVert. d F ( p , p ′ ) = f ∈ F sup ∥ E p [ f ( X )] − E p ′ [ f ( X )] ∥ .
where F \mathcal{F} F is a class of R 3 n → R p + p 2 \mathbb{R}^{3n}\to\mathbb{R}^{p+p^{2}} R 3 n → R p + p 2 test functions such that
{ x ↦ ( F x ; ψ θ , F x ; ψ θ ⊗ 2 ) ∣ θ ∈ R q } ⊆ F . \left\{\mathbf{x}\mapsto\left(F_{\mathbf{x};\psi_{\theta}},F_{\mathbf{x};\psi_{\theta}}^{\otimes2}\right)\mid\theta\in\mathbb{R}^{q}\right\}\subseteq\mathcal{F}. { x ↦ ( F x ; ψ θ , F x ; ψ θ ⊗ 2 ) ∣ θ ∈ R q } ⊆ F .
d F d_{\mathcal{F}} d F is called an integral probability metric (Müller, [38 ] ).
Proposition 4.1. Fix θ ∈ R p \theta\in\mathbb{R}^{p} θ ∈ R p . Then
∥ E [ δ θ ( D A ) ] − E [ δ θ ( O G ) ] ∥ ≤ d F ( p ψ θ ^ ; D A ( m ) , p ψ θ ^ ( m ) ) , \left\lVert\mathbb{E}[\delta\theta^{(\mathrm{DA})}]-\mathbb{E}[\delta\theta^{(\mathrm{OG})}]\right\rVert\leq d_{\mathcal{F}}\left(p_{\psi_{\hat{\theta}};\mathrm{DA}}^{(m)},p_{\psi_{\hat{\theta}}}^{(m)}\right), E [ δ θ ( DA ) ] − E [ δ θ ( OG ) ] ≤ d F ( p ψ θ ^ ; DA ( m ) , p ψ θ ^ ( m ) ) , \left\lVert\operatorname{Var}[\delta\theta^{(\mathrm{DA})}]-\operatorname{Var}[\delta\theta^{(\mathrm{OG})}]-\frac{(k-1)\operatorname{Var}[F_{\mathbf{X}_{1}^{\mathbf{g}};\psi_{\theta}\mid\mathbf{X}_{1}]}{N}\right\rVert}
≤ 1 + 2 ∥ E [ δ θ ( O G ) ] ∥ + d F ( p ψ θ ^ ; D A ( m ) , p ψ θ ^ ( m ) ) N d F ( p ψ θ ^ ; D A ( m ) , p ψ θ ^ ( m ) ) . \leq\frac{1+2\left\lVert\mathbb{E}[\delta\theta^{(\mathrm{OG})}]\right\rVert+d_{\mathcal{F}}\left(p_{\psi_{\hat{\theta}};\mathrm{DA}}^{(m)},p_{\psi_{\hat{\theta}}}^{(m)}\right)}{N}d_{\mathcal{F}}\left(p_{\psi_{\hat{\theta}};\mathrm{DA}}^{(m)},p_{\psi_{\hat{\theta}}}^{(m)}\right). ≤ N 1 + 2 E [ δ θ ( OG ) ] + d F ( p ψ θ ^ ; DA ( m ) , p ψ θ ^ ( m ) ) d F ( p ψ θ ^ ; DA ( m ) , p ψ θ ^ ( m ) ) . In particular, if the distribution p ψ θ ^ ( m ) p_{\psi_{\hat{\theta}}}^{(m)} p ψ θ ^ ( m ) is invariant under G d i a g \mathcal{G}_{\mathrm{diag}} G diag , i.e. g ( X 1 ) = X 1 \mathbf{g}(\mathbf{X}_1)=\mathbf{X}_1 g ( X 1 ) = X 1 for all g ∈ G d i a g \mathbf{g}\in\mathcal{G}_{\mathrm{diag}} g ∈ G diag , we have
E [ δ θ ( D A ) ] = E [ δ θ ( O G ) ] and Var [ δ θ ( D A ) ] ⪰ Var [ δ θ ( O G ) ] , \mathbb{E}[\delta\theta^{(\mathrm{DA})}]=\mathbb{E}[\delta\theta^{(\mathrm{OG})}]\quad\text{and}\quad\operatorname{Var}[\delta\theta^{(\mathrm{DA})}]\succeq\operatorname{Var}[\delta\theta^{(\mathrm{OG})}], E [ δ θ ( DA ) ] = E [ δ θ ( OG ) ] and Var [ δ θ ( DA ) ] ⪰ Var [ δ θ ( OG ) ] , where ⪰ \succeq ⪰ is the Loewner order of non-negative matrices.
The error d F ( p ψ θ ^ ; D A ( m ) , p ψ θ ^ ( m ) ) d_{\mathcal{F}}(p_{\psi_{\hat{\theta}};\mathrm{DA}}^{(m)},p_{\psi_{\hat{\theta}}}^{(m)}) d F ( p ψ θ ^ ; DA ( m ) , p ψ θ ^ ( m ) ) describes how much an augmented sample from m m m -step chain deviates from an unaugmented sample on average, as measured through the gradient F x ; ψ θ F_{x;\psi_\theta} F x ; ψ θ and the squared gradient F x ; ψ θ ⊗ 2 F_{x;\psi_\theta}^{\otimes2} F x ; ψ θ ⊗ 2 . For early training, we expect this error to have small contributions to the overall optimization compared to other sources of noise, e.g., the error incurred by running short chains instead of long chains and by stochastic gradients. At the final steps of training, we expect this error to be small as p ψ θ ( m ) p_{\psi_\theta}^{(m)} p ψ θ ( m ) becomes approximately invariant; Figure 1 (a) shows that this is the case for an unsymmetrized, well-trained neural network.
Several messages follow from Proposition 4.1 :
DA leads to similar gradients in expectation but a possibly worse variance. This is in stark contrast to known analyses of DA in the ML literature, where DA for empirical averages is expected to improve the variance [12 , 24 ] . This surprising difference arises because those analyses focus on statistical errors arising from augmenting a size-N N N real-life dataset to a size N k Nk N k dataset, whereas our analysis pays attention to both statistical errors and computational errors in a setup that compares a size N N N dataset versus a size N / k × k N/k \times k N / k × k augmented dataset. Indeed, since the only computational saving of augmentation is the sampling cost C s a m p ≪ C o p t i m C_{\mathrm{samp}} \ll C_{\mathrm{optim}} C samp ≪ C optim , every augmentation comes at the cost of one i.i.d. sample unused.
Instability of DA is not specific to mean and variance. While Proposition 4.1 only controls the mean and the variance, they do describe the distributions of δ θ ( D A ) \delta\theta^{(\mathrm{DA})} δ θ ( DA ) and δ θ ( O G ) \delta\theta^{(\mathrm{OG})} δ θ ( OG ) well, even in the high-dimensional regime where the number of parameters q q q is large compared to the batch size N N N . This is due to recent results on high-dimensional Central Limit Theorem (CLT): In Theorem D.1 in the appendix, we adapt results from Chernozhukov et al. [13 ] to show that δ θ ( D A ) \delta\theta^{(\mathrm{DA})} δ θ ( DA ) and δ θ ( O G ) \delta\theta^{(\mathrm{OG})} δ θ ( OG ) are approximately normal in an appropriate sense. This shows that the instability of DA parameter update is a general feature of the distribution, and not just specific to the mean and the variance.
Applicability to multi-step updates. Proposition 4.1 concerns one-step gradients, but the analysis applies to multi-step updates: VMC methods draws fresh samples at every step conditionally on the parameter θ \theta θ from the previous step, so the same bounds hold with E [ ∙ ] \mathbb{E}[\bullet] E [ ∙ ] and Var [ ∙ ] \operatorname{Var}[\bullet] Var [ ∙ ] replaced by the conditional counterparts E [ ∙ ∣ θ ] \mathbb{E}[\bullet\mid\theta] E [ ∙ ∣ θ ] and Var [ ∙ ∣ θ ] \operatorname{Var}[\bullet\mid\theta] Var [ ∙ ∣ θ ] .
One limitation of the above analysis is that it is restricted to first-order updates. Extending similar analyses to second-order updates is a known challenge in the literature: Those methods typically pre-multiply the empirical average 1 N ∑ i ≤ N / k ∑ j ≤ k F g i , j ( x i ) ; ψ θ \frac{1}{N}\sum_{i\le N/k}\sum_{j\le k}F_{g_{i,j}(x_i);\psi_\theta} N 1 ∑ i ≤ N / k ∑ j ≤ k F g i , j ( x i ) ; ψ θ by the inverse of an empirical Fisher information matrix, which is also affected by augmentation, and there exist pathological examples where augmentation may increase or decrease the variance depending on problem-specific parameters (see e.g., ridge regression analysis in Huang et al. [24 ] ). Nevertheless, we confirm numerically in Figure 3 that the gradient variance
Figure 3. Normalized variance of differently symmetrized gradient updates against GPU hours. Experiment details in § 7 .
under KFAC, a second-order method used by DeepSolid, is also inflated under DA. Table 1 also verifies that the computational speedup from DA is negligible. Figure 5 and Table 2 show that training with DA leads to a worse performance.
Figure 5. Performance of wavefunctions against GPU hours, obtained with different symmetrization methods. Metrics are defined in § 6 and experiment details in § 7 . (c) is computed via Var[PA/OG] in § 6 but with ψ θ ( O G ) \psi_{\theta}^{(\mathrm{OG})} ψ θ ( OG ) replaced by different ψ θ \psi_{\theta} ψ θ 's.
Group-averaging (GA)
Fix G \mathcal{G} G with ∣ G ∣ = k |\mathcal{G}| = k ∣ G ∣ = k and let ψ θ G \psi_\theta^{\mathcal{G}} ψ θ G be the averaged network obtained from taking ψ = ψ θ \psi= \psi_\theta ψ = ψ θ in (4) . As with DA, since k k k -times more derivatives are computed, we need to use a batch size of N / k N/k N / k to maintain the same computational cost. Draw X 1 G , … , X N / k G ∼ i . i . d . p ψ θ ( m ) X_1^{\mathcal{G}},\ldots,X_{N/k}^{\mathcal{G}} \overset{\mathrm{i.i.d.}}{\sim} p_{\psi_\theta^{(m)}} X 1 G , … , X N / k G ∼ i.i.d. p ψ θ ( m ) . The GA update rule is
θ ↦ θ − δ θ ( G A ) , δ θ ( G A ) : = 1 N / k ∑ i ≤ N / k F X i G ; ψ θ G . \theta\mapsto\theta-\delta\theta^{(\mathrm{GA})},\qquad\delta\theta^{(\mathrm{GA})}:=\frac{1}{N/k}\sum_{i\le N/k}F_{X_i^{\mathcal{G}};\psi_\theta^{\mathcal{G}}}. θ ↦ θ − δ θ ( GA ) , δ θ ( GA ) := N / k 1 i ≤ N / k ∑ F X i G ; ψ θ G .
The mean and variance of δ θ ( G A ) \delta\theta^{(\mathrm{GA})} δ θ ( GA ) is straightforward:
Lemma 4.2. Fix θ ∈ R p \theta\in\mathbb{R}^{p} θ ∈ R p . Then
E [ δ θ ( G A ) ] = E [ F X 1 G ; ψ θ G ] , Var [ δ θ ( G A ) ] = Var [ F X 1 G ; ψ θ G ] N / k . \mathbb{E}\left[\delta\theta^{(\mathrm{GA})}\right]=\mathbb{E}\left[F_{X_1^{\mathcal{G}};\psi_\theta^{\mathcal{G}}}\right],\qquad\operatorname{Var}\left[\delta\theta^{(\mathrm{GA})}\right]=\frac{\operatorname{Var}\left[F_{X_1^{\mathcal{G}};\psi_\theta^{\mathcal{G}}}\right]}{N/k}. E [ δ θ ( GA ) ] = E [ F X 1 G ; ψ θ G ] , Var [ δ θ ( GA ) ] = N / k Var [ F X 1 G ; ψ θ G ] .
Remark 4.3. As with DA, we can mean and variance as proxies to understand the distribution of δ θ ( G A ) \delta\theta^{(\mathrm{GA})} δ θ ( GA ) since a high-dimensional CLT does apply here; see Appendix § 14 .
GA may also destabilize gradients. The decrease in sample size is again visible in the variance in Lemma 4.2, which suffers from a k \sqrt{k} k blowup. Notice that the reference mean and variance are stated in terms of the gradient F X 1 G ; ψ θ G F_{X_1^{\mathcal{G}};\psi_\theta^{\mathcal{G}}} F X 1 G ; ψ θ G , which depends on the GA wavefunction both through the samples X 1 G X_1^{\mathcal{G}} X 1 G and through the gradient evaluation at ψ θ G \psi_\theta^{\mathcal{G}} ψ θ G . Unlike the discussion in Proposition 4.1 , we no longer expect that the mean and variance of F X 1 G ; ψ θ G F_{X_1^{\mathcal{G}};\psi_\theta^{\mathcal{G}}} F X 1 G ; ψ θ G are close to the unsymmetrized analogue F X 1 ; ψ θ F_{X_1;\psi_\theta} F X 1 ; ψ θ , since it does not suffice for X 1 G X_1^{\mathcal{G}} X 1 G and X 1 X_1 X 1 to have similar distributions. In general, Var [ δ θ ( G A ) ] \operatorname{Var}[\delta\theta^{(\mathrm{GA})}] Var [ δ θ ( GA ) ] increases if and only if the ratio
Var [ F X 1 G ; ψ θ G ] Var [ F X 1 ; ψ θ ] > 1 k . \frac{\operatorname{Var}\left[F_{X_1^{\mathcal{G}};\psi_\theta^{\mathcal{G}}}\right]}{\operatorname{Var}\left[F_{X_1;\psi_\theta}\right]}>\frac{1}{k}. Var [ F X 1 ; ψ θ ] Var [ F X 1 G ; ψ θ G ] > k 1 .
Empirically, we see that a variance increase for δ θ ( G A ) \delta\theta^{(\mathrm{GA})} δ θ ( GA ) is visible for KFAC in Figure 3 compared to δ θ ( O G ) \delta\theta^{(\mathrm{OG})} δ θ ( OG ) with similar computational costs (Table 1 ).
We also include two further comparisons:
GA with subsampling (GAs). One way to circumvent this computational hurdle is to average over a size-k k k uniform subsample of G \mathcal{G} G at every training step, and use the full G \mathcal{G} G at inference time. We numerically investigate the effects of keeping N N N constant and uniformly sample k = 1 k=1 k = 1 element: While GAs further destabilizes the gradient (Figure 3 ), Table 2 shows that its performance improves from OG. Yet, it falls short of the performance obtained by post hoc averaging directly on the original wavefunction.
GA with same N N N . Say the batch size N N N is kept the same. While the per-step cost of a k k k -fold averaging increases by ≈ k \approx k ≈ k times Table 1 , one may ask if the number of steps until convergence may be reduced under the symmetrized wavefunction, such that the overall training cost is constant. Figure 3 shows that while such a reduction does appear, it is not enough to offset the increase in per-step cost.
Smoothed canonicalization (SC)
Another symmetrization method that gained traction in the theoretical ML community is canonicalization, defined as the projection to the fundamental region of a given group [27 , 17 ] . For 1d unit translations, an example of canonicalization is the map x ↦ x m o d 1 x \mapsto x \bmod1 x ↦ x mod 1 , illustrated in blue in Figure 4 . Canonicalization for space groups G s p \mathcal{G}_{\mathrm{sp}} G sp is possible but suffers from non-smoothness at the boundary, as visible at 0 and 1 in Figure 4 and as we show in Appendix § 15 . Jastrow factors [46 ] from VMC methods can be viewed as a way to smooth 1d canonicalization along each lattice vector, but the construction is specific to translations. Dym et al. [17 ] proposes a smoothed canonicalization (SC) for large permutation and rotation groups by taking weighted averages at the boundary. In Appendix § 15 , we adapt their idea to develop an SC for diagonal invariance under G s p \mathcal{G}_{\mathrm{sp}} G sp that also respects the anti-symmetry constraint of (1). The orange curve in Figure 4 illustrates our method for 1d translations. However, we demonstrate in Appendix § 15 that SC via weighted averaging requires averaging over n × ∣ G ϵ ∣ n \times|\mathcal{G}_{\epsilon}| n × ∣ G ϵ ∣ elements, where G ϵ \mathcal{G}_{\epsilon} G ϵ is some carefully chosen subset of G s p \mathcal{G}_{\mathrm{sp}} G sp and n n n is the number of electrons; the additional cost of n n n arises from an anti-symmetry requirement. Therefore, SC suffers from similar computational bottlenecks as DA and GA and typically to a worse extent. This renders SC unsuitable for training. Since SC for G d i a g \mathcal{G}_{\mathrm{diag}} G diag may be of independent interest, we provide theoretical guarantees and a discussion of its weaknesses in Appendix § 15 .
Figure 4. Canonicalization functions for 1d unit translations.
SC is an architecture-agnostic invariant map for G d i a g \mathcal{G}_{\mathrm{diag}} G diag , and its shortcomings demonstrate our discussion in § 2 that obtaining invariant maps for such restricted symmetries can be difficult. An alternative approach [21 , 49 , 19 , 20 ] is to introduce invariance implicitly, by building feature maps based on electron-atom distances. One must then ensure the network is invariant under permutations of atomic indices. Introducing this permutation-invariance naively, by averaging, requires an expensive summation over a permutation group, and introduces additional complications with the boundary condition. The works
referenced above address this by building simpler early-layer features that reduce the amount of averaging required. However, this approach restricts the representation power of earlier layers, and is not directly compatible with the DeepSolid architecture. It remains unclear whether those architectures can be adapted to perform well in solid systems. We focus on DeepSolid for the purpose of our investigation.
Post hoc symmetrization
An alternative to in-training symmetrization is post hoc symmetrization: We may first train θ ^ \hat{\theta} θ ^ with unsymmetrized updates (e.g. (6) ), and seek to symmetrize ψ θ ^ \psi_{\hat{\theta}} ψ θ ^ during inference. In contrast to § 4 , post hoc symmetrization no longer incurs the cost of C g r a d C_{\mathrm{grad}} C grad . While computing properties based on Monte Carlo estimates still incurs C s a m p l e C_{\mathrm{sample}} C sample , samples are typically obtained in large batches from long chains only once, before being used for multiple downstream computations. This allows us to perform a moderate amount of averaging without compromising on sample size. Meanwhile, a direct evaluation of the wavefunction, e.g. for producing the visualization in Figure 1 , also does not incur C s a m p l e C_{\mathrm{sample}} C sample .
A wavefunction ψ θ \psi_{\theta} ψ θ , as a physical object, is deterministic and should not involve exogenous randomness. As such, we do not consider DA for post-processing, and discuss only group averaging and canonicalization in this setup.
Post hoc averaging (PA). Since the cost of averaging still scales linearly with ∣ G ∣ |\mathcal{G}| ∣ G ∣ , taking an average over the entire G \mathcal{G} G can still be prohibitive when G \mathcal{G} G is large. Fortunately, Fact 2.1 ensures the validity of averaging over any finite subset G ⊆ G \mathcal{G} \subseteq\mathcal{G} G ⊆ G of our choice. One may choose G \mathcal{G} G to be a subgroup of interest, or a generating set of G \mathcal{G} G . Given a trained wavefunction ψ θ ^ \psi_{\hat{\theta}} ψ θ ^ , the corresponding PA wavefunction reads
ψ θ ^ ( P A ; G ) ( x ) : = 1 ∣ G ∣ ∑ g ∈ G ψ θ ^ ( g ( x ) ) . \psi_{\hat{\theta}}^{(\mathrm{PA};\mathcal{G})}(\mathbf{x}) := \frac{1}{|\mathcal{G}|}\sum_{g\in\mathcal{G}}\psi_{\hat{\theta}}(g(\mathbf{x})) . ψ θ ^ ( PA ; G ) ( x ) := ∣ G ∣ 1 g ∈ G ∑ ψ θ ^ ( g ( x )) .
Estimates of physical quantities are obtained by drawing N N N samples from p ψ θ ^ ( P A ; G ) p_{\psi_{\hat{\theta}}^{(\mathrm{PA};\mathcal{G})}} p ψ θ ^ ( PA ; G ) , incurring a computational cost of N ∣ G ∣ C s a m p l e N|\mathcal{G}|C_{\mathrm{sample}} N ∣ G ∣ C sample per MCMC step. For G \mathcal{G} G of a moderate size, we can compare N N N samples from p ψ θ ^ ( P A ; G ) p_{\psi_{\hat{\theta}}^{(\mathrm{PA};\mathcal{G})}} p ψ θ ^ ( PA ; G ) directly with N N N samples from p ψ θ ^ p_{\psi_{\hat{\theta}}} p ψ θ ^ , without incurring a larger statistical error. ψ θ ^ ( P A ; G ) \psi_{\hat{\theta}}^{(\mathrm{PA};\mathcal{G})} ψ θ ^ ( PA ; G ) offers two clear advantages over ψ θ ^ \psi_{\hat{\theta}} ψ θ ^ :
More symmetry. By construction, ψ θ ^ ( P A ; G ) \psi_{\hat{\theta}}^{(\mathrm{PA};\mathcal{G})} ψ θ ^ ( PA ; G ) exhibits a higher degree of symmetry than ψ θ ^ \psi_{\hat{\theta}} ψ θ ^ , the extent of which depends on the subset G \mathcal{G} G chosen and the degree of symmetry already present in ψ θ ^ \psi_{\hat{\theta}} ψ θ ^ . See Figure 1 for the improved symmetry.
Robustness to outliers. The ground-truth wavefunctions are required to be non-smooth whenever an electron coincides with an atom or an electron. At those regions of x \mathbf{x} x , the Hamiltonian H ψ θ ( x ) H\psi_{\theta}(\mathbf{x}) H ψ θ ( x ) diverges, but for ground-truth wavefunctions as well as carefully constrained clas
sical wavefunction ansatz, the local energy E l o c a l ; ψ θ ( x ) = H ψ θ ( x ) / ψ θ ( x ) E_{\mathrm{local};\psi_\theta}(\mathbf{x}) = H\psi_\theta(\mathbf{x})/\psi_\theta(\mathbf{x}) E local ; ψ θ ( x ) = H ψ θ ( x ) / ψ θ ( x ) in (5) stays finite due to the renormalization by ψ θ ( x ) \psi_\theta(\mathbf{x}) ψ θ ( x ) . The same is not necessarily true for neural network solvers, which are much more flexible by design: The Laplacian term in H ψ θ ( x ) H\psi_\theta(\mathbf{x}) H ψ θ ( x ) may become large even when the magnitude of ψ θ ( x ) \psi_\theta(\mathbf{x}) ψ θ ( x ) stays put. This issue also manifests in the evaluation of other empirical quantities, e.g., the gradient terms computed in MCMC. The averaging in ψ θ ^ ( P A ; G ) \psi_{\hat{\theta}}^{(\mathrm{PA};\mathcal{G})} ψ θ ^ ( PA ; G ) improves robustness in the sense that, if ψ θ ^ ( g ( x ) ) \psi_{\hat{\theta}}(g(\mathbf{x})) ψ θ ^ ( g ( x )) takes a large numerical value for one particular g ∈ G g \in\mathcal{G} g ∈ G and not for the rest, the numerical outlier is rescaled by a factor 1 ∣ G ∣ \frac{1}{|\mathcal{G}|} ∣ G ∣ 1 . The level of robustness also increases with the size of G \mathcal{G} G .
Figure 5 and Table 2 show that ψ θ ^ ( P A ; G ) \psi_{\hat{\theta}}^{(\mathrm{PA};\mathcal{G})} ψ θ ^ ( PA ; G ) outperforms in-training symmetrization with the same computational costs in all metrics considered. Compare, for example, PA with 40k steps of training with N = 1000 N = 1000 N = 1000 versus GA with 10k steps of training, N = 1000 N = 1000 N = 1000 and k = 12 k = 12 k = 12 : PA achieves a lower energy with lower variance as well as perfect symmetry, with only 1 / 4 1/4 1/4 of the training budget. Among methods with similar end-of-training energies, PA also attain a lower energy and variance with fewer training steps (Figure 5 ). We also remark that GA and GAs by default implement PA at inference, so their only difference with PA is from training.
Issues with post hoc canonicalization (PC). One may also use smooth canonicalization post hoc to symmetrize a trained wavefunction ψ θ ^ \psi_{\hat{\theta}} ψ θ ^ . For completeness, we record the performance of PC in Table 2 . The results are significantly worse than other methods, despite being applied to a well-trained wavefunction. The issue might arise from the fact that a weighted averaging near the boundary leads to a blowup in second derivatives, and we examine it in detail in Appendix § 15 . This makes PC unsuitable specifically for our problem, since E l o c a l E_{\mathrm{local}} E local involves the Hamiltonian.
Evaluation and visualization methods
VMC wavefunctions are typically assessed via
E [ E l o c a l ; ψ θ ( X ) ] and Var [ E l o c a l ; ψ θ ( X ) ] , X ∼ p ψ θ . (7) \mathbb{E}[E_{\mathrm{local};\psi_\theta}(\mathbf{X})] \text{ and } \operatorname{Var}[E_{\mathrm{local};\psi_\theta}(\mathbf{X})],\quad\mathbf{X} \sim p_{\psi_\theta}. \tag*{(7)} E [ E local ; ψ θ ( X )] and Var [ E local ; ψ θ ( X )] , X ∼ p ψ θ . (7)
The energy is our optimization objective (5) . The variance is another measure of fit for (1) : It admits a lower bound of 0 that is attained by any true solution ψ ⋆ \psi_\star ψ ⋆ to (1) , since E l o c a l ; ψ ⋆ E_{\mathrm{local};\psi_\star} E local ; ψ ⋆ is everywhere constant. See Kent et al. [28 ] .
To show the amount of approximate symmetry already present in the OG wavefunction ψ θ ^ ( O G ) \psi_{\hat{\theta}}^{(\mathrm{OG})} ψ θ ^ ( OG ) , we compare ψ θ ^ ( O G ) \psi_{\hat{\theta}}^{(\mathrm{OG})} ψ θ ^ ( OG ) against PA wavefunctions averaged over different G \mathcal{G} G 's. This is reported as Var[PA/OG] in Table 2 , which stands for
Var [ 1 ∣ G ∣ ∑ g ∈ G ψ θ ^ ( O G ) ( g ( X ) ) / ψ θ ^ ( O G ) ( X ) ] , X ∼ p ψ θ ^ ( O G ) . \operatorname{Var}\left[\frac{1}{|\mathcal{G}|}\sum_{g\in\mathcal{G}}\psi_{\hat{\theta}}^{(\mathrm{OG})}(g(\mathbf{X}))/\psi_{\hat{\theta}}^{(\mathrm{OG})}(\mathbf{X})\right],\quad\mathbf{X}\sim p_{\psi_{\hat{\theta}}^{(\mathrm{OG})}}. Var ∣ G ∣ 1 g ∈ G ∑ ψ θ ^ ( OG ) ( g ( X )) / ψ θ ^ ( OG ) ( X ) , X ∼ p ψ θ ^ ( OG ) .
We also seek to visualize ψ θ \psi_\theta ψ θ for its symmetry under G d i a g \mathcal{G}_{\mathrm{diag}} G diag . Visualizing diagonal symmetry can be challenging, as G d i a g \mathcal{G}_{\mathrm{diag}} G diag acts on a high-dimensional space R 3 n \mathbb{R}^{3n} R 3 n . We propose a visualization method that exploits the fact that our G d i a g \mathcal{G}_{\mathrm{diag}} G diag is completely described by a group G ~ \widetilde{\mathcal{G}} G of isometries in R 3 \mathbb{R}^{3} R 3 :
Let G ~ \widetilde{\mathcal{G}} G be a group acting on R 3 \mathbb{R}^{3} R 3 defined as G ~ : = { A ∈ R 3 × 3 ∣ A ( ⋅ ) + b ∈ G for some b ∈ R 3 } \widetilde{\mathcal{G}} := \{A \in\mathbb{R}^{3\times3} \mid A(\cdot)+b \in\mathcal{G}\text{ for some }b\in\mathbb{R}^{3}\} G := { A ∈ R 3 × 3 ∣ A ( ⋅ ) + b ∈ G for some b ∈ R 3 } ;
Fix x ~ s y m m ∈ R 3 n \tilde{\mathbf{x}}_{\mathrm{symm}}\in\mathbb{R}^{3n} x ~ symm ∈ R 3 n , a configuration of n n n electrons such that for every g ~ ∈ G ~ \tilde{g}\in\widetilde{\mathcal{G}} g ~ ∈ G , g ~ ( x ~ s y m m ) = x ~ s y m m \tilde{g}(\tilde{\mathbf{x}}_{\mathrm{symm}})=\tilde{\mathbf{x}}_{\mathrm{symm}} g ~ ( x ~ symm ) = x ~ symm ;
Given a function f : R 3 n → R f:\mathbb{R}^{3n}\to\mathbb{R} f : R 3 n → R to visualize, we plot the function f ~ ( t ) : = f ( x ~ s y m m + t ) \tilde{f}(t):=f(\tilde{\mathbf{x}}_{\mathrm{symm}}+t) f ~ ( t ) := f ( x ~ symm + t ) with t ∈ R 3 t\in\mathbb{R}^{3} t ∈ R 3 , i.e. all electrons are translated by t t t simultaneously.
The next result confirms the validity of this method:
Lemma 6.1. Let f : ( R 3 ) n → R f:(\mathbb{R}^{3})^{n}\to\mathbb{R} f : ( R 3 ) n → R be a function invariant under permutations of its n n n arguments. Then for any g ∈ G g\in\mathcal{G} g ∈ G and t ∈ R 3 t\in\mathbb{R}^{3} t ∈ R 3 , f ~ ( g ( t ) ) = f ~ ( t ) = f ( g ( x + t ) ) − f ( x + t ) \tilde{f}(g(t))=\tilde{f}(t)=f(g(\mathbf{x}+t))-f(\mathbf{x}+t) f ~ ( g ( t )) = f ~ ( t ) = f ( g ( x + t )) − f ( x + t ) .
The permutation invariance assumption holds for ∣ ψ θ ∣ |\psi_\theta| ∣ ψ θ ∣ and E l o c a l ; ψ θ E_{\mathrm{local};\psi_\theta} E local ; ψ θ , since ψ θ \psi_\theta ψ θ is anti-symmetric. Figure 1 (a), (b), and (f) are plotted with this method, with f ( x ) = log ∣ ψ θ ( x ) ∣ 2 f(\mathbf{x})=\log|\psi_\theta(\mathbf{x})|^2 f ( x ) = log ∣ ψ θ ( x ) ∣ 2 , x ~ s y m m \tilde{\mathbf{x}}_{\mathrm{symm}} x ~ symm given in Figure 1 (e) and G = G ~ = P 3 m 1 \mathcal{G}=\widetilde{\mathcal{G}}=\mathrm{P3m1} G = G = P3m1 , which illustrates the partial symmetries of the P6mm group of graphene. To see the G = P 6 m m \mathcal{G}=\mathrm{P6mm} G = P6mm symmetry, a different x ~ s y m m \tilde{\mathbf{x}}_{\mathrm{symm}} x ~ symm is required since G ~ ≠ G \widetilde{\mathcal{G}}\ne\mathcal{G} G = G in this case; see Appendix § 13 .
Experimental details
All code and data are available at https://github.com/PrincetonLIPS/invariant-DeepSolid. Experiments are performed with DeepSolid (Li et al., 2022) on crystalline solids. Each network is evaluated by sampling from MCMC chains with 30k length, and the model from the last training step is used unless otherwise specified. Supercell size is included in the first column of Table 2 . Appendix § 12 includes further specifications, experiments and a remark about why energy improvements in the decimals are considered substantial. A few remarks about each system:
Graphene. This is the setup considered in Figure 1 ,3,5 and Table 1 . PA outperforms other methods both in terms of the metrics in Table 2 and speed of convergence in Figure 5 .
LiH. We use nested subgroups of F m 3 ˉ m Fm\bar{3}m F m 3 ˉ m and observe that the performance improves with k k k . For comparison, Li et al. (2022) report the energy − 8.15096 ( 1 ) -8.15096(1) − 8.15096 ( 1 ) for DeepSolid trained with 3e5 steps and batch 4096. PA attains comparable performance in similar systems1 with 3e4 steps and batch 4000.
bcc-Li. This is a known difficult case for ab initio methods including DeepSolid (Yao et al. [48 ] ; Li et al. (2022)) and PA again helps. F m m m Fmmm F mmm and P 4 / m m m P4/mmm P 4/ mmm are subgroups of I m 3 ˉ m Im\bar{3}m I m 3 ˉ m containing different symmetries, and each offers similar improvements. For both LiH and bcc-Li, we also observe a saturation effect: The improvement saturates once sufficiently many symmetries are incorporated. We do not know whether an optimal choice of subgroup exists that balances performance and computational cost.
Discussion
For ML problems that exhibit geometric structure, conventional wisdom holds that incorporating the correct symmetries should improve performance. This is called into question by the recent success of AlphaFold3 (Abramson et al. [1 ] ), where substantial accuracy improvement is obtained without incorporating invariance into its architecture, as well as by other works on atomic potential and ML theory referenced in § 1 . Our work investigates such tradeoffs in the context of wavefunctions, and specifically of VMC. To this end, we compare different model-agnostic symmetrizations. To ensure comparability, we do so on a single architecture, and choose hyperparameters as in the original DeepSolid work (Li et al. (2022)). We do not claim to have exhausted all possible ways in which symmetries may affect VMC.
Regarding the implications of our findings for other settings, we note a number of points:
Computational-statistical tradeoffs for DA and GA. We expect the in-training tradeoffs observed in § 4 to be applicable to other ML setups where sampling is performed in between gradient updates. A non-physics example is the contrastive divergence algorithm for training energy-based models (Hinton [23 ] ; Du et al. (2020)).
Computational cost of SC. The computational cost discussed in § 4.3 is specific to the canonicalization we have adopted from Dym et al. [17 ] , which adopts an “average-near-the-boundary” approach. It is an open question whether more efficient canonicalizations exist for G s p \mathcal{G}_{\mathrm{sp}} G sp .
Applicability beyond DeepSolid and VMC. We conjecture that both our negative in-training findings and positive post-training findings are applicable beyond DeepSolid and VMC, especially in cases where difficult cases of restricted symmetries arise. As discussed in § 2 , if the system of interest possesses simpler symmetries such as translations or E ( 3 ) E(3) E ( 3 ) , there often exist more efficient symmetrizations than averaging or augmentation. These may avoid the in-training statistical-computational tradeoffs we observe.
Theoretical analyses. While our experiments focus on DeepSolid, our theoretical analyses describe more generally how DA, GA and SC interact with VMC.
Acknowledgement
This work was partially supported by NSF OAC 2118201. KHH and PO are supported by the Gatsby Charitable Foundation (GAT3850). NZ acknowledges support from the Princeton AI² initiative. This work used Princeton ionic cluster and Delta GPU at the National Center for Supercomputing Applications through allocation MAT220011 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. We also thank an anonymous reviewer for pointing out regimes where per-step sampling cost could be larger than per-step optimisation cost.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
The appendix is organized as follows:
Appendix A collects the notation used throughout the paper.
Appendix B includes additional figures and details on the experiments.
Appendix C proves the validity of the diagonal visualization method (Lemma 6.1 ) and includes additional details on how to visualize the full symmetry beyond Figure 1 .
Appendix D states the high-dimensional central limit theorem discussed in § 4 .
Appendix E develops and discusses the smoothed canonicalization method discussed in § 4.3 and 5 for diagonal invariance under a space group.
Appendices F to H prove all mathematical results developed in this paper.
Appendix I includes an additional discussion on the case where per-iter sampling cost is greater than per-iter gradient cost.
Notation
n n n : number of electrons
N N N : number of Markov chains / number of Markov chain samples
m m m : length of Markov chain run
k k k : number of symmetry operations
x x x : R 3 \mathbb{R}^{3} R 3 -valued position of a single electron. x ~ = ( x , σ ) ∈ R 3 × { ↑ , ↓ } \tilde{x}=(x,\sigma)\in\mathbb{R}^{3}\times\{\uparrow,\downarrow\} x ~ = ( x , σ ) ∈ R 3 × { ↑ , ↓ } additionally specifies the spin σ \sigma σ of the electron.
x \mathbf{x} x : R 3 n \mathbb{R}^{3n} R 3 n -valued positions of n n n electrons, also called an electron configuration. The diagonally transformed configuration is denoted as g ( x ) = ( g ( x 1 ) , … , g ( x n ) ) g(\mathbf{x})=(g(x_{1}),\ldots,g(x_{n})) g ( x ) = ( g ( x 1 ) , … , g ( x n )) . Similarly, x ~ ∈ ( R 3 × { ↑ , ↓ } ) n \tilde{\mathbf{x}}\in(\mathbb{R}^{3}\times\{\uparrow,\downarrow\})^{n} x ~ ∈ ( R 3 × { ↑ , ↓ } ) n is the n n n -electron analogue of x ~ \tilde{x} x ~ .
G G G : Space group acting on R 3 \mathbb{R}^{3} R 3
G ∗ G^{*} G ∗ : Point group subgroup of G G G , consisting of elements of the form x ↦ A ( x ) x\mapsto A(x) x ↦ A ( x ) where A A A is an orthogonal 3 × 3 3\times3 3 × 3 matrix
G d i a g G_{\mathrm{diag}} G diag : Diagonal group induced by the space group, acting on an electron configuration in R 3 n \mathbb{R}^{3n} R 3 n
G \mathcal{G} G : subset (not necessarily a subgroup) of elements of G d i a g G_{\mathrm{diag}} G diag
P n P_{n} P n : permutation group acting on an electron configuration in R 3 n \mathbb{R}^{3n} R 3 n
T s u p \mathbb{T}_{\mathrm{sup}} T sup : translation group acting on R 3 \mathbb{R}^{3} R 3 that represents the supercell assumption
⊗ \otimes ⊗ : the tensor product
∥ ∙ ∥ \|\bullet\| ∥ ∙ ∥ : Euclidean norm
( ∙ ) l (\bullet)_{l} ( ∙ ) l : l l l -th coordinate of a vector
absolute constant: a number that does not depend on any other variables, and in particular not on n n n , N N N , k k k or M M M
= d \overset{d}{=} = d : equality in distribution
Experimental details and additional results
DeepSolid architecture
All our experiments use DeepSolid (Li et al., 2022) as ψ θ \psi_{\theta} ψ θ , the baseline unsymmetrized network. DeepSolid adapts FermiNet, a molecular neural network ansatz (Pfau et al., [40 ] ), to infinite periodic solids. We briefly review their architectures.
Architecture-wise, FermiNet takes as inputs n n n electron positions and split them into two groups according to spins. For some distance function d d d , the electrons are encoded through feature maps of the form d ( x j − x k ) d(x_{j}-x_{k}) d ( x j − x k ) and d ( x j − τ I ) d(x_{j}-\tau_{I}) d ( x j − τ I ) , i.e. electron-electron interactions and electron-nucleus interactions, which are passed into a fully-connected network with tanh activations. The outputs are orbital information at the single-electron level that depends on electron-electron interactions, which are combined by a weighted sum of Slater determinants to form antisymmetric many-electron wavefunctions. For details, see Figure 1 and Algorithm 1 of [40 ] .
DeepSolid makes several adaptations of FermiNet that are crucial for modelling periodic solids. Of those, the most relevant one to our discussion are their choice of periodic features d ( x j − x k ) d(x_j-x_k) d ( x j − x k ) and d ( x j − x K ) d(x_j-x_K) d ( x j − x K ) , stated in (2) of [40 ] . Their periodic features introduce separate invariance with respect to supercell translations, such that the n n n electrons are effectively restricted within the supercell. The construction is based on Jastrow correlation factors [46 ] and can be viewed as a version of smoothed canonicalization for 1d translations (§ 4.3 ).
Physical systems
Table 3. List of physical systems considered in the experiments, where the number a × b a \times b a × b indicates the size of the supercell for a 2d system and a × b × c a \times b \times c a × b × c is that for a 3d system. The primitive cell (corresponding to 1 × 1 × 1 1 \times1 \times1 1 × 1 × 1 supercell) of graphene is visualized in Figure 1 (d), and those of LiH and bcc-Li are in Figure 6 (a) and Figure 6 (e).
The experimental benchmark used for comparing all symmetrization strategies is graphene with an 1 × 1 1 \times1 1 × 1 supercell for computational convenience. Effects of post-hoc averaging over different choices of G \mathcal{G} G are illustrated through LiH and bcc-Li, which possess substantially more symmetries. Two types of simple symmetries in LiH, P 1 ˉ \mathrm{P}\bar{1} P 1 ˉ and P 2 / m \mathrm{P}2/m P 2/ m , are illustrated in Figure 6 (c) and (d); we refer readers to [8 ] for an exhaustive visualization of all 2d and 3d space groups.
Figure 6. Visualization of the primitive cells of LiH and bcc-Li.
Parameter settings
The symmetrization strategies we consider, i.e. DA, GA, SC, PA and PC, are all implemented as wrapper functions around the original DeepSolid model. Architecturally this can be viewed as inserting one layer each before and after the DeepSolid processing pipeline. Notably, this allows us to leave most of the training parameters from DeepSolid unchanged and retain the performance from the original DeepSolid.
To ensure a fair comparison, we keep the default network and training parameter settings from DeepSolid across all our experiments, except that we vary the training batch size according to k k k , the number of symmetry operations used. Graphene training uses NVIDIA GeForce RTX 2080 Ti (12GB) and LiH and bcc-Li use NVIDIA A100 SXM4 (40GB).
At the inference stage, we collect samples from independent MCMC chains with length 30, 000. The numbers of samples collected are respectively 50, 000 for graphene and 20, 000 for LiH and bcc-Li. The choice of smoothing in post-hoc canonicalization is s ∞ s_{\infty} s ∞ defined in Appendix § 15 .§ 1 ; see Appendix § 15 for details.
Additional results
A remark about the scale of energy improvements. Notice that the energy improvements reported in Table 2 are in the third decimals in Hartree. Improvements at this scale are crucial in physics and chemistry. In the physical systems we consider, core electron binding energies are on the order of keV (1 keV ≈ 37 \approx37 ≈ 37 Hartree), while valence electron binding energies are in the range of 1 − 10 eV (≈ 0.037 \approx0.037 ≈ 0.037 Hartree). To obtain wavefunctions with accurate ground state energy, it is therefore crucial to obtain energy improvements on the order of 10 − 3 10^{-3} 1 0 − 3 Hartree or smaller. In fact, the term “chemical accuracy” – used in computational chemistry to describe the level of precision needed for calculated energies to provide meaningful predictions of chemical phenomena, is given by 1 kcal / mol (≈ 0.00159 \approx0.00159 ≈ 0.00159 Hartree). For more references, see [47 , 39 ] .
Varying number of subsamples in GAs. In Table 2 and Figure 5 , group-averaging with subsampling (GAs) achieves the closest performance to post hoc averaging on graphene 1 × 1 1 \times1 1 × 1 . In Figure 7 below, we verify that the performance does not improve if one varies k k k , the number of subsampled group elements, and that a larger k k k also leads to gradient destabilization.
Figure 7. Performance of GAs plotted against GPU hours and across different subsample size k k k .
Averaging over diagonal translations. The results in Table 2 focus on averaging over point groups, and one may ask whether averaging additionally over translations helps. We perform a preliminary investigation in LiH and bcc-Li, and find that incorporating translations does not lead to significant performance improvement. The results are indicated in Table 4 by X T \mathsf{X}\mathsf{T} XT , and we include the corresponding results for point groups without translations for comparison. Note that as averaging over translations incurs 8 × 8\times 8 × of the computational cost, we have computed the statistics only on 10, 000 samples.
Averaging over subsets of group elements. Fact 2.1 ensures the validity of averaging over any subset of group elements
caption: tbl:4 . Performance of post hoc averaging with translations.
:::
and not just subgroups. The only caveat is that for a general finite subset G \mathcal{G} G , the average 1 ∣ G ∣ ∑ g ∈ G ψ ( g ( x ) ) \frac{1}{|\mathcal{G}|}\sum_{g\in\mathcal{G}}\psi(g(x)) ∣ G ∣ 1 ∑ g ∈ G ψ ( g ( x )) is no longer guaranteed to be invariant under G \mathcal{G} G . Nevertheless, one may still ask if averaging over subsets of a group G \mathcal{G} G that are “sufficiently representative”, e.g. the generators of the group G \mathcal{G} G , can be helpful. We perform a preliminary investigation on graphene and LiH for PA with Gen ( G ) \operatorname{Gen}(\mathcal{G}) Gen ( G ) , a fixed set of generators of G \mathcal{G} G plus the identity element. The statistics are computed on 40,000 samples and reported in Table 5 . The results are inconclusive: We find that for graphene, PA with Gen ( G ) \operatorname{Gen}(\mathcal{G}) Gen ( G ) improves energy but significantly inflates variance, whereas for LiH, PA with Gen ( G ) \operatorname{Gen}(\mathcal{G}) Gen ( G ) has worse energy and variance. As a sanity check, we also compute Var [ P A G / ψ θ ] \operatorname{Var}[\mathrm{PA}^{\mathcal{G}}/\psi_\theta] Var [ PA G / ψ θ ] for each wavefunction to verify that the PA with Gen ( G ) \operatorname{Gen}(\mathcal{G}) Gen ( G ) is closer to PA computed on G \mathcal{G} G compared to the original wavefunction.
Table 5. Performance of post hoc averaging with subsets of group elements.
Additional details on Lemma 6.1 and visualizing diagonal invariance
We first state the proof of Lemma 6.1 , which is illustrative for understanding the necessity of using a modified group
G ~ = { A ∈ R 3 × 3 ∣ A ( ⋅ ) + b ∈ G for some b ∈ R 3 } . \widetilde{\mathcal{G}}=\{A\in\mathbb{R}^{3\times3}\mid A(\mathord{\cdot})+b\in\mathcal{G}\text{ for some }b\in\mathbb{R}^{3}\}. G = { A ∈ R 3 × 3 ∣ A ( ⋅ ) + b ∈ G for some b ∈ R 3 } .
Proof of Lemma 6.1. By the definition of f ~ \tilde{f} f ~ ,
f ~ ( g ( t ) ) = f ( x ~ s y m m + A ( t ) + b ) = ( a ) f ( A ( x ~ s y m m ) + A ( t ) + b ) = ( b ) f ( A ( x ~ s y m m + t ) + b ) = f ( g ( x + t ) ) . \tilde{f}(g(t))=f(\tilde{x}_{\mathrm{symm}}+A(t)+b)\overset{(a)}{=}f(A(\tilde{x}_{\mathrm{symm}})+A(t)+b)\overset{(b)}{=}f(A(\tilde{x}_{\mathrm{symm}}+t)+b)=f(g(x+t)). f ~ ( g ( t )) = f ( x ~ symm + A ( t ) + b ) = ( a ) f ( A ( x ~ symm ) + A ( t ) + b ) = ( b ) f ( A ( x ~ symm + t ) + b ) = f ( g ( x + t )) .
In ( a ) (a) ( a ) , we have used that x ~ s y m m \tilde{x}_{\mathrm{symm}} x ~ symm is invariant under A ∈ G ~ A\in\widetilde{\mathcal{G}} A ∈ G . Using the definition of f ~ \tilde{f} f ~ again to note that f ~ ( t ) = f ( x + t ) \tilde{f}(t)=f(x+t) f ~ ( t ) = f ( x + t ) finishes the proof.
As mentioned in § 6 , G ~ \widetilde{\mathcal{G}} G and the point group G ∗ \mathcal{G}_* G ∗ of G \mathcal{G} G are in general two different groups. When G ∗ \mathcal{G}_* G ∗ is the P 6 m m P6mm P 6 mm point group, both G ∗ \mathcal{G}_* G ∗ and G ~ \widetilde{\mathcal{G}} G consist of 12 elements. However, G ∗ \mathcal{G}_* G ∗ is generated by P 3 m 1 P3m1 P 3 m 1 and the reflection R R R indicated in Figure 8 , whereas G ~ \widetilde{\mathcal{G}} G is generated by P 3 m 1 P3m1 P 3 m 1 and the reflection R ~ \widetilde{R} R indicated in Figure 8 . Applying our method to G ~ \widetilde{\mathcal{G}} G allows for visualizing the full P 6 m m P6mm P 6 mm symmetry of the wavefunction (Figure 9 ).
Figure 8. A G ~ \widetilde{\mathcal{G}} G -symmetric configuration of 12 electrons, where G ~ \widetilde{\mathcal{G}} G is obtained from P 6 m m P6mm P 6 mm
Figure 9. (a) log ∣ ψ θ ( O G ) ∣ 2 \log|\psi_{\theta}^{(\mathrm{OG})}|^2 log ∣ ψ θ ( OG ) ∣ 2 , original DeepSolid; (b) log ∣ ψ θ ( P A ; G ) ∣ 2 \log|\psi_{\theta}^{(\mathrm{PA;G})}|^2 log ∣ ψ θ ( PA ; G ) ∣ 2 , post hoc averaging. Visualization of the full P6mm-diagonal invariance of ψ θ ( O G ) \psi_{\theta}^{(\mathrm{OG})} ψ θ ( OG ) versus ψ θ ( P A ; G ) \psi_{\theta}^{(\mathrm{PA;G})} ψ θ ( PA ; G ) . Same setup and wavefunctions as Figure 1 , except that the configuration to be translated, x s y m m \mathbf{x}_{\mathrm{symm}} x symm , is given by Figure 8 .
CLT results for DA and GA
We present results that characterize the distributions of the parameter update under data augmentation and group-averaging. First focus on the parameter update under DA:
δ θ ( D A ) = 1 N ∑ i ≤ N / k ∑ j ≤ k F g i , j ( X i ) ; ψ θ , \delta\theta^{(\mathrm{DA})} = \frac{1}{N}\sum_{i\leq N/k}\sum_{j\leq k} F_{g_{i,j}}(X_i);\psi_\theta, δ θ ( DA ) = N 1 i ≤ N / k ∑ j ≤ k ∑ F g i , j ( X i ) ; ψ θ ,
where X 1 , … , X N / k ∼ i . i . d . p ψ θ ( m ) X_1,\ldots,X_{N/k}\mathrel{\mathop{\sim}^{\mathrm{i.i.d.}}}p_{\psi_\theta}^{(m)} X 1 , … , X N / k ∼ i.i.d. p ψ θ ( m ) and g 1 , 1 , … , g N / k , k g_{1,1},\ldots,g_{N/k,k} g 1 , 1 , … , g N / k , k are i.i.d. samples from some distribution on G \mathbb{G} G .
Notice that due to augmentations, δ θ ( D A ) \delta\theta^{(\mathrm{DA})} δ θ ( DA ) involves a correlated sum. Nevertheless, for each l ≤ p l\leq p l ≤ p , we can re-express the l l l -th coordinate of the DA parameter update as
δ θ l ( D A ) = 1 N / k ∑ i ≤ N / k F i l ( D A ) where F i l ( D A ) : = 1 k ∑ j = 1 k ( F g i , j ( X i ) ; ψ θ ) l , \delta\theta_l^{(\mathrm{DA})}=\frac{1}{N/k}\sum_{i\leq N/k}F_{il}^{(\mathrm{DA})}
\qquad\text{where}\qquad
F_{il}^{(\mathrm{DA})}:=\frac{1}{k}\sum_{j=1}^{k}\left(F_{g_{i,j}}(X_i);\psi_\theta\right)_l , δ θ l ( DA ) = N / k 1 i ≤ N / k ∑ F i l ( DA ) where F i l ( DA ) := k 1 j = 1 ∑ k ( F g i , j ( X i ) ; ψ θ ) l ,
an empirical average of i.i.d. univariate random variables across 1 ≤ i ≤ N / k 1\leq i\leq N/k 1 ≤ i ≤ N / k , which allows for the application of a CLT. A coordinate-wise CLT, i.e. the normal approximation of δ θ l ( D A ) \delta\theta_l^{(\mathrm{DA})} δ θ l ( DA ) for any fixed l ≤ p l\leq p l ≤ p , is straightforward from classical CLT results. Since we are concerned about stability of the gradient estimate, it is more crucial to study the behavior of the coordinate of maximum deviation, i.e.
max l ≤ p ∣ δ θ l ( D A ) − E [ δ θ l ( D A ) ] ∣ , \max_{l\leq p}\left|\delta\theta_l^{(\mathrm{DA})}-\mathbb{E}\left[\delta\theta_l^{(\mathrm{DA})}\right]\right|, l ≤ p max δ θ l ( DA ) − E [ δ θ l ( DA ) ] ,
and verify that its behavior is completely described by the mean and the variance. The next result complements Proposition 4.1 by providing CLT results for both individual coordinates and the coordinates of maximum deviation in the high-dimensional regime , where parameter dimension p p p — the number of weights in our neural network — is allowed to be much larger than the batch size N / k N/k N / k — the number of Markov chains per training step. For convenience, we denote the standard deviation of F i l ( D A ) F_{il}^{(\mathrm{DA})} F i l ( DA ) as
σ l ( D A ) : = Var [ F i l ( D A ) ] = Var [ ( F g 1 , 1 ( X 1 ) ; ψ θ ) l ] k + ( k − 1 ) Cov [ ( F g 1 , 1 ( X 1 ) ; ψ θ ) l , ( F g 1 , 2 ( X 1 ) ; ψ θ ) l ] k . \sigma_l^{(\mathrm{DA})}:=\sqrt{\operatorname{Var}\left[F_{il}^{(\mathrm{DA})}\right]}=\sqrt{\frac{\operatorname{Var}\left[(F_{g_{1,1}}(X_1);\psi_\theta)_l\right]}{k}+\frac{(k-1)\operatorname{Cov}\left[(F_{g_{1,1}}(X_1);\psi_\theta)_l,(F_{g_{1,2}}(X_1);\psi_\theta)_l\right]}{k}}. σ l ( DA ) := Var [ F i l ( DA ) ] = k Var [ ( F g 1 , 1 ( X 1 ) ; ψ θ ) l ] + k ( k − 1 ) Cov [ ( F g 1 , 1 ( X 1 ) ; ψ θ ) l , ( F g 1 , 2 ( X 1 ) ; ψ θ ) l ] .
Theorem D.1. Fix θ ∈ R q \theta\in\mathbb{R}^{q} θ ∈ R q . Let Z ∼ N ( 0 , I p ) Z \sim\mathcal{N}(0,I_p) Z ∼ N ( 0 , I p ) be a standard Gaussian vector and Z l Z_l Z l be its l l l -th coordinate. Then there exists some absolute constant C 1 C_1 C 1 such that for every l ≤ p l \le p l ≤ p ,
sup t ∈ R ∣ P ( δ θ l ( D A ) ≤ t ) − P ( [ E [ δ θ l ( D A ) ] + ( Var [ δ θ l ( D A ) ] ) 1 / 2 Z l ≤ t ] ) ∣ ≤ C 1 E ∣ F l ( D A ) ∣ 3 N / k ( σ l ( D A ) ) 3 . \sup_{t\in\mathbb{R}}\left|\mathbb{P}\left(\delta\theta_l^{(\mathrm{DA})}\le t\right)-\mathbb{P}\left(\left[\mathbb{E}\left[\delta\theta_l^{(\mathrm{DA})}\right]+\left(\operatorname{Var}\left[\delta\theta_l^{(\mathrm{DA})}\right]\right)^{1/2}Z_l\le t\right]\right)\right|\le\frac{C_1\mathbb{E}\left|F_{l}^{(\mathrm{DA})}\right|^3}{\sqrt{N/k}\left(\sigma_l^{(\mathrm{DA})}\right)^3}. t ∈ R sup P ( δ θ l ( DA ) ≤ t ) − P ( [ E [ δ θ l ( DA ) ] + ( Var [ δ θ l ( DA ) ] ) 1/2 Z l ≤ t ] ) ≤ N / k ( σ l ( DA ) ) 3 C 1 E F l ( DA ) 3 . Moreover, assume a mild tail condition that for every l ′ ≤ p l' \le p l ′ ≤ p with σ l ′ ( D A ) > 0 \sigma_{l'}^{(\mathrm{DA})}>0 σ l ′ ( DA ) > 0 , we have
E [ exp ( ( σ l ′ ( D A ) ) − 1 ∣ F l ( D A ) ∣ F ~ ( D A ) ) ] ≤ 2 where F ~ ( D A ) : = max l ≤ p with σ l ( D A ) > 0 max q ∈ { 3 , 4 } ( E [ ( σ l ( D A ) ) − q ∣ F l ( D A ) ∣ q ] ) 1 / q . \mathbb{E}\left[\exp\left(\frac{\left(\sigma_{l'}^{(\mathrm{DA})}\right)^{-1}\left|F_{l}^{(\mathrm{DA})}\right|}{\widetilde{F}^{(\mathrm{DA})}}\right)\right]\le2
\qquad\text{where}\qquad
\widetilde{F}^{(\mathrm{DA})}:=\max_{\substack{l\le p\\\text{with }\sigma_l^{(\mathrm{DA})}>0}}\max_{q\in\{3,4\}}\left(\mathbb{E}\left[\left(\sigma_l^{(\mathrm{DA})}\right)^{-q}\left|F_l^{(\mathrm{DA})}\right|^q\right]\right)^{1/q}. E exp F ( DA ) ( σ l ′ ( DA ) ) − 1 F l ( DA ) ≤ 2 where F ( DA ) := l ≤ p with σ l ( DA ) > 0 max q ∈ { 3 , 4 } max ( E [ ( σ l ( DA ) ) − q F l ( DA ) q ] ) 1/ q . Then the coordinate of maximum deviation of θ ( D A ) \theta^{(\mathrm{DA})} θ ( DA ) also satisfies a CLT: There is some absolute constant C 2 > 0 C_2>0 C 2 > 0 such that
sup t ∈ R ∣ P ( max l ≤ p ∣ δ θ l ( D A ) − E [ δ θ l ( D A ) ] ∣ ≤ t ) − P ( max l ≤ p ∣ ( Var [ δ θ l ( D A ) ] ) 1 / 2 Z l ∣ ≤ t ) ∣ ≤ C 2 ( ( F ~ ( D A ) ) 2 ( log ( p N / k ) ) 7 N / k ) 1 / 6 . \sup_{t\in\mathbb{R}}\left|\mathbb{P}\left(\max_{l\le p}\left|\delta\theta_l^{(\mathrm{DA})}-\mathbb{E}\left[\delta\theta_l^{(\mathrm{DA})}\right]\right|\le t\right)-\mathbb{P}\left(\max_{l\le p}\left|\left(\operatorname{Var}\left[\delta\theta_l^{(\mathrm{DA})}\right]\right)^{1/2}Z_l\right|\le t\right)\right|\le C_2\left(\frac{\left(\widetilde{F}^{(\mathrm{DA})}\right)^2(\log(pN/k))^7}{N/k}\right)^{1/6}. t ∈ R sup P ( l ≤ p max δ θ l ( DA ) − E [ δ θ l ( DA ) ] ≤ t ) − P ( l ≤ p max ( Var [ δ θ l ( DA ) ] ) 1/2 Z l ≤ t ) ≤ C 2 N / k ( F ( DA ) ) 2 ( log ( pN / k ) ) 7 1/6 .
The bounds in Theorem D.1 each control a difference in distribution function between δ θ ( D A ) \delta\theta^{(\mathrm{DA})} δ θ ( DA ) and a normal distribution, measured through either an arbitrarily fixed coordinate or the coordinate of maximum deviation from its mean. In particular, they say that the distribution of δ θ ( D A ) \delta\theta^{(\mathrm{DA})} δ θ ( DA ) is approximately normal and therefore completely characterized by the mean and the variance studied in Proposition 4.1 . To interpret them in details:
The coordinate-wise bound follows from the classical Berry–Esséen Theorem (see e.g. Theorem 3.7 of Chen et al. [11 ] ), and the normal approximation error does not involve the dimension p p p .
For the coordinate of maximum deviation, the normal approximation error is small so long as p p p is much smaller than a constant multiple of exp ( ( N / k ) 1 / 7 ) \exp((N/k)^{1/7}) exp (( N / k ) 1/7 ) , which is in particular true even if the parameter dimension p p p is larger than the batch size N / k N/k N / k . The moment assumption amounts to a light-tailed condition on the distribution of the updates, and they can be relaxed at the cost of more complicated bounds — see Chernozhukov et al. [13 ] .
We also remark that Theorem D.1 is a special case of the universality result of Huang et al. [24 ] for data augmentation, but with a sharper bound and in Kolmogorov distance.
Analogous CLTs hold for both the unaugmented update and the group-averaging update. Recall that these updates are
δ θ ( O G ) : = 1 N ∑ i ≤ N F X i ; ψ θ and δ θ ( G A ) : = 1 N / k ∑ i ≤ N / k F X i G ; ψ θ . \delta\theta^{(\mathrm{OG})}:=\frac{1}{N}\sum_{i\le N}F_{X_i;\psi_\theta}
\qquad\text{and}\qquad
\delta\theta^{(\mathrm{GA})}:=\frac{1}{N/k}\sum_{i\le N/k}F_{X_i^G;\psi_\theta}. δ θ ( OG ) := N 1 i ≤ N ∑ F X i ; ψ θ and δ θ ( GA ) := N / k 1 i ≤ N / k ∑ F X i G ; ψ θ .
where X 1 G , … , X N / k G ∼ i . i . d . p ψ θ ( m ) X_1^G,\ldots,X_{N/k}^G\mathrel{\overset{\mathrm{i.i.d.}}{\sim}}p_{\psi_\theta}^{(m)} X 1 G , … , X N / k G ∼ i.i.d. p ψ θ ( m ) . In particular, δ θ ( O G ) \delta\theta^{(\mathrm{OG})} δ θ ( OG ) is the same as δ θ ( D A ) \delta\theta^{(\mathrm{DA})} δ θ ( DA ) with k = 1 k=1 k = 1 and g 1 , 1 g_{1,1} g 1 , 1 set to the identity transformation, and δ θ ( G A ) \delta\theta^{(\mathrm{GA})} δ θ ( GA ) is the same as δ θ ( O G ) \delta\theta^{(\mathrm{OG})} δ θ ( OG ) with N N N replaced by N / k N/k N / k and p ψ θ ( m ) p_{\psi_\theta}^{(m)} p ψ θ ( m ) replaced by p ψ θ ( m ) p_{\psi_\theta}^{(m)} p ψ θ ( m ) . Therefore the CLT results for δ θ ( O G ) \delta\theta^{(\mathrm{OG})} δ θ ( OG ) and δ θ ( G A ) \delta\theta^{(\mathrm{GA})} δ θ ( GA ) are direct consequences of Theorem D.1 by defining δ θ l ( O G ) \delta\theta_l^{(\mathrm{OG})} δ θ l ( OG ) , F l ( O G ) F_l^{(\mathrm{OG})} F l ( OG ) , σ l ( O G ) \sigma_l^{(\mathrm{OG})} σ l ( OG ) , F ~ ( O G ) \widetilde{F}^{(\mathrm{OG})} F ( OG ) , δ θ l ( G A ) \delta\theta_l^{(\mathrm{GA})} δ θ l ( GA ) , F l ( G A ) F_l^{(\mathrm{GA})} F l ( GA ) , σ l ( G A ) \sigma_l^{(\mathrm{GA})} σ l ( GA ) and F ~ ( G A ) \widetilde{F}^{(\mathrm{GA})} F ( GA ) as the analogous quantities. We do not state these results for brevity.
Smoothed canonicalization
This appendix expands on the discussion in § 4.3 and § 5 . We restrict our attention to a diagonal group G d i a g \mathbb{G}_{\mathrm{diag}} G diag induced by a space group G = G s p \mathbb{G}=\mathbb{G}_{\mathrm{sp}} G = G sp .
The essential idea behind canonicalization is that, to construct an invariant layer under G d i a g \mathbb{G}_{\mathrm{diag}} G diag , one may perform a “projection” onto a “smallest invariant set” Π ⊂ R 3 n \Pi\subset\mathbb{R}^{3n} Π ⊂ R 3 n , called the fundamental region . Formally, Π \Pi Π is a fixed set that contains exactly one point from each orbit of G d i a g \mathbb{G}_{\mathrm{diag}} G diag —called the representative point of the orbit—and canonicalization is an R 3 n → Π \mathbb{R}^{3n}\to\Pi R 3 n → Π map that brings an input x x x to its corresponding representative. In the simple case of unit translations and n = 1 n=1 n = 1 , a standard canonicalization is x ↦ x m o d 1 x\mapsto x\bmod1 x ↦ x mod 1 with Π = [ 0 , 1 ) \Pi=[0,1) Π = [ 0 , 1 ) , as visualized in Figure 4 . A naive canonicalization suffers from boundary discontinuity that violates physical constraints: In the case of mod, this arises because
lim ϵ → 0 + ( ϵ m o d 1 ) = 0 ≠ 1 = lim ϵ → 0 − ( ϵ m o d 1 ) . (8) \lim_{\epsilon\to0+}(\epsilon\bmod1)=0\ne1=\lim_{\epsilon\to0-}(\epsilon\bmod1). \tag*{(8)} ϵ → 0 + lim ( ϵ mod 1 ) = 0 = 1 = ϵ → 0 − lim ( ϵ mod 1 ) . (8)
Resolving this requires introducing smoothing at the boundary. As an example, consider the supercell assumption [42 , 29 ] , which corresponds to separate invariance under translations along each 1d lattice vector. Jastrow factors ([46 ] ; also see (Li et al., 2022) for its implementation in DeepSolid) exactly perform a smoothed version of canonicalization along each 1d lattice vector: These factors take the form h ( x m o d 1 ) h(x \bmod1) h ( x mod 1 ) , where x ∈ R x \in\mathbb{R} x ∈ R is a suitably rescaled input along one lattice vector and h h h is a piecewise polynomial chosen such that x ↦ h ( x m o d 1 ) x \mapsto h(x \bmod1) x ↦ h ( x mod 1 ) is twice continuously differentiable. However, the construction of h h h is easy here only because the fundamental regions of 1d translations have simple boundaries (two endpoints), and does not generalize well to more complicated symmetries.
For groups such as permutations and rotations, Dym, Lawrence, and Siegel [17 ] proposes a smoothing method by taking weighted averages at the boundary. We adapt this method for our G d i a g G_{\mathrm{diag}} G diag induced by a space group, and show how it can be achieved by operating with the group G G G acting on single electrons. This requires two ingredients:
Fundamental region of G d i a g G_{\mathrm{diag}} G diag . There are infinitely many choices of fundamental regions, and we shall fix one that is convenient for our canonicalization. Fix Π 0 ⊂ R 3 \Pi_0 \subset\mathbb{R}^3 Π 0 ⊂ R 3 , a fundamental region of R 3 \mathbb{R}^3 R 3 under G G G . Then Π : = Π 0 × ( R 3 ) n − 1 \Pi:= \Pi_0 \times(\mathbb{R}^3)^{n-1} Π := Π 0 × ( R 3 ) n − 1 forms a fundamental region under G d i a g G_{\mathrm{diag}} G diag , as it is intersected exactly once by any orbit { ( g ( x 1 ) , … , g ( x n ) ) ∣ g ∈ G } \{(g(x_1), \ldots, g(x_n)) \mid g \in G\} {( g ( x 1 ) , … , g ( x n )) ∣ g ∈ G } . An advantage of this choice is that, since the boundary of Π \Pi Π is completely determined by the boundary of Π 0 \Pi_0 Π 0 in the space of the first electron, we may perform smoothing directly near the boundary of Π 0 \Pi_0 Π 0 . Also notice that we may equivalently choose the fundamental region Π ( k ) : = ( R 3 ) k − 1 × Π 0 × ( R 3 ) n − k − 1 \Pi^{(k)} := (\mathbb{R}^3)^{k-1} \times\Pi_0 \times(\mathbb{R}^3)^{n-k-1} Π ( k ) := ( R 3 ) k − 1 × Π 0 × ( R 3 ) n − k − 1 for any 1 ≤ k ≤ n 1 \leq k \leq n 1 ≤ k ≤ n .
G ϵ ( x ) : = { g ∈ G ∣ d ( g ( x ) , Π 0 ) ≤ ϵ } , G_\epsilon(x) := \left\{g \in G \mid d(g(x), \Pi_0) \leq\epsilon\right\}, G ϵ ( x ) := { g ∈ G ∣ d ( g ( x ) , Π 0 ) ≤ ϵ } ,
where d d d is some distance preserved by the isometries in G G G :
d ( g ( x ) , g ( Π 0 ) ) = d ( x , Π 0 ) for all g ∈ G and x ∈ R 3 . d(g(x),g(\Pi_0)) = d(x,\Pi_0) \quad\text{for all } g \in G \text{ and } x \in\mathbb{R}^3. d ( g ( x ) , g ( Π 0 )) = d ( x , Π 0 ) for all g ∈ G and x ∈ R 3 .
Adapting the weighted averaging idea from Dym et al. [17 ] , we perform smoothing in the ϵ \epsilon ϵ -neighborhood of Π 0 \Pi_0 Π 0 by assigning weights to the different group operations: For each g ∈ G g \in G g ∈ G , we define
w ϵ g ( x ) : = λ ϵ ( d ( g ( x ) , Π 0 ) ) ∑ g ′ ∈ G ϵ ( x ) λ ϵ ( d ( g ′ ( x ) , Π 0 ) ) ∈ [ 0 , 1 ] , w_\epsilon^g(x) := \frac{\lambda_\epsilon(d(g(x),\Pi_0))}{\sum_{g' \in G_\epsilon(x)} \lambda_\epsilon(d(g'(x),\Pi_0))} \in[0,1], w ϵ g ( x ) := ∑ g ′ ∈ G ϵ ( x ) λ ϵ ( d ( g ′ ( x ) , Π 0 )) λ ϵ ( d ( g ( x ) , Π 0 )) ∈ [ 0 , 1 ] ,
where λ ϵ : R → [ 0 , 1 ] \lambda_\epsilon: \mathbb{R} \to[0,1] λ ϵ : R → [ 0 , 1 ] is a strictly decreasing function such that λ ϵ ( w ) = 1 \lambda_\epsilon(w) = 1 λ ϵ ( w ) = 1 for all w ≤ 0 w \leq0 w ≤ 0 and λ ϵ ( w ) = 0 \lambda_\epsilon(w) = 0 λ ϵ ( w ) = 0 for all w ≥ ϵ w \geq\epsilon w ≥ ϵ . If d ( g ( x ) , Π 0 ) ≥ ϵ d(g(x),\Pi_0) \geq\epsilon d ( g ( x ) , Π 0 ) ≥ ϵ , g g g moves x x x “too far”, and is assigned zero weights. If d ( g ( x ) , Π 0 ) = 0 d(g(x),\Pi_0) = 0 d ( g ( x ) , Π 0 ) = 0 , g g g keeps x x x within the closure of Π 0 \Pi_0 Π 0 , and is assigned a weight of 1. In particular, if x ∈ Π 0 x \in\Pi_0 x ∈ Π 0 is a point of high symmetry and ϵ \epsilon ϵ is small enough such that g ( x ) = x g(x) = x g ( x ) = x for all g ∈ G ϵ ( x ) g \in G_\epsilon(x) g ∈ G ϵ ( x ) , the weight of every g ∈ G ϵ ( x ) g \in G_\epsilon(x) g ∈ G ϵ ( x ) is 1 ∣ G ϵ ( x ) ∣ \frac{1}{|G_\epsilon(x)|} ∣ G ϵ ( x ) ∣ 1 . Here, all operations that preserve x x x are assigned equal weights. See Appendix § 15 .§ 1 for explicit choices of ϵ \epsilon ϵ and λ ϵ \lambda_\epsilon λ ϵ .
With these tools at hand, we define the smoothed canonicalization (SC) of an unsymmetrized wavefunction f θ f_\theta f θ as
ψ θ ; ϵ ( S C ) ( x ) : = 1 n ∑ k = 1 n ψ θ ; ϵ ( S C ; k ) ( x ) , where ψ θ ; ϵ ( S C ; k ) ( x ) : = ∑ g ∈ G ϵ ( x k ) w ϵ g ( x k ) ψ θ ( g ( x ) ) . (9) \psi_{\theta;\epsilon}^{(\mathrm{SC})}(x) := \frac{1}{n}\sum_{k=1}^{n}\psi_{\theta;\epsilon}^{(\mathrm{SC};k)}(x), \qquad\text{where} \qquad\psi_{\theta;\epsilon}^{(\mathrm{SC};k)}(x) := \sum_{g \in G_\epsilon(x_k)} w_\epsilon^g(x_k)\psi_\theta(g(x)). \tag*{(9)} ψ θ ; ϵ ( SC ) ( x ) := n 1 k = 1 ∑ n ψ θ ; ϵ ( SC ; k ) ( x ) , where ψ θ ; ϵ ( SC ; k ) ( x ) := g ∈ G ϵ ( x k ) ∑ w ϵ g ( x k ) ψ θ ( g ( x )) . (9)
Appendix § 15 .§ 2 illustrates (9) through an one-electron example. The next result guarantees the validity of ψ θ ; ϵ ( S C ) ( x ) = 1 n ∑ k = 1 n ψ θ ; ϵ ( S C ; k ) ( x ) \psi_{\theta;\epsilon}^{(\mathrm{SC})}(x) = \frac{1}{n}\sum_{k=1}^{n}\psi_{\theta;\epsilon}^{(\mathrm{SC};k)}(x) ψ θ ; ϵ ( SC ) ( x ) = n 1 ∑ k = 1 n ψ θ ; ϵ ( SC ; k ) ( x ) as a symmetrized wavefunction: Theorem E.1(i) proves diagonal invariance, (ii) proves anti-symmetry and (iii) ensures smoothness under appropriate choices of λ ϵ \lambda_\epsilon λ ϵ and d d d .
Theorem E.1. Fix θ ∈ R q \theta\in\mathbb{R}^q θ ∈ R q and ϵ > 0 \epsilon> 0 ϵ > 0 . The following properties hold for ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) and ψ θ ; ϵ ( S C ; k ) \psi_{\theta;\epsilon}^{(\mathrm{SC};k)} ψ θ ; ϵ ( SC ; k ) for all 1 ≤ k ≤ n 1 \leq k \leq n 1 ≤ k ≤ n :
(i) ψ θ ; ϵ ( S C ; k ) ( g ( x ) ) = ψ θ ; ϵ ( S C ; k ) ( x ) \psi_{\theta;\epsilon}^{(\mathrm{SC};k)}(g(x)) = \psi_{\theta;\epsilon}^{(\mathrm{SC};k)}(x) ψ θ ; ϵ ( SC ; k ) ( g ( x )) = ψ θ ; ϵ ( SC ; k ) ( x ) for all g ∈ G g \in G g ∈ G and x ∈ R 3 n x \in\mathbb{R}^{3n} x ∈ R 3 n ;
(ii) Let the spin-dependence in ψ θ \psi_\theta ψ θ be explicit, i.e. we view ψ θ \psi_\theta ψ θ as an ( R 3 × { ↑ , ↓ } ) n → C (\mathbb{R}^3 \times\{\uparrow,\downarrow\})^n \to\mathbb{C} ( R 3 × { ↑ , ↓ } ) n → C function. If ψ θ \psi_\theta ψ θ is anti-symmetric with respect to permutations of ( x i , σ i ) (x_i,\sigma_i) ( x i , σ i ) , then so is ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) .
(iii) Suppose λ ϵ \lambda_{\epsilon} λ ϵ and d ( ⋅ , g ( Π 0 ) ) d(\mathord{\cdot}, g(\Pi_0)) d ( ⋅ , g ( Π 0 )) are p p p -times continuously differentiable for all g ∈ G g \in G g ∈ G , and that ψ θ \psi_{\theta} ψ θ is p p p -times continuously differentiable at g ( x ) ∈ R 3 n g(x) \in\mathbb{R}^{3n} g ( x ) ∈ R 3 n for all g ∈ G g \in G g ∈ G . Then ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) is also p p p -times continuously differentiable at x x x . Moreover for 0 ≤ q ≤ p 0 \le q \le p 0 ≤ q ≤ p , the q q q -th derivative of ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) at x = ( x 1 , … , x n ) x = (x_1,\ldots,x_n) x = ( x 1 , … , x n ) can be computed as
∇ q ψ θ ; ϵ ( S C ) ( x 1 , … , x n ) = 1 n ∑ k = 1 n ∑ g ∈ G ϵ ( x k ) ∇ q ψ θ ; ϵ ; g , x k ( S C ; k ) ( x ) , \nabla^q\psi_{\theta;\epsilon}^{(\mathrm{SC})}(x_1,\ldots,x_n)=\frac{1}{n}\sum_{k=1}^{n}\sum_{g\in G_{\epsilon}(x_k)}\nabla^q\psi_{\theta;\epsilon;g,x_k}^{(\mathrm{SC};k)}(x), ∇ q ψ θ ; ϵ ( SC ) ( x 1 , … , x n ) = n 1 k = 1 ∑ n g ∈ G ϵ ( x k ) ∑ ∇ q ψ θ ; ϵ ; g , x k ( SC ; k ) ( x ) ,
where we have defined, for 1 ≤ k ≤ n 1 \le k \le n 1 ≤ k ≤ n , g ∈ G g \in G g ∈ G and x , y 1 , … , y n ∈ R 3 x,y_1,\ldots,y_n \in\mathbb{R}^3 x , y 1 , … , y n ∈ R 3 ,
ψ θ ; ϵ ; g , x ( S C ; k ) ( y 1 , … , y n ) : = λ ϵ ( d ( y k , g ( Π 0 ) ) ) ∑ g ′ ∈ G ϵ ( x ) λ ϵ ( d ( y k , g ′ ( Π 0 ) ) ) ψ θ ( g − 1 ( y 1 ) , … , g − 1 ( y n ) ) . \psi_{\theta;\epsilon;g,x}^{(\mathrm{SC};k)}(y_1,\ldots,y_n):=\frac{\lambda_{\epsilon}(d(y_k,g(\Pi_0)))}{\sum_{g'\in G_{\epsilon}(x)}\lambda_{\epsilon}(d(y_k,g'(\Pi_0)))}\psi_{\theta}(g^{-1}(y_1),\ldots,g^{-1}(y_n)). ψ θ ; ϵ ; g , x ( SC ; k ) ( y 1 , … , y n ) := ∑ g ′ ∈ G ϵ ( x ) λ ϵ ( d ( y k , g ′ ( Π 0 ))) λ ϵ ( d ( y k , g ( Π 0 ))) ψ θ ( g − 1 ( y 1 ) , … , g − 1 ( y n )) .
Computational cost compared to DA and GA. As mentioned in § 4.3 , SC suffers from a similar computational issue as DA and GA as it involves averaging. However, the amount of averaging SC requires is different. The additional averaging over n n n fundamental regions is crucial for ensuring anti-symmetry, which is made precise in Theorem E.1 and its proof in Appendix H. Averaging is also required over the set G ϵ G_{\epsilon} G ϵ , which necessarily includes all point group operations and unit translations of G G G (note that G ϵ G_{\epsilon} G ϵ is not a group). For a fair comparison, compare SC to DA and GA that average over the group G G G . For small systems with 1 × 1 × 1 1 \times1 \times1 1 × 1 × 1 supercell, G ϵ G_{\epsilon} G ϵ is strictly larger than the point group G G G , and the amount of averaging required by SC, n ∣ G ϵ ∣ n|G_{\epsilon}| n ∣ G ϵ ∣ , is strictly larger. For large systems, the cost of SC scales differently from DA and GA: While the cost of DA and GA increase as a function of the increasing group size ∣ G ∣ |G| ∣ G ∣ , for SC, G ϵ G_{\epsilon} G ϵ is fixed and the cost increases as a function of the number of electrons n n n .
Post hoc smoothed canonicalization (PC). As discussed in § 5 , we may perform canonicalization only at evaluation. However, there is an inherent tradeoff in the choice of ϵ \epsilon ϵ : If ϵ \epsilon ϵ is large, averaging happens in a large neighborhood of the boundary, which can cause the performance of ψ θ ^ ; ϵ ( S C ) \psi_{\hat{\theta};\epsilon}^{(\mathrm{SC})} ψ θ ^ ; ϵ ( SC ) to deviate significantly from the well-trained ψ θ ^ ( O G ) \psi_{\hat{\theta}}^{(\mathrm{OG})} ψ θ ^ ( OG ) (see Appendix § 15 .§ 2 ). If ϵ \epsilon ϵ is small, there may be a blowup in the derivatives of the weights via the smoothing function:
Lemma E.2. If λ ϵ \lambda_{\epsilon} λ ϵ is twice continuously differentiable, there are y 1 , y 2 ∈ [ 0 , ϵ ] y_1,y_2 \in[0,\epsilon] y 1 , y 2 ∈ [ 0 , ϵ ] s.t. ∂ λ ϵ ( y 1 ) = ϵ − 1 \partial\lambda_{\epsilon}(y_1)=\epsilon^{-1} ∂ λ ϵ ( y 1 ) = ϵ − 1 , ∂ 2 λ ϵ ( y 2 ) = ϵ − 2 \partial^2\lambda_{\epsilon}(y_2)=\epsilon^{-2} ∂ 2 λ ϵ ( y 2 ) = ϵ − 2 .
Since the local energy calculation involves ∂ 2 λ ϵ \partial^2\lambda_{\epsilon} ∂ 2 λ ϵ , this can lead to an ϵ − 2 \epsilon^{-2} ϵ − 2 blowup of local energy in a region of size O ( ϵ ) O(\epsilon) O ( ϵ ) , resulting in larger fluctuations of local energy. This inflation in the variance is clearly visible in Table 2 .
Choices of λ ϵ \lambda_{\epsilon} λ ϵ and d d d
To obtain a p p p -times continuously differentiable ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) , Theorem E.1 (iii) requires a smoothing function λ ϵ \lambda_{\epsilon} λ ϵ and a distance function d ( ⋅ , g ( Π 0 ) ) d(\mathord{\cdot},g(\Pi_0)) d ( ⋅ , g ( Π 0 )) that are p p p -times continuous differentiable for all g ∈ G g \in G g ∈ G .
Construction of λ ϵ \lambda_\epsilon λ ϵ . Since λ ϵ : R → [ 0 , 1 ] \lambda_\epsilon: \mathbb{R} \to[0,1] λ ϵ : R → [ 0 , 1 ] is required to be strictly decreasing with λ ϵ ( w ) = 1 \lambda_\epsilon(w)=1 λ ϵ ( w ) = 1 for w ≤ 0 w \le0 w ≤ 0 and λ ( w ) = 0 \lambda(w)=0 λ ( w ) = 0 for all w ≥ ϵ w \ge\epsilon w ≥ ϵ , the function
s ( w ) : = λ ϵ ( ϵ ( 1 − w ) ) s(w) := \lambda_\epsilon\bigl(\epsilon(1-w)\bigr) s ( w ) := λ ϵ ( ϵ ( 1 − w ) )
is a p p p -times continuously differentiable approximation of a step function: s s s is strictly increasing, s ( w ) = 0 s(w)=0 s ( w ) = 0 for all w ≤ 0 w \le0 w ≤ 0 and s ( w ) = 1 s(w)=1 s ( w ) = 1 for all w ≥ 1 w \ge1 w ≥ 1 . Many choices are available for such a function: To have an infinitely differentiable λ ϵ \lambda_\epsilon λ ϵ , one may consider the smoothed step function
s ∞ ( w ) : = ϕ ( w ) ϕ ( w ) + ϕ ( 1 − w ) where ϕ ( w ) : = { exp ( − w − 1 ) if w > 0 , 0 if w ≤ 0 , s_\infty(w) := \frac{\phi(w)}{\phi(w)+\phi(1-w)}
\qquad\text{where}\qquad
\phi(w) :=
\begin{cases}
\exp(-w^{-1}) & \text{if } w>0,\\
0 & \text{if } w\le0,
\end{cases} s ∞ ( w ) := ϕ ( w ) + ϕ ( 1 − w ) ϕ ( w ) where ϕ ( w ) := { exp ( − w − 1 ) 0 if w > 0 , if w ≤ 0 ,
and use the relationship λ ϵ ( w ) = s ( 1 − w / ϵ ) \lambda_\epsilon(w)=s(1-w/\epsilon) λ ϵ ( w ) = s ( 1 − w / ϵ ) to obtain
λ ϵ ; ∞ ( w ) : = ϕ ( 1 − w / ϵ ) ϕ ( 1 − w / ϵ ) + ϕ ( w / ϵ ) . \lambda_{\epsilon;\infty}(w) := \frac{\phi(1-w/\epsilon)}{\phi(1-w/\epsilon)+\phi(w/\epsilon)}. λ ϵ ; ∞ ( w ) := ϕ ( 1 − w / ϵ ) + ϕ ( w / ϵ ) ϕ ( 1 − w / ϵ ) .
As we only need to evaluate the Hamiltonian, twice continuous differentiability typically suffices for our problem. Another choice of the step function s s s and the corresponding λ ϵ \lambda_\epsilon λ ϵ are the degree-three polynomial splines
s 2 ( w ) : = { 0 if w ≤ 0 , 9 w 3 2 if w ∈ ( 0 , 1 3 ] , 9 ( 1 − w ) 3 2 + ( 2 − 3 w ) 3 2 + 1 if w ∈ ( 1 3 , 2 3 ] , − 9 ( 1 − w ) 3 2 + 1 if w ∈ ( 2 3 , 1 ] , 1 if w > 1 , λ ϵ ; 2 ( w ) : = { 1 if w ≤ 0 , 1 − 9 w 3 2 ϵ 3 if w ∈ ( 0 , ϵ 3 ] , − 9 w 3 2 ϵ 3 + ( 3 w − ϵ ) 3 2 ϵ 3 + 1 if w ∈ ( ϵ 3 , 2 ϵ 3 ] , 9 ( ϵ − w ) 3 2 ϵ 3 if w ∈ ( 2 ϵ 3 , ϵ ] , 0 if w > ϵ . s_2(w) :=
\begin{cases}
0 & \text{if } w\le0,\\
\frac{9w^3}{2} & \text{if } w\in\left(0,\frac{1}{3}\right],\\
\frac{9(1-w)^3}{2}+\frac{(2-3w)^3}{2}+1 & \text{if } w\in\left(\frac{1}{3},\frac{2}{3}\right],\\
-\frac{9(1-w)^3}{2}+1 & \text{if } w\in\left(\frac{2}{3},1\right],\\
1 & \text{if } w>1,
\end{cases}
\qquad
\lambda_{\epsilon;2}(w) :=
\begin{cases}
1 & \text{if } w\le0,\\
1-\frac{9w^3}{2\epsilon^3} & \text{if } w\in\left(0,\frac{\epsilon}{3}\right],\\
-\frac{9w^3}{2\epsilon^3}+\frac{(3w-\epsilon)^3}{2\epsilon^3}+1 & \text{if } w\in\left(\frac{\epsilon}{3},\frac{2\epsilon}{3}\right],\\
\frac{9(\epsilon-w)^3}{2\epsilon^3} & \text{if } w\in\left(\frac{2\epsilon}{3},\epsilon\right],\\
0 & \text{if } w>\epsilon.
\end{cases} s 2 ( w ) := ⎩ ⎨ ⎧ 0 2 9 w 3 2 9 ( 1 − w ) 3 + 2 ( 2 − 3 w ) 3 + 1 − 2 9 ( 1 − w ) 3 + 1 1 if w ≤ 0 , if w ∈ ( 0 , 3 1 ] , if w ∈ ( 3 1 , 3 2 ] , if w ∈ ( 3 2 , 1 ] , if w > 1 , λ ϵ ; 2 ( w ) := ⎩ ⎨ ⎧ 1 1 − 2 ϵ 3 9 w 3 − 2 ϵ 3 9 w 3 + 2 ϵ 3 ( 3 w − ϵ ) 3 + 1 2 ϵ 3 9 ( ϵ − w ) 3 0 if w ≤ 0 , if w ∈ ( 0 , 3 ϵ ] , if w ∈ ( 3 ϵ , 3 2 ϵ ] , if w ∈ ( 3 2 ϵ , ϵ ] , if w > ϵ .
Figure 10 plots s ∞ s_\infty s ∞ , s 2 s_2 s 2 , λ ϵ ; ∞ \lambda_{\epsilon;\infty} λ ϵ ; ∞ , λ ϵ ; 2 \lambda_{\epsilon;2} λ ϵ ; 2 and their derivatives. Note that the derivative plots for λ ϵ \lambda_\epsilon λ ϵ ’s verify Lemma E.2. To achieve general p p p -times continuous differentiability, we refer readers to the many constructions of such step functions available in the statistics literature, e.g. Theorem 3.3 of Chen et al. [11 ] or Lemma 34 of Huang et al. [25 ] . In particular, our s 2 s_2 s 2 is related to h m ; τ ; δ h_{m;\tau;\delta} h m ; τ ; δ constructed in Lemma 34 of Huang et al. [25 ] via s 2 ( w ) = h 2 ; 1 ; 1 ( w ) s_2(w)=h_{2;1;1}(w) s 2 ( w ) = h 2 ; 1 ; 1 ( w ) ; one can similarly take their h p ; 1 ; 1 h_{p;1;1} h p ; 1 ; 1 to obtain a p p p -times continuously differentiable s p s_p s p and obtain the corresponding λ ϵ ; p ( w ) : = 1 − s p ( w / ϵ ) \lambda_{\epsilon;p}(w):=1-s_p(w/\epsilon) λ ϵ ; p ( w ) := 1 − s p ( w / ϵ ) .
Figure 10. Illustration of different choices of step functions s s s , the corresponding λ ϵ \lambda_{\epsilon} λ ϵ ’s and their first two derivatives. ϵ = 0.1 \epsilon= 0.1 ϵ = 0.1 above.
Constructing d ( ⋅ , g ( Π 0 ) ) d(\mathord{\cdot},g(\Pi_0)) d ( ⋅ , g ( Π 0 )) via the step function s s s . A p p p -times continuously differentiable step function s s s , as discussed above, induces a p p p -times continuously differentiable approximation of max { ⋅ , 0 } \max\{\mathord{\cdot},0\} max { ⋅ , 0 } , given as
s ~ ( w ) : = w s ( w ) . \widetilde{s}(w) := w s(w). s ( w ) := w s ( w ) .
In particular s ~ ( w ) = 0 \widetilde{s}(w)=0 s ( w ) = 0 if and only if w ≤ 0 w\le0 w ≤ 0 , and s ~ ( w ) = w \widetilde{s}(w)=w s ( w ) = w for w ≥ 1 w\ge1 w ≥ 1 . To utilize s ~ \widetilde{s} s to construct d ( ⋅ , g ( Π 0 ) ) d(\mathord{\cdot},g(\Pi_0)) d ( ⋅ , g ( Π 0 )) , we notice that each g ( Π 0 ) g(\Pi_0) g ( Π 0 ) itself is a fundamental region, and its closure is a simplex completely characterized by the relation
x ∈ g ( Π 0 ) ⟺ ( x − g ( c 0 ) ) ⊤ n l ∥ n l ∥ 2 ≤ 1 for 1 ≤ l ≤ m , x\in g(\Pi_0)\Longleftrightarrow\frac{(x-g(c_0))^\top n_l}{\lVert n_l\rVert^2}\le1\quad\text{for }1\le l\le m, x ∈ g ( Π 0 ) ⟺ ∥ n l ∥ 2 ( x − g ( c 0 ) ) ⊤ n l ≤ 1 for 1 ≤ l ≤ m ,
where c 0 c_0 c 0 is the center of Π 0 \Pi_0 Π 0 and n l n_l n l is the normal vector starting from c 0 c_0 c 0 that is normal to the l l l -th face of Π 0 \Pi_0 Π 0 . This allows us to define
d ( x , g ( Π 0 ) ) : = ∑ l = 1 m ( s ~ ( ( x − g ( c 0 ) ) ⊤ n l ∥ n l ∥ 2 − 1 ) ) 2 . d(x,g(\Pi_0)) := \sum_{l=1}^{m}\left(\widetilde{s}\left(\frac{(x-g(c_0))^\top n_l}{\lVert n_l\rVert^2}-1\right)\right)^2. d ( x , g ( Π 0 )) := l = 1 ∑ m ( s ( ∥ n l ∥ 2 ( x − g ( c 0 ) ) ⊤ n l − 1 ) ) 2 .
This distance function satisfies that
d ( g ( x ) , g ( Π 0 ) ) = d ( x , Π 0 ) d(g(x),g(\Pi_0))=d(x,\Pi_0) d ( g ( x ) , g ( Π 0 )) = d ( x , Π 0 ) for all g ∈ G g\in G g ∈ G , which can be verified by noting that g ( ⋅ ) = A ( ⋅ ) + b g(\mathord{\cdot})=A(\mathord{\cdot})+b g ( ⋅ ) = A ( ⋅ ) + b for an orthogonal matrix A ∈ R 3 × 3 A\in\mathbb{R}^{3\times3} A ∈ R 3 × 3 and a translation vector b ∈ R 3 b\in\mathbb{R}^3 b ∈ R 3 ;
d ( x , Π ) = 0 d(x,\Pi)=0 d ( x , Π ) = 0 if and only if x ∈ Π x\in\Pi x ∈ Π , and the map x ↦ d ( x , Π ) x\mapsto d(x,\Pi) x ↦ d ( x , Π ) is p p p -times continuously differentiable.
To visualize this distance function in the case of a 1d system, see Figure 14 .
Figure 14. Plots of d ( x , [ 0 , 1 ) ) d(x,[0,1)) d ( x , [ 0 , 1 )) and its derivatives under different choices of the step function s s s .
An one-electron example of ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC )
For simplicity, we first illustrate ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) for a single 1d-electron system with unit translation invariance. In this case, the group G G G is generated by translations of length 1, and the fundamental region is Π 0 = [ 0 , 1 ) \Pi_0=[0,1) Π 0 = [ 0 , 1 ) , as illustrated in Figure 13 . As discussed at (8) , a standard canonicalization is to take x ↦ x m o d 1 x\mapsto x\bmod1 x ↦ x mod 1 , which suffers from discontinuity at the boundary. Our proposed smoothed canonicalization in this case becomes
Figure 13. Illustration of smoothed canonicalization ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) for a single 1d electron
ψ θ ; ϵ ( S C ) ( x ) = ∑ t ∈ Z s . t . d ( x + t , [ 0 , 1 ) ) λ ϵ ( d ( x + t , [ 0 , 1 ) ) ) ∑ t ′ ∈ Z s . t . d ( x + t ′ , [ 0 , 1 ) ) λ ϵ ( d ( x + t ′ , [ 0 , 1 ) ) ) ψ θ ( x + t ) . \psi_{\theta;\epsilon}^{(\mathrm{SC})}(x)=\sum_{\substack{t\in\mathbb{Z}\ \mathrm{s.t.}\ d(x+t,[0,1))}} \frac{\lambda_{\epsilon}(d(x+t,[0,1)))}{\sum_{\substack{t'\in\mathbb{Z}\ \mathrm{s.t.}\ d(x+t',[0,1))}} \lambda_{\epsilon}(d(x+t',[0,1)))}\psi_{\theta}(x+t). ψ θ ; ϵ ( SC ) ( x ) = t ∈ Z s.t. d ( x + t , [ 0 , 1 )) ∑ ∑ t ′ ∈ Z s.t. d ( x + t ′ , [ 0 , 1 )) λ ϵ ( d ( x + t ′ , [ 0 , 1 ))) λ ϵ ( d ( x + t , [ 0 , 1 ))) ψ θ ( x + t ) .
Choosing the distance function d ( ∙ , [ 0 , 1 ) ) d(\bullet,[0,1)) d ( ∙ , [ 0 , 1 )) as in Appendix § 15 .§ 1 , we have
d ( x , [ 0 , 1 ) ) = s ~ ( 2 ( x − 1 ) ) + s ~ ( − 2 x ) d(x,[0,1))=\tilde{s}(2(x-1))+\tilde{s}(-2x) d ( x , [ 0 , 1 )) = s ~ ( 2 ( x − 1 )) + s ~ ( − 2 x )
where
s ~ ( w ) = w s ( w ) \tilde{s}(w)=ws(w) s ~ ( w ) = w s ( w )
and s s s is a p p p -times differentiable approximation of a step function. Figure 14 plots d ( x , [ 0 , 1 ) ) d(x,[0,1)) d ( x , [ 0 , 1 )) and its derivatives under s = s ∞ s=s_{\infty} s = s ∞ and s = s 2 s=s_2 s = s 2 from Appendix § 15 .§ 1 , and verifies that d ( x , [ 0 , 1 ) ) d(x,[0,1)) d ( x , [ 0 , 1 )) is twice continuously differentiable under either case.
Notice by construction that ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) is 1-periodic (also see Theorem E.1 for the proof in the general case), so it suffices to consider the value of ψ θ ; ϵ ( S C ) ( x ) \psi_{\theta;\epsilon}^{(\mathrm{SC})}(x) ψ θ ; ϵ ( SC ) ( x ) within the interval x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] . Clearly d ( x , [ 0 , 1 ) ) = 0 d(x,[0,1))=0 d ( x , [ 0 , 1 )) = 0 . By construction of the distance function, the only possible translations t t t with non-zero (unnormalized) weight, defined as
w ~ ϵ ( x + t ) : = λ ϵ ( d ( x + t , [ 0 , 1 ) ) ) , \tilde{w}_{\epsilon}(x+t):=\lambda_{\epsilon}(d(x+t,[0,1))), w ~ ϵ ( x + t ) := λ ϵ ( d ( x + t , [ 0 , 1 ))) ,
are t = ± 1 t=\pm1 t = ± 1 . In other words, for x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] , we can express
ψ θ ; ϵ ( S C ) ( x ) = w ~ ϵ ( x − 1 ) S ϵ ( x ) ψ θ ( x − 1 ) + w ~ ϵ ( x ) S ϵ ( x ) ψ θ ( x ) + w ~ ϵ ( x + 1 ) S ϵ ( x ) ψ θ ( x + 1 ) , \psi_{\theta;\epsilon}^{(\mathrm{SC})}(x)=\frac{\tilde{w}_{\epsilon}(x-1)}{S_{\epsilon}(x)}\psi_{\theta}(x-1)+\frac{\tilde{w}_{\epsilon}(x)}{S_{\epsilon}(x)}\psi_{\theta}(x)+\frac{\tilde{w}_{\epsilon}(x+1)}{S_{\epsilon}(x)}\psi_{\theta}(x+1), ψ θ ; ϵ ( SC ) ( x ) = S ϵ ( x ) w ~ ϵ ( x − 1 ) ψ θ ( x − 1 ) + S ϵ ( x ) w ~ ϵ ( x ) ψ θ ( x ) + S ϵ ( x ) w ~ ϵ ( x + 1 ) ψ θ ( x + 1 ) ,
where we have denoted the normalizing term as S ϵ ( x ) : = w ~ ϵ ( x − 1 ) + w ~ ϵ ( x ) + w ~ ϵ ( x + 1 ) S_{\epsilon}(x) := \tilde{w}_{\epsilon}(x-1) + \tilde{w}_{\epsilon}(x) + \tilde{w}_{\epsilon}(x+1) S ϵ ( x ) := w ~ ϵ ( x − 1 ) + w ~ ϵ ( x ) + w ~ ϵ ( x + 1 ) . Figure 11 plots the values of the three unnormalized weights as x x x varies, under different choices of ϵ \epsilon ϵ . Several observations follow:
Figure 11. Plots of w ~ ϵ ( x − 1 ) \tilde{w}_{\epsilon}(x-1) w ~ ϵ ( x − 1 ) , w ~ ϵ ( x ) \tilde{w}_{\epsilon}(x) w ~ ϵ ( x ) and w ~ ϵ ( x + 1 ) \tilde{w}_{\epsilon}(x+1) w ~ ϵ ( x + 1 ) defined via either s ∞ s_{\infty} s ∞ or s 2 s_2 s 2 as the smoothed step function used in λ ϵ \lambda_{\epsilon} λ ϵ and d d d , and for different values of ϵ \epsilon ϵ .
For x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] , we always have w ~ ϵ ( x ) = 1 \tilde{w}_{\epsilon}(x) = 1 w ~ ϵ ( x ) = 1 , whereas w ~ ϵ ( x − 1 ) \tilde{w}_{\epsilon}(x-1) w ~ ϵ ( x − 1 ) is non-zero only for x x x close to 1, and w ~ ϵ ( x + 1 ) \tilde{w}_{\epsilon}(x+1) w ~ ϵ ( x + 1 ) is non-zero only for x x x close to 0. In particular, ψ θ ; ϵ ( S C ) ( x ) \psi_{\theta;\epsilon}^{(\mathrm{SC})}(x) ψ θ ; ϵ ( SC ) ( x ) is exactly the original wavefunction ψ θ ( x ) \psi_{\theta}(x) ψ θ ( x ) when x x x is far away from the edge of the interval, whereas an weighted average is taken with ψ θ ( x + 1 ) \psi_{\theta}(x+1) ψ θ ( x + 1 ) if x x x is close to 0, and with ψ θ ( x − 1 ) \psi_{\theta}(x-1) ψ θ ( x − 1 ) if x x x is close to 1.
At x = 0 x = 0 x = 0 and x = 1 x = 1 x = 1 , ψ θ ; ϵ ( S C ) ( x ) \psi_{\theta;\epsilon}^{(\mathrm{SC})}(x) ψ θ ; ϵ ( SC ) ( x ) is exactly the arithmetic mean of ψ θ ( 0 ) \psi_{\theta}(0) ψ θ ( 0 ) and ψ θ ( 1 ) \psi_{\theta}(1) ψ θ ( 1 ) . Indeed, smooth canonicalization allows the wavefunction to smoothly interpolate to an average exactly at the boundary of the fundamental region.
Effect on the performance of ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) as ϵ \epsilon ϵ increases. Compare the exact canonicalization ψ θ ( x m o d 1 ) \psi_{\theta}(x \bmod1) ψ θ ( x mod 1 ) with ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) : In the former case, the wavefunction value depends only on the values of ψ θ \psi_{\theta} ψ θ within the fundamental region Π 0 = [ 0 , 1 ) \Pi_{0} = [0,1) Π 0 = [ 0 , 1 ) , whereas in the latter case, ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) depends on ψ θ \psi_{\theta} ψ θ evaluated on a slightly larger set { x ∈ R ∣ d ( x , [ 0 , 1 ) ) ≤ ϵ } \{x \in\mathbb{R} \mid d(x,[0,1)) \leq\epsilon\} { x ∈ R ∣ d ( x , [ 0 , 1 )) ≤ ϵ } , represented as the interval [ x − ϵ , x 1 + ϵ ] [x_{-\epsilon},x_{1+\epsilon}] [ x − ϵ , x 1 + ϵ ] in both Figure 13 and Figure 11 . Meanwhile, ψ θ ( x m o d 1 ) = ψ θ ( x ) \psi_{\theta}(x \bmod1) = \psi_{\theta}(x) ψ θ ( x mod 1 ) = ψ θ ( x ) for all x ∈ ( 0 , 1 ) x \in(0,1) x ∈ ( 0 , 1 ) , whereas ψ θ ; ϵ ( S C ) ( x ) = ψ θ ( x ) \psi_{\theta;\epsilon}^{(\mathrm{SC})}(x) = \psi_{\theta}(x) ψ θ ; ϵ ( SC ) ( x ) = ψ θ ( x ) only for a slightly smaller set x ∈ [ x ϵ , x 1 − ϵ ] x \in[x_{\epsilon},x_{1-\epsilon}] x ∈ [ x ϵ , x 1 − ϵ ] , labelled in Figure 13 and Figure 11 . Notably for post hoc canonicalization, as ϵ \epsilon ϵ increases, the performance of ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) relies on ψ θ \psi_{\theta} ψ θ to be a well-trained on a larger and larger region, and ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) only recovers the trained ψ θ \psi_{\theta} ψ θ on a smaller and smaller region, thus retaining fewer benefits from training.
Gradient blowup as ϵ \epsilon ϵ decreases. Lemma E.2 and Figure 10 both show that the derivatives of λ ϵ \lambda_{\epsilon} λ ϵ blow up as ϵ \epsilon ϵ gets small. This also leads to a blowup in derivatives of w ~ ϵ ( x ) \tilde{w}_{\epsilon}(x) w ~ ϵ ( x ) and hence those of ψ θ ; ϵ ( S C ) \psi_{\theta;\epsilon}^{(\mathrm{SC})} ψ θ ; ϵ ( SC ) near the boundary. Since the derivatives of w ~ ϵ ( x ) \tilde{w}_{\epsilon}(x) w ~ ϵ ( x ) involve products of derivatives of λ ϵ \lambda_{\epsilon} λ ϵ and derivatives of d d d by the chain rule, the difference in the magnitude of blowup across s = s 2 s = s_{2} s = s 2 and s = s ∞ s = s_{\infty} s = s ∞ is more pronounced, as illustrated in Figure 12 : defining the weight function w ~ ϵ \tilde{w}_{\epsilon} w ~ ϵ via s ∞ s_{\infty} s ∞ leads to a higher degree of smoothness, but at the cost of a larger blowup in the derivative.
Figure 12. Plots of derivatives of w ~ ϵ ( x ) \tilde{w}_{\epsilon}(x) w ~ ϵ ( x ) for different values of ϵ \epsilon ϵ .
Proof of Fact 2.1
Fix g ∈ G g \in G g ∈ G with its action on x ∈ R 3 x \in\mathbb{R}^{3} x ∈ R 3 represented by g ( x ) = A x + b g(x) = Ax + b g ( x ) = A x + b . Given that ( ψ , E ) (\psi, E) ( ψ , E ) solves (1) , we seek to show that under the stated conditions, ψ g ( x ) : = ψ ( g ( x ) ) \psi_g(x) := \psi(g(x)) ψ g ( x ) := ψ ( g ( x )) is also an anti-symmetric solution to (1) with respect to the same energy E E E . For x ∈ R 3 n x \in\mathbb{R}^{3n} x ∈ R 3 n and g ∈ G g \in G g ∈ G , denote x g = g ( x ) x_g = g(x) x g = g ( x ) . Then for every x ∈ R 3 n x \in\mathbb{R}^{3n} x ∈ R 3 n ,
( − 1 2 ∇ 2 + V ( x ) ) ψ g ( x ) = ( − 1 2 ∇ 2 + V ( x ) ) ψ ( A x 1 + b , … , A x n + b ) = − 1 2 ( A T A ) ∇ 2 ψ ( x g ) + V ( x ) ψ ( x g ) = ( a ) − 1 2 ∇ 2 ψ ( x g ) + V ( x g ) ψ ( x g ) = ( b ) E ψ ( x g ) = E ψ g ( x ) . \begin{aligned}
\left(-\frac{1}{2}\nabla^{2}+V(x)\right)\psi_g(x)
&=\left(-\frac{1}{2}\nabla^{2}+V(x)\right)\psi(Ax_1+b,\ldots,Ax_n+b)\\
&=-\frac{1}{2}(A^{\mathsf{T}}A)\nabla^{2}\psi(x_g)+V(x)\psi(x_g)\\
&\overset{(a)}{=}-\frac{1}{2}\nabla^{2}\psi(x_g)+V(x_g)\psi(x_g)
\overset{(b)}{=}E\psi(x_g)=E\psi_g(x).
\end{aligned} ( − 2 1 ∇ 2 + V ( x ) ) ψ g ( x ) = ( − 2 1 ∇ 2 + V ( x ) ) ψ ( A x 1 + b , … , A x n + b ) = − 2 1 ( A T A ) ∇ 2 ψ ( x g ) + V ( x ) ψ ( x g ) = ( a ) − 2 1 ∇ 2 ψ ( x g ) + V ( x g ) ψ ( x g ) = ( b ) E ψ ( x g ) = E ψ g ( x ) .
In (a), we have used the G d i a g G_{\mathrm{diag}} G diag -invariance of V V V ; in (b), we used that ( ψ , E ) (\psi,E) ( ψ , E ) solve (1) . Since the above holds for all x ∈ R 3 n x \in\mathbb{R}^{3n} x ∈ R 3 n , we get that ( ψ g , E ) (\psi_g,E) ( ψ g , E ) also solves the Schrödinger’s equation. To verify the anti-symmetric requirement, we write the wavefunction ψ \psi ψ as ψ ( x ~ ) = ψ ( x ~ 1 , … , x ~ n ) \psi(\tilde{x})=\psi(\tilde{x}_1,\ldots,\tilde{x}_n) ψ ( x ~ ) = ψ ( x ~ 1 , … , x ~ n ) , where each x ~ i = ( x i , σ i ) \tilde{x}_i=(x_i,\sigma_i) x ~ i = ( x i , σ i ) now additionally depends on the spin σ i ∈ { ↑ , ↓ } \sigma_i\in\{\uparrow,\downarrow\} σ i ∈ { ↑ , ↓ } . Since G G G only acts on the spatial position in R 3 \mathbb{R}^{3} R 3 and leaves the spins invariant, we can WLOG express the g g g -transformed version of x ~ i \tilde{x}_i x ~ i as g ( x ~ i ) = ( g ( x i ) , σ i ) g(\tilde{x}_i)=(g(x_i),\sigma_i) g ( x ~ i ) = ( g ( x i ) , σ i ) . Moreover, since σ ∈ P n \sigma\in P_n σ ∈ P n commutes with the diagonal action of g ∈ G g\in G g ∈ G on ( R 3 × { ↑ , ↓ } ) n (\mathbb{R}^{3}\times\{\uparrow,\downarrow\})^n ( R 3 × { ↑ , ↓ } ) n ,
ψ g ( σ ( x ~ ) ) = ψ ( g ∘ σ ( x ~ ) ) = ψ ( σ ∘ g ( x ~ ) ) = ( c ) sgn ( σ ) ψ ( g ( x ~ ) ) = sgn ( σ ) ψ g ( x ~ ) . \psi_g(\sigma(\tilde{x}))=\psi(g\circ\sigma(\tilde{x}))=\psi(\sigma\circ g(\tilde{x}))\overset{(c)}{=}\operatorname{sgn}(\sigma)\psi(g(\tilde{x}))=\operatorname{sgn}(\sigma)\psi_g(\tilde{x}). ψ g ( σ ( x ~ )) = ψ ( g ∘ σ ( x ~ )) = ψ ( σ ∘ g ( x ~ )) = ( c ) sgn ( σ ) ψ ( g ( x ~ )) = sgn ( σ ) ψ g ( x ~ ) .
In (c), we have used the anti-symmetry of ψ \psi ψ . This proves that ( ψ g , E ) (\psi_g,E) ( ψ g , E ) solves (1) with the correct anti-symmetric requirement. To prove the theorem statement, we see that ψ G \psi^G ψ G is a linear combination of finitely many eigenfunctions ( ψ g ) g ∈ G (\psi_g)_{g\in G} ( ψ g ) g ∈ G with the same eigenvalue E E E and therefore yields an anti-symmetric solution with the same energy E E E .
Proofs for data augmentation and group-averaging
Proof of Proposition 4.1
We first compute the difference in expectation as
∥ E [ δ θ ( D A ) ] − E [ δ θ ( O G ) ] ∥ = ∥ E [ 1 N ∑ i ≤ N / k ∑ j ≤ k F g i , j ( X i ) ; ψ θ ] − E [ 1 N ∑ i ≤ N F X i ; ψ θ ] ∥ = ∥ E [ F g 1 , 1 ( X 1 ) ; ψ θ ] − E [ F X 1 ; ψ θ ] ∥ = ∥ E X ∼ p ψ θ ; D A ( m ) [ F X ; ψ θ ] − E Y ∼ p ψ θ ( m ) [ F Y ; ψ θ ] ∥ . \begin{aligned}
\left\|\mathbb{E}[\delta\theta(\mathrm{DA})]-\mathbb{E}[\delta\theta(\mathrm{OG})]\right\|
&=\left\|\mathbb{E}\left[\frac{1}{N}\sum_{i\leq N/k}\sum_{j\leq k}F_{g_{i,j}}(X_i);\psi_\theta\right]-\mathbb{E}\left[\frac{1}{N}\sum_{i\leq N}F_{X_i};\psi_\theta\right]\right\|\\
&=\left\|\mathbb{E}[F_{g_{1,1}}(X_1);\psi_\theta]-\mathbb{E}[F_{X_1};\psi_\theta]\right\|\\
&=\left\|\mathbb{E}_{X\sim p_{\psi_\theta;\mathrm{DA}}^{(m)}}[F_{X;\psi_\theta}]-\mathbb{E}_{Y\sim p_{\psi_\theta}^{(m)}}[F_{Y;\psi_\theta}]\right\|.
\end{aligned} ∥ E [ δ θ ( DA )] − E [ δ θ ( OG )] ∥ = E N 1 i ≤ N / k ∑ j ≤ k ∑ F g i , j ( X i ) ; ψ θ − E [ N 1 i ≤ N ∑ F X i ; ψ θ ] = E [ F g 1 , 1 ( X 1 ) ; ψ θ ] − E [ F X 1 ; ψ θ ] = E X ∼ p ψ θ ; DA ( m ) [ F X ; ψ θ ] − E Y ∼ p ψ θ ( m ) [ F Y ; ψ θ ] .
In the last line, we used linearity of expectation, the fact that g i , j ( X i ) g_{i,j}(X_i) g i , j ( X i ) ’s are identically distributed and that X i X_i X i ’s are identically distributed. Recall that { x ↦ ( F x ; ψ θ , F x ; ψ θ ⊗ 2 ) ∣ θ ∈ R q } ⊆ F \{x\mapsto(F_{x;\psi_\theta},F_{x;\psi_\theta}^{\otimes2})\mid\theta\in\mathbb{R}^{q}\}\subseteq\mathcal{F} { x ↦ ( F x ; ψ θ , F x ; ψ θ ⊗ 2 ) ∣ θ ∈ R q } ⊆ F and d F ( p , q ) = sup f ∈ F ∥ E X ∼ p [ f ( X ) ] − E Y ∼ q [ f ( Y ) ] ∥ d_{\mathcal{F}}(p,q)=\sup_{f\in\mathcal{F}}\|\mathbb{E}_{X\sim p}[f(X)]-\mathbb{E}_{Y\sim q}[f(Y)]\| d F ( p , q ) = sup f ∈ F ∥ E X ∼ p [ f ( X )] − E Y ∼ q [ f ( Y )] ∥ . This implies that
∥ E [ δ θ ( D A ) ] − E [ δ θ ( O G ) ] ∥ = ∥ E X ∼ p ψ θ ; D A ( m ) [ F X ; ψ θ ] − E Y ∼ p ψ θ ( m ) [ F Y ; ψ θ ] ∥ ≤ d F ( p ψ θ ; D A ( m ) , p ψ θ ( m ) ) . \left\|\mathbb{E}[\delta\theta(\mathrm{DA})]-\mathbb{E}[\delta\theta(\mathrm{OG})]\right\|
=\left\|\mathbb{E}_{X\sim p_{\psi_\theta;\mathrm{DA}}^{(m)}}[F_{X;\psi_\theta}]-\mathbb{E}_{Y\sim p_{\psi_\theta}^{(m)}}[F_{Y;\psi_\theta}]\right\|
\leq d_{\mathcal{F}}\left(p_{\psi_\theta;\mathrm{DA}}^{(m)},p_{\psi_\theta}^{(m)}\right). ∥ E [ δ θ ( DA )] − E [ δ θ ( OG )] ∥ = E X ∼ p ψ θ ; DA ( m ) [ F X ; ψ θ ] − E Y ∼ p ψ θ ( m ) [ F Y ; ψ θ ] ≤ d F ( p ψ θ ; DA ( m ) , p ψ θ ( m ) ) .
which proves the first bound. For the second bound, notice that
Var [ δ θ ( D A ) ] = Var [ 1 N ∑ i ≤ N / k ∑ j ≤ k F g i , j ( X i ) ; ψ θ ] \operatorname{Var}[\delta\theta^{(\mathrm{DA})}]
= \operatorname{Var}\left[\frac{1}{N}\sum_{i\le N/k}\sum_{j\le k}F_{g_{i,j}}(X_i);\psi_\theta\right] Var [ δ θ ( DA ) ] = Var N 1 i ≤ N / k ∑ j ≤ k ∑ F g i , j ( X i ) ; ψ θ
= ( a ) 1 N / k Var [ 1 k ∑ j ≤ k F g 1 , j ( X 1 ) ; ψ θ ] \mathrel{\overset{(a)}{=}} \frac{1}{N/k}\operatorname{Var}\left[\frac{1}{k}\sum_{j\le k}F_{g_{1,j}}(X_1);\psi_\theta\right] = ( a ) N / k 1 Var k 1 j ≤ k ∑ F g 1 , j ( X 1 ) ; ψ θ
= ( b ) 1 N Var [ F g 1 , 1 ( X 1 ) ; ψ θ ] + k − 1 N Cov [ F g 1 , 1 ( X 1 ) ; ψ θ , F g 1 , 2 ( X 1 ) ; ψ θ ] \mathrel{\overset{(b)}{=}} \frac{1}{N}\operatorname{Var}[F_{g_{1,1}}(X_1);\psi_\theta]
+\frac{k-1}{N}\operatorname{Cov}[F_{g_{1,1}}(X_1);\psi_\theta,F_{g_{1,2}}(X_1);\psi_\theta] = ( b ) N 1 Var [ F g 1 , 1 ( X 1 ) ; ψ θ ] + N k − 1 Cov [ F g 1 , 1 ( X 1 ) ; ψ θ , F g 1 , 2 ( X 1 ) ; ψ θ ]
= ( c ) 1 N Var [ F g 1 , 1 ( X 1 ) ; ψ θ ] + k − 1 N Var E [ F g 1 , 1 ( X 1 ) ; ψ θ ∣ X 1 ] . \mathrel{\overset{(c)}{=}} \frac{1}{N}\operatorname{Var}[F_{g_{1,1}}(X_1);\psi_\theta]
+\frac{k-1}{N}\operatorname{Var}\mathbb{E}[F_{g_{1,1}}(X_1);\psi_\theta\mid X_1]. = ( c ) N 1 Var [ F g 1 , 1 ( X 1 ) ; ψ θ ] + N k − 1 Var E [ F g 1 , 1 ( X 1 ) ; ψ θ ∣ X 1 ] .
In (a), we have noted that the summands are i.i.d. across i ≤ N / k i\le N/k i ≤ N / k ; in (b), we have computed the variance of the sum explicitly by expanding the expectation of a double-sum; in (c), we have applied the law of total covariance to obtain that
Cov [ F g 1 , 1 ( X 1 ) ; ψ θ , F g 1 , 2 ( X 1 ) ; ψ θ ] = Cov [ E [ F g 1 , 1 ( X 1 ) ; ψ θ ∣ X 1 ] , E [ F g 1 , 2 ( X 1 ) ; ψ θ ∣ X 1 ] ] ⏟ = Var E [ F g 1 , 1 ( X 1 ) ; ψ θ ∣ X 1 ] + E Cov [ F g 1 , 1 ( X 1 ) ; ψ θ , F g 1 , 2 ( X 1 ) ; ψ θ ∣ X 1 ] ⏟ = 0 . \operatorname{Cov}[F_{g_{1,1}}(X_1);\psi_\theta,F_{g_{1,2}}(X_1);\psi_\theta]
=
\underbrace{\operatorname{Cov}\left[\mathbb{E}[F_{g_{1,1}}(X_1);\psi_\theta\mid X_1],\mathbb{E}[F_{g_{1,2}}(X_1);\psi_\theta\mid X_1]\right]}_{=\operatorname{Var}\mathbb{E}[F_{g_{1,1}}(X_1);\psi_\theta\mid X_1]}
+
\underbrace{\mathbb{E}\operatorname{Cov}[F_{g_{1,1}}(X_1);\psi_\theta,F_{g_{1,2}}(X_1);\psi_\theta\mid X_1]}_{=0}. Cov [ F g 1 , 1 ( X 1 ) ; ψ θ , F g 1 , 2 ( X 1 ) ; ψ θ ] = = Var E [ F g 1 , 1 ( X 1 ) ; ψ θ ∣ X 1 ] Cov [ E [ F g 1 , 1 ( X 1 ) ; ψ θ ∣ X 1 ] , E [ F g 1 , 2 ( X 1 ) ; ψ θ ∣ X 1 ] ] + = 0 E Cov [ F g 1 , 1 ( X 1 ) ; ψ θ , F g 1 , 2 ( X 1 ) ; ψ θ ∣ X 1 ] .
Meanwhile, the same calculation with k = 1 k=1 k = 1 and g 1 , 1 g_{1,1} g 1 , 1 replaced by identity gives
Var [ δ θ ( O G ) ] = 1 N Var [ F X 1 ; ψ θ ] . \operatorname{Var}[\delta\theta^{(\mathrm{OG})}]
=\frac{1}{N}\operatorname{Var}[F_{X_1};\psi_\theta]. Var [ δ θ ( OG ) ] = N 1 Var [ F X 1 ; ψ θ ] .
Taking a difference and applying the triangle inequality twice, we have
∥ Var [ δ θ ( D A ) ] − Var [ δ θ ( O G ) ] − k − 1 N Var E [ F g 1 , 1 ( X 1 ) ; ψ θ ∣ X 1 ] ∥ = 1 N ∥ Var [ F g 1 , 1 ( X 1 ) ; ψ θ ] − Var [ F X 1 ; ψ θ ] ∥ ≤ 1 N ∥ E [ F g 1 , 1 ( X 1 ) ; ψ θ ⊗ 2 ] − E [ F X 1 ; ψ θ ⊗ 2 ] ∥ + 1 N ∥ E [ F g 1 , 1 ( X 1 ) ; ψ θ ] ⊗ 2 − E [ F X 1 ; ψ θ ] ⊗ 2 ∥ ≤ 1 N ∥ E [ F g 1 , 1 ( X 1 ) ; ψ θ ⊗ 2 ] − E [ F X 1 ; ψ θ ⊗ 2 ] ∥ + 1 N ∥ E [ F g 1 , 1 ( X 1 ) ; ψ θ ] + E [ F X 1 ; ψ θ ] ∥ ∥ E [ F g 1 , 1 ( X 1 ) ; ψ θ ] − E [ F X 1 ; ψ θ ] ∥ ≤ d F ( p ψ θ ; D A ( m ) , p ψ θ ( m ) ) N + 1 N ( 2 ∥ E [ δ θ ( O G ) ] ∥ + d F ( p ψ θ ; D A ( m ) , p ψ θ ( m ) ) ) × d F ( p ψ θ ; D A ( m ) , p ψ θ ( m ) ) = 1 + 2 ∥ E [ δ θ ( O G ) ] ∥ + d F ( p ψ θ ; D A ( m ) , p ψ θ ( m ) ) N × d F ( p ψ θ ; D A ( m ) , p ψ θ ( m ) ) . (10) \begin{aligned}
\left\|\operatorname{Var}[\delta\theta^{(\mathrm{DA})}]
-\operatorname{Var}[\delta\theta^{(\mathrm{OG})}]
-\frac{k-1}{N}\operatorname{Var}\mathbb{E}[F_{g_{1,1}}(X_1);\psi_\theta\mid X_1]\right\|
=
\frac{1}{N}\left\|\operatorname{Var}[F_{g_{1,1}}(X_1);\psi_\theta]
-\operatorname{Var}[F_{X_1};\psi_\theta]\right\| \\
&\le\frac{1}{N}\left\|\mathbb{E}[F_{g_{1,1}}(X_1);\psi_\theta^{\otimes2}]
-\mathbb{E}[F_{X_1};\psi_\theta^{\otimes2}]\right\|
+\frac{1}{N}\left\|\mathbb{E}[F_{g_{1,1}}(X_1);\psi_\theta]^{\otimes2}
-\mathbb{E}[F_{X_1};\psi_\theta]^{\otimes2}\right\| \\
&\le\frac{1}{N}\left\|\mathbb{E}[F_{g_{1,1}}(X_1);\psi_\theta^{\otimes2}]
-\mathbb{E}[F_{X_1};\psi_\theta^{\otimes2}]\right\|
+\frac{1}{N}\left\|\mathbb{E}[F_{g_{1,1}}(X_1);\psi_\theta]
+\mathbb{E}[F_{X_1};\psi_\theta]\right\|
\left\|\mathbb{E}[F_{g_{1,1}}(X_1);\psi_\theta]
-\mathbb{E}[F_{X_1};\psi_\theta]\right\| \\
&\le\frac{d_{F}(p_{\psi_\theta;\mathrm{DA}}^{(m)},p_{\psi_\theta}^{(m)})}{N}
+\frac{1}{N}\left(2\|\mathbb{E}[\delta\theta^{(\mathrm{OG})}]\|
+d_{F}(p_{\psi_\theta;\mathrm{DA}}^{(m)},p_{\psi_\theta}^{(m)})\right)
\times d_{F}(p_{\psi_\theta;\mathrm{DA}}^{(m)},p_{\psi_\theta}^{(m)}) \\
&= \frac{1+2\|\mathbb{E}[\delta\theta^{(\mathrm{OG})}]\|
+d_{F}(p_{\psi_\theta;\mathrm{DA}}^{(m)},p_{\psi_\theta}^{(m)})}{N}
\times d_{F}(p_{\psi_\theta;\mathrm{DA}}^{(m)},p_{\psi_\theta}^{(m)}). \tag*{(10)}
\end{aligned} Var [ δ θ ( DA ) ] − Var [ δ θ ( OG ) ] − N k − 1 Var E [ F g 1 , 1 ( X 1 ) ; ψ θ ∣ X 1 ] = N 1 Var [ F g 1 , 1 ( X 1 ) ; ψ θ ] − Var [ F X 1 ; ψ θ ] ≤ N 1 E [ F g 1 , 1 ( X 1 ) ; ψ θ ⊗ 2 ] − E [ F X 1 ; ψ θ ⊗ 2 ] + N 1 E [ F g 1 , 1 ( X 1 ) ; ψ θ ] ⊗ 2 − E [ F X 1 ; ψ θ ] ⊗ 2 ≤ N 1 E [ F g 1 , 1 ( X 1 ) ; ψ θ ⊗ 2 ] − E [ F X 1 ; ψ θ ⊗ 2 ] + N 1 E [ F g 1 , 1 ( X 1 ) ; ψ θ ] + E [ F X 1 ; ψ θ ] E [ F g 1 , 1 ( X 1 ) ; ψ θ ] − E [ F X 1 ; ψ θ ] ≤ N d F ( p ψ θ ; DA ( m ) , p ψ θ ( m ) ) + N 1 ( 2∥ E [ δ θ ( OG ) ] ∥ + d F ( p ψ θ ; DA ( m ) , p ψ θ ( m ) ) ) × d F ( p ψ θ ; DA ( m ) , p ψ θ ( m ) ) = N 1 + 2∥ E [ δ θ ( OG ) ] ∥ + d F ( p ψ θ ; DA ( m ) , p ψ θ ( m ) ) × d F ( p ψ θ ; DA ( m ) , p ψ θ ( m ) ) . (10)
This proves the second bound. In the case when g ( X 1 ) = d X 1 g(X_1)\mathrel{\overset{d}{=}}X_1 g ( X 1 ) = d X 1 for all g ∈ G d i a g g\in\mathbb{G}_{\mathrm{diag}} g ∈ G diag , we have p ψ θ ; D A ( m ) = p ψ θ ( m ) p_{\psi_\theta;\mathrm{DA}}^{(m)}=p_{\psi_\theta}^{(m)} p ψ θ ; DA ( m ) = p ψ θ ( m ) and therefore the bounds above all evaluate to zero. In this case we have E [ δ θ ( D A ) ] = E [ δ θ ( O G ) ] \mathbb{E}[\delta\theta^{(\mathrm{DA})}]=\mathbb{E}[\delta\theta^{(\mathrm{OG})}] E [ δ θ ( DA ) ] = E [ δ θ ( OG ) ] and
Var [ δ θ ( D A ) ] − Var [ δ θ ( O G ) ] = k − 1 N Var E [ F g 1 , 1 ( X 1 ) ; ψ θ ∣ X 1 ] , \operatorname{Var}[\delta\theta^{(\mathrm{DA})}]
-\operatorname{Var}[\delta\theta^{(\mathrm{OG})}]
=\frac{k-1}{N}\operatorname{Var}\mathbb{E}[F_{g_{1,1}}(X_1);\psi_\theta\mid X_1], Var [ δ θ ( DA ) ] − Var [ δ θ ( OG ) ] = N k − 1 Var E [ F g 1 , 1 ( X 1 ) ; ψ θ ∣ X 1 ] ,
which is positive semi-definite.
Proof of Lemma 4.2
By construction, δ θ ( G A ) = 1 N / k ∑ i ≤ N / k F X i G ; ψ θ \delta\theta^{(\mathrm{GA})}=\frac{1}{N/k}\sum_{i\le N/k}F_{X_i^{G}};\psi_\theta δ θ ( GA ) = N / k 1 ∑ i ≤ N / k F X i G ; ψ θ is a size-N / k N/k N / k empirical average of i.i.d. quantities. The mean and variance formulas thus follows directly from a standard computation:
E [ δ θ ( G A ) ] = E [ F X 1 G ; ψ θ ] and Var [ δ θ ( G A ) ] = Var [ F X 1 G ; ψ θ ] N / k . \mathbb{E}[\delta\theta^{(\mathrm{GA})}]
=\mathbb{E}[F_{X_1^{G}};\psi_\theta]
\qquad\text{and}\qquad
\operatorname{Var}[\delta\theta^{(\mathrm{GA})}]
=\frac{\operatorname{Var}[F_{X_1^{G}};\psi_\theta]}{N/k}. E [ δ θ ( GA ) ] = E [ F X 1 G ; ψ θ ] and Var [ δ θ ( GA ) ] = N / k Var [ F X 1 G ; ψ θ ] .
Proof of Theorem D.1
To prove the coordinate-wise bound, fix l ≤ p l\le p l ≤ p . Note that if σ l ( D A ) = 0 \sigma_l^{(\mathrm{DA})}=0 σ l ( DA ) = 0 , then δ θ l ( D A ) = E [ δ θ l ( D A ) ] \delta\theta_l^{(\mathrm{DA})}=\mathbb{E}[\delta\theta_l^{(\mathrm{DA})}] δ θ l ( DA ) = E [ δ θ l ( DA ) ] with probability 1, implying that the distribution difference is zero and hence the bound is satisfied. In the case σ l ( D A ) > 0 \sigma_l^{(\mathrm{DA})}>0 σ l ( DA ) > 0 , δ θ l ( D A ) = 1 N / k ∑ i ≤ N / k F i l ( D A ) \delta\theta_l^{(\mathrm{DA})}=\frac{1}{N/k}\sum_{i\le N/k}F_{il}^{(\mathrm{DA})} δ θ l ( DA ) = N / k 1 ∑ i ≤ N / k F i l ( DA ) is an average of i.i.d. univariate random variables with positive variance. By renormalizing t t t and applying the Berry-Esseen theorem (see Theorem 3.7 of Chen et al. [11 ] or Shevtsova [44 ] for the version with a tight constant C 1 = 0.469 C_1 = 0.469 C 1 = 0.469 ) applied to 1 N / k ∑ i ≤ N / k F i l \frac{1}{N/k}\sum_{i\leq N/k} F_{il} N / k 1 ∑ i ≤ N / k F i l , we get that
sup t ∈ R ∣ P ( δ θ 1 l ( D A ) ≤ t ) − P ( E [ δ θ 1 l ( D A ) ] + ( Var [ δ θ 1 l ( D A ) ] ) 1 / 2 Z l ≤ t ) ∣ = sup t ∈ R ∣ P ( 1 N / k ∑ i ≤ N / k ( F i l ( D A ) − E [ F i l ( D A ) ] ) ≤ t ) − P ( σ l ( D A ) N / k Z l ≤ t ) ∣ ≤ C 1 E ∣ F 1 l ( D A ) ∣ 3 N / k ( σ l ( D A ) ) 3 . \begin{aligned}
\sup_{t\in\mathbb{R}} \left| \mathbb{P}\left(\delta\theta_{1l}^{(\mathrm{DA})} \leq t\right) - \mathbb{P}\left(\mathbb{E}\left[\delta\theta_{1l}^{(\mathrm{DA})}\right] + \left(\operatorname{Var}\left[\delta\theta_{1l}^{(\mathrm{DA})}\right]\right)^{1/2} Z_l \leq t\right) \right| \\
&= \sup_{t\in\mathbb{R}} \left| \mathbb{P}\left(\frac{1}{\sqrt{N/k}}\sum_{i\leq N/k}\left(F_{il}^{(\mathrm{DA})}-\mathbb{E}\left[F_{il}^{(\mathrm{DA})}\right]\right)\leq t\right)-\mathbb{P}\left(\frac{\sigma_l^{(\mathrm{DA})}}{\sqrt{N/k}}Z_l\leq t\right)\right|
\leq\frac{C_1\mathbb{E}\left|F_{1l}^{(\mathrm{DA})}\right|^3}{\sqrt{N/k}\left(\sigma_l^{(\mathrm{DA})}\right)^3}.
\end{aligned} t ∈ R sup P ( δ θ 1 l ( DA ) ≤ t ) − P ( E [ δ θ 1 l ( DA ) ] + ( Var [ δ θ 1 l ( DA ) ] ) 1/2 Z l ≤ t ) = t ∈ R sup P N / k 1 i ≤ N / k ∑ ( F i l ( DA ) − E [ F i l ( DA ) ] ) ≤ t − P ( N / k σ l ( DA ) Z l ≤ t ) ≤ N / k ( σ l ( DA ) ) 3 C 1 E F 1 l ( DA ) 3 .
This proves the first set of bounds. To prove the second set of bounds, we first denote the mean-zero variable F ~ i l : = − F i l ( D A ) + E [ F i l ( D A ) ] \widetilde{F}_{il}:=-F_{il}^{(\mathrm{DA})}+\mathbb{E}\left[F_{il}^{(\mathrm{DA})}\right] F i l := − F i l ( DA ) + E [ F i l ( DA ) ] , and let ( Z ~ 11 , … , Z ~ n p ) (\widetilde{Z}_{11},\ldots,\widetilde{Z}_{np}) ( Z 11 , … , Z n p ) be an R n p \mathbb{R}^{np} R n p -valued Gaussian vector with the same mean and variance as ( F ~ 11 , … , F ~ n p ) (\widetilde{F}_{11},\ldots,\widetilde{F}_{np}) ( F 11 , … , F n p ) . The difference in distribution function can be re-expressed as
sup t ∈ R ∣ P ( max l ≤ p ∣ δ θ 1 l ( D A ) − E [ δ θ 1 l ( D A ) ] ∣ ≤ t ) − P ( max l ≤ p ∣ ( Var [ δ θ 1 l ( D A ) ] ) 1 / 2 Z l ∣ ≤ t ) ∣ = sup t ∈ R ∣ P ( max l ≤ p ∣ 1 N / k ∑ i ≤ N / k F ~ i l ∣ ≤ t ) − P ( max l ≤ p ∣ 1 N / k ∑ i ≤ N / k Z ~ i l ∣ ≤ t ) ∣ . (11) \begin{aligned}
\sup_{t\in\mathbb{R}}\left|\mathbb{P}\left(\max_{l\leq p}\left|\delta\theta_{1l}^{(\mathrm{DA})}-\mathbb{E}\left[\delta\theta_{1l}^{(\mathrm{DA})}\right]\right|\leq t\right)-\mathbb{P}\left(\max_{l\leq p}\left|\left(\operatorname{Var}\left[\delta\theta_{1l}^{(\mathrm{DA})}\right]\right)^{1/2}Z_l\right|\leq t\right)\right| \\
&= \sup_{t\in\mathbb{R}}\left|\mathbb{P}\left(\max_{l\leq p}\left|\frac{1}{\sqrt{N/k}}\sum_{i\leq N/k}\widetilde{F}_{il}\right|\leq t\right)-\mathbb{P}\left(\max_{l\leq p}\left|\frac{1}{\sqrt{N/k}}\sum_{i\leq N/k}\widetilde{Z}_{il}\right|\leq t\right)\right|. \tag*{(11)}
\end{aligned} t ∈ R sup P ( l ≤ p max δ θ 1 l ( DA ) − E [ δ θ 1 l ( DA ) ] ≤ t ) − P ( l ≤ p max ( Var [ δ θ 1 l ( DA ) ] ) 1/2 Z l ≤ t ) = t ∈ R sup P l ≤ p max N / k 1 i ≤ N / k ∑ F i l ≤ t − P l ≤ p max N / k 1 i ≤ N / k ∑ Z i l ≤ t . (11)
where we have used the definition of θ 1 l ( D A ) \theta_{1l}^{(\mathrm{DA})} θ 1 l ( DA ) and also replaced t t t by t / N / k t/\sqrt{N/k} t / N / k . Note that F ~ i l \widetilde{F}_{il} F i l 's are i.i.d. mean-zero across 1 ≤ i ≤ N / k 1\leq i\leq N/k 1 ≤ i ≤ N / k and Z ~ i l \widetilde{Z}_{il} Z i l 's are i.i.d. mean-zero across 1 ≤ i ≤ N / k 1\leq i\leq N/k 1 ≤ i ≤ N / k . As before, if σ l ( D A ) = 0 \sigma_l^{(\mathrm{DA})}=0 σ l ( DA ) = 0 for all 1 ≤ l ≤ p 1\leq l\leq p 1 ≤ l ≤ p , the two random variables to be compared are both 0 with probability 1 and the distributional difference above evaluates to zero. If there is at least one l l l such that σ l ( D A ) = 0 \sigma_l^{(\mathrm{DA})}=0 σ l ( DA ) = 0 , we can restrict both maxima above to be over l ≤ p l\leq p l ≤ p such that σ l ( D A ) = 0 \sigma_l^{(\mathrm{DA})}=0 σ l ( DA ) = 0 and ignore the coordinates with zero variance. As such, we can WLOG assume that σ l ( D A ) > 0 \sigma_l^{(\mathrm{DA})}>0 σ l ( DA ) > 0 for all l ≤ p l\leq p l ≤ p . Now write
F ~ i : = ( σ 1 − 1 F ~ i 1 , … , σ p − 1 F ~ i p ) , Z ~ i : = ( σ 1 − 1 Z ~ i 1 , … , σ p − 1 Z ~ i p ) , \widetilde{F}_i := \left(\sigma_1^{-1}\widetilde{F}_{i1},\ldots,\sigma_p^{-1}\widetilde{F}_{ip}\right), \qquad\widetilde{Z}_i := \left(\sigma_1^{-1}\widetilde{Z}_{i1},\ldots,\sigma_p^{-1}\widetilde{Z}_{ip}\right), F i := ( σ 1 − 1 F i 1 , … , σ p − 1 F i p ) , Z i := ( σ 1 − 1 Z i 1 , … , σ p − 1 Z i p ) ,
and denote the hyper-rectangular set A ( t ) : = [ − σ 1 − 1 t , + σ 1 − 1 t ] × ⋯ × [ − σ p − 1 t , + σ p − 1 t ] ⊆ R q \mathcal{A}(t):=[-\sigma_1^{-1}t,+\sigma_1^{-1}t]\times\cdots\times[-\sigma_p^{-1}t,+\sigma_p^{-1}t]\subseteq\mathbb{R}^q A ( t ) := [ − σ 1 − 1 t , + σ 1 − 1 t ] × ⋯ × [ − σ p − 1 t , + σ p − 1 t ] ⊆ R q . We can now express the difference above further as
( 11 ) = sup t ∈ R ∣ P ( 1 N / k ∑ i ≤ N / k F ~ i ∈ A ( t ) ) − P ( 1 N / k ∑ i ≤ N / k Z ~ i ∈ A ( t ) ) ∣ . (11)=\sup_{t\in\mathbb{R}}\left|\mathbb{P}\left(\frac{1}{\sqrt{N/k}}\sum_{i\leq N/k}\widetilde{F}_i\in\mathcal{A}(t)\right)-\mathbb{P}\left(\frac{1}{\sqrt{N/k}}\sum_{i\leq N/k}\widetilde{Z}_i\in\mathcal{A}(t)\right)\right|. ( 11 ) = t ∈ R sup P N / k 1 i ≤ N / k ∑ F i ∈ A ( t ) − P N / k 1 i ≤ N / k ∑ Z i ∈ A ( t ) .
This is a difference in distribution functions between a normalized empirical average of p p p -dimensional vectors with zero mean and identity covariance and a standard Gaussian in R q \mathbb{R}^q R q , measured through a subset of hyperrectangles. In particular, this is the quantity controlled by Chernozhukov et al. [13 ] : Under the stated moment conditions and applying their Proposition 2.1, we have that for some absolute constant C 2 > 0 C_2>0 C 2 > 0 ,
( 11 ) ≤ C 2 ( ( F ~ ( D A ) ) 2 ( log ( p N / k ) ) 7 N / k ) 1 / 6 . (11)\leq C_2\left(\frac{\left(\widetilde{F}^{(\mathrm{DA})}\right)^2\left(\log(pN/k)\right)^7}{N/k}\right)^{1/6}. ( 11 ) ≤ C 2 N / k ( F ( DA ) ) 2 ( log ( pN / k ) ) 7 1/6 .
Proofs for results on canonicalization
Proof of Theorem E.1
To prove (i), we fix any g ~ ∈ G \widetilde{g}\in G g ∈ G , and WLOG let k = 1 k=1 k = 1 . Then by the definition of ψ θ ; ϵ ( S C ; 1 ) \psi_{\theta;\epsilon}^{(\mathrm{SC;1})} ψ θ ; ϵ ( SC ; 1 ) ,
ψ θ ; ϵ ( S C ; 1 ) ( g ~ ( x 1 ) , … , g ~ ( x n ) ) = ∑ g ∈ G s . t . d ( g ~ ( x 1 ) , g ( Π 0 ) ) ≤ ϵ ( λ ϵ ( d ( g ~ ( x 1 ) , g ( Π 0 ) ) ) ∑ g ′ ∈ G s . t . d ( g ~ ( x 1 ) , g ′ ( Π 0 ) ) ≤ ϵ λ ϵ ( d ( g ~ ( x 1 ) , g ′ ( Π 0 ) ) ) × ψ θ ( g − 1 g ~ ( x 1 ) , … , g − 1 g ~ ( x n ) ) ) . \psi_{\theta;\epsilon}^{(\mathrm{SC;1})}\left(\widetilde{g}(x_1),\ldots,\widetilde{g}(x_n)\right)
=
\sum_{\substack{g\in G\ \mathrm{s.t.}\\ d(\widetilde{g}(x_1),g(\Pi_0))\leq\epsilon}}
\left(
\frac{\lambda_\epsilon(d(\widetilde{g}(x_1),g(\Pi_0)))}
{\sum_{\substack{g'\in G\ \mathrm{s.t.}\\d(\widetilde{g}(x_1),g'(\Pi_0))\leq\epsilon}}\lambda_\epsilon(d(\widetilde{g}(x_1),g'(\Pi_0)))}
\times\psi_\theta\left(g^{-1}\widetilde{g}(x_1),\ldots,g^{-1}\widetilde{g}(x_n)\right)
\right). ψ θ ; ϵ ( SC ; 1 ) ( g ( x 1 ) , … , g ( x n ) ) = g ∈ G s.t. d ( g ( x 1 ) , g ( Π 0 )) ≤ ϵ ∑ ∑ g ′ ∈ G s.t. d ( g ( x 1 ) , g ′ ( Π 0 )) ≤ ϵ λ ϵ ( d ( g ( x 1 ) , g ′ ( Π 0 ))) λ ϵ ( d ( g ( x 1 ) , g ( Π 0 ))) × ψ θ ( g − 1 g ( x 1 ) , … , g − 1 g ( x n ) ) .
Relabelling g g g by g ~ g \widetilde{g}g g g and g ′ g' g ′ by g ~ g ′ \widetilde{g}g' g g ′ in the sums above, and noting that d ( g ~ ( x 1 ) , g ~ g ( Π 0 ) ) = d ( x 1 , g ( Π 0 ) ) d(\widetilde{g}(x_1),\widetilde{g}g(\Pi_0))=d(x_1,g(\Pi_0)) d ( g ( x 1 ) , g g ( Π 0 )) = d ( x 1 , g ( Π 0 )) , we obtain that
ψ θ ; ϵ ( S C ; 1 ) ( g ~ ( x 1 ) , … , g ~ ( x n ) ) = ∑ g ∈ G s . t . d ( g ~ ( x 1 ) , g ~ g ( Π 0 ) ) ≤ ϵ ( λ ϵ ( d ( g ~ ( x 1 ) , g ~ g ( Π 0 ) ) ) ∑ g ′ ∈ G s . t . d ( g ~ ( x 1 ) , g ~ g ′ ( Π 0 ) ) ≤ ϵ λ ϵ ( d ( g ~ ( x 1 ) , g ~ g ′ ( Π 0 ) ) ) × ψ θ ( g − 1 g ~ − 1 g ~ ( x 1 ) , … , g − 1 g ~ − 1 g ~ ( x n ) ) ) = ∑ g ∈ G s . t . d ( x 1 , g ( Π 0 ) ) ≤ ϵ ( λ ϵ ( d ( x 1 , g ( Π 0 ) ) ) ∑ g ′ ∈ G s . t . d ( x 1 , g ′ ( Π 0 ) ) ≤ ϵ λ ϵ ( d ( x 1 , g ′ ( Π 0 ) ) ) ψ θ ( g − 1 ( x 1 ) , … , g − 1 ( x n ) ) ) = ψ θ ; ϵ ( S C ; 1 ) ( x 1 , … , x n ) . \begin{aligned}
\psi_{\theta;\epsilon}^{(\mathrm{SC;1})}\left(\widetilde{g}(x_1),\ldots,\widetilde{g}(x_n)\right)
=
\sum_{\substack{g\in G\ \mathrm{s.t.}\\d(\widetilde{g}(x_1),\widetilde{g}g(\Pi_0))\leq\epsilon}}
\left(
\frac{\lambda_\epsilon(d(\widetilde{g}(x_1),\widetilde{g}g(\Pi_0)))}
{\sum_{\substack{g'\in G\ \mathrm{s.t.}\\d(\widetilde{g}(x_1),\widetilde{g}g'(\Pi_0))\leq\epsilon}}\lambda_\epsilon(d(\widetilde{g}(x_1),\widetilde{g}g'(\Pi_0)))}
\times\psi_\theta\left(g^{-1}\widetilde{g}^{-1}\widetilde{g}(x_1),\ldots,g^{-1}\widetilde{g}^{-1}\widetilde{g}(x_n)\right)
\right) \\
&=
\sum_{\substack{g\in G\ \mathrm{s.t.}\\d(x_1,g(\Pi_0))\leq\epsilon}}
\left(
\frac{\lambda_\epsilon(d(x_1,g(\Pi_0)))}
{\sum_{\substack{g'\in G\ \mathrm{s.t.}\\d(x_1,g'(\Pi_0))\leq\epsilon}}\lambda_\epsilon(d(x_1,g'(\Pi_0)))}
\psi_\theta\left(g^{-1}(x_1),\ldots,g^{-1}(x_n)\right)
\right) \\
&=\psi_{\theta;\epsilon}^{(\mathrm{SC;1})}(x_1,\ldots,x_n).
\end{aligned} ψ θ ; ϵ ( SC ; 1 ) ( g ( x 1 ) , … , g ( x n ) ) = g ∈ G s.t. d ( g ( x 1 ) , g g ( Π 0 )) ≤ ϵ ∑ ∑ g ′ ∈ G s.t. d ( g ( x 1 ) , g g ′ ( Π 0 )) ≤ ϵ λ ϵ ( d ( g ( x 1 ) , g g ′ ( Π 0 ))) λ ϵ ( d ( g ( x 1 ) , g g ( Π 0 ))) × ψ θ ( g − 1 g − 1 g ( x 1 ) , … , g − 1 g − 1 g ( x n ) ) = g ∈ G s.t. d ( x 1 , g ( Π 0 )) ≤ ϵ ∑ ∑ g ′ ∈ G s.t. d ( x 1 , g ′ ( Π 0 )) ≤ ϵ λ ϵ ( d ( x 1 , g ′ ( Π 0 ))) λ ϵ ( d ( x 1 , g ( Π 0 ))) ψ θ ( g − 1 ( x 1 ) , … , g − 1 ( x n ) ) = ψ θ ; ϵ ( SC ; 1 ) ( x 1 , … , x n ) .
This proves diagonal G \mathcal{G} G -invariance.
To prove (ii), we shall make the spin-dependence explicit and write x ~ i = ( x i , σ i ) \widetilde{x}_i=(x_i,\sigma_i) x i = ( x i , σ i ) and g ( x ~ i ) = ( g ( x i ) , σ i ) g(\widetilde{x}_i)=(g(x_i),\sigma_i) g ( x i ) = ( g ( x i ) , σ i ) . First consider π \pi π , a transposition that swaps the indices 1 and 2. Then
ψ θ ; ϵ ( S C ) ( x ~ π ( 1 ) , … , x ~ π ( n ) ) = 1 n ∑ k = 1 n ψ θ ; ϵ ( S C ; k ) ( x ~ 2 , x ~ 1 , x ~ 3 , … , x ~ n ) . \psi_{\theta;\epsilon}^{(\mathrm{SC})}(\widetilde{x}_{\pi(1)},\ldots,\widetilde{x}_{\pi(n)})=\frac{1}{n}\sum_{k=1}^{n}\psi_{\theta;\epsilon}^{(\mathrm{SC};k)}(\widetilde{x}_2,\widetilde{x}_1,\widetilde{x}_3,\ldots,\widetilde{x}_n). ψ θ ; ϵ ( SC ) ( x π ( 1 ) , … , x π ( n ) ) = n 1 k = 1 ∑ n ψ θ ; ϵ ( SC ; k ) ( x 2 , x 1 , x 3 , … , x n ) .
By the anti-symmetry of ψ θ \psi_\theta ψ θ , we see that for k ≥ 3 k\geq3 k ≥ 3 ,
ψ θ ; ϵ ( S C ; k ) ( x ~ 2 , x ~ 1 , x ~ 3 , … , x ~ n ) = ∑ g ∈ G ϵ ( x k ) w ϵ g ( x k ) ψ θ ( g − 1 ( x ~ 2 ) , g − 1 ( x ~ 1 ) , g − 1 ( x ~ 3 ) , … , g − 1 ( x ~ n ) ) = − ∑ g ∈ G ϵ ( x k ) w ϵ g ( x k ) ψ θ ( g − 1 ( x 1 ) , g − 1 ( x 2 ) , g − 1 ( x 3 ) , … , g − 1 ( x n ) ) = − ψ θ ; ϵ ( S C ; k ) ( x ~ 1 , … , x ~ n ) . \begin{aligned}
\psi_{\theta;\epsilon}^{(\mathrm{SC};k)}(\widetilde{x}_2,\widetilde{x}_1,\widetilde{x}_3,\ldots,\widetilde{x}_n)
&=\sum_{g\in G_\epsilon(x_k)}w_\epsilon^g(x_k)\psi_\theta\left(g^{-1}(\widetilde{x}_2),g^{-1}(\widetilde{x}_1),g^{-1}(\widetilde{x}_3),\ldots,g^{-1}(\widetilde{x}_n)\right)\\
&=-\sum_{g\in G_\epsilon(x_k)}w_\epsilon^g(x_k)\psi_\theta\left(g^{-1}(x_1),g^{-1}(x_2),g^{-1}(x_3),\ldots,g^{-1}(x_n)\right)\\
&=-\psi_{\theta;\epsilon}^{(\mathrm{SC};k)}(\widetilde{x}_1,\ldots,\widetilde{x}_n).
\end{aligned} ψ θ ; ϵ ( SC ; k ) ( x 2 , x 1 , x 3 , … , x n ) = g ∈ G ϵ ( x k ) ∑ w ϵ g ( x k ) ψ θ ( g − 1 ( x 2 ) , g − 1 ( x 1 ) , g − 1 ( x 3 ) , … , g − 1 ( x n ) ) = − g ∈ G ϵ ( x k ) ∑ w ϵ g ( x k ) ψ θ ( g − 1 ( x 1 ) , g − 1 ( x 2 ) , g − 1 ( x 3 ) , … , g − 1 ( x n ) ) = − ψ θ ; ϵ ( SC ; k ) ( x 1 , … , x n ) .
For the case k = 1 , 2 k=1,2 k = 1 , 2 , we can apply a similar calculation while noting that the weights remain unchanged, and obtain
ψ θ ; ϵ ( S C ; 1 ) ( x ~ 2 , x ~ 1 , x ~ 3 , … , x ~ n ) = − ψ θ ; ϵ ( S C ; 2 ) ( x ~ 1 , x ~ 2 , x ~ 3 , … , x ~ n ) , \psi_{\theta;\epsilon}^{(\mathrm{SC};1)}(\widetilde{x}_2,\widetilde{x}_1,\widetilde{x}_3,\ldots,\widetilde{x}_n)=-\psi_{\theta;\epsilon}^{(\mathrm{SC};2)}(\widetilde{x}_1,\widetilde{x}_2,\widetilde{x}_3,\ldots,\widetilde{x}_n), ψ θ ; ϵ ( SC ; 1 ) ( x 2 , x 1 , x 3 , … , x n ) = − ψ θ ; ϵ ( SC ; 2 ) ( x 1 , x 2 , x 3 , … , x n ) ,
ψ θ ( S C ; 2 ) ( x ~ 2 , x ~ 1 , x ~ 3 , … , x ~ n ) = − ψ θ ; ϵ ( S C ; 1 ) ( x ~ 1 , x ~ 2 , x ~ 3 , … , x ~ n ) . \psi_{\theta}^{(\mathrm{SC};2)}(\widetilde{x}_2,\widetilde{x}_1,\widetilde{x}_3,\ldots,\widetilde{x}_n)=-\psi_{\theta;\epsilon}^{(\mathrm{SC};1)}(\widetilde{x}_1,\widetilde{x}_2,\widetilde{x}_3,\ldots,\widetilde{x}_n). ψ θ ( SC ; 2 ) ( x 2 , x 1 , x 3 , … , x n ) = − ψ θ ; ϵ ( SC ; 1 ) ( x 1 , x 2 , x 3 , … , x n ) .
This implies that
ψ θ ; ϵ ( S C ) ( x ~ π ( 1 ) , … , x ~ π ( n ) ) = − 1 n ∑ k = 1 n ψ θ ; ϵ ( S C ; k ) ( x ~ 1 , x ~ 2 , x ~ 3 , … , x ~ n ) = sgn ( π ) ψ θ ; ϵ ( S C ) ( x ~ 1 , … , x ~ n ) . \psi_{\theta;\epsilon}^{(\mathrm{SC})}(\widetilde{x}_{\pi(1)},\ldots,\widetilde{x}_{\pi(n)})=-\frac{1}{n}\sum_{k=1}^{n}\psi_{\theta;\epsilon}^{(\mathrm{SC};k)}(\widetilde{x}_1,\widetilde{x}_2,\widetilde{x}_3,\ldots,\widetilde{x}_n)=\operatorname{sgn}(\pi)\psi_{\theta;\epsilon}^{(\mathrm{SC})}(\widetilde{x}_1,\ldots,\widetilde{x}_n). ψ θ ; ϵ ( SC ) ( x π ( 1 ) , … , x π ( n ) ) = − n 1 k = 1 ∑ n ψ θ ; ϵ ( SC ; k ) ( x 1 , x 2 , x 3 , … , x n ) = sgn ( π ) ψ θ ; ϵ ( SC ) ( x 1 , … , x n ) .
Since the choice of indices 1 and 2 are arbitrary, the above in fact holds for all transpositions π \pi π , which implies
ψ θ ; ϵ ( S C ) ( x ~ τ ( 1 ) , … , x ~ τ ( n ) ) = sgn ( τ ) ψ θ ; ϵ ( S C ) ( x ~ 1 , … , x ~ n ) \psi_{\theta;\epsilon}^{(\mathrm{SC})}(\widetilde{x}_{\tau(1)},\ldots,\widetilde{x}_{\tau(n)})=\operatorname{sgn}(\tau)\psi_{\theta;\epsilon}^{(\mathrm{SC})}(\widetilde{x}_1,\ldots,\widetilde{x}_n) ψ θ ; ϵ ( SC ) ( x τ ( 1 ) , … , x τ ( n ) ) = sgn ( τ ) ψ θ ; ϵ ( SC ) ( x 1 , … , x n )
for all permutations τ \tau τ of the n n n electrons. This proves (ii).
To prove (iii), it suffices to show that for every k ≤ n k\leq n k ≤ n , the function ψ θ ; ϵ ( S C ; k ) \psi_{\theta;\epsilon}^{(\mathrm{SC};k)} ψ θ ; ϵ ( SC ; k ) defined above is p p p -times continuously differentiable at x x x , i.e. ∇ p ψ θ ; ϵ ( S C ; k ) \nabla^p\psi_{\theta;\epsilon}^{(\mathrm{SC};k)} ∇ p ψ θ ; ϵ ( SC ; k ) exists and is continuous at x x x . Again it suffices to show this for the case k = 1 k=1 k = 1 . Let x ~ : = ( x ~ 1 , … , x ~ n ) \widetilde{x}:=(\widetilde{x}_1,\ldots,\widetilde{x}_n) x := ( x 1 , … , x n ) be a vector in a sufficiently small neighborhood of a fixed x : = ( x 1 , … , x n ) x:=(x_1,\ldots,x_n) x := ( x 1 , … , x n ) , and recall that
G ϵ ( x ~ 1 ) = { g ∈ G ∣ d ( x ~ 1 , g ( Π 0 ) ) ≤ ϵ } . G_\epsilon(\widetilde{x}_1)=\{g\in\mathcal{G}\mid d(\widetilde{x}_1,g(\Pi_0))\leq\epsilon\}. G ϵ ( x 1 ) = { g ∈ G ∣ d ( x 1 , g ( Π 0 )) ≤ ϵ } .
For x ~ \widetilde{x} x in a sufficiently small neighborhood of x x x , G ϵ ( x ~ 1 ) G_\epsilon(\widetilde{x}_1) G ϵ ( x 1 ) takes value in { G l } 0 ≤ l ≤ M \{G_l\}_{0\leq l\leq M} { G l } 0 ≤ l ≤ M , where
G l : = G ϵ ( x 1 ) ∖ { g 1 ϵ , … , g l ϵ } , G_l:=G_\epsilon(x_1)\setminus\{g_1^\epsilon,\ldots,g_l^\epsilon\}, G l := G ϵ ( x 1 ) ∖ { g 1 ϵ , … , g l ϵ } ,
and g 1 ϵ , … , g M ϵ ∈ G g_1^\epsilon,\ldots,g_M^\epsilon\in\mathcal{G} g 1 ϵ , … , g M ϵ ∈ G is an enumeration of all group elements such that
d ( x 1 , g l ϵ ( Π 0 ) ) = ϵ for 0 ≤ l ≤ M . d(x_1,g_l^\epsilon(\Pi_0))=\epsilon\qquad\text{for }0\leq l\leq M. d ( x 1 , g l ϵ ( Π 0 )) = ϵ for 0 ≤ l ≤ M .
Therefore ψ θ ; ϵ ( S C ; 1 ) ( x ~ ) \psi_{\theta;\epsilon}^{(\mathrm{SC};1)}(\widetilde{x}) ψ θ ; ϵ ( SC ; 1 ) ( x ) takes values in { ψ G l ( x ~ ) } 0 ≤ l ≤ M \{\psi_{G_l}(\widetilde{x})\}_{0\leq l\leq M} { ψ G l ( x ) } 0 ≤ l ≤ M , where
ψ G l ( x ~ ) : = ∑ g ∈ G l λ ϵ ( ϵ − d ( x ~ 1 , g ( Π 0 ) ) ϵ ) ∑ g ′ ∈ G l λ ϵ ( ϵ − d ( x ~ 1 , g ′ ( Π 0 ) ) ϵ ) ψ θ ( g − 1 ( x ~ 1 ) , … , g − 1 ( x ~ n ) ) . \psi_{G_l}(\widetilde{x}):=\sum_{g\in G_l}\frac{\lambda_\epsilon\left(\frac{\epsilon-d(\widetilde{x}_1,g(\Pi_0))}{\epsilon}\right)}{\sum_{g'\in G_l}\lambda_\epsilon\left(\frac{\epsilon-d(\widetilde{x}_1,g'(\Pi_0))}{\epsilon}\right)}\psi_\theta\left(g^{-1}(\widetilde{x}_1),\ldots,g^{-1}(\widetilde{x}_n)\right). ψ G l ( x ) := g ∈ G l ∑ ∑ g ′ ∈ G l λ ϵ ( ϵ ϵ − d ( x 1 , g ′ ( Π 0 )) ) λ ϵ ( ϵ ϵ − d ( x 1 , g ( Π 0 )) ) ψ θ ( g − 1 ( x 1 ) , … , g − 1 ( x n ) ) .
Notice that at x ~ = x \widetilde{x}=x x = x , by the definition of g l ϵ g_l^\epsilon g l ϵ , we have
ψ G l ( x ) = ∑ g ∈ G l λ ϵ ( d ( x 1 , g ( Π 0 ) ) ϵ ) ∑ g ′ ∈ G l λ ϵ ( d ( x 1 , g ′ ( Π 0 ) ) ϵ ) × ψ θ ( g − 1 ( x 1 ) , … , g − 1 ( x n ) ) = ( a ) ∑ g ∈ G ϵ ( x 1 ) λ ϵ ( d ( x 1 , g ( Π 0 ) ) ϵ ) ∑ g ′ ∈ G ϵ ( x 1 ) λ ϵ ( d ( x 1 , g ′ ( Π 0 ) ) ϵ ) × ψ θ ( g − 1 ( x 1 ) , … , g − 1 ( x n ) ) = ψ θ ; ϵ ( S C ; 1 ) ( x ) . \begin{aligned}
\psi_{G_l}(x)
&=\sum_{g\in G_l}\frac{\lambda_\epsilon\left(\frac{d(x_1,g(\Pi_0))}{\epsilon}\right)}{\sum_{g'\in G_l}\lambda_\epsilon\left(\frac{d(x_1,g'(\Pi_0))}{\epsilon}\right)}\times\psi_\theta\left(g^{-1}(x_1),\ldots,g^{-1}(x_n)\right)\\
&\overset{(a)}{=}\sum_{g\in G_\epsilon(x_1)}\frac{\lambda_\epsilon\left(\frac{d(x_1,g(\Pi_0))}{\epsilon}\right)}{\sum_{g'\in G_\epsilon(x_1)}\lambda_\epsilon\left(\frac{d(x_1,g'(\Pi_0))}{\epsilon}\right)}\times\psi_\theta\left(g^{-1}(x_1),\ldots,g^{-1}(x_n)\right)=\psi_{\theta;\epsilon}^{(\mathrm{SC};1)}(x).
\end{aligned} ψ G l ( x ) = g ∈ G l ∑ ∑ g ′ ∈ G l λ ϵ ( ϵ d ( x 1 , g ′ ( Π 0 )) ) λ ϵ ( ϵ d ( x 1 , g ( Π 0 )) ) × ψ θ ( g − 1 ( x 1 ) , … , g − 1 ( x n ) ) = ( a ) g ∈ G ϵ ( x 1 ) ∑ ∑ g ′ ∈ G ϵ ( x 1 ) λ ϵ ( ϵ d ( x 1 , g ′ ( Π 0 )) ) λ ϵ ( ϵ d ( x 1 , g ( Π 0 )) ) × ψ θ ( g − 1 ( x 1 ) , … , g − 1 ( x n ) ) = ψ θ ; ϵ ( SC ; 1 ) ( x ) .
Since the above argument works with x x x replaced by x ~ \tilde{x} x ~ , we also have
ψ θ ; ϵ ( S C ; 1 ) ( x ~ ) = ∑ g ∈ G ~ l λ ϵ ( d ( x 1 , g ( Π 0 ) ) ) ∑ g ′ ∈ G ~ l λ ϵ ( d ( x 1 , g ′ ( Π 0 ) ) ) × ψ θ ( g − 1 ( x 1 ) , … , g − 1 ( x n ) ) (12) \psi_{\theta;\epsilon}^{(\mathrm{SC};1)}(\tilde{x})=\sum_{g\in\widetilde{\mathcal{G}}_l}\frac{\lambda_{\epsilon}(d(x_1,g(\Pi_0)))}{\sum_{g'\in\widetilde{\mathcal{G}}_l}\lambda_{\epsilon}(d(x_1,g'(\Pi_0)))}\times\psi_{\theta}(g^{-1}(x_1),\ldots,g^{-1}(x_n)) \tag*{(12)} ψ θ ; ϵ ( SC ; 1 ) ( x ~ ) = g ∈ G l ∑ ∑ g ′ ∈ G l λ ϵ ( d ( x 1 , g ′ ( Π 0 ))) λ ϵ ( d ( x 1 , g ( Π 0 ))) × ψ θ ( g − 1 ( x 1 ) , … , g − 1 ( x n )) (12)
for 0 ≤ l ≤ M ~ 0\le l\le\widetilde{M} 0 ≤ l ≤ M , where we have defined
G ~ l : = G ϵ ( x ~ 1 ) ∖ { g ~ 1 ϵ , … , g ~ l ϵ } , \widetilde{\mathcal{G}}_l:=\mathcal{G}_{\epsilon}(\tilde{x}_1)\setminus\{\tilde{g}_1^{\epsilon},\ldots,\tilde{g}_l^{\epsilon}\}, G l := G ϵ ( x ~ 1 ) ∖ { g ~ 1 ϵ , … , g ~ l ϵ } ,
and g ~ 1 ϵ , … , g ~ M ~ ϵ ∈ G \tilde{g}_1^{\epsilon},\ldots,\tilde{g}_{\widetilde{M}}^{\epsilon}\in\mathcal{G} g ~ 1 ϵ , … , g ~ M ϵ ∈ G is an enumeration of all group elements such that
d ( x ~ 1 , g ~ l ϵ ( Π 0 ) ) = ϵ for 0 ≤ l ≤ M . d(\tilde{x}_1,\tilde{g}_l^{\epsilon}(\Pi_0))=\epsilon\qquad\text{for }0\le l\le M. d ( x ~ 1 , g ~ l ϵ ( Π 0 )) = ϵ for 0 ≤ l ≤ M .
Notice that for x ~ \tilde{x} x ~ in a sufficiently small neighborhood of x x x , we have { G ~ l } l ≤ M ~ = { G l } l ≤ M \{\widetilde{\mathcal{G}}_l\}_{l\le\widetilde{M}}=\{\mathcal{G}_l\}_{l\le M} { G l } l ≤ M = { G l } l ≤ M , in which case (12) implies
ψ θ ; ϵ ( S C ; 1 ) ( x ~ ) = ψ G l ( x ~ ) for all 0 ≤ l ≤ M . \psi_{\theta;\epsilon}^{(\mathrm{SC};1)}(\tilde{x})=\psi_{\mathcal{G}_l}(\tilde{x})\qquad\text{for all }0\le l\le M. ψ θ ; ϵ ( SC ; 1 ) ( x ~ ) = ψ G l ( x ~ ) for all 0 ≤ l ≤ M .
In other words, we have shown that in a sufficiently small neighborhood of x x x , F 1 F_1 F 1 equals ψ G l \psi_{\mathcal{G}_l} ψ G l for all 1 ≤ l ≤ M 1\le l\le M 1 ≤ l ≤ M . Recall that λ \lambda λ and d ( ⋅ , g ( Π 0 ) ) d(\mathord{\cdot},g(\Pi_0)) d ( ⋅ , g ( Π 0 )) are p p p -times continuously differentiable for all g ∈ G g\in\mathcal{G} g ∈ G , and ψ θ \psi_{\theta} ψ θ is p p p -times continuously differentiable at g ( x ) g(x) g ( x ) for all g ∈ G g\in\mathcal{G} g ∈ G by assumption. This implies that ψ G l \psi_{\mathcal{G}_l} ψ G l is also p p p -times continuously differentiable at x x x and so is ψ θ ; ϵ ( S C ; 1 ) \psi_{\theta;\epsilon}^{(\mathrm{SC};1)} ψ θ ; ϵ ( SC ; 1 ) . Moreover, for 0 ≤ q ≤ p 0\le q\le p 0 ≤ q ≤ p , the derivative can be computed as
∇ q ψ θ ; ϵ ( S C ; 1 ) ( x ) = ∇ q ψ G l ( x ) = ∑ g ∈ G ϵ ( x 1 ) ∇ q ψ θ , g , x 1 ( S C ; k ) ( x ) , \nabla^q\psi_{\theta;\epsilon}^{(\mathrm{SC};1)}(x)=\nabla^q\psi_{\mathcal{G}_l}(x)=\sum_{g\in\mathcal{G}_{\epsilon}(x_1)}\nabla^q\psi_{\theta,g,x_1}^{(\mathrm{SC};k)}(x), ∇ q ψ θ ; ϵ ( SC ; 1 ) ( x ) = ∇ q ψ G l ( x ) = g ∈ G ϵ ( x 1 ) ∑ ∇ q ψ θ , g , x 1 ( SC ; k ) ( x ) ,
where we recall that for 1 ≤ k ≤ n 1\le k\le n 1 ≤ k ≤ n , g ∈ G g\in\mathcal{G} g ∈ G and x , y 1 , … , y n ∈ R 3 x,y_1,\ldots,y_n\in\mathbb{R}^3 x , y 1 , … , y n ∈ R 3 ,
ψ θ , ϵ ; g , x ( S C ; k ) ( y 1 , … , y n ) : = λ ϵ ( d ( y k , g ( Π 0 ) ) ) ∑ g ′ ∈ G ϵ ( x ) λ ϵ ( d ( y k , g ′ ( Π 0 ) ) ) ψ θ ( g − 1 ( y 1 ) , … , g − 1 ( y n ) ) . \psi_{\theta,\epsilon;g,x}^{(\mathrm{SC};k)}(y_1,\ldots,y_n):=\frac{\lambda_{\epsilon}(d(y_k,g(\Pi_0)))}{\sum_{g'\in\mathcal{G}_{\epsilon}(x)}\lambda_{\epsilon}(d(y_k,g'(\Pi_0)))}\psi_{\theta}(g^{-1}(y_1),\ldots,g^{-1}(y_n)). ψ θ , ϵ ; g , x ( SC ; k ) ( y 1 , … , y n ) := ∑ g ′ ∈ G ϵ ( x ) λ ϵ ( d ( y k , g ′ ( Π 0 ))) λ ϵ ( d ( y k , g ( Π 0 ))) ψ θ ( g − 1 ( y 1 ) , … , g − 1 ( y n )) .
The same argument applies to all ψ θ ; ϵ ( S C ; k ) \psi_{\theta;\epsilon}^{(\mathrm{SC};k)} ψ θ ; ϵ ( SC ; k ) ’s with k ≤ n k\le n k ≤ n and therefore for 0 ≤ q ≤ p 0\le q\le p 0 ≤ q ≤ p ,
∇ q ψ θ ; ϵ ( S C ) ( x 1 , … , x n ) = 1 n ∑ k = 1 n ∇ q ψ θ ; ϵ ( S C ; k ) ( x 1 , … , x n ) = 1 n ∑ k = 1 n ∑ g ∈ G ϵ ( x k ) ∇ q ψ θ , ϵ ; g , x k ( S C ; k ) ( x ) , \nabla^q\psi_{\theta;\epsilon}^{(\mathrm{SC})}(x_1,\ldots,x_n)=\frac{1}{n}\sum_{k=1}^{n}\nabla^q\psi_{\theta;\epsilon}^{(\mathrm{SC};k)}(x_1,\ldots,x_n)=\frac{1}{n}\sum_{k=1}^{n}\sum_{g\in\mathcal{G}_{\epsilon}(x_k)}\nabla^q\psi_{\theta,\epsilon;g,x_k}^{(\mathrm{SC};k)}(x), ∇ q ψ θ ; ϵ ( SC ) ( x 1 , … , x n ) = n 1 k = 1 ∑ n ∇ q ψ θ ; ϵ ( SC ; k ) ( x 1 , … , x n ) = n 1 k = 1 ∑ n g ∈ G ϵ ( x k ) ∑ ∇ q ψ θ , ϵ ; g , x k ( SC ; k ) ( x ) ,
which proves (iii).
Proof of Lemma E.2
Since λ ϵ ( 0 ) = 1 \lambda_{\epsilon}(0)=1 λ ϵ ( 0 ) = 1 , λ ϵ ( ϵ ) = 0 \lambda_{\epsilon}(\epsilon)=0 λ ϵ ( ϵ ) = 0 and λ ϵ \lambda_{\epsilon} λ ϵ is continuously differentiable, by the mean value theorem, there is y 1 ∈ ( 0 , ϵ ) y_1\in(0,\epsilon) y 1 ∈ ( 0 , ϵ ) such that
∂ λ ϵ ( y 1 ) = ( 1 − 0 ) / ϵ = ϵ − 1 . \partial\lambda_{\epsilon}(y_1)=(1-0)/\epsilon=\epsilon^{-1}. ∂ λ ϵ ( y 1 ) = ( 1 − 0 ) / ϵ = ϵ − 1 .
Meanwhile, since λ ϵ ( w ) = 1 \lambda_{\epsilon}(w)=1 λ ϵ ( w ) = 1 for all w ≤ 0 w\le0 w ≤ 0 , ∂ λ ϵ ( w ) = 0 \partial\lambda_{\epsilon}(w)=0 ∂ λ ϵ ( w ) = 0 for all w < 0 w<0 w < 0 , and since ∂ λ ϵ \partial\lambda_{\epsilon} ∂ λ ϵ is continuous, ∂ λ ϵ ( 0 ) = 0 \partial\lambda_{\epsilon}(0)=0 ∂ λ ϵ ( 0 ) = 0 . By the twice continuous differentiability of λ ϵ \lambda_{\epsilon} λ ϵ and the mean value theorem again, there exists y 1 ′ ∈ ( 0 , y 1 ) ⊂ ( 0 , ϵ ) y'_1\in(0,y_1)\subset(0,\epsilon) y 1 ′ ∈ ( 0 , y 1 ) ⊂ ( 0 , ϵ ) such that
∂ 2 λ ϵ ( y 1 ′ ) = ( ϵ − 1 − 0 ) / y 1 = 1 ϵ y 1 ′ ≥ 1 ϵ 2 . \partial^2\lambda_{\epsilon}(y'_1)=(\epsilon^{-1}-0)/y_1=\frac{1}{\epsilon y'_1}\ge\frac{1}{\epsilon^2}. ∂ 2 λ ϵ ( y 1 ′ ) = ( ϵ − 1 − 0 ) / y 1 = ϵ y 1 ′ 1 ≥ ϵ 2 1 .
Since ∂ 2 λ ϵ ( w ) = 0 \partial^2\lambda_{\epsilon}(w)=0 ∂ 2 λ ϵ ( w ) = 0 for all w < 0 w<0 w < 0 and ∂ 2 λ ϵ \partial^2\lambda_{\epsilon} ∂ 2 λ ϵ is continuous, ∂ 2 λ ϵ ( 0 ) = 0 \partial^2\lambda_{\epsilon}(0)=0 ∂ 2 λ ϵ ( 0 ) = 0 . As ∂ 2 λ ϵ ( y 1 ′ ) ≥ 1 ϵ 2 \partial^2\lambda_{\epsilon}(y'_1)\ge\frac{1}{\epsilon^2} ∂ 2 λ ϵ ( y 1 ′ ) ≥ ϵ 2 1 , by the intermediate value theorem, there exists some y 2 ∈ [ 0 , y 1 ] ⊂ [ 0 , ϵ ] y_2\in[0,y_1]\subset[0,\epsilon] y 2 ∈ [ 0 , y 1 ] ⊂ [ 0 , ϵ ] such that ∂ 2 λ ϵ ( y 2 ) = ϵ − 2 \partial^2\lambda_{\epsilon}(y_2)=\epsilon^{-2} ∂ 2 λ ϵ ( y 2 ) = ϵ − 2 .
Discussion on the case where per-iter sampling cost is greater than per-iter gradient cost
An anonymous reviewer has pointed out regimes where, unlike § 3 , C s a m p C_{\mathrm{samp}} C samp may take larger values than C g r a d C_{\mathrm{grad}} C grad . We briefly discuss this point. There are two cases where C s a m p ≫ C g r a d C_{\mathrm{samp}}\gg C_{\mathrm{grad}} C samp ≫ C grad may occur: (i) There exists significantly accelerated implementations of gradient computation (Li et al., 2024) such that it becomes more efficient than sampling; (ii) One typically needs to scale the number of sampling steps and neural network size both with the physical system size (von Glehn et al., 2023; Li et al., 2022), and the increase in per-step sampling cost could outweigh the increase in per-step gradient cost. In these regimes, our theoretical claims do hold, but come with some additional nuances:
DA instability. When C s a m p ≪ C g r a d C_{\mathrm{samp}} \ll C_{\mathrm{grad}} C samp ≪ C grad , our DA batch size in § 4.1 was chosen as k × N ′ = N k \times N^{\prime} = N k × N ′ = N , where N ′ : = N / k N^{\prime} := N/k N ′ := N / k is the number of i.i.d. drawn samples and k k k is the number of augmentations. The reduction in the number of i.i.d. drawn samples was key to the destabilization effect. If instead C s a m p ≫ C g r a d C_{\mathrm{samp}} \gg C_{\mathrm{grad}} C samp ≫ C grad , one may increase N ′ N^{\prime} N ′ , the number of i.i.d. samples drawn, though the same conclusion holds as long as N ′ < N N^{\prime} < N N ′ < N . Meanwhile if N ′ = N N^{\prime} = N N ′ = N , DA does not destabilize, but the k k k augmentations always increase the computational cost. The statistical-computational tradeoff now depends on how large N ′ N^{\prime} N ′ is and therefore has a more intricate dependence on the computational cost ratio C s a m p / C g r a d C_{\mathrm{samp}}/C_{\mathrm{grad}} C samp / C grad .
GA instability. The destabilization effect still occurs, as group-averaging always increases both sampling and gradient evaluation cost, which necessitates a reduction in the number of i.i.d. samples drawn.
Empirically, we do not observe C s a m p ≫ C g r a d C_{\mathrm{samp}} \gg C_{\mathrm{grad}} C samp ≫ C grad for the DeepSolid experiments we have performed.