betaFeedback

WorksPhysics and Astronomy

Self-attention is a Dyson equation for many-body machine learning

Abstract

Why does symmetric sum-pooling (DeepSets) predict ground-state potentials to sub-meV accuracy, yet fail for correlated wavefunctions and excited states? Sum-pooling is a mean-field (Hartree) model: its O(1/N)O(1/N) off-diagonal Jacobian suffices for nearsighted ground-state energies, but erases nonlocal two-body correlations. We show that self-attention is a real-space Dyson iteration whose Boltzmann (softmax) kernel acts as a two-body self-energy. Because physical Hamiltonians act pairwise, Löwdin's 2-RDM theorem and the Galitskii–Migdal relation guarantee that iterating a two-body propagator captures the full energetics of correlated matter. When fitting a fractal backflow determinant, DeepSets blurs the nodal surface onto a mean-field envelope, whereas self-attention recovers the fractal attractor. Seen through this lens, the unreasonable effectiveness of attention is not an accident; in a universe built from pairwise interactions, two-body attention is all you need.

Introduction

Neural network wavefunction Ansätze often feed electron-pair separations directly into the network [1], a brute-force O(N2)O(N^2) approach that bloats the architecture, while raw single-particle coordinates lose translational symmetry unless embedded periodically [2]. Symmetric sum-pooling (often via equivariant DeepSets) scalarises interactions between particles into one-body terms. By the universal approximation theorem, this is formally sufficient to encode everything about the system, but with finite capacity it is not tractable to extract many-body correlations from a single pooled vector. Yet in machine learning interatomic potentials for a system size of NN atoms, finite-cutoff equivariant message passing followed by DeepSets sum-pooling over local atomic energies EiE_i (E=∑i=1NEiE = \sum_{i=1}^{N} E_i) achieves sub-meV [3] accuracy; whereas PsiFormer [4] replaces the pair features in FermiNet [5] with more simple multi-head self-attention over single-particle features, and achieves superior electronic correlation in molecular dimers.

Attention [6] now underpins almost all areas of machine learning, improving performance while simplifying complex architectures. Conventionally framed in computer-science database terminology, the mechanism pairs queries with keys and normalises their dot products via a softmax to perform a “fuzzy” lookup over values. Following the example of atom-density representations written in Dirac notation [7], we recast this database abstraction into the operator language of many-body physics. At its core, self-attention applies learned linear projection matrices WQ,K,VW_{Q,K,V} to map single-particle states ∣hi⟩|h_i\rangle into queries ∣qi⟩=WQ∣hi⟩|q_i\rangle= W_Q|h_i\rangle, keys ∣kj⟩=WK∣hj⟩|k_j\rangle= W_K|h_j\rangle and values ∣vj⟩=WV∣hj⟩|v_j\rangle= W_V|h_j\rangle, before normalising their query–key inner products with a Boltzmann (softmax) factor.

This reduces to a state-dependent two-body propagator ∑j=1NZi−1exp⁡(β⟨qi∣kj⟩)∣vj⟩\sum_{j=1}^{N} Z_i^{-1}\exp(\beta\langle q_i|k_j\rangle)|v_j\rangle, normalised by the local partition function Zi=∑n=1Nexp⁡(β⟨qi∣kn⟩)Z_i = \sum_{n=1}^{N}\exp(\beta\langle q_i|k_n\rangle). In this paper we show that DeepSets (scalar) pooling is a one-body mean-field (Hartree) description, explaining why it accurately captures additive, locally nearsighted quantities such as ground-state energies and forces. Remarkably, self-attention maps directly onto Dyson resummation of the nonlocal two-body propagator. Because physical Hamiltonians are governed by pairwise interactions and exact energetics reduce to two-body correlators [8], iterative self-attention is formally sufficient to describe the one- and two-body observables (and hence exact energetics) of pairwise (physical, Coulomb) Hamiltonians.

Consequently, machine learning architectures directly targeting nonlocal quantum mechanical observables require self-attention rather than scalar sum-pooling. This equivalence to foundational completeness theorems of quantum matter explains why, across so many physical systems, “attention is all you need” [6]. Here we make this connection between machine-learning architectures and many-body theory explicit, bridging the disparate notation and terminology of the two communities.

Theory

Consider a system of NN quantum particles or atoms at collective coordinates R=(r1,…,rN)\mathbf{R} = (\mathbf{r}_1,\ldots,\mathbf{r}_N) with ri∈RD\mathbf{r}_i \in\mathbb{R}^{D} in DD spatial dimensions, represented at layer ll (of LL) of a machine-learning model by the set of permutation-equivariant single-particle states h(l)={∣hi(l)⟩}i=1Nh^{(l)} = \{|h_i^{(l)}\rangle\}_{i=1}^{N}. Both DeepSets and self-attention update operations can be written as a discrete two-body Dyson equation,

∣hi(l+1)⟩=∣hi(l)⟩+∑j=1NΣij[h(l)]∣vj(l)⟩,(1)|h_i^{(l+1)}\rangle= |h_i^{(l)}\rangle+ \sum_{j=1}^{N}\Sigma_{ij}[h^{(l)}]|v_j^{(l)}\rangle, \tag*{(1)}

where ∣vj(l)⟩\left|v_j^{(l)}\right\rangle is the single-particle value (message) sent by particle jj and Σij[h(l)]\Sigma_{ij}\left[h^{(l)}\right] is the scalar effective two-body self-energy kernel coupling particle ii to particle jj as a function of the effective layer state h(l)h^{(l)}.

Each layer in the machine-learning network performs one step of Dyson iteration; the total depth LL sets the order of resummation.

Scalar DeepSets is the mean-field (Hartree) approximation

In DeepSets[9], interactions are aggregated via an unweighted symmetric sum (sum pooling) over all particles,

∣hi(l+1)⟩=ρ(∣hi(l)⟩,∣vˉ(l)⟩),∣vˉ(l)⟩=1N∑j=1Nϕ(∣hj(l)⟩),(2)\left|h_i^{(l+1)}\right\rangle= \rho\left(\left|h_i^{(l)}\right\rangle,\left|\bar{v}^{(l)}\right\rangle\right), \qquad\left|\bar{v}^{(l)}\right\rangle= \frac{1}{N}\sum_{j=1}^{N}\phi\left(\left|h_j^{(l)}\right\rangle\right), \tag*{(2)}

where ϕ\phi and ρ\rho are learned single-particle embedding and update functions (multilayer perceptrons), and ∣vˉ(l)⟩\left|\bar{v}^{(l)}\right\rangle is the globally pooled value state. Setting individual particle values ∣vj(l)⟩=ϕ(∣hj(l)⟩)\left|v_j^{(l)}\right\rangle=\phi\left(\left|h_j^{(l)}\right\rangle\right) and comparing directly with Eqn. (2) reveals the physical loss: the pooled state ∣vˉ(l)⟩\left|\bar{v}^{(l)}\right\rangle carries no target index ii. Every particle experiences an identical background potential, corresponding to a uniform mean-field (Hartree) kernel Σˉij=1/N\bar{\Sigma}_{ij}=1/N. Consequently, the off-diagonal Jacobian measuring how particle ii’s dressed state responds to particle jj’s position scales as

Jˉij=∂∂rjhi(l+1)=1N∂∂vˉ(l)ρi∂∂rjvj(l)∼O(1N)(i≠j),(3)\bar{J}_{ij}=\frac{\partial}{\partial r_j}h_i^{(l+1)}=\frac{1}{N}\frac{\partial}{\partial\bar{v}^{(l)}}\rho_i\frac{\partial}{\partial r_j}v_j^{(l)}\sim\mathcal{O}\left(\frac{1}{N}\right)\quad(i\ne j), \tag*{(3)}

where ρi=ρ(∣hi(l)⟩,∣vˉ(l)⟩)\rho_i=\rho\left(\left|h_i^{(l)}\right\rangle,\left|\bar{v}^{(l)}\right\rangle\right). When these pooled states are used downstream, specific pairwise correlations are diluted by 1/N1/N; worse, because Eqn. (3) factorises into an ii-only factor ∂∂vˉ(l)ρi\frac{\partial}{\partial\bar{v}^{(l)}}\rho_i times a jj-only factor ∂∂rjvj(l)\frac{\partial}{\partial r_j}v_j^{(l)} independent of pairwise separations rij=∣ri−rj∣r_{ij}=\lvert r_i-r_j\rvert, any steep local gradient at rjr_j broadcasts spurious crosstalk to every distant spectator ii.

Physically this pathology means that a close approach between two particles jogs all the other particles in the system, as the large local interaction has to be modelled by the mean-field. As with any mean-field theory, this cannot support strong correlation.

Self-attention is a Dyson equation

Self-attention[6] replaces the uniform Σˉij=1/N\bar{\Sigma}_{ij}=1/N weight in Eqn. (2) with a state-dependent Boltzmann kernel. Written in the motivating form of Dirac bra–ket notation, learned linear operators WQ,WK∈Rdk×dmodelW_Q,W_K\in\mathbb{R}^{d_k\times d_{\mathrm{model}}} and WV∈Rdmodel×dmodelW_V\in\mathbb{R}^{d_{\mathrm{model}}\times d_{\mathrm{model}}} project each dmodeld_{\mathrm{model}}-dimensional

state ∣hi(l)⟩\left|h_i^{(l)}\right\rangle into query ∣qi(l)⟩=WQ∣hi(l)⟩\left|q_i^{(l)}\right\rangle=W_Q\left|h_i^{(l)}\right\rangle, key ∣kj(l)⟩=WK∣hj(l)⟩\left|k_j^{(l)}\right\rangle=W_K\left|h_j^{(l)}\right\rangle (both of feature dimension dkd_k), and value ∣vj(l)⟩=WV∣hj(l)⟩\left|v_j^{(l)}\right\rangle=W_V\left|h_j^{(l)}\right\rangle (model dimension dmodeld_{\mathrm{model}}) states. The update operation is then

∣hi(l+1)⟩=∣hi(l)⟩+∑j=1Nexp⁡(β⟨qi(l)|kj(l)⟩)Zi(l)∣vj(l)⟩,(4)\left|h_i^{(l+1)}\right\rangle=\left|h_i^{(l)}\right\rangle+\sum_{j=1}^{N}\frac{\exp\left(\beta\left\langle q_i^{(l)}\middle|k_j^{(l)}\right\rangle\right)}{Z_i^{(l)}}\left|v_j^{(l)}\right\rangle, \tag*{(4)}

where Zi(l)=∑n=1Nexp⁡(β⟨qi(l)|kn(l)⟩)Z_i^{(l)}=\sum_{n=1}^{N}\exp\left(\beta\left\langle q_i^{(l)}\middle|k_n^{(l)}\right\rangle\right) is the local canonical partition function at inverse effective temperature β=dk−1/2\beta=d_k^{-1/2}. This immediately generalises to HH parallel interaction channels in multi-head attention.

In the high-temperature limit β→0\beta\to0, ΣijAttn[h(l)]→1/N\Sigma_{ij}^{\mathrm{Attn}}\left[h^{(l)}\right]\to1/N and Eqn. (4) collapses to the DeepSets mean-field sum of Eqn. (2); whereas β→∞\beta\to\infty freezes into a nearest-neighbour graph. The standard scaling β=dk−1/2\beta=d_k^{-1/2} keeps β⟨qi(l)|kj(l)⟩∼O(1)\beta\left\langle q_i^{(l)}\middle|k_j^{(l)}\right\rangle\sim\mathcal{O}(1) in the correlated regime between mean-field melting and glassy freezing[11].

Here the inner product ⟨qi(l)|kj(l)⟩=⟨hi(l)∣WQ†WK∣hj(l)⟩\left\langle q_i^{(l)}\middle|k_j^{(l)}\right\rangle=\left\langle h_i^{(l)}\right|W_Q^\dagger W_K\left|h_j^{(l)}\right\rangle (where WQ†W_Q^\dagger denotes the adjoint of WQW_Q) acts as a learned, generally non-symmetric (WQ≠WKW_Q\ne W_K) two-body interaction vertex that dynamically reconstructs translationally invariant pair kernels (such as Gaussian radial basis or periodic plane waves) from single-particle states without storing explicit ri−rj\mathbf{r}_i-\mathbf{r}_j pair vectors.2

Crucially, for strongly correlated pairs (i,j)(i,j), the Boltzmann factor concentrates O(1)O(1) weight on partner jj while silencing distant spectators (this is attention, ΣikAttn→0\Sigma^{\mathrm{Attn}}_{ik} \to0 for k≠jk \ne j), preserving an O(1)O(1) off-diagonal Jacobian JijAttn=∂∂rjhi(l+1)∝ΣijAttn[h(l)]J^{\mathrm{Attn}}_{ij} = \frac{\partial}{\partial\mathbf{r}_j}h_i^{(l+1)} \propto\Sigma^{\mathrm{Attn}}_{ij}[h^{(l)}] independent of system size NN.

In continuum operator learning, Tsai et al. [12] and Cao [13] recently proved that self-attention acts as a discretised Fredholm integral operator whose kernel approximates a differential equation Green’s function (with unnormalised linear attention forming a Petrov–Galerkin projection). Reformulating in explicitly quantum-mechanical notation, the residual skip connection (δij)(\delta_{ij}) acts as the bare one-body propagator G0G_0, the attention kernel ΣijAttn[h(l)]\Sigma^{\mathrm{Attn}}_{ij}[h^{(l)}] acts as the pairwise self-energy functional Σ[G1]\Sigma[G_1], and the LL-layer map dressing bare inputs into quasiparticles, ∣h(L)⟩=G1∣h(0)⟩|h^{(L)}\rangle= G_1|h^{(0)}\rangle, iterates Dyson’s equation[14],

G1=G0+G0Σ[G1]G1,⟨V^⟩=12Tr⁡(Σ[G1]G1).(5)G_1 = G_0 + G_0\Sigma[G_1]G_1,\qquad\langle\hat{V}\rangle= \frac{1}{2}\operatorname{Tr}\bigl(\Sigma[G_1]G_1\bigr). \tag*{(5)}

We express this diagrammatically in Figure 1. Unrolling LL attention layers (with queries ∣qi(l)⟩\lvert q_i^{(l)}\rangle and keys ∣kj(l)⟩\lvert k_j^{(l)}\rangle recomputed from dressed states at each steps) generates the self-consistent Dyson series G1=G0+G0Σ[G1]G0+G0Σ[G1]G0Σ[G1]G0+…G_1 = G_0 + G_0\Sigma[G_1]G_0 + G_0\Sigma[G_1]G_0\Sigma[G_1]G_0 + \ldots, dressing single-particle coordinates into quasiparticles. Because physical Hamiltonians H^=∑i=1Nh^(i)+∑i<jNv^(i,j)\hat{H} = \sum_{i=1}^{N}\hat{h}(i) + \sum_{i<j}^{N}\hat{v}(i,j) contain at most pairwise (v^(i,j))(\hat{v}(i,j)) interactions, Löwdin’s 2-RDM theorem[8] and the Galitskii–Migdal relation[15] for the interaction energy ⟨V^⟩\langle\hat{V}\rangle guarantee that a dressed one-body propagator G1G_1 coupled to a two-body self-energy Σ[G1]\Sigma[G_1] suffices to determine exact energetics and two-body observables.

DeepSets and self-attention Dyson diagrams

Figure 1. Following Mattuck[10], we express the two-body Dyson equation iteration as a Feynman diagram. (a) In DeepSets Eqn. (2), summing ϕ(∣hj(l)⟩)\phi\left(\left|h_j^{(l)}\right\rangle\right) inside a closed Hartree tadpole loop forms a mean-field average ∣vˉ(l)⟩\left|\bar{v}^{(l)}\right\rangle that dresses particle ii via readout vertex ρi\rho_i with uniform weight 1/N1/N. Vertex operators ρ\rho and ϕ\phi are multilayer perceptrons (MLPs). (b) In self-attention Eqn. (4), the residual skip stream acts as the unscattered propagator G0G_0 (single line), while linear projections (vertex operators) ⟨qi(l)∣\left\langle q_i^{(l)}\right| and ∣kj(l)⟩\left|k_j^{(l)}\right\rangle form a directed Boltzmann self-energy ΣijAttn[h(l)]\Sigma_{ij}^{\mathrm{Attn}}\left[h^{(l)}\right] (dashed line) that combines partner jj’s dressed value stream ∣vj(l)⟩\left|v_j^{(l)}\right\rangle (double line) into particle ii with O(1)\mathcal{O}(1) weight, dressing the outgoing propagator Eqn. (5).

This correspondence explains both the architectural split across atomistic machine learning and its computational scaling. By Kohn’s nearsightedness principle[16], screened, extensive ground-state potential energy surfaces E0(R)E_0(\mathbf{R}) depend only on short-range local environments, so finite-cutoff message passing followed by O(N)O(N)

Hartree sum-pooling (E=∑iEiE=\sum_i E_i, as in MACE[3] and other MLPs) is physically complete.

Conversely, nonlocal quantum observables depend on all (N2)\binom{N}{2} (N2N^2) off-diagonal two-body propagators and relative fragment orientations, which are erased by mean-field sum-pooling but preserved by ⟨qi∣kj⟩\langle q_i|k_j\rangle. Both PsiFormer[4] in quantum Monte Carlo and PET[17] (albeit limited to local clusters) in interatomic potentials have empirically discovered the physical rewards of self-attention. The clearest next frontier lies wherever models must reconstruct nonlocal quantum operators directly: two-index tight-binding and Kohn–Sham Hamiltonian matrix elements Hij(R)H_{ij}(\mathbf{R}) (such as SchNOrb[18], DeePH[19] and MACE-H[20]), or the multi-state companion Hamiltonians Mmn(R)M_{mn}(\mathbf{R}) that govern excited-state crossings in X-MACE[21], where pairwise attention weights naturally mirror off-diagonal couplings.

The O(N2)O(N^2) scaling of dense self-attention is thus not an architectural flaw, but the irreducible cost of unscreened pairwise interactions when nearsightedness fails. By further analogy with computational condensed matter, sparse and hierarchical attention mirror the development of fast-multipole methods.

Results: Fractal Strawberry

The insights in this paper stem from fitting the fractal determinant[22] of a hydrodynamic backflow wavefunction[23] to verify the expressive power of a continuous normalising flow (CNF) backflow model (SpinDrift[24]). To model Fermionic quantum criticality, Krüger and Zaanen[22] dressed free-Fermion plane waves of occupied momenta in a Slater determinant ΨBF(R)=det⁡[exp⁡(ikm⋅rn)]\Psi_{\mathrm{BF}}(\mathbf{R}) = \det[\exp(i\mathbf{k}_m\cdot\mathbf{r}_n)], evaluated at quasiparticle backflow coordinates r~i\tilde{\mathbf{r}}_i, displaced by an instantaneous two-body velocity field vi(R)\mathbf{v}_i(\mathbf{R}),

r~i=ri+vi(R),\tilde{\mathbf{r}}_i = \mathbf{r}_i + \mathbf{v}_i(\mathbf{R}),
vi(R)=∑j≠iη(rij)(ri−rj),\mathbf{v}_i(\mathbf{R}) = \sum_{j\ne i}\eta(r_{ij})(\mathbf{r}_i-\mathbf{r}_j),
η(rij)=a3rij3+r03.(6)\eta(r_{ij}) = \frac{a^3}{r_{ij}^3+r_0^3}. \tag*{(6)}

where rij=∣ri−rj∣r_{ij} = \lvert\mathbf{r}_i-\mathbf{r}_j\rvert, aa controls the backflow strength, and r0r_0 sets the short-range length scale. Beyond a critical coupling strength in periodic systems, the nodal surface ΨBF(R)=0\Psi_{\mathrm{BF}}(\mathbf{R})=0 undergoes a topological transition from smoothly varying (Fermi-liquid like) into a scale-invariant fractal, driven by what Krüger and Zaanen identified diagrammatically as infinite-order vertex corrections[22]. This wavefunction was a putative suggestion based on scaling arguments, with no construction of the underlying Hamiltonian offered. The form of the backflow comes from hydrodynamic flow arguments in the original setting of liquid helium[23].

Although this fractal nodal surface embodies infinite-order many-body criticality, the instantaneous backflow velocity field has the exact pairwise Dyson structure of Eqn. (1), with self-energy kernel ΣijBF=η(rij)\Sigma_{ij}^{\mathrm{BF}} = \eta(r_{ij}) weighting the pairwise relative-coordinate message ∣vij⟩=ri−rj|v_{ij}\rangle= r_i - r_j. In Spindrift[24], we evaluate an open-boundary thermal Slater determinant (N=6N = 6 spin-polarised fermions in D=2D = 2) after calculating quasi-position (backflow) from integrating a single-layer (L=1L = 1) neural velocity field over pseudo-time s∈[0,1]s \in[0,1]. In order to induce fractal phase space folding with open boundaries, we integrate with only two coarse RK4 steps (Δs=0.5\Delta s = 0.5).

When we attempted to learn this backflow mapping using an equivariant DeepSets architecture (Eqn. (2)), the O(1/N)O(1/N) mean-field sum washed out the pairwise kernel, killed the local shear ridges in the finite-time Lyapunov exponent (FTLE) map (Figure 3a), and collapsed the fractal structure (Figure 2) into a smooth, featureless envelope—a “squashed strawberry” (Figure 3b). The model was simply not expressive enough to hold the inter-particle correlation which generated the fractal structure. Replacing DeepSets pooling with a PsiFormer-inspired self-attention layer (∼20\sim20 lines of Julia code) restores the O(1)O(1) nonlocal Dyson kernel Σij\Sigma_{ij}, recovering both the dipolar FTLE rosettes (Figure 3d) and the self-similar fractal nodal pockets (Figure 3e).

Reference fractal backflow determinant

Figure 2. The reference fractal backflow determinant which we used for fitting, r0=0.2r_0 = 0.2 and a=1a = 1 (strongly fractal). Visualised for N=6N = 6 by holding 5 particles stationary (yellow dots) on a ring of radius 1.5 and then using the sixth particle as a probe (red/blue: determinant sign; brightness: ∣det⁡∣0.07|\det|^{0.07}, nodes in black). In this open-boundary model, the nested fractal pockets emerge from phase-space folding across the two intentionally coarse RK4 steps (Δs=0.5\Delta s = 0.5) that integrate the singular 1/r31/r^3 velocity field into quasi-positions.

Width-matched DeepSets and self-attention fits to the fractal backflow target

Figure 3. Width-matched DeepSets (top) and self-attention (bottom) fits to the fractal backflow target (Figure 2), trained by L₁ regression on the instantaneous velocity field of Eqn. (6), using the same coarse two-step RK4 integrator. (a, d) Finite-time Lyapunov exponent (FTLE) maps ln⁡σmax⁡(∇RR~)\ln\sigma_{\max}(\nabla_{\mathbf{R}}\widetilde{\mathbf{R}}), where σmax⁡\sigma_{\max} denotes the maximum singular value of the flow-map Jacobian ∇RR~\nabla_{\mathbf{R}}\widetilde{\mathbf{R}}: DeepSets smears sensitivity (albeit rather prettily) across the trap, whereas self-attention resolves dipolar rosettes around the stationary particles (black dots). (b, e) Slater determinants: DeepSets collapses the nodal surface into a smooth mean-field envelope, whilst self-attention recovers the nested fractal pockets. (c, f) Off-diagonal two-body velocity Jacobian spectral norm ∥∂vi/∂rj∥2=σmax⁡(∂vi/∂rj)\lVert\partial\mathbf{v}_i/\partial\mathbf{r}_j\rVert_2 = \sigma_{\max}(\partial\mathbf{v}_i/\partial\mathbf{r}_j) (the largest singular value of each 2×22 \times2 pair block i≠ji \ne j of ∂v\partial\mathbf{v}) versus pair separation rijr_{ij} across 3,600 random pairs (points are raw samples; solid line and shaded band show median and interquartile range; black dashed line shows the exact analytic derivative): DeepSets suffers O(1/N)O(1/N) saturation at short range alongside spurious crosstalk at long range, whereas self-attention tracks the exact pairwise kernel over four decades. Both velocity networks share identical encoders and decoders (hidden dimension dmodel=128d_{\mathrm{model}} = 128, GELU activations, zero-initialised output layer; 50,690 Float64 parameters). DeepSets then uses parameter-free mean-pooling, whereas a H=4H = 4-head dk=32d_k = 32 attention block adds 65,536 parameters for a total of 116,226. Both models were trained by online stochastic sampling for 10,000 minibatch steps (batch size 512), with the L₁ loss saturating for DeepSets at ≈1.0\approx1.0 after 200 steps, while self-attention descended smoothly to 0.17.

Differentiating the fitted velocity networks vi(R)v_i(\mathbf{R}) directly provides our key quantitative test of Eqn. (3) (Figure 3c, Figure 3f). As the probe particle approaches a stationary ring particle (rij=0.05<r0r_{ij} = 0.05 < r_0), the exact off-diagonal velocity Jacobian norm reaches ∥∂v1/∂r6∥2=123\left\lVert\partial v_1/\partial r_6 \right\rVert_2 = 123: self-attention reproduces this within 1.6% (125), whereas DeepSets saturates at ∼8\sim8 from rij<0.8r_{ij} < 0.8. Simultaneously, for the diametrically opposite spectator particle (rij=3.05r_{ij} = 3.05), the true rij−3r_{ij}^{-3} coupling decays to 0.070 (the self-attention fit is 0.021), yet DeepSets predicts 0.325, producing the vertical spray of long-range outliers in Figure 3c: inflating ∂∂rivj\frac{\partial}{\partial r_i}v_j to chase a short-range collision broadcasts a global shift through Σˉ\bar{\Sigma} to every particle in the system.

From a renormalisation group (RG) perspective, Krüger and Zaaneen’s fractal nodal surface is the geometric signature of a quantum critical fixed point, where scale-invariant vertex corrections fold phase space across all length scales[22]. Under DeepSets sum-pooling, the O(1/N)O(1/N) Jacobian collapses acts as a hard ultraviolet cut-off that amputates short-range gradients (Figure 3c), trapping the flow at a trivial mean-field fixed point: the smooth “squashed strawberry” (Figure 3b). Self-attention preserves the power-law two-body kernel across four decades of separation (Figure 3f), allowing repeated composition of the flow map to sustain criticality and generate the self-similar fractal attractor (Figure 3e).

Conclusions

Taking this Dyson equation correspondence seriously explains specific Transformer design choices. First, the exponential Boltzmann factor in Eqn. (4) is essential: whereas O(N)O(N) “linear attention” (Σij∼⟨qi∣kj⟩\Sigma_{ij} \sim\langle q_i|k_j\rangle) reduces to a separable low-rank kernel (i.e. a finite-order perturbation theory), expanding exp⁡(β⟨qi∣kj⟩)\exp(\beta\langle q_i|k_j\rangle) generates all powers of the two-body vertex, providing a full-rank kernel. Second, had this kernel been evaluated solely on bare input coordinates, stacking LL layers would merely unroll a bare Born perturbation series G0+G0Σ0G0+⋯G_0 + G_0\Sigma_0G_0 + \cdots. Because a Transformer instead recomputes queries and keys from the already-dressed states at every layer, it feeds the dressed propagator back inside the self-energy functional, Σ[{h[l]}]\Sigma[\{h^{[l]}\}]. Diagrammatically, network depth executes a self-consistent skeleton expansion (summing polarisation rings and vertex dressings in a discrete functional RG flow). Multi-head attention decomposes this self-energy action into HH parallel interaction channels, ∑h=1HΣij(h)∣vj(l,h)⟩\sum_{h=1}^{H}\Sigma_{ij}^{(h)}|v_j^{(l,h)}\rangle (where ∣vj(l,h)⟩|v_j^{(l,h)}\rangle absorbs each head’s output projection, as seen in PsiFormer on the benzene dimer[4], where separate heads spontaneously partition short-range intramolecular exchange from long-range intermolecular dispersion).

This perturbative structure likewise explains why residual skip connections and zero-initialised output layers are often required to train deep attention networks. Switching on a strong self-energy Σ\Sigma instantaneously drives the Dyson series (G0−1−Σ)−1(G_0^{-1} - \Sigma)^{-1} across poles and branch cuts; bare states deform smoothly into dressed quasiparticles only when the interaction is turned on adiabatically from the non-interacting propagator G0G_0. Deep Transformers and continuous normalising flows rediscover this exact safeguard. The skip path supplies the bare propagator G0=IG_0 = I, while zero-initialising the output projection pins the initial model at the non-interacting limit, Σ=0\Sigma= 0, allowing gradient descent to dress coordinates adiabatically without crossing singularities. (Replacing spatial indices with time steps immediately extends this picture to open quantum systems, where causal self-attention acts as a non-

Markovian Nakajima–Zwanzig memory kernel.)

Equivariant sum-pooling and self-attention are not just two competing heuristics; they mirror the two foundational approximations that have structured condensed-matter physics for a century. Sum-pooling is a mean-field (Hartree) approximation. Where the nearsightedness principle holds (as in ground-state interatomic potentials) summing local one-body energies is both efficient and physically complete. Where near-sightedness breaks down (such as correlated wavefunction determinants, or long-range excited states), sum-pooling suffers an O(1/N)O(1/N) Jacobian collapse that washes out off-diagonal structure into a featureless mean-field.

Self-attention fixes this by replacing a one-body sum with a real-space Dyson iteration. Its softmaxnormalised inner product constructs a state-dependent, two-body self-energy Σij\Sigma_{ij} on the fly, preserving the O(1)O(1) pairwise sensitivity required to reconstruct scale-invariant fractal attractors and nonlocal excitonic couplings. Because physical Hamiltonians are built from pairwise interactions, Löwdin’s 2-RDM theorem and the Galitskii–Migdal relation guarantee that a dressed one-body propagator coupled to a two-body self-energy is formally complete for many-body energetics. The rise of the Transformer is therefore no accident; it rediscovers the minimal complete representation demanded by pairwise interactions. Mean-field pooling suffices for local potentials, but for correlated matter, attention is all we ever needed.

Acknowledgments

I thank Christoph Ortner, Matthew Foulkes and Frank Krüger for fruitful discussion that led directly to the realisations described in this letter. J.M.F. is supported by a Royal Society University Research Fellowship (URF-R1-191292). Julia codes implementing Spindrift and the fractal backflow benchmarks are available on GitHub as part of Halcyon.jl[25]. I gratefully acknowledge the use of the Imperial College Research Computing Service[26].

References

  1. [1]J. Herrmann, J. Spencer, K. Choo, A. Mezzacapo, W. M. C. Foulkes, D. Pfau, G. Carleo, and F. Noé, Ab initio quantum chemistry with neural-network wavefunctions, Nature Reviews Chemistry 7, 692 (2023).
  2. [2]G. Cassella, H. Sutterud, S. Azadi, N. D. Drummond, D. Pfau, J. S. Spencer, and W. M. C. Foulkes, Discovering quantum phase transitions with fermionic neural networks, Physical Review Letters 130, 036401 (2023).DOI
  3. [3]I. Batatia, D. P. Kovacs, G. N. C. Simm, C. Ortner, and G. Csányi, MACE: Higher order equivariant message passing neural networks for fast and accurate force fields, in Advances in Neural Information Processing Systems, Vol. 35 (Curran Associates, Inc., 2022) pp. 11423–11436.
  4. [4]I. von Glehn, J. S. Spencer, and D. Pfau, A self-attention ansatz for ab-initio quantum chemistry, in International Conference on Learning Representations (ICLR) (2023).arxiv.org/abs/2211.13672
  5. [5]D. Pfau, J. S. Spencer, A. G. D. G. Matthews, and W. M. C. Foulkes, Ab initio solution of the many-electron Schrödinger equation with deep neural networks, Physical Review Research 2, 033429 (2020).
  6. [6]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention is all you need, in Advances in Neural Information Processing Systems, Vol. 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Curran Associates, Inc., 2017).arxiv.org/abs/1706.03762
  7. [7]M. J. Willatt, F. Musil, and M. Ceriotti, Atom-density representations for machine learning, The Journal of Chemical Physics 150, 154110 (2019).DOI
  8. [8]P.-O. Löwdin, Quantum theory of many-particle systems. I. physical interpretations by means of density matrices, natural spin-orbitals, and convergence problems in the method of configurational interaction, Physical Review 97, 1474 (1955).
  9. [9]M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola, Deep sets, in Advances in Neural Information Processing Systems, Vol. 30 (Curran Associates, Inc., 2017).
  10. [10]R. D. Mattuck, A Guide to Feynman Diagrams in the Many-Body Problem, 2nd ed., Dover Books on Physics and Chemistry (Dover Publications, New York, 1992).
  11. [11]H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve, V. Greiff, D. Kreil, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter, Hopfield networks is all you need, in International Conference on Learning Representations (ICLR) (2021).arxiv.org/abs/2008.02217
  12. [12]Y.-H. H. Tsai, S. Bai, M. Yamada, L.-P. Morency, and R. Salakhutdinov, Transformer dissection: An unified understanding of transformer’s attention via the lens of kernel, in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (Association for Computational Linguistics, 2019) pp. 4343–4352.arxiv.org/abs/1908.11775
  13. [13]S. Cao, Choose a transformer: Fourier or Galerkin, in Advances in Neural Information Processing Systems, Vol. 34 (Curran Associates, Inc., 2021) pp. 24924–24940.arxiv.org/abs/2105.14995
  14. [14]F. J. Dyson, The S matrix in quantum electrodynamics, Physical Review 75, 1736 (1949).DOI
  15. [15]V. M. Galitskii and A. B. Migdal, Application of quantum field theory methods to the many body problem, Soviet Physics JETP 7, 96 (1958).
  16. [16]E. Prodan and W. Kohn, Nearsightedness of electronic matter, Proceedings of the National Academy of Sciences 102, 11635 (2005).pubmed.ncbi.nlm.nih.gov/16087868
  17. [17]S. Podznyakov and M. Ceriotti, Smooth, exact rotational symmetrization for deep learning on point clouds, in Advances in Neural Information Processing Systems 36, NeurIPS 2023 (Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2023) p. 79469–79501.
  18. [18]K. T. Schütt, M. Gastegger, A. Tkatchenko, K.-R. Müller, and R. J. Maurer, Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions, Nature Communications 10, 10.1038/s41467-019-12875-0 (2019).
  19. [19]H. Li, Z. Wang, N. Zou, M. Ye, R. Xu, X. Gong, W. Duan, and Y. Xu, Deep-learning density functional theory hamiltonian for efficient ab initio electronic-structure calculation, Nature Computational Science 2, 367–377 (2022).DOI
  20. [20]C. Qian, V. Vitartas, J. R. Kermode, and R. J. Maurer, Equivariant electronic hamiltonian prediction with many-body message passing, npj Computational Materials 12, 10.1038/s41524-026-02020-1 (2026).DOI
  21. [21]R. Barrett, C. Ortner, and J. Westermayr, Transferable machine learning potential x-MACE for excited states using integrated DeepSets (2025), arXiv:2502.12870 [physics.chem-ph].arxiv.org/abs/2502.12870
  22. [22]F. Krüger and J. Zaanen, Fermionic quantum criticality and the fractal nodal surface, Physical Review B 78, 035104 (2008).
  23. [23]R. P. Feynman and M. Cohen, Energy spectrum of the excitations in liquid helium, Physical Review 102, 1189 (1956).DOI
  24. [24]J. M. Frost, Spindrift: Learning quantum degeneracy from thermal purity in restricted path integral Monte Carlo (2026), arXiv:2607.29590 [cond-mat.stat-mech].arxiv.org/abs/2607.29590
  25. [25]J. M. Frost, HALCYON.jl: Path integral Monte Carlo and continuous normalising flows in Julia, https://github.com/Frost-group/Halcyon.jl (2026).
  26. [26]Imperial College Research Computing Service, Research computing service facility at Imperial College London, Imperial College London, https://doi.org/10.14469/hpc/2232 (2016).

Paper details

vlnc:W_2ggg5d99
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents