WorksPhysics and Astronomy
Self-attention is a Dyson equation for many-body machine learning
Abstract
Why does symmetric sum-pooling (DeepSets) predict ground-state potentials to sub-meV accuracy, yet fail for correlated wavefunctions and excited states? Sum-pooling is a mean-field (Hartree) model: its off-diagonal Jacobian suffices for nearsighted ground-state energies, but erases nonlocal two-body correlations. We show that self-attention is a real-space Dyson iteration whose Boltzmann (softmax) kernel acts as a two-body self-energy. Because physical Hamiltonians act pairwise, Löwdin's 2-RDM theorem and the Galitskii–Migdal relation guarantee that iterating a two-body propagator captures the full energetics of correlated matter. When fitting a fractal backflow determinant, DeepSets blurs the nodal surface onto a mean-field envelope, whereas self-attention recovers the fractal attractor. Seen through this lens, the unreasonable effectiveness of attention is not an accident; in a universe built from pairwise interactions, two-body attention is all you need.
Introduction
Neural network wavefunction Ansätze often feed electron-pair separations directly into the network [1], a brute-force approach that bloats the architecture, while raw single-particle coordinates lose translational symmetry unless embedded periodically [2]. Symmetric sum-pooling (often via equivariant DeepSets) scalarises interactions between particles into one-body terms. By the universal approximation theorem, this is formally sufficient to encode everything about the system, but with finite capacity it is not tractable to extract many-body correlations from a single pooled vector. Yet in machine learning interatomic potentials for a system size of atoms, finite-cutoff equivariant message passing followed by DeepSets sum-pooling over local atomic energies () achieves sub-meV [3] accuracy; whereas PsiFormer [4] replaces the pair features in FermiNet [5] with more simple multi-head self-attention over single-particle features, and achieves superior electronic correlation in molecular dimers.
Attention [6] now underpins almost all areas of machine learning, improving performance while simplifying complex architectures. Conventionally framed in computer-science database terminology, the mechanism pairs queries with keys and normalises their dot products via a softmax to perform a “fuzzy” lookup over values. Following the example of atom-density representations written in Dirac notation [7], we recast this database abstraction into the operator language of many-body physics. At its core, self-attention applies learned linear projection matrices to map single-particle states into queries , keys and values , before normalising their query–key inner products with a Boltzmann (softmax) factor.
This reduces to a state-dependent two-body propagator , normalised by the local partition function . In this paper we show that DeepSets (scalar) pooling is a one-body mean-field (Hartree) description, explaining why it accurately captures additive, locally nearsighted quantities such as ground-state energies and forces. Remarkably, self-attention maps directly onto Dyson resummation of the nonlocal two-body propagator. Because physical Hamiltonians are governed by pairwise interactions and exact energetics reduce to two-body correlators [8], iterative self-attention is formally sufficient to describe the one- and two-body observables (and hence exact energetics) of pairwise (physical, Coulomb) Hamiltonians.
Consequently, machine learning architectures directly targeting nonlocal quantum mechanical observables require self-attention rather than scalar sum-pooling. This equivalence to foundational completeness theorems of quantum matter explains why, across so many physical systems, “attention is all you need” [6]. Here we make this connection between machine-learning architectures and many-body theory explicit, bridging the disparate notation and terminology of the two communities.
Theory
Consider a system of quantum particles or atoms at collective coordinates with in spatial dimensions, represented at layer (of ) of a machine-learning model by the set of permutation-equivariant single-particle states . Both DeepSets and self-attention update operations can be written as a discrete two-body Dyson equation,
where is the single-particle value (message) sent by particle and is the scalar effective two-body self-energy kernel coupling particle to particle as a function of the effective layer state .
Each layer in the machine-learning network performs one step of Dyson iteration; the total depth sets the order of resummation.
Scalar DeepSets is the mean-field (Hartree) approximation
In DeepSets[9], interactions are aggregated via an unweighted symmetric sum (sum pooling) over all particles,
where and are learned single-particle embedding and update functions (multilayer perceptrons), and is the globally pooled value state. Setting individual particle values and comparing directly with Eqn. (2) reveals the physical loss: the pooled state carries no target index . Every particle experiences an identical background potential, corresponding to a uniform mean-field (Hartree) kernel . Consequently, the off-diagonal Jacobian measuring how particle ’s dressed state responds to particle ’s position scales as
where . When these pooled states are used downstream, specific pairwise correlations are diluted by ; worse, because Eqn. (3) factorises into an -only factor times a -only factor independent of pairwise separations , any steep local gradient at broadcasts spurious crosstalk to every distant spectator .
Physically this pathology means that a close approach between two particles jogs all the other particles in the system, as the large local interaction has to be modelled by the mean-field. As with any mean-field theory, this cannot support strong correlation.
Self-attention is a Dyson equation
Self-attention[6] replaces the uniform weight in Eqn. (2) with a state-dependent Boltzmann kernel. Written in the motivating form of Dirac bra–ket notation, learned linear operators and project each -dimensional
state into query , key (both of feature dimension ), and value (model dimension ) states. The update operation is then
where is the local canonical partition function at inverse effective temperature . This immediately generalises to parallel interaction channels in multi-head attention.
In the high-temperature limit , and Eqn. (4) collapses to the DeepSets mean-field sum of Eqn. (2); whereas freezes into a nearest-neighbour graph. The standard scaling keeps in the correlated regime between mean-field melting and glassy freezing[11].
Here the inner product (where denotes the adjoint of ) acts as a learned, generally non-symmetric () two-body interaction vertex that dynamically reconstructs translationally invariant pair kernels (such as Gaussian radial basis or periodic plane waves) from single-particle states without storing explicit pair vectors.2
Crucially, for strongly correlated pairs , the Boltzmann factor concentrates weight on partner while silencing distant spectators (this is attention, for ), preserving an off-diagonal Jacobian independent of system size .
In continuum operator learning, Tsai et al. [12] and Cao [13] recently proved that self-attention acts as a discretised Fredholm integral operator whose kernel approximates a differential equation Green’s function (with unnormalised linear attention forming a Petrov–Galerkin projection). Reformulating in explicitly quantum-mechanical notation, the residual skip connection acts as the bare one-body propagator , the attention kernel acts as the pairwise self-energy functional , and the -layer map dressing bare inputs into quasiparticles, , iterates Dyson’s equation[14],
We express this diagrammatically in Figure 1. Unrolling attention layers (with queries and keys recomputed from dressed states at each steps) generates the self-consistent Dyson series , dressing single-particle coordinates into quasiparticles. Because physical Hamiltonians contain at most pairwise interactions, Löwdin’s 2-RDM theorem[8] and the Galitskii–Migdal relation[15] for the interaction energy guarantee that a dressed one-body propagator coupled to a two-body self-energy suffices to determine exact energetics and two-body observables.

Figure 1. Following Mattuck[10], we express the two-body Dyson equation iteration as a Feynman diagram. (a) In DeepSets Eqn. (2), summing inside a closed Hartree tadpole loop forms a mean-field average that dresses particle via readout vertex with uniform weight . Vertex operators and are multilayer perceptrons (MLPs). (b) In self-attention Eqn. (4), the residual skip stream acts as the unscattered propagator (single line), while linear projections (vertex operators) and form a directed Boltzmann self-energy (dashed line) that combines partner ’s dressed value stream (double line) into particle with weight, dressing the outgoing propagator Eqn. (5).
This correspondence explains both the architectural split across atomistic machine learning and its computational scaling. By Kohn’s nearsightedness principle[16], screened, extensive ground-state potential energy surfaces depend only on short-range local environments, so finite-cutoff message passing followed by
Hartree sum-pooling (, as in MACE[3] and other MLPs) is physically complete.
Conversely, nonlocal quantum observables depend on all () off-diagonal two-body propagators and relative fragment orientations, which are erased by mean-field sum-pooling but preserved by . Both PsiFormer[4] in quantum Monte Carlo and PET[17] (albeit limited to local clusters) in interatomic potentials have empirically discovered the physical rewards of self-attention. The clearest next frontier lies wherever models must reconstruct nonlocal quantum operators directly: two-index tight-binding and Kohn–Sham Hamiltonian matrix elements (such as SchNOrb[18], DeePH[19] and MACE-H[20]), or the multi-state companion Hamiltonians that govern excited-state crossings in X-MACE[21], where pairwise attention weights naturally mirror off-diagonal couplings.
The scaling of dense self-attention is thus not an architectural flaw, but the irreducible cost of unscreened pairwise interactions when nearsightedness fails. By further analogy with computational condensed matter, sparse and hierarchical attention mirror the development of fast-multipole methods.
Results: Fractal Strawberry
The insights in this paper stem from fitting the fractal determinant[22] of a hydrodynamic backflow wavefunction[23] to verify the expressive power of a continuous normalising flow (CNF) backflow model (SpinDrift[24]). To model Fermionic quantum criticality, Krüger and Zaanen[22] dressed free-Fermion plane waves of occupied momenta in a Slater determinant , evaluated at quasiparticle backflow coordinates , displaced by an instantaneous two-body velocity field ,
where , controls the backflow strength, and sets the short-range length scale. Beyond a critical coupling strength in periodic systems, the nodal surface undergoes a topological transition from smoothly varying (Fermi-liquid like) into a scale-invariant fractal, driven by what Krüger and Zaanen identified diagrammatically as infinite-order vertex corrections[22]. This wavefunction was a putative suggestion based on scaling arguments, with no construction of the underlying Hamiltonian offered. The form of the backflow comes from hydrodynamic flow arguments in the original setting of liquid helium[23].
Although this fractal nodal surface embodies infinite-order many-body criticality, the instantaneous backflow velocity field has the exact pairwise Dyson structure of Eqn. (1), with self-energy kernel weighting the pairwise relative-coordinate message . In Spindrift[24], we evaluate an open-boundary thermal Slater determinant ( spin-polarised fermions in ) after calculating quasi-position (backflow) from integrating a single-layer () neural velocity field over pseudo-time . In order to induce fractal phase space folding with open boundaries, we integrate with only two coarse RK4 steps ().
When we attempted to learn this backflow mapping using an equivariant DeepSets architecture (Eqn. (2)), the mean-field sum washed out the pairwise kernel, killed the local shear ridges in the finite-time Lyapunov exponent (FTLE) map (Figure 3a), and collapsed the fractal structure (Figure 2) into a smooth, featureless envelope—a “squashed strawberry” (Figure 3b). The model was simply not expressive enough to hold the inter-particle correlation which generated the fractal structure. Replacing DeepSets pooling with a PsiFormer-inspired self-attention layer ( lines of Julia code) restores the nonlocal Dyson kernel , recovering both the dipolar FTLE rosettes (Figure 3d) and the self-similar fractal nodal pockets (Figure 3e).

Figure 2. The reference fractal backflow determinant which we used for fitting, and (strongly fractal). Visualised for by holding 5 particles stationary (yellow dots) on a ring of radius 1.5 and then using the sixth particle as a probe (red/blue: determinant sign; brightness: , nodes in black). In this open-boundary model, the nested fractal pockets emerge from phase-space folding across the two intentionally coarse RK4 steps () that integrate the singular velocity field into quasi-positions.

Figure 3. Width-matched DeepSets (top) and self-attention (bottom) fits to the fractal backflow target (Figure 2), trained by L₁ regression on the instantaneous velocity field of Eqn. (6), using the same coarse two-step RK4 integrator. (a, d) Finite-time Lyapunov exponent (FTLE) maps , where denotes the maximum singular value of the flow-map Jacobian : DeepSets smears sensitivity (albeit rather prettily) across the trap, whereas self-attention resolves dipolar rosettes around the stationary particles (black dots). (b, e) Slater determinants: DeepSets collapses the nodal surface into a smooth mean-field envelope, whilst self-attention recovers the nested fractal pockets. (c, f) Off-diagonal two-body velocity Jacobian spectral norm (the largest singular value of each pair block of ) versus pair separation across 3,600 random pairs (points are raw samples; solid line and shaded band show median and interquartile range; black dashed line shows the exact analytic derivative): DeepSets suffers saturation at short range alongside spurious crosstalk at long range, whereas self-attention tracks the exact pairwise kernel over four decades. Both velocity networks share identical encoders and decoders (hidden dimension , GELU activations, zero-initialised output layer; 50,690 Float64 parameters). DeepSets then uses parameter-free mean-pooling, whereas a -head attention block adds 65,536 parameters for a total of 116,226. Both models were trained by online stochastic sampling for 10,000 minibatch steps (batch size 512), with the L₁ loss saturating for DeepSets at after 200 steps, while self-attention descended smoothly to 0.17.
Differentiating the fitted velocity networks directly provides our key quantitative test of Eqn. (3) (Figure 3c, Figure 3f). As the probe particle approaches a stationary ring particle (), the exact off-diagonal velocity Jacobian norm reaches : self-attention reproduces this within 1.6% (125), whereas DeepSets saturates at from . Simultaneously, for the diametrically opposite spectator particle (), the true coupling decays to 0.070 (the self-attention fit is 0.021), yet DeepSets predicts 0.325, producing the vertical spray of long-range outliers in Figure 3c: inflating to chase a short-range collision broadcasts a global shift through to every particle in the system.
From a renormalisation group (RG) perspective, Krüger and Zaaneen’s fractal nodal surface is the geometric signature of a quantum critical fixed point, where scale-invariant vertex corrections fold phase space across all length scales[22]. Under DeepSets sum-pooling, the Jacobian collapses acts as a hard ultraviolet cut-off that amputates short-range gradients (Figure 3c), trapping the flow at a trivial mean-field fixed point: the smooth “squashed strawberry” (Figure 3b). Self-attention preserves the power-law two-body kernel across four decades of separation (Figure 3f), allowing repeated composition of the flow map to sustain criticality and generate the self-similar fractal attractor (Figure 3e).
Conclusions
Taking this Dyson equation correspondence seriously explains specific Transformer design choices. First, the exponential Boltzmann factor in Eqn. (4) is essential: whereas “linear attention” () reduces to a separable low-rank kernel (i.e. a finite-order perturbation theory), expanding generates all powers of the two-body vertex, providing a full-rank kernel. Second, had this kernel been evaluated solely on bare input coordinates, stacking layers would merely unroll a bare Born perturbation series . Because a Transformer instead recomputes queries and keys from the already-dressed states at every layer, it feeds the dressed propagator back inside the self-energy functional, . Diagrammatically, network depth executes a self-consistent skeleton expansion (summing polarisation rings and vertex dressings in a discrete functional RG flow). Multi-head attention decomposes this self-energy action into parallel interaction channels, (where absorbs each head’s output projection, as seen in PsiFormer on the benzene dimer[4], where separate heads spontaneously partition short-range intramolecular exchange from long-range intermolecular dispersion).
This perturbative structure likewise explains why residual skip connections and zero-initialised output layers are often required to train deep attention networks. Switching on a strong self-energy instantaneously drives the Dyson series across poles and branch cuts; bare states deform smoothly into dressed quasiparticles only when the interaction is turned on adiabatically from the non-interacting propagator . Deep Transformers and continuous normalising flows rediscover this exact safeguard. The skip path supplies the bare propagator , while zero-initialising the output projection pins the initial model at the non-interacting limit, , allowing gradient descent to dress coordinates adiabatically without crossing singularities. (Replacing spatial indices with time steps immediately extends this picture to open quantum systems, where causal self-attention acts as a non-
Markovian Nakajima–Zwanzig memory kernel.)
Equivariant sum-pooling and self-attention are not just two competing heuristics; they mirror the two foundational approximations that have structured condensed-matter physics for a century. Sum-pooling is a mean-field (Hartree) approximation. Where the nearsightedness principle holds (as in ground-state interatomic potentials) summing local one-body energies is both efficient and physically complete. Where near-sightedness breaks down (such as correlated wavefunction determinants, or long-range excited states), sum-pooling suffers an Jacobian collapse that washes out off-diagonal structure into a featureless mean-field.
Self-attention fixes this by replacing a one-body sum with a real-space Dyson iteration. Its softmaxnormalised inner product constructs a state-dependent, two-body self-energy on the fly, preserving the pairwise sensitivity required to reconstruct scale-invariant fractal attractors and nonlocal excitonic couplings. Because physical Hamiltonians are built from pairwise interactions, Löwdin’s 2-RDM theorem and the Galitskii–Migdal relation guarantee that a dressed one-body propagator coupled to a two-body self-energy is formally complete for many-body energetics. The rise of the Transformer is therefore no accident; it rediscovers the minimal complete representation demanded by pairwise interactions. Mean-field pooling suffices for local potentials, but for correlated matter, attention is all we ever needed.
Acknowledgments
I thank Christoph Ortner, Matthew Foulkes and Frank Krüger for fruitful discussion that led directly to the realisations described in this letter. J.M.F. is supported by a Royal Society University Research Fellowship (URF-R1-191292). Julia codes implementing Spindrift and the fractal backflow benchmarks are available on GitHub as part of Halcyon.jl[25]. I gratefully acknowledge the use of the Imperial College Research Computing Service[26].
References
- [1]J. Herrmann, J. Spencer, K. Choo, A. Mezzacapo, W. M. C. Foulkes, D. Pfau, G. Carleo, and F. Noé, Ab initio quantum chemistry with neural-network wavefunctions, Nature Reviews Chemistry 7, 692 (2023).
- [2]G. Cassella, H. Sutterud, S. Azadi, N. D. Drummond, D. Pfau, J. S. Spencer, and W. M. C. Foulkes, Discovering quantum phase transitions with fermionic neural networks, Physical Review Letters 130, 036401 (2023).DOI
- [3]I. Batatia, D. P. Kovacs, G. N. C. Simm, C. Ortner, and G. Csányi, MACE: Higher order equivariant message passing neural networks for fast and accurate force fields, in Advances in Neural Information Processing Systems, Vol. 35 (Curran Associates, Inc., 2022) pp. 11423–11436.
- [4]I. von Glehn, J. S. Spencer, and D. Pfau, A self-attention ansatz for ab-initio quantum chemistry, in International Conference on Learning Representations (ICLR) (2023).arxiv.org/abs/2211.13672
- [5]D. Pfau, J. S. Spencer, A. G. D. G. Matthews, and W. M. C. Foulkes, Ab initio solution of the many-electron Schrödinger equation with deep neural networks, Physical Review Research 2, 033429 (2020).
- [6]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention is all you need, in Advances in Neural Information Processing Systems, Vol. 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Curran Associates, Inc., 2017).arxiv.org/abs/1706.03762
- [7]M. J. Willatt, F. Musil, and M. Ceriotti, Atom-density representations for machine learning, The Journal of Chemical Physics 150, 154110 (2019).DOI
- [8]P.-O. Löwdin, Quantum theory of many-particle systems. I. physical interpretations by means of density matrices, natural spin-orbitals, and convergence problems in the method of configurational interaction, Physical Review 97, 1474 (1955).
- [9]M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola, Deep sets, in Advances in Neural Information Processing Systems, Vol. 30 (Curran Associates, Inc., 2017).
- [10]R. D. Mattuck, A Guide to Feynman Diagrams in the Many-Body Problem, 2nd ed., Dover Books on Physics and Chemistry (Dover Publications, New York, 1992).
- [11]H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve, V. Greiff, D. Kreil, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter, Hopfield networks is all you need, in International Conference on Learning Representations (ICLR) (2021).arxiv.org/abs/2008.02217
- [12]Y.-H. H. Tsai, S. Bai, M. Yamada, L.-P. Morency, and R. Salakhutdinov, Transformer dissection: An unified understanding of transformer’s attention via the lens of kernel, in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (Association for Computational Linguistics, 2019) pp. 4343–4352.arxiv.org/abs/1908.11775
- [13]S. Cao, Choose a transformer: Fourier or Galerkin, in Advances in Neural Information Processing Systems, Vol. 34 (Curran Associates, Inc., 2021) pp. 24924–24940.arxiv.org/abs/2105.14995
- [14]F. J. Dyson, The S matrix in quantum electrodynamics, Physical Review 75, 1736 (1949).DOI
- [15]V. M. Galitskii and A. B. Migdal, Application of quantum field theory methods to the many body problem, Soviet Physics JETP 7, 96 (1958).
- [16]E. Prodan and W. Kohn, Nearsightedness of electronic matter, Proceedings of the National Academy of Sciences 102, 11635 (2005).pubmed.ncbi.nlm.nih.gov/16087868
- [17]S. Podznyakov and M. Ceriotti, Smooth, exact rotational symmetrization for deep learning on point clouds, in Advances in Neural Information Processing Systems 36, NeurIPS 2023 (Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2023) p. 79469–79501.
- [18]K. T. Schütt, M. Gastegger, A. Tkatchenko, K.-R. Müller, and R. J. Maurer, Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions, Nature Communications 10, 10.1038/s41467-019-12875-0 (2019).
- [19]H. Li, Z. Wang, N. Zou, M. Ye, R. Xu, X. Gong, W. Duan, and Y. Xu, Deep-learning density functional theory hamiltonian for efficient ab initio electronic-structure calculation, Nature Computational Science 2, 367–377 (2022).DOI
- [20]C. Qian, V. Vitartas, J. R. Kermode, and R. J. Maurer, Equivariant electronic hamiltonian prediction with many-body message passing, npj Computational Materials 12, 10.1038/s41524-026-02020-1 (2026).DOI
- [21]R. Barrett, C. Ortner, and J. Westermayr, Transferable machine learning potential x-MACE for excited states using integrated DeepSets (2025), arXiv:2502.12870 [physics.chem-ph].arxiv.org/abs/2502.12870
- [22]F. Krüger and J. Zaanen, Fermionic quantum criticality and the fractal nodal surface, Physical Review B 78, 035104 (2008).
- [23]R. P. Feynman and M. Cohen, Energy spectrum of the excitations in liquid helium, Physical Review 102, 1189 (1956).DOI
- [24]J. M. Frost, Spindrift: Learning quantum degeneracy from thermal purity in restricted path integral Monte Carlo (2026), arXiv:2607.29590 [cond-mat.stat-mech].arxiv.org/abs/2607.29590
- [25]J. M. Frost, HALCYON.jl: Path integral Monte Carlo and continuous normalising flows in Julia, https://github.com/Frost-group/Halcyon.jl (2026).
- [26]Imperial College Research Computing Service, Research computing service facility at Imperial College London, Imperial College London, https://doi.org/10.14469/hpc/2232 (2016).