betaFeedback

WorksMathematics

Free energy of the Hopfield model

Abstract

We identify the limit of the free energy of the Hopfield model with independent symmetric Bernoulli patterns, for every temperature and loading ratio. A Hubbard-Stratonovich transformation reduces the problem to that of identifying the limit free energy of a bipartite spin glass with ±1\pm 1 and spherical spins. One key difficulty is that the patterns, which become the coupling coefficients in the bipartite model, are Bernoulli instead of Gaussian, and the limit free energy is sensitive to the law of these coefficients. We show that the model is in fact equivalent to one involving Gaussian patterns and also subject to a finite number of Bernoulli random fields. We then solve this model by adapting the techniques recently introduced to identify the limit free energies of spin glasses with either ±1\pm 1 or spherical spins.

Introduction

The Hopfield model [35, 38, 45] describes the behavior of a collection of NN binary variables σ=(σ1,…,σN)∈{−1,1}N\sigma=(\sigma_1,\ldots,\sigma_N)\in\{-1,1\}^N representing neurons that interact with a given collection of patterns x1,…,xMN∈{−1,1}Nx^1,\ldots,x^{M_N}\in\{-1,1\}^N meant to be memorized. To be precise, we give ourselves a sequence (MN)N∈N(M_N)_{N\in\mathbb{N}} of nonnegative integers, and let (xiμ)i≤N,μ≤MN(x_i^\mu)_{i\leq N,\mu\leq M_N} be independent random variables, each uniform on {−1,1}\{-1,1\}; we call such random variables Bernoulli variables. We assume that

lim⁡N→∞MNN=α∈(0,∞),(1)\lim_{N\to\infty}\frac{M_N}{N}=\alpha\in(0,\infty), \tag*{(1)}

and call the parameter α\alpha the loading ratio. We write XN:=(xiμ)i≤N,μ≤MNX_N:=(x_i^\mu)_{i\leq N,\mu\leq M_N} for the NN-by-MNM_N matrix of patterns, so that

∣XNTσ∣2=∑μ=1MN(∑i=1Nxiμσi)2=∑i,j=1N∑μ=1MNxiμxjμσiσj,\lvert X_N^{\mathsf{T}}\sigma\rvert^2=\sum_{\mu=1}^{M_N}\left(\sum_{i=1}^N x_i^\mu\sigma_i\right)^2=\sum_{i,j=1}^N\sum_{\mu=1}^{M_N}x_i^\mu x_j^\mu\sigma_i\sigma_j,

and we let PNP_N denote the uniform probability measure on {−1,1}N\{-1,1\}^N. For every inverse temperature β≥0\beta\geq0, we define the random free energy FN(β)F_N(\beta) and its expectation F‾N(β)\overline{F}_N(\beta) by

FN(β):=1Nlog⁡∫exp⁡(β2N∣XNTσ∣2) dPN(σ)andF‾N(β):=EFN(β).(2)F_N(\beta):=\frac{1}{N}\log\int\exp\left(\frac{\beta}{2N}\lvert X_N^{\mathsf{T}}\sigma\rvert^2\right)\,\mathrm{d}P_N(\sigma) \qquad\text{and}\qquad \overline{F}_N(\beta):=\mathbb{E}F_N(\beta). \tag*{(2)}

The idea of the model is that the associated Gibbs measure represents the equilibrium state of the set of neurons, with the synaptic weight between neurons ii and jj being fixed by the patterns as ∑μ=1MNxiμxjμ\sum_{\mu=1}^{M_N}x_i^\mu x_j^\mu, in agreement with Hebb’s rule [33].

Our main result is an explicit variational formula for the limit of the free energy of this model. We first describe the ingredients featured in this formula. For a∈[0,1)a\in[0,1), let P([0,a])\mathcal{P}([0,a]) denote the set of Borel probability measures on [0,a][0,a], and let

P<1:=⋃a∈[0,1)P([0,a])\mathcal{P}_{<1}:=\bigcup_{a\in[0,1)}\mathcal{P}([0,a])

be the set of probability measures on [0,1)[0,1) with support bounded away from 1. For ζ∈P<1\zeta\in\mathcal{P}_{<1}, we define Dζ:[0,1)→(0,∞)D_\zeta:[0,1)\to(0,\infty) and the function E:P<1→R\mathcal{E}:\mathcal{P}_{<1}\to\mathbb{R} by

Dζ(x):=∫x1ζ([0,s]) dsandE(ζ):=12∫01(1Dζ(x)−11−x)dx.(3)D_\zeta(x):=\int_x^1\zeta([0,s])\,\mathrm{d}s \qquad\text{and}\qquad \mathcal{E}(\zeta):=\frac{1}{2}\int_0^1\left(\frac{1}{D_\zeta(x)}-\frac{1}{1-x}\right)\mathrm{d}x. \tag*{(3)}

If supp⁡ζ⊂[0,a]\operatorname{supp}\zeta\subset[0,a] with a<1a<1, then Dζ(x)=1−xD_\zeta(x)=1-x for x∈[a,1)x\in[a,1), so the integrand in (3) vanishes on [a,1)[a,1). We will see in Lemma 2.3 that E\mathcal{E} is the convex conjugate of the spherical initial condition of [29], and that it is strictly convex.

For a≥0a \ge0 and ζ∈P<1\zeta\in\mathcal{P}_{<1}, let ϕa,ζ:[0,1]×R→R\phi_{a,\zeta} : [0,1] \times\mathbb{R} \to\mathbb{R} be the solution of the Parisi equation

−∂sϕa,ζ=a(∂x2ϕa,ζ+ζ([0,s])(∂xϕa,ζ)2),ϕa,ζ(1,x)=log⁡cosh⁡x.(4)-\partial_s\phi_{a,\zeta}=a\left(\partial_x^2\phi_{a,\zeta}+\zeta([0,s])(\partial_x\phi_{a,\zeta})^2\right),\qquad\phi_{a,\zeta}(1,x)=\log\cosh x. \tag*{(4)}

When ζ\zeta is finitely supported, the function s↦ζ([0,s])s \mapsto\zeta([0,s]) is a step function, and the solution of (4) is given by a finite recursion of Gaussian expectations; for a general ζ\zeta, the solution is the limit of these, see § 2. We define Pa:P<1×R→R\mathcal{P}_a:\mathcal{P}_{<1}\times\mathbb{R}\to\mathbb{R} and R:(0,∞)→R\mathcal{R}:(0,\infty)\to\mathbb{R} by

Pa(ζ,h):=ϕa,ζ(0,h)andR(r):=log⁡r−r2−12.(5)\mathcal{P}_a(\zeta,h):=\phi_{a,\zeta}(0,h)\qquad\text{and}\qquad\mathcal{R}(r):=\log r-\frac{r^2-1}{2}. \tag*{(5)}

In particular, P0(ζ,h)=log⁡cosh⁡h\mathcal{P}_0(\zeta,h)=\log\cosh h for every ζ\zeta. The function Pa(⋅,h)\mathcal{P}_a(\cdot,h) is the Parisi functional of the ±1\pm1 spins; the function R\mathcal{R} is the exponential rate per pattern, as N→∞N\to\infty, of the density at rr of the normalized radius ∣z∣/MN\lvert z\rvert/\sqrt{M_N} of a standard Gaussian vector z∈RMNz\in\mathbb{R}^{M_N}, see (129) and (132).

Theorem 1.1 (Free energy of the Hopfield model). Under (1), for every β>0\beta>0, the limit

F(α,β):=lim⁡N→∞F‾N(β)(6)F(\alpha,\beta):=\lim_{N\to\infty}\overline{F}_N(\beta) \tag*{(6)}

exists and is given by

F(α,β)=sup⁡r>00≤h≤βinf⁡ζ∈P<1{αR(r)−h22β+Pαβr2/2(ζ,h)+αE(ζ)}.(7)F(\alpha,\beta)=\sup_{\substack{r>0\\0\le h\le\beta}}\inf_{\zeta\in\mathcal{P}_{<1}}\left\{\alpha\mathcal{R}(r)-\frac{h^2}{2\beta}+\mathcal{P}_{\alpha\beta r^2/2}(\zeta,h)+\alpha\mathcal{E}(\zeta)\right\}. \tag*{(7)}

Moreover, F‾N(β)\overline{F}_N(\beta) converges to F(α,β)F(\alpha,\beta) almost surely and in L1L^1 as NN tends to infinity.

The next proposition collects the properties of the variational problem in (7). We stress that the uniqueness statement concerns the minimizing ζ\zeta for a fixed pair (r,h)(r,h); we do not claim that the maximizing pair (r,h)(r,h) is unique.

Proposition 1.2 (Structure of the variational problem). Let α,β>0\alpha,\beta>0, r>0r>0 and h∈Rh\in\mathbb{R}, and set a:=αβr2/2a:=\alpha\beta r^2/2.

(1) For every b∈[0,1)b\in[0,1), the function ζ↦Pa(ζ,h)+αE(ζ)\zeta\mapsto\mathcal{P}_a(\zeta,h)+\alpha\mathcal{E}(\zeta) is continuous on P([0,b])\mathcal{P}([0,b]) for the topology of weak convergence, and it is strictly convex on P<1\mathcal{P}_{<1}.

(2) The infimum of this function over ζ∈P<1\zeta\in\mathcal{P}_{<1} is attained at a unique ζ\zeta, and this minimizer belongs to P([0,βr2/(1+βr2)])\mathcal{P}([0,\beta r^2/(1+\beta r^2)]).

(3) The supremum in (7) is attained, and its value is unchanged if the constraint 0≤h≤β0\le h\le\beta is replaced by h∈Rh\in\mathbb{R}.

The starting point of our analysis is the observation (often called the Hubbard–Stratonovich transformation) that, denoting by γd\gamma_d the standard Gaussian measure on Rd\mathbb{R}^d, we have

F‾N(β)=1NElog⁡∫exp⁡(βNσTXNz) dγMN(z) dPN(σ).(8)\overline{F}_N(\beta)=\frac{1}{N}\mathbb{E}\log\int\exp\left(\sqrt{\frac{\beta}{N}}\sigma^\mathsf{T}X_Nz\right)\,d\gamma_{M_N}(z)\,dP_N(\sigma). \tag*{(8)}

This turns the Hopfield model into a bipartite spin glass. By decomposing the Gaussian variable into a radial and a spherical part, we can reduce the problem to that of identifying the limit free energy of a bipartite spin glass involving both ±1\pm1 and spherical spins. The identification of the limit free energy of general multi-species spin glasses has recently witnessed significant progress, with the cases of spherical and of ±1\pm1 spins being now fully resolved in [29] and [18, 27] respectively. It may thus seem that, in order to fully resolve the problem, one only needs to combine these results and extend them to a mixed ±1\pm1/spherical setting.

References

  1. [1]Robert J. Adler and Jonathan E. Taylor. Random fields and geometry. Springer Monographs in Mathematics. Springer, New York, 2007.DOI
  2. [2]Elena Agliari, Adriano Barra, Pierluigi Bianco, Alberto Fachechi, and Diego Pallara. The infinite volume limit of the Hopfield neural networks in the replica-symmetric regime. Ann. Henri Poincaré, 2026. Online first.arxiv.org/abs/2409.10145
  3. [3]Elena Agliari, Adriano Barra, Chiara Longo, and Daniele Tantari. Neural networks retrieving Boolean patterns in a sea of Gaussian ones. J. Stat. Phys., 168:1085–1104, 2017.DOI
  4. [4]Daniel J. Amit, Hanoch Gutfreund, and Haim Sompolinsky. Spin-glass models of neural networks. Phys. Rev. A, 32(2):1007–1018, 1985.
  5. [5]Daniel J. Amit, Hanoch Gutfreund, and Haim Sompolinsky. Storing infinite numbers of patterns in a spin-glass model of neural networks. Phys. Rev. Lett., 55(14):1530–1533, 1985.
  6. [6]Daniel J. Amit, Hanoch Gutfreund, and Haim Sompolinsky. Statistical mechanics of neural networks near saturation. Ann. Physics, 173(1):30–67, 1987.DOI
  7. [7]Peter M. Aronow and Patrick Lopatto. Quantitative Parisi formulas and fluctuations in the Sherrington–Kirkpatrick model. Preprint, arXiv:2609.09103, 2026.arxiv.org/abs/2609.09103
  8. [8]Antonio Auffinger and Wei-Kuo Chen. The Parisi formula has a unique minimizer. Comm. Math. Phys., 335(3):1429–1444, 2015.DOI
  9. [9]Adriano Barra, Alberto Bernacchia, Enrica Santucci, and Pierluigi Contucci. On the equivalence of Hopfield networks and Boltzmann machines. Neural Netw., 34:1–9, 2012.arxiv.org/abs/1105.2790
  10. [10]Adriano Barra, Pierluigi Contucci, Emanuele Mingione, and Daniele Tantari. Multi-species mean field spin glasses. Rigorous results. Ann. Henri Poincaré, 16(3):691–708, 2015.arxiv.org/abs/1307.5154
  11. [11]Adriano Barra, Gino Del Ferraro, and Daniele Tantari. Mean field spin glasses treated with PDE techniques. Eur. Phys. J. B, 86(7):Art. 332, 10, 2013.arxiv.org/abs/1304.2623
  12. [12]Adriano Barra, Aldo Di Biasio, and Francesco Guerra. Replica symmetry breaking in mean-field spin glasses through the Hamilton–Jacobi technique. J. Stat. Mech. Theory Exp., 2010(9):P09006, 22, 2010.DOI
  13. [13]Adriano Barra and Fabrizio Durante. Existence of the thermodynamic limit for the Hopfield model. Preprint, arXiv:2610.41938, 2026.arxiv.org/abs/2610.41938
  14. [14]Adriano Barra, Giuseppe Genovese, and Francesco Guerra. The replica symmetric approximation of the analogical neural network. J. Stat. Phys., 140(4):784–796, 2010.DOI
  15. [15]Adriano Barra, Giuseppe Genovese, Francesco Guerra, and Daniele Tantari. How glassy are neural networks? J. Stat. Mech. Theory Exp., 2012(7):P07009, 2012.DOI
  16. [16]Adriano Barra, Giuseppe Genovese, Peter Sollich, and Daniele Tantari. Phase diagram of restricted Boltzmann machines and generalized Hopfield networks with arbitrary priors. Phys. Rev. E, 97(2):022310, 2018.DOI
  17. [17]Adriano Barra and Francesco Guerra. About the ergodic regime in the analogical Hopfield neural networks: moments of the partition function. J. Math. Phys., 49(12):125217, 2008.arxiv.org/abs/0911.3515
  18. [18]Jeanne Boursier and Jean-Christophe Mourrat. Free energy of non-convex multi-species Ising spin glasses with external fields. Preprint, arXiv:2610.18350, 2026.arxiv.org/abs/2610.18350
  19. [19]Anton Bovier and Véronique Gayrard. The retrieval phase of the Hopfield model: a rigorous analysis of the overlap distribution. Probab. Theory Related Fields, 107(1):61–98, 1997.DOI
  20. [20]Anton Bovier and Véronique Gayrard. Hopfield models as generalized random mean field models. In Anton Bovier and Pierre Picco, editors, Mathematical aspects of spin glasses and neural networks, volume 41 of Progr. Probab., pages 3–89. Birkhäuser Boston, Boston, MA, 1998.
  21. [21]Anton Bovier, Véronique Gayrard, and Pierre Picco. Gibbs states of the Hopfield model with extensively many patterns. J. Statist. Phys., 79(1–2):395–414, 1995.DOI
  22. [22]Francesco Camilli, Pierluigi Contucci, and Emanuele Mingione. The solution of the Hopfield model with Gaussian patterns. Preprint, arXiv:2610.32618, 2026.arxiv.org/abs/2610.32618
  23. [23]Philippe Carmona and Yueyun Hu. Universality in Sherrington–Kirkpatrick’s spin glass model. Ann. Inst. H. Poincaré Probab. Statist., 42(2):215–222, 2006.DOI
  24. [24]Sourav Chatterjee. A simple invariance theorem. Preprint, arXiv:math/0508213, 2005.DOI
  25. [25]Hong-Bin Chen. On free energy of non-convex multi-species spin glasses. ALEA Lat. Am. J. Probab. Math. Stat., 23:429–473, 2026.arxiv.org/abs/2411.13342
  26. [26]Hong-Bin Chen, Fu-Hsuan Ho, and Victor Issa. A formula for the free energy of non-convex vector spin glasses. Preprint, arXiv:2610.18744, 2026.arxiv.org/abs/2610.18744
  27. [27]Hong-Bin Chen, Victor Issa, and Jean-Christophe Mourrat. Free energy of non-convex multi-species spin glasses with centered Ising spins. Preprint, arXiv:2606.16636, 2026.arxiv.org/abs/2606.16636
  28. [28]Hong-Bin Chen and Jean-Christophe Mourrat. On the free energy of vector spin glasses with non-convex interactions. Probab. Math. Phys., 6:1–80, 2025.DOI
  29. [29]Hong-Bin Chen and Jean-Christophe Mourrat. Free energy of non-convex multi-species spherical spin glasses. Preprint, arXiv:2609.32710, 2026.arxiv.org/abs/2609.32710
  30. [30]Hong-Bin Chen and Jiaming Xia. Hamilton–Jacobi equations from mean-field spin glasses. Probab. Theory Related Fields, 192(3–4):803–873, 2025.DOI
  31. [31]Andrea Crisanti, Daniel J. Amit, and Hanoch Gutfreund. Saturation level of the Hopfield model for neural network. Europhys. Lett., 2(4):337–341, 1986.
  32. [32]Francesco Guerra. Sum rules for the free energy in the mean field spin glass model. In Mathematical physics in mathematics and physics (Siena, 2000), volume 30 of Fields Inst. Commun., pages 161–170. Amer. Math. Soc., Providence, RI, 2001.DOI
  33. [33]Donald O. Hebb. The organization of behavior: a neuropsychological theory. Wiley, New York, 1949.DOI
  34. [34]Fu-Hsuan Ho. Convergence of the free energy of vector spin glasses with non-convex interactions. Preprint, arXiv:2609.39307, 2026.arxiv.org/abs/2609.39307
  35. [35]John J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A., 79(8):2554–2558, 1982.
  36. [36]Aukosh Jagannath and Ian Tobasco. A dynamic programming approach to the Parisi functional. Proc. Amer. Math. Soc., 144(7):3135–3150, 2016.DOI
  37. [37]Taegyun Kim. A sharp universality dichotomy for the free energy of spherical spin glasses. Preprint, arXiv:2601.08599, 2026.arxiv.org/abs/2601.08599
  38. [38]William A. Little. The existence of persistent states in the brain. Math. Biosci., 19(1–2):101–120, 1974.
  39. [39]Jean-Christophe Mourrat. Nonconvex interactions in mean-field spin glasses. Probab. Math. Phys., 2(2):281–339, 2021.
  40. [40]Jean-Christophe Mourrat. The Parisi formula is a Hamilton–Jacobi equation in Wasserstein space. Canad. J. Math., 74(3):607–629, 2022.DOI
  41. [41]Jean-Christophe Mourrat. Free energy upper bound for mean-field vector spin glasses. Ann. Inst. Poincaré Probab. Stat., 59(3):1143–1182, 2023.arxiv.org/abs/2010.09114
  42. [42]Dmitry Panchenko. The free energy in a multi-species Sherrington–Kirkpatrick model. Ann. Probab., 43(6):3494–3513, 2015.DOI
  43. [43]Leonid Pastur, Mariya Shcherbina, and Brunello Tirozzi. The replica-symmetric solution without replica trick for the Hopfield model. J. Statist. Phys., 74(5–6):1161–1183, 1994.DOI
  44. [44]Leonid A. Pastur and Alexander L. Figotin. Rigorously solvable model of spin glass. JETP Lett., 25(8):323–326, 1977.
  45. [45]Leonid A. Pastur and Alexander L. Figotin. Theory of disordered spin systems. Theoret. and Math. Phys., 35(2):403–414, 1978.DOI
  46. [46]Melhem Sawhney and Mark Sellke. Free energy universality of spherical spin glasses. Preprint, arXiv:2408.13701, 2024.arxiv.org/abs/2408.13701
  47. [47]Mariya Shcherbina and Brunello Tirozzi. The free energy of a class of Hopfield models. J. Statist. Phys., 72(1–2):113–125, 1993.DOI
  48. [48]Maurice Sion. On general minimax theorems. Pacific J. Math., 8(1):171–176, 1958.DOI
  49. [49]Helmut Steffan and Reimer Kühn. Replica symmetry breaking in attractor neural network models. Z. Phys. B, 95(2):249–260, 1994.DOI
  50. [50]Eliran Subag. Geometry of spherical spin glasses. In Susan Friedlander and Yuri Tschinkel, editors, Proceedings of the International Congress of Mathematicians 2026, Volume 6: Invited Lectures (Sections 12–14), pages 102–122. SIAM, 2026.
  51. [51]Michel Talagrand. Concentration of measure and isoperimetric inequalities in product spaces. Inst. Hautes Études Sci. Publ. Math., 81:73–205, 1995.arxiv.org/abs/math/9406212
  52. [52]Michel Talagrand. Rigorous results for the Hopfield model with many patterns. Probab. Theory Related Fields, 110(2):177–276, 1998.DOI
  53. [53]Michel Talagrand. Exponential inequalities and convergence of moments in the replica-symmetric regime of the Hopfield model. Ann. Probab., 28(4):1393–1469, 2000.DOI
  54. [54]Michel Talagrand. Mean field models for spin glasses. Volume I: Basic examples, volume 54 of Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge. Springer-Verlag, Berlin, 2011.
  55. [55]Michel Talagrand. Mean field models for spin glasses. Volume II: Advanced replica-symmetry and low temperature, volume 55 of Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge. Springer, Heidelberg, 2011.
  56. [56]Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. In Yonina C. Eldar and Gitta Kutyniok, editors, Compressed sensing: theory and applications, pages 210–268. Cambridge University Press, 2012.arxiv.org/abs/1011.3027

Paper details

vlnc:W_hefm8qh6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents