betaFeedback

WorksComputer Science

A Geometric Theory for Nonequilibrium Transformer Dynamics

Abstract

We develop a variational and geometric theory based on statistical physics for the joint evolution of Transformer attention and token representations at fixed model parameters. To this end, we introduce a free energy for the attention and model the evolution of the attention weights by its Fisher–Rao gradient flow. At fixed token representations, we prove that the softmax matrix is the unique free-energy minimiser and that every positive attention trajectory converges to it. For prescribed time-dependent representations, we show that the attention dynamics becomes linear after a change to logit variables. This result yields tracking bounds relative to the instantaneous softmax and sufficient conditions for attention to converge to a fixed matrix. Hamiltonian dynamics for the representations and their momenta are then coupled to the attention relaxation driven by the same energy. This coupling results in an energy-dissipation law. We also develop two alternative Hamiltonians incorporating the attention value map. We prove global well-posedness of the attention flow and all three coupled systems.

Introduction

Transformers underpin large language models (LLMs) and occupy a central place in modern artificial intelligence [7]. Their attention mechanism determines how token representations interact. Namely, pairwise scores are converted by softmax into normalised weights, which are then used to combine information from different positions [14]. As the token representations change, so do the scores and the resulting interactions.

Here, we develop a variational and geometric theory based on statistical physics for the joint evolution of attention weights and token representations. We first model attention as a Fisher–Rao gradient flow whose equilibrium solution at fixed representations recovers softmax. When the representations vary, attention retains information from past scores and can depart from the current softmax weights. To close the system, the attention dynamics is coupled to Hamiltonian dynamics for the representations through a common energy.

Framework and main results

We represent each row of the attention matrix AA as a probability distribution over source positions. We define a free energy combining the negative expected attention score with a negative-entropy term, which penalises concentrated attention. For each fixed representation matrix XX, we prove that its unique minimiser is the usual softmax matrix. We express the free-energy gap in terms of relative entropy, whose local quadratic expansion motivates our choice of the Fisher–Rao metric for the attention gradient flow.

For a prescribed C1C^1 representation path X(t)X(t), we evolve A(t)A(t) by the Fisher–Rao gradient flow of the attention free energy. We show that a change to logit variables linearises the dynamics and results in an explicit solution. We prove global existence and uniqueness for positive initial attention, whose positivity and row normalisation are preserved. The solution shows how the attention at a given time depends on an exponentially weighted history of scores. We bound attention’s departure from the instantaneous softmax and give sufficient conditions for convergence to a fixed attention matrix. In particular, for fixed XX, every positive attention trajectory converges to the softmax equilibrium.

To determine the joint dynamics of the representations X(t)X(t) together with attention, we introduce canonical momenta P(t)P(t) and a Hamiltonian comprising the sum of kinetic energy and the free energy of attention. Thus, we couple canonical Hamiltonian dynamics for (X,P)(X,P) to Fisher–Rao gradient descent of the same Hamiltonian in AA. To incorporate the attention value map into the representation dynamics, we develop two alternative Hamiltonian formulations: one penalises discrepancies between representations and attended values, while the other sets X˙\dot{X} equal to the attention output. For all three systems, we derive an energy-dissipation law and prove global existence and uniqueness for positive initial attention, while preserving positivity and row normalisation.

Relation to existing work

Continuous-time models of self-attention treat token representations as interacting particles, with attention weights computed from their current features. This approach has revealed clustering and metastability of the representations [5, 6, 7]. Burger et al. [2025] develop a Wasserstein-type gradient-flow interpretation of mean-field self-attention with normalised representations, under symmetry and compatibility conditions on the interaction and value matrices. Their geometry acts on distributions of representations on the sphere. In our formulation, by contrast, the Fisher–Rao geometry acts on distributions over source positions, namely the attention rows. We give these rows their own evolution and couple it to Hamiltonian dynamics for the representations through a common energy. Our construction also requires no symmetry of the query–key interaction matrix.

Our attention flow (14) is an entropy-adjusted replicator equation in each row. Coucheney et al. [2015] show that these equations are equivalent to accumulating payoffs with exponential discounting and then applying a logit (softmax) choice map. At the level of this row equation, source positions correspond to actions, attention weights to action probabilities, and the scores sij(X(t))s_{ij}(X(t)) to payoffs. Mertikopoulos and Sandholm [2018] place replicator dynamics in a broader class of game dynamics constructed from Riemannian metrics on strategy simplices. The Shahshahani metric yielding replicator dynamics in their framework is the Fisher–Rao metric used here. In these game models, payoffs depend on strategy distributions. Our scores, however, are generated by token representations X(t)X(t), which we either prescribe or evolve together with attention and momenta through a common Hamiltonian.

Scope and organisation

The structure of the paper is as follows.

  • Section 2 defines the attention model and free energy, proves the softmax minimisation result, and motivates the Fisher–Rao metric through relative entropy.

  • Section 3 introduces the geometric tools, derives the Fisher–Rao gradient on the attention space, and relates Fisher–Rao gradient flows to logit dynamics.

  • Section 4 derives the explicit attention solution for prescribed representations, establishes global well-posedness and tracking estimates, and studies convergence and the free-energy balance.

  • Section 5 develops the canonical Hamiltonian coupling and two alternatives incorporating the value map, derives their energy-dissipation laws, and proves global well-posedness for all three systems.

  • Section 6 summarises the results and discusses directions for further work.

Appendix A collects supporting geometric proofs and entropy interpretations of the Fisher–Rao metric. Appendix B defines the Transformer residual blocks and establishes the one-step consistency of their scaled updates with a continuous-depth equation.

The attention model

Features and scores

A Transformer processes a sequence of N≥2N \ge2 tokens. For example, a language model splits an input text into tokens, such as words, parts of words or punctuation, and represents each token by a vector in a feature space Rd\mathbb{R}^{d}, with d≥1d \ge1. Successive Transformer layers update these vectors using information from the token sequence. We call them token representations or feature vectors.

At a given Transformer layer, we collect the current representations of the token sequence in X=(x1,…,xN)∈X:=Rd×NX = (x_1,\ldots,x_N) \in\mathcal{X} := \mathbb{R}^{d \times N}, where column ii is the feature vector of token ii, for i=1,…,Ni = 1,\ldots,N. Fix a query/key dimension dk≥1d_k \ge1. We consider a single unmasked attention head, with matrices

WQ,WK∈Rdk×d,WV∈Rd×d.W_Q, W_K \in\mathbb{R}^{d_k \times d}, \qquad W_V \in\mathbb{R}^{d \times d}.

These matrices remain fixed throughout the attention and Hamiltonian models developed in the main text. At that layer, the attention mechanism defines the query, key and value at position ii, respectively, by

qi=WQxi∈Rdk,ki=WKxi∈Rdk,vi=WVxi∈Rd,i=1,…,N.q_i = W_Qx_i \in\mathbb{R}^{d_k}, \qquad k_i = W_Kx_i \in\mathbb{R}^{d_k}, \qquad v_i = W_Vx_i \in\mathbb{R}^{d}, \qquad i = 1,\ldots,N.

Informally, the query qiq_i describes what information the position ii seeks, the key kik_i provides a descriptor against which queries are matched, and the value viv_i carries the information to be combined in the output. The scaled inner product of the query at position ii and the key at position jj defines their attention score sij(X)s_{ij}(X). Collecting these scores gives the map S:X→RN×NS : \mathcal{X} \to\mathbb{R}^{N \times N}, defined by

M:=WQTWKdk,S(X):=XTMX,M := \frac{W_Q^{\mathsf{T}}W_K}{\sqrt{d_k}}, \qquad S(X) := X^{\mathsf{T}}MX,
sij(X)=xiTMxj=qiTkjdk,i,j=1,…,N.s_{ij}(X) = x_i^{\mathsf{T}}Mx_j = \frac{q_i^{\mathsf{T}}k_j}{\sqrt{d_k}}, \qquad i,j = 1,\ldots,N.

Here ii is called the receiving position and jj is called the source position. For a fixed temperature τ>0\tau> 0, softmax converts each row of scores into attention weights:

Aij∗(X)=exp⁡(sij(X)/τ)∑k=1Nexp⁡(sik(X)/τ),i,j=1,…,N.(1)A_{ij}^{*}(X) = \frac{\exp(s_{ij}(X)/\tau)}{\sum_{k=1}^{N} \exp(s_{ik}(X)/\tau)}, \qquad i,j = 1,\ldots,N. \tag*{(1)}

For fixed scores, lower temperatures τ\tau favour the largest scores more strongly, while higher temperatures give more uniform weights.

For each receiving position ii, the weights Aij∗(X)A_{ij}^{*}(X) sum to one over source positions, and the output is the weighted average of the values, ∑j=1NAij∗(X)vj\sum_{j=1}^{N} A_{ij}^{*}(X)v_j. Since attention is unmasked, all positions, including ii itself, contribute. Figure 1 summarises the attention computation, and Figure 2 illustrates one receiving row. Appendix B describes the residual blocks that update XX and their continuous-depth scaling.

Self-attention for a fixed receiving position

Figure 1. Self-attention for a fixed receiving position ii. Its query is compared with the keys at all source positions j=1,…,Nj = 1,\ldots,N. Softmax converts the scores into weights, which are then used to combine the value vectors.

An illustrative attention row for the phrase “This geometric formulation deserves attention”

Figure 2. An illustrative attention row for the phrase “This geometric formulation deserves attention”, treating each word as one token. The receiving position is i=4i = 4; the arrow labels give the chosen weights A4j∗A^*_{4j}.

Independent attention and its energy

We allow attention to adjust over time instead of imposing (1) at each instant. We consider the following state space.

Definition 2.1 (Attention state).

A={A∈RN×N:Aij>0,∑j=1NAij=1 for every i}.\mathcal{A} = \left\{ A \in\mathbb{R}^{N \times N} : A_{ij} > 0,\quad\sum_{j=1}^{N} A_{ij} = 1\ \text{for every } i \right\}.

Each row of an attention matrix A∈AA \in\mathcal{A} is a strictly positive probability vector: its entries are positive and sum to one. It therefore specifies a probability distribution over source positions. Formula (1) defines the map A∗:X→AA^* : \mathcal{X} \to\mathcal{A}. For the independent state AA, the feature attention output is WVXATWV X A^{\mathsf{T}}. The star in A∗(X)A^*(X) distinguishes the algebraic softmax weights from AA, which we will model as a dynamical variable.

For each receiving position i=1,…,Ni = 1,\ldots,N, define

a(i)=(Ai1,…,AiN)T,si(X)=(si1(X),…,siN(X))T.a^{(i)} = (A_{i1},\ldots,A_{iN})^{\mathsf{T}}, \qquad s_i(X) = (s_{i1}(X),\ldots,s_{iN}(X))^{\mathsf{T}}.

For a source-dependent quantity b=(b1,…,bN)b = (b_1,\ldots,b_N), define its expectation under attention row ii by

Ei[b]:=∑j=1NAijbj.\mathbb{E}_i[b] := \sum_{j=1}^{N} A_{ij} b_j.

Logarithms of vectors are taken entrywise.

Definition 2.2 (Attention free energy). For A∈AA \in\mathcal{A} and X∈XX \in\mathcal{X}, define the attention free energy by

F(A;X)=−∑i,j=1NAijsij(X)+τ∑i,j=1NAijlog⁡Aij(2)\mathcal{F}(A;X)=-\sum_{i,j=1}^{N}A_{ij}s_{ij}(X)+\tau\sum_{i,j=1}^{N}A_{ij}\log A_{ij} \tag*{(2)}
=∑i=1N(−Ei[si(X)]+τEi[log⁡a(i)]).=\sum_{i=1}^{N}\left(-\mathbb{E}_{i}[s_i(X)]+\tau\mathbb{E}_{i}[\log a^{(i)}]\right).

For each receiving row i∈{1,…,N}i \in\{1,\ldots,N\}, minimising the score term −Ei[si(X)]-\mathbb{E}_{i}[s_i(X)] is equivalent to maximising the attention-weighted score Ei[si(X)]\mathbb{E}_{i}[s_i(X)]. The term τEi[log⁡a(i)]\tau\mathbb{E}_{i}[\log a^{(i)}] is minus τ\tau times the row entropy, so minimising it favours higher entropy and discourages concentrated attention. It acts as an entropic regulariser, with τ\tau controlling its strength relative to score maximisation.

Consider now each term on its own, with XX fixed. The score term has infimum −max⁡1≤j≤Nsij(X)-\max_{1\le j\le N}s_{ij}(X), approached by concentrating attention on sources with the largest score. The entropy term alone is uniquely minimised by the uniform row Aij=1/NA_{ij}=1/N for j=1,…,Nj=1,\ldots,N, with value −τlog⁡N-\tau\log N.

Variational softmax

The equilibrium state of the free energy may be characterised by relative entropy.

Definition 2.3 (Relative entropy). Let n≥2n \ge2 and let p,q∈(0,∞)np,q \in(0,\infty)^n satisfy ∑j=1npj=∑j=1nqj=1\sum_{j=1}^{n}p_j=\sum_{j=1}^{n}q_j=1. Define relative entropy DKL(p∥q)\mathrm{D}_{\mathrm{KL}}(p\|q) as

DKL(p∥q):=∑j=1npjlog⁡pjqj.\mathrm{D}_{\mathrm{KL}}(p\|q):=\sum_{j=1}^{n}p_j\log\frac{p_j}{q_j}.

For outcomes drawn from pp, relative entropy measures the average extra information required by an ideal encoding based on qq rather than pp. With natural logarithms, it is measured in nats. It is nonnegative, with equality only for p=qp=q, but is generally asymmetric and is not a metric; see Appendix § 9.§ 3 for the nonnegativity proof.

References

  1. [1]S.-i. Amari. Information geometry and its applications. Springer, 2016.
  2. [2]S.-i. Amari and H. Nagaoka. Methods of information geometry, volume 191. American Mathematical Soc., 2000.
  3. [4]P. Coucheney, B. Gauj al, and P. Mertikopoulos. Penalty-regulated dynamics and robust learning procedures in games. Mathematics of Operations Research, 40(3):611–633, 2015. URL https://arxiv.org/abs/1303.2270.
  4. [5]B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet. The emergence of clusters in self-attention dynamics. In Advances in Neural Information Processing Systems, volume 36, pages 57026–57037, 2023.DOI
  5. [6]B. Geshkovski, H. Koubbi, Y. Polyanskiy, and P. Rigollet. Dynamic metastability in the self-attention model. arXiv preprint arXiv:2410.06833, 2024.arxiv.org/abs/2410.06833
  6. [7]B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet. A mathematical perspective on transformers. Bulletin (New Series) of the American Mathematical Society, 62(3):427–479, 2025.arxiv.org/abs/2312.10794
  7. [8]B. Hoover, Y. Liang, B. Pham, R. Panda, H. Strobel t, D. H. Chau, M. J. Zaki, and D. Krotov. Energy Transformer. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://arxiv.org/abs/2302.07253.
  8. [10]T. Nguyen, R. G. Baraniuk, R. M. Kirby, S. J. Osher, and B. Wang. Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization. arXiv preprint arXiv:2208.00579, 2022. URL https://arxiv.org/abs/2208.00579.
  9. [11]F. Nielsen, K. Sun, and S. Marchand-Maillet. On Hölder projective divergences. Entropy, 19(3): 122, 2017. URL https://franknielsen.github.io/papers/entropy-19-00122-v2.pdf.
  10. [12]S. Pathiraja and P. Wacker. Connections between sequential Bayesian inference and evolutionary dynamics. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 383(2298):20240241, 2025. doi: 10.1098/rsta.2024.0241.
  11. [13]V. Stein, L. Maier, W. Li, and G. Steidl. SympFormer: Inertial Attention Blocks from Damped Hamiltonian Dynamics on Spaces of Probability Measures. arXiv preprint arXiv:2610.31600, 2026. URL https://arxiv.org/abs/2610.31600.
  12. [14]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1706.03762

Paper details

vlnc:W_q73h4w3t
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

large language models

View statement
Looking for related papers…
Contents