Introduction
Transformers underpin large language models (LLMs) and occupy a central place in modern artificial intelligence [7 ] . Their attention mechanism determines how token representations interact. Namely, pairwise scores are converted by softmax into normalised weights, which are then used to combine information from different positions [14 ] . As the token representations change, so do the scores and the resulting interactions.
Here, we develop a variational and geometric theory based on statistical physics for the joint evolution of attention weights and token representations. We first model attention as a Fisher–Rao gradient flow whose equilibrium solution at fixed representations recovers softmax. When the representations vary, attention retains information from past scores and can depart from the current softmax weights. To close the system, the attention dynamics is coupled to Hamiltonian dynamics for the representations through a common energy.
Framework and main results
We represent each row of the attention matrix A A A as a probability distribution over source positions. We define a free energy combining the negative expected attention score with a negative-entropy term, which penalises concentrated attention. For each fixed representation matrix X X X , we prove that its unique minimiser is the usual softmax matrix. We express the free-energy gap in terms of relative entropy, whose local quadratic expansion motivates our choice of the Fisher–Rao metric for the attention gradient flow.
For a prescribed C 1 C^1 C 1 representation path X ( t ) X(t) X ( t ) , we evolve A ( t ) A(t) A ( t ) by the Fisher–Rao gradient flow of the attention free energy. We show that a change to logit variables linearises the dynamics and results in an explicit solution. We prove global existence and uniqueness for positive initial attention, whose positivity and row normalisation are preserved. The solution shows how the attention at a given time depends on an exponentially weighted history of scores. We bound attention’s departure from the instantaneous softmax and give sufficient conditions for convergence to a fixed attention matrix. In particular, for fixed X X X , every positive attention trajectory converges to the softmax equilibrium.
To determine the joint dynamics of the representations X ( t ) X(t) X ( t ) together with attention, we introduce canonical momenta P ( t ) P(t) P ( t ) and a Hamiltonian comprising the sum of kinetic energy and the free energy of attention. Thus, we couple canonical Hamiltonian dynamics for ( X , P ) (X,P) ( X , P ) to Fisher–Rao gradient descent of the same Hamiltonian in A A A . To incorporate the attention value map into the representation dynamics, we develop two alternative Hamiltonian formulations: one penalises discrepancies between representations and attended values, while the other sets X ˙ \dot{X} X ˙ equal to the attention output. For all three systems, we derive an energy-dissipation law and prove global existence and uniqueness for positive initial attention, while preserving positivity and row normalisation.
Relation to existing work
Continuous-time models of self-attention treat token representations as interacting particles, with attention weights computed from their current features. This approach has revealed clustering and metastability of the representations [5 , 6 , 7 ] . Burger et al. [2025] develop a Wasserstein-type gradient-flow interpretation of mean-field self-attention with normalised representations, under symmetry and compatibility conditions on the interaction and value matrices. Their geometry acts on distributions of representations on the sphere. In our formulation, by contrast, the Fisher–Rao geometry acts on distributions over source positions, namely the attention rows. We give these rows their own evolution and couple it to Hamiltonian dynamics for the representations through a common energy. Our construction also requires no symmetry of the query–key interaction matrix.
Our attention flow (14) is an entropy-adjusted replicator equation in each row. Coucheney et al. [2015] show that these equations are equivalent to accumulating payoffs with exponential discounting and then applying a logit (softmax) choice map. At the level of this row equation, source positions correspond to actions, attention weights to action probabilities, and the scores s i j ( X ( t ) ) s_{ij}(X(t)) s ij ( X ( t )) to payoffs. Mertikopoulos and Sandholm [2018] place replicator dynamics in a broader class of game dynamics constructed from Riemannian metrics on strategy simplices. The Shahshahani metric yielding replicator dynamics in their framework is the Fisher–Rao metric used here. In these game models, payoffs depend on strategy distributions. Our scores, however, are generated by token representations X ( t ) X(t) X ( t ) , which we either prescribe or evolve together with attention and momenta through a common Hamiltonian.
Scope and organisation
The structure of the paper is as follows.
Section 2 defines the attention model and free energy, proves the softmax minimisation result, and motivates the Fisher–Rao metric through relative entropy.
Section 3 introduces the geometric tools, derives the Fisher–Rao gradient on the attention space, and relates Fisher–Rao gradient flows to logit dynamics.
Section 4 derives the explicit attention solution for prescribed representations, establishes global well-posedness and tracking estimates, and studies convergence and the free-energy balance.
Section 5 develops the canonical Hamiltonian coupling and two alternatives incorporating the value map, derives their energy-dissipation laws, and proves global well-posedness for all three systems.
Appendix A collects supporting geometric proofs and entropy interpretations of the Fisher–Rao metric. Appendix B defines the Transformer residual blocks and establishes the one-step consistency of their scaled updates with a continuous-depth equation.
The attention model
Features and scores
A Transformer processes a sequence of N ≥ 2 N \ge2 N ≥ 2 tokens. For example, a language model splits an input text into tokens, such as words, parts of words or punctuation, and represents each token by a vector in a feature space R d \mathbb{R}^{d} R d , with d ≥ 1 d \ge1 d ≥ 1 . Successive Transformer layers update these vectors using information from the token sequence. We call them token representations or feature vectors.
At a given Transformer layer, we collect the current representations of the token sequence in X = ( x 1 , … , x N ) ∈ X : = R d × N X = (x_1,\ldots,x_N) \in\mathcal{X} := \mathbb{R}^{d \times N} X = ( x 1 , … , x N ) ∈ X := R d × N , where column i i i is the feature vector of token i i i , for i = 1 , … , N i = 1,\ldots,N i = 1 , … , N . Fix a query/key dimension d k ≥ 1 d_k \ge1 d k ≥ 1 . We consider a single unmasked attention head, with matrices
W Q , W K ∈ R d k × d , W V ∈ R d × d . W_Q, W_K \in\mathbb{R}^{d_k \times d}, \qquad W_V \in\mathbb{R}^{d \times d}. W Q , W K ∈ R d k × d , W V ∈ R d × d .
These matrices remain fixed throughout the attention and Hamiltonian models developed in the main text. At that layer, the attention mechanism defines the query, key and value at position i i i , respectively, by
q i = W Q x i ∈ R d k , k i = W K x i ∈ R d k , v i = W V x i ∈ R d , i = 1 , … , N . q_i = W_Qx_i \in\mathbb{R}^{d_k}, \qquad k_i = W_Kx_i \in\mathbb{R}^{d_k}, \qquad v_i = W_Vx_i \in\mathbb{R}^{d}, \qquad i = 1,\ldots,N. q i = W Q x i ∈ R d k , k i = W K x i ∈ R d k , v i = W V x i ∈ R d , i = 1 , … , N .
Informally, the query q i q_i q i describes what information the position i i i seeks, the key k i k_i k i provides a descriptor against which queries are matched, and the value v i v_i v i carries the information to be combined in the output. The scaled inner product of the query at position i i i and the key at position j j j defines their attention score s i j ( X ) s_{ij}(X) s ij ( X ) . Collecting these scores gives the map S : X → R N × N S : \mathcal{X} \to\mathbb{R}^{N \times N} S : X → R N × N , defined by
M : = W Q T W K d k , S ( X ) : = X T M X , M := \frac{W_Q^{\mathsf{T}}W_K}{\sqrt{d_k}}, \qquad S(X) := X^{\mathsf{T}}MX, M := d k W Q T W K , S ( X ) := X T M X ,
s i j ( X ) = x i T M x j = q i T k j d k , i , j = 1 , … , N . s_{ij}(X) = x_i^{\mathsf{T}}Mx_j = \frac{q_i^{\mathsf{T}}k_j}{\sqrt{d_k}}, \qquad i,j = 1,\ldots,N. s ij ( X ) = x i T M x j = d k q i T k j , i , j = 1 , … , N .
Here i i i is called the receiving position and j j j is called the source position. For a fixed temperature τ > 0 \tau> 0 τ > 0 , softmax converts each row of scores into attention weights:
A i j ∗ ( X ) = exp ( s i j ( X ) / τ ) ∑ k = 1 N exp ( s i k ( X ) / τ ) , i , j = 1 , … , N . (1) A_{ij}^{*}(X) = \frac{\exp(s_{ij}(X)/\tau)}{\sum_{k=1}^{N} \exp(s_{ik}(X)/\tau)}, \qquad i,j = 1,\ldots,N. \tag*{(1)} A ij ∗ ( X ) = ∑ k = 1 N exp ( s ik ( X ) / τ ) exp ( s ij ( X ) / τ ) , i , j = 1 , … , N . (1)
For fixed scores, lower temperatures τ \tau τ favour the largest scores more strongly, while higher temperatures give more uniform weights.
For each receiving position i i i , the weights A i j ∗ ( X ) A_{ij}^{*}(X) A ij ∗ ( X ) sum to one over source positions, and the output is the weighted average of the values, ∑ j = 1 N A i j ∗ ( X ) v j \sum_{j=1}^{N} A_{ij}^{*}(X)v_j ∑ j = 1 N A ij ∗ ( X ) v j . Since attention is unmasked, all positions, including i i i itself, contribute. Figure 1 summarises the attention computation, and Figure 2 illustrates one receiving row. Appendix B describes the residual blocks that update X X X and their continuous-depth scaling.
Figure 1. Self-attention for a fixed receiving position i i i . Its query is compared with the keys at all source positions j = 1 , … , N j = 1,\ldots,N j = 1 , … , N . Softmax converts the scores into weights, which are then used to combine the value vectors.
Figure 2. An illustrative attention row for the phrase “This geometric formulation deserves attention”, treating each word as one token. The receiving position is i = 4 i = 4 i = 4 ; the arrow labels give the chosen weights A 4 j ∗ A^*_{4j} A 4 j ∗ .
Independent attention and its energy
We allow attention to adjust over time instead of imposing (1) at each instant. We consider the following state space.
Definition 2.1 (Attention state).
A = { A ∈ R N × N : A i j > 0 , ∑ j = 1 N A i j = 1 for every i } . \mathcal{A} = \left\{ A \in\mathbb{R}^{N \times N} : A_{ij} > 0,\quad\sum_{j=1}^{N} A_{ij} = 1\ \text{for every } i \right\}. A = { A ∈ R N × N : A ij > 0 , j = 1 ∑ N A ij = 1 for every i } .
Each row of an attention matrix A ∈ A A \in\mathcal{A} A ∈ A is a strictly positive probability vector: its entries are positive and sum to one. It therefore specifies a probability distribution over source positions. Formula (1) defines the map A ∗ : X → A A^* : \mathcal{X} \to\mathcal{A} A ∗ : X → A . For the independent state A A A , the feature attention output is W V X A T WV X A^{\mathsf{T}} W V X A T . The star in A ∗ ( X ) A^*(X) A ∗ ( X ) distinguishes the algebraic softmax weights from A A A , which we will model as a dynamical variable.
For each receiving position i = 1 , … , N i = 1,\ldots,N i = 1 , … , N , define
a ( i ) = ( A i 1 , … , A i N ) T , s i ( X ) = ( s i 1 ( X ) , … , s i N ( X ) ) T . a^{(i)} = (A_{i1},\ldots,A_{iN})^{\mathsf{T}}, \qquad s_i(X) = (s_{i1}(X),\ldots,s_{iN}(X))^{\mathsf{T}}. a ( i ) = ( A i 1 , … , A i N ) T , s i ( X ) = ( s i 1 ( X ) , … , s i N ( X ) ) T .
For a source-dependent quantity b = ( b 1 , … , b N ) b = (b_1,\ldots,b_N) b = ( b 1 , … , b N ) , define its expectation under attention row i i i by
E i [ b ] : = ∑ j = 1 N A i j b j . \mathbb{E}_i[b] := \sum_{j=1}^{N} A_{ij} b_j. E i [ b ] := j = 1 ∑ N A ij b j .
Logarithms of vectors are taken entrywise.
Definition 2.2 (Attention free energy). For A ∈ A A \in\mathcal{A} A ∈ A and X ∈ X X \in\mathcal{X} X ∈ X , define the attention free energy by
F ( A ; X ) = − ∑ i , j = 1 N A i j s i j ( X ) + τ ∑ i , j = 1 N A i j log A i j (2) \mathcal{F}(A;X)=-\sum_{i,j=1}^{N}A_{ij}s_{ij}(X)+\tau\sum_{i,j=1}^{N}A_{ij}\log A_{ij} \tag*{(2)} F ( A ; X ) = − i , j = 1 ∑ N A ij s ij ( X ) + τ i , j = 1 ∑ N A ij log A ij (2) = ∑ i = 1 N ( − E i [ s i ( X ) ] + τ E i [ log a ( i ) ] ) . =\sum_{i=1}^{N}\left(-\mathbb{E}_{i}[s_i(X)]+\tau\mathbb{E}_{i}[\log a^{(i)}]\right). = i = 1 ∑ N ( − E i [ s i ( X )] + τ E i [ log a ( i ) ] ) .
For each receiving row i ∈ { 1 , … , N } i \in\{1,\ldots,N\} i ∈ { 1 , … , N } , minimising the score term − E i [ s i ( X ) ] -\mathbb{E}_{i}[s_i(X)] − E i [ s i ( X )] is equivalent to maximising the attention-weighted score E i [ s i ( X ) ] \mathbb{E}_{i}[s_i(X)] E i [ s i ( X )] . The term τ E i [ log a ( i ) ] \tau\mathbb{E}_{i}[\log a^{(i)}] τ E i [ log a ( i ) ] is minus τ \tau τ times the row entropy, so minimising it favours higher entropy and discourages concentrated attention. It acts as an entropic regulariser, with τ \tau τ controlling its strength relative to score maximisation.
Consider now each term on its own, with X X X fixed. The score term has infimum − max 1 ≤ j ≤ N s i j ( X ) -\max_{1\le j\le N}s_{ij}(X) − max 1 ≤ j ≤ N s ij ( X ) , approached by concentrating attention on sources with the largest score. The entropy term alone is uniquely minimised by the uniform row A i j = 1 / N A_{ij}=1/N A ij = 1/ N for j = 1 , … , N j=1,\ldots,N j = 1 , … , N , with value − τ log N -\tau\log N − τ log N .
Variational softmax
The equilibrium state of the free energy may be characterised by relative entropy.
Definition 2.3 (Relative entropy). Let n ≥ 2 n \ge2 n ≥ 2 and let p , q ∈ ( 0 , ∞ ) n p,q \in(0,\infty)^n p , q ∈ ( 0 , ∞ ) n satisfy ∑ j = 1 n p j = ∑ j = 1 n q j = 1 \sum_{j=1}^{n}p_j=\sum_{j=1}^{n}q_j=1 ∑ j = 1 n p j = ∑ j = 1 n q j = 1 . Define relative entropy D K L ( p ∥ q ) \mathrm{D}_{\mathrm{KL}}(p\|q) D KL ( p ∥ q ) as
D K L ( p ∥ q ) : = ∑ j = 1 n p j log p j q j . \mathrm{D}_{\mathrm{KL}}(p\|q):=\sum_{j=1}^{n}p_j\log\frac{p_j}{q_j}. D KL ( p ∥ q ) := j = 1 ∑ n p j log q j p j .
For outcomes drawn from p p p , relative entropy measures the average extra information required by an ideal encoding based on q q q rather than p p p . With natural logarithms, it is measured in nats. It is nonnegative, with equality only for p = q p=q p = q , but is generally asymmetric and is not a metric; see Appendix § 9 .§ 3 for the nonnegativity proof.
Proposition 2.4 (Softmax minimises the total free energy). Fix X ∈ X X \in\mathcal{X} X ∈ X and τ > 0 \tau>0 τ > 0 . The matrix
A i j ∗ ( X ) = e s i j ( X ) / τ Z i ( X ) , i , j = 1 , … , N , where Z i ( X ) = ∑ k = 1 N e s i k ( X ) / τ , A^{*}_{ij}(X)=\frac{e^{s_{ij}(X)/\tau}}{Z_i(X)},\qquad i,j=1,\ldots,N,\qquad\text{where } Z_i(X)=\sum_{k=1}^{N}e^{s_{ik}(X)/\tau}, A ij ∗ ( X ) = Z i ( X ) e s ij ( X ) / τ , i , j = 1 , … , N , where Z i ( X ) = k = 1 ∑ N e s ik ( X ) / τ , is the unique minimiser of F ( ⋅ ; X ) \mathcal{F}(\cdot;X) F ( ⋅ ; X ) on A \mathcal{A} A . Writing a ( i ) = ( A i 1 , … , A i N ) T a^{(i)}=(A_{i1},\ldots,A_{iN})^{\mathsf{T}} a ( i ) = ( A i 1 , … , A i N ) T and a ∗ ( i ) = ( A i 1 ∗ , … , A i N ∗ ) T a^{*(i)}=(A^{*}_{i1},\ldots,A^{*}_{iN})^{\mathsf{T}} a ∗ ( i ) = ( A i 1 ∗ , … , A i N ∗ ) T for the corresponding rows as column vectors, one has
F ( A ∗ ( X ) ; X ) = − τ ∑ i = 1 N log Z i ( X ) , \mathcal{F}(A^{*}(X);X)=-\tau\sum_{i=1}^{N}\log Z_i(X), F ( A ∗ ( X ) ; X ) = − τ i = 1 ∑ N log Z i ( X ) , F ( A ; X ) − F ( A ∗ ( X ) ; X ) = τ ∑ i = 1 N D K L ( a ( i ) ∥ a ∗ ( i ) ) . (3) \mathcal{F}(A;X)-\mathcal{F}(A^{*}(X);X)=\tau\sum_{i=1}^{N}\mathrm{D}_{\mathrm{KL}}\left(a^{(i)}\|a^{*(i)}\right). \tag*{(3)} F ( A ; X ) − F ( A ∗ ( X ) ; X ) = τ i = 1 ∑ N D KL ( a ( i ) ∥ a ∗ ( i ) ) . (3)
Proof. By (2.2), the objective and its constraints separate by rows. For each receiving row i ∈ { 1 , … , N } i\in\{1,\ldots,N\} i ∈ { 1 , … , N } , define its contribution to the free energy on the positive orthant by
F i ( a ; X ) : = ∑ j = 1 N a j [ − s i j ( X ) + τ log a j ] , a ∈ ( 0 , ∞ ) N . \mathcal{F}_i(a;X):=\sum_{j=1}^{N}a_j[-s_{ij}(X)+\tau\log a_j],\qquad a\in(0,\infty)^N. F i ( a ; X ) := j = 1 ∑ N a j [ − s ij ( X ) + τ log a j ] , a ∈ ( 0 , ∞ ) N .
Then F ( A ; X ) = ∑ i = 1 N F i ( a ( i ) ; X ) \mathcal{F}(A;X)=\sum_{i=1}^{N}\mathcal{F}_i(a^{(i)};X) F ( A ; X ) = ∑ i = 1 N F i ( a ( i ) ; X ) . Fix i i i for the row calculation and write s j = s i j ( X ) s_j=s_{ij}(X) s j = s ij ( X ) and π j = A i j ∗ ( X ) \pi_j=A^*_{ij}(X) π j = A ij ∗ ( X ) . To find the stationary points of F i \mathcal{F}_i F i under the constraint ∑ j = 1 N a j = 1 \sum_{j=1}^{N}a_j=1 ∑ j = 1 N a j = 1 , apply Lagrange multipliers to
F ~ i ( a , λ i ; X ) = F i ( a ; X ) + λ i ( ∑ j = 1 N a j − 1 ) . \widetilde{\mathcal{F}}_i(a,\lambda_i;X)=\mathcal{F}_i(a;X)+\lambda_i\left(\sum_{j=1}^{N}a_j-1\right). F i ( a , λ i ; X ) = F i ( a ; X ) + λ i ( j = 1 ∑ N a j − 1 ) .
Its stationary equations are
− s j + τ ( 1 + log a j ) + λ i = 0 ( j = 1 , … , N ) , ∑ j = 1 N a j = 1. -s_j+\tau(1+\log a_j)+\lambda_i=0\quad(j=1,\ldots,N),\qquad\sum_{j=1}^{N}a_j=1. − s j + τ ( 1 + log a j ) + λ i = 0 ( j = 1 , … , N ) , j = 1 ∑ N a j = 1.
The first equations give
log a j = s j τ − 1 − λ i τ , j = 1 , … , N . \log a_j=\frac{s_j}{\tau}-1-\frac{\lambda_i}{\tau},\qquad j=1,\ldots,N. log a j = τ s j − 1 − τ λ i , j = 1 , … , N .
Exponentiating yields a j = C i e s j / τ a_j=C_i e^{s_j/\tau} a j = C i e s j / τ , where C i = e − 1 − λ i / τ C_i=e^{-1-\lambda_i/\tau} C i = e − 1 − λ i / τ is independent of j j j . With Z i = ∑ j = 1 N e s j / τ Z_i=\sum_{j=1}^{N}e^{s_j/\tau} Z i = ∑ j = 1 N e s j / τ , normalisation gives
1 = ∑ j = 1 N a j = C i ∑ j = 1 N e s j / τ = C i Z i . 1=\sum_{j=1}^{N}a_j=C_i\sum_{j=1}^{N}e^{s_j/\tau}=C_iZ_i. 1 = j = 1 ∑ N a j = C i j = 1 ∑ N e s j / τ = C i Z i .
Hence C i = Z i − 1 C_i=Z_i^{-1} C i = Z i − 1 and λ i = τ ( log Z i − 1 ) \lambda_i=\tau(\log Z_i-1) λ i = τ ( log Z i − 1 ) . The stationary point is therefore the softmax row a = π a=\pi a = π from (2.1).
To establish global minimality, we substitute log π j = s j / τ − log Z i \log\pi_j=s_j/\tau-\log Z_i log π j = s j / τ − log Z i in the definition of F i \mathcal{F}_i F i above. Since ∑ j = 1 N π j = 1 \sum_{j=1}^{N}\pi_j=1 ∑ j = 1 N π j = 1 , we have
F i ( π ; X ) = ∑ j = 1 N π j ( − s j + τ log π j ) = − τ log Z i ∑ j = 1 N π j = − τ log Z i . \mathcal{F}_i(\pi;X)=\sum_{j=1}^{N}\pi_j(-s_j+\tau\log\pi_j)=-\tau\log Z_i\sum_{j=1}^{N}\pi_j=-\tau\log Z_i. F i ( π ; X ) = j = 1 ∑ N π j ( − s j + τ log π j ) = − τ log Z i j = 1 ∑ N π j = − τ log Z i .
For any positive probability vector a a a , the same identity and Definition 2.3 give
F i ( a ; X ) − F i ( π ; X ) = τ ∑ j = 1 N a j ( log a j − s j τ + log Z i ) = τ ∑ j = 1 N a j log a j π j = τ D K L ( a ∥ π ) . \begin{aligned}
\mathcal{F}_i(a;X)-\mathcal{F}_i(\pi;X)=\tau\sum_{j=1}^{N}a_j\left(\log a_j-\frac{s_j}{\tau}+\log Z_i\right) \\
&=\tau\sum_{j=1}^{N}a_j\log\frac{a_j}{\pi_j}=\tau D_{\mathrm{KL}}(a\Vert\pi).
\end{aligned} F i ( a ; X ) − F i ( π ; X ) = τ j = 1 ∑ N a j ( log a j − τ s j + log Z i ) = τ j = 1 ∑ N a j log π j a j = τ D KL ( a ∥ π ) .
Proposition A.1 gives nonnegativity of this gap, with equality only for a = π a=\pi a = π . Summing over receiving rows proves both identities in (2.3) and uniqueness of the minimiser on A \mathcal{A} A . □
From free energy to Fisher–Rao geometry
For fixed X ∈ X X\in\mathcal{X} X ∈ X , (2.3) expresses F ( A ; X ) − F ( A ∗ ( X ) ; X ) \mathcal{F}(A;X)-\mathcal{F}(A^*(X);X) F ( A ; X ) − F ( A ∗ ( X ) ; X ) as τ ∑ i = 1 N D K L ( a ( i ) ∥ a ∗ ( i ) ) \tau\sum_{i=1}^{N}D_{\mathrm{KL}}(a^{(i)}\Vert a^{*(i)}) τ ∑ i = 1 N D KL ( a ( i ) ∥ a ∗ ( i ) ) . A variation A ε = A ∗ ( X ) + ε B A_\varepsilon=A^*(X)+\varepsilon B A ε = A ∗ ( X ) + εB preserves row normalisation precisely when
∑ j = 1 N B i j = 0 , i = 1 , … , N , \sum_{j=1}^{N}B_{ij}=0,\qquad i=1,\ldots,N, j = 1 ∑ N B ij = 0 , i = 1 , … , N ,
since each row of A ∗ ( X ) A^*(X) A ∗ ( X ) already sums to one. For such a matrix B ∈ R N × N B\in\mathbb{R}^{N\times N} B ∈ R N × N , strict positivity of A ∗ ( X ) A^*(X) A ∗ ( X ) ensures that A ε ∈ A A_\varepsilon\in\mathcal{A} A ε ∈ A for sufficiently small ∣ ε ∣ |\varepsilon| ∣ ε ∣ . Write b ( i ) = ( B i 1 , … , B i N ) T b^{(i)}=(B_{i1},\ldots,B_{iN})^{\mathsf{T}} b ( i ) = ( B i 1 , … , B i N ) T . Using the local expansion from Proposition A.1 in Appendix § 9 .§ 3 , we obtain
F ( A ε ; X ) − F ( A ∗ ( X ) ; X ) = τ ∑ i = 1 N D K L ( a ∗ ( i ) + ε b ( i ) ∥ a ∗ ( i ) ) = τ ε 2 2 ∑ i = 1 N g a ∗ ( i ) ( b ( i ) , b ( i ) ) + O ( ε 3 ) = τ ε 2 2 ∑ i , j = 1 N B i j 2 A i j ∗ ( X ) + O ( ε 3 ) = τ ε 2 2 g A ∗ ( X ) ( B , B ) + O ( ε 3 ) , (4) \begin{aligned}
\mathcal{F}(A_\varepsilon; X) - \mathcal{F}(A^*(X); X) = \tau\sum_{i=1}^{N} D_{\mathrm{KL}}(a^{*(i)}+\varepsilon b^{(i)} \mathbin\Vert a^{*(i)}) \\
&= \frac{\tau\varepsilon^2}{2}\sum_{i=1}^{N} g_{a^{*(i)}}(b^{(i)},b^{(i)})+O(\varepsilon^3) \\
&= \frac{\tau\varepsilon^2}{2}\sum_{i,j=1}^{N}\frac{B_{ij}^{2}}{A_{ij}^{*}(X)}+O(\varepsilon^3) \\
&= \frac{\tau\varepsilon^2}{2}g_{A^*(X)}(B,B)+O(\varepsilon^3), \tag*{(4)}
\end{aligned} F ( A ε ; X ) − F ( A ∗ ( X ) ; X ) = τ i = 1 ∑ N D KL ( a ∗ ( i ) + ε b ( i ) ∥ a ∗ ( i ) ) = 2 τ ε 2 i = 1 ∑ N g a ∗ ( i ) ( b ( i ) , b ( i ) ) + O ( ε 3 ) = 2 τ ε 2 i , j = 1 ∑ N A ij ∗ ( X ) B ij 2 + O ( ε 3 ) = 2 τ ε 2 g A ∗ ( X ) ( B , B ) + O ( ε 3 ) , (4)
where g p ( v , v ) = ∑ j = 1 N v j 2 / p j g_p(v,v)=\sum_{j=1}^{N}v_j^2/p_j g p ( v , v ) = ∑ j = 1 N v j 2 / p j for a positive probability vector p ∈ R N p\in\mathbb{R}^{N} p ∈ R N and v ∈ R N v\in\mathbb{R}^{N} v ∈ R N with ∑ j = 1 N v j = 0 \sum_{j=1}^{N}v_j=0 ∑ j = 1 N v j = 0 . This is the Fisher–Rao squared length for one probability vector, and
g A ∗ ( X ) ( B , B ) : = ∑ i = 1 N g a ∗ ( i ) ( b ( i ) , b ( i ) ) = ∑ i , j = 1 N B i j 2 A i j ∗ ( X ) g_{A^*(X)}(B,B):=\sum_{i=1}^{N}g_{a^{*(i)}}(b^{(i)},b^{(i)})=\sum_{i,j=1}^{N}\frac{B_{ij}^{2}}{A_{ij}^{*}(X)} g A ∗ ( X ) ( B , B ) := i = 1 ∑ N g a ∗ ( i ) ( b ( i ) , b ( i ) ) = i , j = 1 ∑ N A ij ∗ ( X ) B ij 2
is the corresponding squared length on the attention space. § 3 constructs these metrics; the full attention metric is given in (11) . Thus the leading term of the free-energy gap is proportional to the squared Fisher–Rao length of the variation. This motivates our choice of the Fisher–Rao gradient flow for attention relaxation.
Geometry of attention
The expression g A ∗ ( X ) ( B , B ) g_{A^*(X)}(B,B) g A ∗ ( X ) ( B , B ) in (4) motivates using the Fisher–Rao metric to measure changes of attention. We now construct this metric and its gradient on the attention space A A A . The geometric constructions below are classical; Appendix § 9 collects their supporting proofs and entropy interpretations. The Fisher–Rao conventions follow Amari and Nagaoka [ [2 ] and Amari [ [1 ] .
Riemannian metrics and gradients
A smooth state space is represented by a manifold M M M . Its tangent space T p M T_pM T p M consists of tangent vectors: the instantaneous velocities of curves through p ∈ M p\in M p ∈ M . A Riemannian metric is a smoothly varying inner product on each T p M T_pM T p M . It generalises the Euclidean dot product and makes lengths, angles and gradients depend on the state p ∈ M p\in M p ∈ M .
Definition 3.1 (Riemannian metric). Let M M M be a smooth finite-dimensional manifold. A Riemannian metric g g g assigns to each p ∈ M p\in M p ∈ M a symmetric bilinear form g p : T p M × T p M → R g_p:T_pM\times T_pM\to\mathbb{R} g p : T p M × T p M → R such that g p ( v , v ) > 0 g_p(v,v)>0 g p ( v , v ) > 0 for every nonzero v ∈ T p M v\in T_pM v ∈ T p M , and the assignment is smooth in local coordinates.
We write ∥ v ∥ g , p 2 = g p ( v , v ) \lVert v\rVert_{g,p}^{2}=g_p(v,v) ∥ v ∥ g , p 2 = g p ( v , v ) . In a coordinate chart z = ( z 1 , … , z r ) : U → z ( U ) ⊂ R r z=(z^1,\ldots,z^r):U\to z(U)\subset\mathbb{R}^r z = ( z 1 , … , z r ) : U → z ( U ) ⊂ R r , with U ⊂ M U\subset M U ⊂ M and r = dim M r=\dim M r = dim M , the metric at p ∈ U p\in U p ∈ U is represented by
G ( z ( p ) ) = ( g α β ( p ) ) α , β = 1 r , g α β ( p ) = g p ( ∂ z α ∣ p , ∂ z β ∣ p ) . G(z(p))=(g_{\alpha\beta}(p))_{\alpha,\beta=1}^{r},\qquad g_{\alpha\beta}(p)=g_p(\partial_{z^\alpha}|_p,\partial_{z^\beta}|_p). G ( z ( p )) = ( g α β ( p ) ) α , β = 1 r , g α β ( p ) = g p ( ∂ z α ∣ p , ∂ z β ∣ p ) . Thus G G G is symmetric and positive definite at every point, and its entries depend smoothly on the coordinates.
Definition 3.2 (Metric identifications). Let T p ∗ M T_p^*M T p ∗ M denote the dual vector space of T p M T_pM T p M . For each p ∈ M p\in M p ∈ M , define
g p ♭ : T p M ⟶ T p ∗ M , v ⟼ g p ( v , ⋅ ) , g p ♯ : = ( g p ♭ ) − 1 . g_p^\flat:T_pM\longrightarrow T_p^*M,\qquad v\longmapsto g_p(v,\cdot),\qquad g_p^\sharp:=(g_p^\flat)^{-1}. g p ♭ : T p M ⟶ T p ∗ M , v ⟼ g p ( v , ⋅ ) , g p ♯ := ( g p ♭ ) − 1 .
Bilinearity makes g p ♭ g_p^\flat g p ♭ linear, and positive definiteness makes it injective. Therefore, since T p M T_pM T p M and T p ∗ M T_p^*M T p ∗ M have the same finite dimension, it is an isomorphism. We also write v ♭ = g p ♭ ( v ) v^\flat= g_p^\flat(v) v ♭ = g p ♭ ( v ) and α ♯ = g p ♯ ( α ) \alpha^\sharp= g_p^\sharp(\alpha) α ♯ = g p ♯ ( α ) for v ∈ T p M v \in T_pM v ∈ T p M and α ∈ T p ∗ M \alpha\in T_p^*M α ∈ T p ∗ M .
For a C 1 C^1 C 1 function F : M → R F:M \to\mathbb{R} F : M → R , the differential d F p : T p M → R dF_p:T_pM \to\mathbb{R} d F p : T p M → R is the linear map defined by
d F p ( v ) = d d s F ( σ ( s ) ) ∣ s = 0 , σ ( 0 ) = p , σ ˙ ( 0 ) = v . dF_p(v) = \left.\frac{d}{ds}F(\sigma(s))\right|_{s=0}, \qquad\sigma(0)=p,\quad\dot{\sigma}(0)=v. d F p ( v ) = d s d F ( σ ( s )) s = 0 , σ ( 0 ) = p , σ ˙ ( 0 ) = v .
where σ : ( − ε , ε ) → M \sigma:(-\varepsilon,\varepsilon)\to M σ : ( − ε , ε ) → M , ε > 0 \varepsilon>0 ε > 0 , is any C 1 C^1 C 1 curve with the indicated initial point and velocity. On an open set U ⊂ R r U \subset\mathbb{R}^r U ⊂ R r , the Euclidean gradient of a C 1 C^1 C 1 function F : U → R F:U \to\mathbb{R} F : U → R is characterised by the identity
d F x ( v ) = ∇ F ( x ) ⋅ v , x ∈ U , v ∈ R r . dF_x(v) = \nabla F(x)\cdot v,\qquad x\in U,\quad v\in\mathbb{R}^r. d F x ( v ) = ∇ F ( x ) ⋅ v , x ∈ U , v ∈ R r .
The Riemannian gradient extends this identity by replacing the Euclidean dot product with the metric.
Definition 3.3 (Riemannian gradient). Let ( M , g ) (M,g) ( M , g ) be a Riemannian manifold and F : M → R F:M \to\mathbb{R} F : M → R a C 1 C^1 C 1 function. Its Riemannian gradient is the vector field grad g F \operatorname{grad}_g F grad g F characterised by
g p ( grad g F ( p ) , v ) = d F p ( v ) g_p(\operatorname{grad}_g F(p),v)=dF_p(v) g p ( grad g F ( p ) , v ) = d F p ( v ) for every p ∈ M p\in M p ∈ M and v ∈ T p M v\in T_pM v ∈ T p M .
Equivalently, grad g F ( p ) = g p ♯ ( d F p ) = ( d F p ) ♯ \operatorname{grad}_g F(p)=g_p^\sharp(dF_p)=(dF_p)^\sharp grad g F ( p ) = g p ♯ ( d F p ) = ( d F p ) ♯ .
In the local coordinates above, applying the defining identity to each coordinate tangent vector gives
[ grad g F ( p ) ] z = G ( z ( p ) ) − 1 ∇ F ^ ( z ( p ) ) , F ^ : = F ∘ z − 1 . (5) [\operatorname{grad}_g F(p)]_z=G(z(p))^{-1}\nabla\widehat{F}(z(p)),\qquad\widehat{F}:=F\circ z^{-1}. \tag*{(5)} [ grad g F ( p ) ] z = G ( z ( p ) ) − 1 ∇ F ( z ( p )) , F := F ∘ z − 1 . (5)
In particular, grad g F \operatorname{grad}_g F grad g F is continuous when F F F is C 1 C^1 C 1 .
Gradient flows and dissipation
Definition 3.4 (Gradient flow). Fix γ > 0 \gamma>0 γ > 0 . Let ( M , g ) (M,g) ( M , g ) be a Riemannian manifold, I ⊂ R I\subset\mathbb{R} I ⊂ R a time interval, and F : M × I → R F:M\times I\to\mathbb{R} F : M × I → R a function that is C 1 C^1 C 1 jointly in ( p , t ) (p,t) ( p , t ) . For each t ∈ I t\in I t ∈ I , define F t : M → R F_t:M\to\mathbb{R} F t : M → R by F t ( p ) : = F ( p , t ) F_t(p):=F(p,t) F t ( p ) := F ( p , t ) . The gradient flow of F F F with mobility γ \gamma γ is the differential equation
p ˙ ( t ) = − γ grad g F t ( p ( t ) ) . (6) \dot{p}(t)=-\gamma\operatorname{grad}_g F_t(p(t)). \tag*{(6)} p ˙ ( t ) = − γ grad g F t ( p ( t )) . (6) An initial-value problem also prescribes p ( t 0 ) = p 0 ∈ M p(t_0)=p_0\in M p ( t 0 ) = p 0 ∈ M .
Proposition 3.5 (Energy balance along a gradient flow). Every solution of (6) satisfies the following identity on its interval of existence:
d d t F ( p ( t ) , t ) = ∂ t F ( p ( t ) , t ) − γ ∥ grad g F t ( p ( t ) ) ∥ g , p ( t ) 2 . \frac{d}{dt}F(p(t),t)=\partial_tF(p(t),t)-\gamma\left\|\operatorname{grad}_g F_t(p(t))\right\|_{g,p(t)}^2. d t d F ( p ( t ) , t ) = ∂ t F ( p ( t ) , t ) − γ grad g F t ( p ( t )) g , p ( t ) 2 . In particular, if F F F is time independent, it is nonincreasing along the flow.
Proof. The chain rule, (6) and Definition 3.3 give
d d t F ( p ( t ) , t ) = ∂ t F ( p ( t ) , t ) + d ( F t ) p ( t ) ( p ˙ ( t ) ) = ∂ t F ( p ( t ) , t ) − γ g p ( t ) ( grad g F t ( p ( t ) ) , grad g F t ( p ( t ) ) ) . \begin{aligned}
\frac{d}{dt}F(p(t),t)=\partial_tF(p(t),t)+d(F_t)_{p(t)}\bigl(\dot{p}(t)\bigr) \\
&=\partial_tF(p(t),t)-\gamma g_{p(t)}\bigl(\operatorname{grad}_g F_t(p(t)),\operatorname{grad}_g F_t(p(t))\bigr).
\end{aligned} d t d F ( p ( t ) , t ) = ∂ t F ( p ( t ) , t ) + d ( F t ) p ( t ) ( p ˙ ( t ) ) = ∂ t F ( p ( t ) , t ) − γ g p ( t ) ( grad g F t ( p ( t )) , grad g F t ( p ( t )) ) .
Projection and constrained gradients
The orthogonal projection in Euclidean space extends to Riemannian manifolds by using the inner products on their tangent spaces.
Definition 3.6 (Orthogonal projection). Let ( E , g ) (E,g) ( E , g ) be a finite-dimensional inner-product space and V ⊂ E V \subset E V ⊂ E a linear subspace. The g g g -orthogonal projection Π : E → V \Pi: E \to V Π : E → V is determined by
Π v ∈ V , g ( v − Π v , w ) = 0 for every v ∈ E and w ∈ V . \Pi v \in V,\qquad g(v-\Pi v,w)=0\quad\text{for every }v\in E\text{ and }w\in V. Π v ∈ V , g ( v − Π v , w ) = 0 for every v ∈ E and w ∈ V . The orthogonal projection fixes vectors in V V V , satisfies Π 2 = Π \Pi^2=\Pi Π 2 = Π , and is self-adjoint. These properties and existence follow from the orthogonal decomposition E = V ⊕ V ⊥ E=V\oplus V^\perp E = V ⊕ V ⊥ . Figure 3 illustrates this decomposition.
Figure 3. Projection in coordinates orthonormal for g g g . The projected vector Π v \Pi v Π v lies in V V V , and the remainder v − Π v v-\Pi v v − Π v is g g g -orthogonal to V V V .
Definition 3.7 (Induced metric). Let M M M be a smooth embedded submanifold of a Riemannian manifold ( N , h ) (\mathcal{N},h) ( N , h ) , with inclusion ι : M ↪ N \iota:M\hookrightarrow\mathcal{N} ι : M ↪ N . The induced metric is the pullback g = ι ∗ h g=\iota^*h g = ι ∗ h , defined by
g p ( v , w ) = h ι ( p ) ( d ι p ( v ) , d ι p ( w ) ) , p ∈ M , v , w ∈ T p M . g_p(v,w)=h_{\iota(p)}\bigl(d\iota_p(v),d\iota_p(w)\bigr),\qquad p\in M,\quad v,w\in T_pM. g p ( v , w ) = h ι ( p ) ( d ι p ( v ) , d ι p ( w ) ) , p ∈ M , v , w ∈ T p M . Identifying T p M T_pM T p M with its image under d ι p d\iota_p d ι p , Definition 3.6 applies with E = T p N E=T_p\mathcal{N} E = T p N , V = T p M V=T_pM V = T p M and inner product h p h_p h p . From now on, the word “ambient” will refer to the containing manifold N \mathcal{N} N .
Proposition 3.8 (Gradient on an embedded submanifold). Let ( N , h ) (\mathcal{N},h) ( N , h ) be a Riemannian manifold, and let M ⊂ N M\subset\mathcal{N} M ⊂ N be a smooth embedded submanifold equipped with the metric g g g induced by h h h . Let F ‾ : N → R \overline{F}:\mathcal{N}\to\mathbb{R} F : N → R be C 1 C^1 C 1 and set F = F ‾ ∣ M F=\left.\overline{F}\right|_M F = F M . For each p ∈ M p\in M p ∈ M , let Π p : T p N → T p M \Pi_p:T_p\mathcal{N}\to T_pM Π p : T p N → T p M be the h p h_p h p -orthogonal projection. Then
grad g F ( p ) = Π p ( grad h F ‾ ( p ) ) . \operatorname{grad}_g F(p)=\Pi_p\bigl(\operatorname{grad}_h\overline{F}(p)\bigr). grad g F ( p ) = Π p ( grad h F ( p ) ) . See Appendix A.1 for the proof.
Probability vectors and the Fisher–Rao metric
Fix an integer n ≥ 2 n\ge2 n ≥ 2 . Denote the outcome set by Ω n = { 1 , … , n } \Omega_n=\{1,\ldots,n\} Ω n = { 1 , … , n } .
Definition 3.9 (Open probability simplex). The space of strictly positive probability vectors is
P n = { p ∈ ( 0 , ∞ ) n : ∑ j = 1 n p j = 1 } . \mathcal{P}_n=\left\{p\in(0,\infty)^n:\sum_{j=1}^{n}p_j=1\right\}. P n = { p ∈ ( 0 , ∞ ) n : j = 1 ∑ n p j = 1 } . We identify p ∈ P n p\in\mathcal{P}_n p ∈ P n with the probability measure on ( Ω n , 2 Ω n ) (\Omega_n,2^{\Omega_n}) ( Ω n , 2 Ω n ) assigning mass p j p_j p j to each outcome j j j .
Lemma 3.10 (Dimension and tangent space). The simplex P n \mathcal{P}_n P n is a smooth embedded submanifold of U n = ( 0 , ∞ ) n \mathcal{U}_n=(0,\infty)^n U n = ( 0 , ∞ ) n of dimension n − 1 n-1 n − 1 . At each p ∈ P n p\in\mathcal{P}_n p ∈ P n , we have
T p P n = { v ∈ R n : ∑ j = 1 n v j = 0 } . (7) T_p\mathcal{P}_n=\left\{v\in\mathbb{R}^n:\sum_{j=1}^{n}v_j=0\right\}. \tag*{(7)} T p P n = { v ∈ R n : j = 1 ∑ n v j = 0 } . (7)
See Appendix § 9 .§ 2 for the proof.
Definition 3.11 (Expectation). Let V V V be a finite-dimensional real vector space. For p ∈ P n p \in\mathcal{P}_n p ∈ P n and a tuple b = ( b 1 , … , b n ) ∈ V n b=(b_1,\ldots,b_n)\in V^n b = ( b 1 , … , b n ) ∈ V n , define its expectation by
E p [ b ] = ∑ j = 1 n p j b j ∈ V . \mathbb{E}_p[b]=\sum_{j=1}^{n}p_jb_j\in V. E p [ b ] = j = 1 ∑ n p j b j ∈ V .
For a fixed outcome j ∈ Ω n j\in\Omega_n j ∈ Ω n , define the log-probability function
ℓ j : P n ⟶ R , ℓ j ( p ) = log p j . \ell_j:\mathcal{P}_n\longrightarrow\mathbb{R},\qquad\ell_j(p)=\log p_j. ℓ j : P n ⟶ R , ℓ j ( p ) = log p j .
At p ∈ P n p\in\mathcal{P}_n p ∈ P n , its differential is the linear map
d ℓ j ∣ p : T p P n ⟶ R , v ⟼ v j p j . d\ell_j|_p:T_p\mathcal{P}_n\longrightarrow\mathbb{R},\qquad v\longmapsto\frac{v_j}{p_j}. d ℓ j ∣ p : T p P n ⟶ R , v ⟼ p j v j .
Definition 3.12 (Fisher–Rao metric). For p ∈ P n p\in\mathcal{P}_n p ∈ P n and v , w ∈ T p P n v,w\in T_p\mathcal{P}_n v , w ∈ T p P n , set
g p ( v , w ) = E p [ ( d ℓ j ∣ p ( v ) d ℓ j ∣ p ( w ) ) j = 1 n ] = ∑ j = 1 n v j w j p j . (8) g_p(v,w)=\mathbb{E}_p\left[\left(d\ell_j|_p(v)d\ell_j|_p(w)\right)_{j=1}^{n}\right]=\sum_{j=1}^{n}\frac{v_jw_j}{p_j}. \tag*{(8)} g p ( v , w ) = E p [ ( d ℓ j ∣ p ( v ) d ℓ j ∣ p ( w ) ) j = 1 n ] = j = 1 ∑ n p j v j w j . (8)
For each p ∈ P n p\in\mathcal{P}_n p ∈ P n , g p g_p g p is bilinear and symmetric, and satisfies
g p ( v , v ) = ∑ j = 1 n v j 2 p j > 0 for v ≠ 0. g_p(v,v)=\sum_{j=1}^{n}\frac{v_j^2}{p_j}>0\qquad\text{for }v\ne0. g p ( v , v ) = j = 1 ∑ n p j v j 2 > 0 for v = 0.
Its coefficients depend smoothly on p p p , so g g g defines a Riemannian metric on P n \mathcal{P}_n P n . In this setting we also write grad F R = grad g \operatorname{grad}_{\mathrm{FR}}=\operatorname{grad}_g grad FR = grad g and ∥ v ∥ F R , p 2 = g p ( v , v ) \|v\|_{\mathrm{FR},p}^{2}=g_p(v,v) ∥ v ∥ FR , p 2 = g p ( v , v ) .
Computing the Fisher–Rao gradient
We now apply the induced-metric construction to P n ⊂ U n \mathcal{P}_n\subset U_n P n ⊂ U n . Equip the positive orthant with the ambient Riemannian metric
g ˉ x ( v , w ) = v T D x w , D x = diag ( x 1 − 1 , … , x n − 1 ) , \bar{g}_x(v,w)=v^{\mathsf{T}}D_xw,\qquad D_x=\operatorname{diag}(x_1^{-1},\ldots,x_n^{-1}), g ˉ x ( v , w ) = v T D x w , D x = diag ( x 1 − 1 , … , x n − 1 ) ,
where x ∈ U n x\in U_n x ∈ U n and v , w ∈ T x U n ≃ R n v,w\in T_xU_n\simeq\mathbb{R}^n v , w ∈ T x U n ≃ R n . Its restriction to T p P n T_p\mathcal{P}_n T p P n is exactly (3.4). By (3.1), the inverse matrix D p − 1 = diag ( p 1 , … , p n ) D_p^{-1}=\operatorname{diag}(p_1,\ldots,p_n) D p − 1 = diag ( p 1 , … , p n ) converts the Euclidean gradient on U n U_n U n into the gradient for g ˉ \bar{g} g ˉ .
Proposition 3.13 (Fisher–Rao gradient on the simplex). Let F : P n → R F:\mathcal{P}_n\to\mathbb{R} F : P n → R be C 1 C^1 C 1 , and choose a C 1 C^1 C 1 F ‾ : U n → R \overline{F}:U_n\to\mathbb{R} F : U n → R with F ‾ ∣ P n = F \overline{F}|_{\mathcal{P}_n}=F F ∣ P n = F . Define f : P n → R n f:\mathcal{P}_n\to\mathbb{R}^n f : P n → R n by f ( p ) : = ∇ F ‾ ( p ) f(p):=\nabla\overline{F}(p) f ( p ) := ∇ F ( p ) , where ∇ \nabla ∇ denotes the Euclidean gradient. For each p ∈ P n p\in\mathcal{P}_n p ∈ P n ,
( grad F R F ( p ) ) j = p j ( f j ( p ) − E p [ f ( p ) ] ) = p j ( f j ( p ) − ∑ k = 1 n p k f k ( p ) ) , j = 1 , … , n . \left(\operatorname{grad}_{\mathrm{FR}}F(p)\right)_j=p_j\left(f_j(p)-\mathbb{E}_p[f(p)]\right)=p_j\left(f_j(p)-\sum_{k=1}^{n}p_kf_k(p)\right),\qquad j=1,\ldots,n. ( grad FR F ( p ) ) j = p j ( f j ( p ) − E p [ f ( p )] ) = p j ( f j ( p ) − k = 1 ∑ n p k f k ( p ) ) , j = 1 , … , n . Equivalently,
grad F R F ( p ) = C ( p ) f ( p ) , C ( p ) : = diag ( p ) − p p T . (9) \operatorname{grad}_{\mathrm{FR}}F(p)=C(p)f(p),\qquad C(p):=\operatorname{diag}(p)-pp^{\mathsf{T}}. \tag*{(9)} grad FR F ( p ) = C ( p ) f ( p ) , C ( p ) := diag ( p ) − p p T . (9) These formulas are independent of the ambient extension.
Proof. Such an extension exists: for example, F ‾ ( x ) = F ( x / ∑ k = 1 n x k ) \overline{F}(x) = F(x/\sum_{k=1}^{n} x_k) F ( x ) = F ( x / ∑ k = 1 n x k ) is C 1 C^1 C 1 on U n U_n U n and agrees with F F F on P n \mathcal{P}_n P n . Let Π p : T p U n → T p P n \Pi_p : T_pU_n \to T_p\mathcal{P}_n Π p : T p U n → T p P n be the g ‾ p \overline{g}_p g p -orthogonal projection, with T p U n ≃ R n T_pU_n \simeq\mathbb{R}^n T p U n ≃ R n . The vector p p p is an ambient unit normal to the simplex, since
g ‾ p ( p , v ) = ∑ j = 1 n v j = 0 ( v ∈ T p P n ) , g ‾ p ( p , p ) = ∑ j = 1 n p j = 1. \overline{g}_p(p,v)=\sum_{j=1}^{n}v_j=0\quad(v\in T_p\mathcal{P}_n),\qquad\overline{g}_p(p,p)=\sum_{j=1}^{n}p_j=1. g p ( p , v ) = j = 1 ∑ n v j = 0 ( v ∈ T p P n ) , g p ( p , p ) = j = 1 ∑ n p j = 1.
Subtracting the normal component gives, for z ∈ T p U n z\in T_pU_n z ∈ T p U n ,
Π p z = z − g ‾ p ( z , p ) p = z − ( ∑ k = 1 n z k ) p . \Pi_pz=z-\overline{g}_p(z,p)p=z-\left(\sum_{k=1}^{n}z_k\right)p. Π p z = z − g p ( z , p ) p = z − ( k = 1 ∑ n z k ) p .
The ambient coordinate formula (5) gives grad g ‾ F ‾ ( p ) = D p − 1 f ( p ) \operatorname{grad}_{\overline{g}}\overline{F}(p)=D_p^{-1}f(p) grad g F ( p ) = D p − 1 f ( p ) . Applying Proposition 3.8 ,
grad F R F ( p ) = Π p D p − 1 f ( p ) = diag ( p ) f ( p ) − p ( p T f ( p ) ) = C ( p ) f ( p ) . \operatorname{grad}_{\mathrm{FR}}F(p)=\Pi_pD_p^{-1}f(p)=\operatorname{diag}(p)f(p)-p\left(p^{\mathsf{T}}f(p)\right)=C(p)f(p). grad FR F ( p ) = Π p D p − 1 f ( p ) = diag ( p ) f ( p ) − p ( p T f ( p ) ) = C ( p ) f ( p ) .
Independence of the extension also follows from this proposition.
Log-odds coordinates and the softmax Jacobian
We now introduce global log-odds coordinates on the ( n − 1 ) (n-1) ( n − 1 ) -dimensional manifold P n \mathcal{P}_n P n ; see Nielsen et al. [ [11 ] .
Proposition 3.14 (Log-odds coordinates). With outcome n n n as reference, the map
η : P n ⟶ R n − 1 , η j ( p ) = log p j p n , j = 1 , … , n − 1 , \eta:\mathcal{P}_n\longrightarrow\mathbb{R}^{n-1},\qquad\eta_j(p)=\log\frac{p_j}{p_n},\qquad j=1,\ldots,n-1, η : P n ⟶ R n − 1 , η j ( p ) = log p n p j , j = 1 , … , n − 1 , is a diffeomorphism. Its inverse is
p n = 1 1 + ∑ k = 1 n − 1 e η k , p j = e η j 1 + ∑ k = 1 n − 1 e η k ( j = 1 , … , n − 1 ) . (10) p_n=\frac{1}{1+\sum_{k=1}^{n-1}e^{\eta_k}},\qquad p_j=\frac{e^{\eta_j}}{1+\sum_{k=1}^{n-1}e^{\eta_k}}\qquad(j=1,\ldots,n-1). \tag*{(10)} p n = 1 + ∑ k = 1 n − 1 e η k 1 , p j = 1 + ∑ k = 1 n − 1 e η k e η j ( j = 1 , … , n − 1 ) . (10)
Proof. The relations p j = e η j p n p_j=e^{\eta_j}p_n p j = e η j p n , for j = 1 , … , n − 1 j=1,\ldots,n-1 j = 1 , … , n − 1 , and ∑ k = 1 n p k = 1 \sum_{k=1}^{n}p_k=1 ∑ k = 1 n p k = 1 give (10) . The forward and inverse maps are smooth.
In affine simplex coordinates, negative entropy E ( p ) = ∑ j = 1 n p j log p j E(p)=\sum_{j=1}^{n}p_j\log p_j E ( p ) = ∑ j = 1 n p j log p j has the log-odds as its first derivatives and the Fisher–Rao metric matrix as its Hessian. The calculation appears in Appendix A.4.
Log-odds use n − 1 n-1 n − 1 coordinates relative to a reference outcome. Softmax gives a representation treating all n n n outcomes symmetrically.
Definition 3.15 (Softmax and logits). The softmax map softmax : R n → P n \operatorname{softmax}:\mathbb{R}^{n}\to\mathcal{P}_n softmax : R n → P n is defined by
softmax j ( u ) = e u j ∑ k = 1 n e u k , j = 1 , … , n . \operatorname{softmax}_j(u)=\frac{e^{u_j}}{\sum_{k=1}^{n}e^{u_k}},\qquad j=1,\ldots,n. softmax j ( u ) = ∑ k = 1 n e u k e u j , j = 1 , … , n . The components of u u u are called logits.
Proposition 3.16 (Logit representation). The softmax map is surjective. For every p ∈ P n p\in\mathcal{P}_n p ∈ P n ,
softmax − 1 ( p ) = { log p + c 1 : c ∈ R } , 1 = ( 1 , … , 1 ) T , \operatorname{softmax}^{-1}(p)=\{\log p+c\mathbf{1}:c\in\mathbb{R}\},\qquad\mathbf{1}=(1,\ldots,1)^{\mathsf{T}}, softmax − 1 ( p ) = { log p + c 1 : c ∈ R } , 1 = ( 1 , … , 1 ) T , where the logarithm is componentwise. If p = softmax ( u ) p=\operatorname{softmax}(u) p = softmax ( u ) , then
log p j p k = u j − u k , j , k = 1 , … , n . \log\frac{p_j}{p_k}=u_j-u_k,\qquad j,k=1,\ldots,n. log p k p j = u j − u k , j , k = 1 , … , n . Consequently the logits form a redundant representation; imposing u n = 0 u_n=0 u n = 0 recovers the log-odds coordinates u j = η j ( p ) u_j=\eta_j(p) u j = η j ( p ) for j = 1 , … , n − 1 j=1,\ldots,n-1 j = 1 , … , n − 1 .
Proof. Since ∑ j = 1 n p j = 1 \sum_{j=1}^{n} p_j = 1 ∑ j = 1 n p j = 1 , we have softmax ( log p ) = p \operatorname{softmax}(\log p) = p softmax ( log p ) = p . If p = softmax ( u ) p = \operatorname{softmax}(u) p = softmax ( u ) , taking logarithms gives u j = log p j + log ∑ k = 1 n e u k u_j = \log p_j + \log\sum_{k=1}^{n} e^{u_k} u j = log p j + log ∑ k = 1 n e u k , so every representative has the stated form. Conversely, adding a common constant to all logits leaves softmax unchanged. Dividing p j p_j p j by p k p_k p k , for j , k ∈ { 1 , … , n } j,k \in\{1,\ldots,n\} j , k ∈ { 1 , … , n } , gives the log-odds identity, and setting k = n k=n k = n proves the last assertion.
For a C 1 C^1 C 1 scalar function a ( t ) a(t) a ( t ) , the shift u ( t ) ↦ u ( t ) + a ( t ) 1 u(t) \mapsto u(t) + a(t)\mathbf{1} u ( t ) ↦ u ( t ) + a ( t ) 1 leaves the probability path unchanged and adds a ˙ ( t ) 1 \dot{a}(t)\mathbf{1} a ˙ ( t ) 1 to the logit velocity. This explains the scalar freedom in Lemma 3.18 below.
The softmax derivative acts on vectors in R n \mathbb{R}^{n} R n , whereas the metric map g p ♯ g_p^\sharp g p ♯ acts on covectors (Definition 3.2 ). To compare them, for p ∈ P n p \in\mathcal{P}_n p ∈ P n and f ∈ R n f \in\mathbb{R}^{n} f ∈ R n , write
[ f ] : T p P n ⟶ R , [ f ] ( v ) : = f T v . [f] : T_p\mathcal{P}_n \longrightarrow\mathbb{R}, \qquad[f](v) := f^{\mathsf{T}}v. [ f ] : T p P n ⟶ R , [ f ] ( v ) := f T v .
Proposition 3.17 (Softmax Jacobian and the metric isomorphism). For the softmax map softmax : R n → P n \operatorname{softmax} : \mathbb{R}^{n} \to\mathcal{P}_n softmax : R n → P n , fix a logit vector u ∈ R n u \in\mathbb{R}^{n} u ∈ R n and set p = softmax ( u ) p = \operatorname{softmax}(u) p = softmax ( u ) . Then its Jacobian has entries
∂ softmax j ∂ u k ( u ) = p j ( δ j k − p k ) , j , k = 1 , … , n . \frac{\partial\operatorname{softmax}_j}{\partial u_k}(u) = p_j(\delta_{jk} - p_k), \qquad j,k = 1,\ldots,n. ∂ u k ∂ softmax j ( u ) = p j ( δ j k − p k ) , j , k = 1 , … , n . Equivalently, the derivative D softmax ( u ) : R n → T p P n D\operatorname{softmax}(u) : \mathbb{R}^{n} \to T_p\mathcal{P}_n D softmax ( u ) : R n → T p P n is represented by the matrix C ( p ) C(p) C ( p ) in (3.5). Every covector in T p ∗ P n T_p^*\mathcal{P}_n T p ∗ P n has the form [ f ] [f] [ f ] , with f f f determined up to addition of a multiple of 1 \mathbf{1} 1 . In this notation, the inverse of g p ♯ g_p^\sharp g p ♯ is
g p ♯ : T p ∗ P n ⟶ T p P n , [ f ] ⟼ C ( p ) f . g_p^\sharp: T_p^*\mathcal{P}_n \longrightarrow T_p\mathcal{P}_n, \qquad[f] \longmapsto C(p)f. g p ♯ : T p ∗ P n ⟶ T p P n , [ f ] ⟼ C ( p ) f . Thus the softmax Jacobian and the map taking covectors to Fisher gradients are represented by the same matrix C ( p ) C(p) C ( p ) .
Proof. Differentiating the quotient gives
∂ softmax j ∂ u k ( u ) = δ j k e u j ∑ ℓ = 1 n e u ℓ − e u j e u k ( ∑ ℓ = 1 n e u ℓ ) 2 = p j ( δ j k − p k ) . \frac{\partial\operatorname{softmax}_j}{\partial u_k}(u) = \frac{\delta_{jk}e^{u_j}\sum_{\ell=1}^{n}e^{u_\ell} - e^{u_j}e^{u_k}}{\left(\sum_{\ell=1}^{n}e^{u_\ell}\right)^2} = p_j(\delta_{jk} - p_k). ∂ u k ∂ softmax j ( u ) = ( ∑ ℓ = 1 n e u ℓ ) 2 δ j k e u j ∑ ℓ = 1 n e u ℓ − e u j e u k = p j ( δ j k − p k ) .
The Euclidean inner product on T p P n T_p\mathcal{P}_n T p P n represents each covector α ∈ T p ∗ P n \alpha\in T_p^*\mathcal{P}_n α ∈ T p ∗ P n by a vector f ∈ T p P n ⊂ R n f \in T_p\mathcal{P}_n \subset\mathbb{R}^{n} f ∈ T p P n ⊂ R n satisfying
α ( v ) = f T v for every v ∈ T p P n . \alpha(v) = f^{\mathsf{T}}v \qquad\text{for every } v \in T_p\mathcal{P}_n. α ( v ) = f T v for every v ∈ T p P n .
Two vectors in R n \mathbb{R}^{n} R n represent the same covector exactly when their difference is orthogonal to T p P n T_p\mathcal{P}_n T p P n in the Euclidean inner product. By (3.3), this means that their difference is a multiple of 1 \mathbf{1} 1 . Since C ( p ) 1 = 0 C(p)\mathbf{1}=0 C ( p ) 1 = 0 and 1 T C ( p ) = 0 \mathbf{1}^{\mathsf{T}}C(p)=0 1 T C ( p ) = 0 , the assignment [ f ] ↦ C ( p ) f [f] \mapsto C(p)f [ f ] ↦ C ( p ) f is independent of the representative and takes values in T p P n T_p\mathcal{P}_n T p P n . For every v ∈ T p P n v \in T_p\mathcal{P}_n v ∈ T p P n ,
g p ( C ( p ) f , v ) = ∑ j = 1 n ( f j − E p [ f ] ) v j = f T v . g_p(C(p)f,v) = \sum_{j=1}^{n}(f_j-\mathbb{E}_p[f])v_j = f^{\mathsf{T}}v. g p ( C ( p ) f , v ) = j = 1 ∑ n ( f j − E p [ f ]) v j = f T v .
This is the defining identity of g p ♯ g_p^\sharp g p ♯ .
Lemma 3.18 (Lifting Fisher–Rao flow to logits). Fix γ > 0 \gamma> 0 γ > 0 . Let I ⊂ R I \subset\mathbb{R} I ⊂ R be a time interval and F : P n × I → R F : \mathcal{P}_n \times I \to\mathbb{R} F : P n × I → R a jointly C 1 C^1 C 1 function. Choose a jointly C 1 C^1 C 1 extension F ‾ : U n × I → R \overline{F} : U_n \times I \to\mathbb{R} F : U n × I → R and define f : P n × I → R n f : \mathcal{P}_n \times I \to\mathbb{R}^{n} f : P n × I → R n by f ( p , t ) : = ∇ p F ‾ ( p , t ) f(p,t) := \nabla_p\overline{F}(p,t) f ( p , t ) := ∇ p F ( p , t ) . A C 1 C^1 C 1 logit path u : I → R n u : I \to\mathbb{R}^{n} u : I → R n projects by p ( t ) = softmax ( u ( t ) ) p(t)=\operatorname{softmax}(u(t)) p ( t ) = softmax ( u ( t )) to the Fisher–Rao gradient flow of F F F (see Definition 3.4 ) if and only if, for some continuous c : I → R c : I \to\mathbb{R} c : I → R ,
u ˙ ( t ) = − γ f ( p ( t ) , t ) + c ( t ) 1. \dot{u}(t) = -\gamma f(p(t),t) + c(t)\mathbf{1}. u ˙ ( t ) = − γ f ( p ( t ) , t ) + c ( t ) 1 . Every positive gradient-flow solution has such a lift.
Proof. Differentiating p ( t ) = softmax ( u ( t ) ) p(t) = \operatorname{softmax}(u(t)) p ( t ) = softmax ( u ( t )) gives, by Proposition 3.17 ,
p ˙ ( t ) = C ( p ( t ) ) u ˙ ( t ) . \dot{p}(t) = C(p(t))\dot{u}(t). p ˙ ( t ) = C ( p ( t )) u ˙ ( t ) .
Since f f f is the Euclidean gradient of the extension, d ( F t ) p ( t ) ( v ) = f ( p ( t ) , t ) ⊤ v d(F_t)_{p(t)}(v) = f(p(t), t)^\top v d ( F t ) p ( t ) ( v ) = f ( p ( t ) , t ) ⊤ v for every v ∈ T p ( t ) P n v \in T_{p(t)}\mathcal{P}_n v ∈ T p ( t ) P n . Hence d ( F t ) p ( t ) = [ f ( p ( t ) , t ) ] d(F_t)_{p(t)} = [f(p(t), t)] d ( F t ) p ( t ) = [ f ( p ( t ) , t )] . Definition 3.3 and Proposition 3.17 therefore give
grad F R F t ( p ( t ) ) = g p ( t ) ♯ ( d ( F t ) p ( t ) ) = g p ( t ) ♯ ( [ f ( p ( t ) , t ) ] ) = C ( p ( t ) ) f ( p ( t ) , t ) . \operatorname{grad}_{\mathrm{FR}} F_t(p(t)) = g^\sharp_{p(t)}(d(F_t)_{p(t)}) = g^\sharp_{p(t)}([f(p(t), t)]) = C(p(t))f(p(t), t). grad FR F t ( p ( t )) = g p ( t ) ♯ ( d ( F t ) p ( t ) ) = g p ( t ) ♯ ([ f ( p ( t ) , t )]) = C ( p ( t )) f ( p ( t ) , t ) .
For p ∈ P n p \in\mathcal{P}_n p ∈ P n and z ∈ R n z \in\mathbb{R}^n z ∈ R n , ( C ( p ) z ) j = p j ( z j − E p [ z ] ) (C(p)z)_j = p_j(z_j - \mathbb{E}_p[z]) ( C ( p ) z ) j = p j ( z j − E p [ z ]) . Since p j > 0 p_j > 0 p j > 0 , C ( p ) z = 0 C(p)z = 0 C ( p ) z = 0 if and only if z j = E p [ z ] z_j = \mathbb{E}_p[z] z j = E p [ z ] for every j = 1 , … , n j = 1,\ldots,n j = 1 , … , n , so ker C ( p ) = span { 1 } \ker C(p) = \operatorname{span}\{1\} ker C ( p ) = span { 1 } . Therefore,
p ˙ ( t ) = − γ grad F R F t ( p ( t ) ) ⟺ C ( p ( t ) ) ( u ˙ ( t ) + γ f ( p ( t ) , t ) ) = 0 \dot{p}(t) = -\gamma\operatorname{grad}_{\mathrm{FR}} F_t(p(t)) \Longleftrightarrow C(p(t))(\dot{u}(t) + \gamma f(p(t),t)) = 0 p ˙ ( t ) = − γ grad FR F t ( p ( t )) ⟺ C ( p ( t )) ( u ˙ ( t ) + γ f ( p ( t ) , t )) = 0
⟺ u ˙ ( t ) + γ f ( p ( t ) , t ) ∈ span { 1 } \Longleftrightarrow\dot{u}(t) + \gamma f(p(t),t) \in\operatorname{span}\{1\} ⟺ u ˙ ( t ) + γ f ( p ( t ) , t ) ∈ span { 1 }
⟺ u ˙ ( t ) = − γ f ( p ( t ) , t ) + c ( t ) 1. \Longleftrightarrow\dot{u}(t) = -\gamma f(p(t),t) + c(t)1. ⟺ u ˙ ( t ) = − γ f ( p ( t ) , t ) + c ( t ) 1.
Here c ( t ) = u ˙ 1 ( t ) + γ f 1 ( p ( t ) , t ) c(t) = \dot{u}_1(t) + \gamma f_1(p(t),t) c ( t ) = u ˙ 1 ( t ) + γ f 1 ( p ( t ) , t ) is continuous because u u u is C 1 C^1 C 1 and f f f is continuous. Finally, every positive gradient-flow solution p p p has the C 1 C^1 C 1 lift u ( t ) = log p ( t ) u(t) = \log p(t) u ( t ) = log p ( t ) , since softmax ( log p ( t ) ) = p ( t ) \operatorname{softmax}(\log p(t)) = p(t) softmax ( log p ( t )) = p ( t ) .
From one row to the attention matrix
The row representation A ↔ ( a ( 1 ) , … , a ( N ) ) A \leftrightarrow(a^{(1)},\ldots,a^{(N)}) A ↔ ( a ( 1 ) , … , a ( N ) ) identifies A \mathcal{A} A with ( P N ) N (\mathcal{P}_N)^N ( P N ) N .
Definition 3.19 (Product metric). For Riemannian manifolds ( M r , g ( r ) ) (M_r,g^{(r)}) ( M r , g ( r ) ) , r = 1 , … , b r = 1,\ldots,b r = 1 , … , b , the sum product metric on M 1 × ⋯ × M b M_1 \times\cdots\times M_b M 1 × ⋯ × M b is
g p p r o d ( U , V ) = ∑ r = 1 b g p ( r ) ( r ) ( U ( r ) , V ( r ) ) , p = ( p ( 1 ) , … , p ( b ) ) . g^{\mathrm{prod}}_p(U,V) = \sum_{r=1}^{b} g^{(r)}_{p^{(r)}}(U^{(r)},V^{(r)}), \qquad p = (p^{(1)},\ldots,p^{(b)}). g p prod ( U , V ) = r = 1 ∑ b g p ( r ) ( r ) ( U ( r ) , V ( r ) ) , p = ( p ( 1 ) , … , p ( b ) ) .
Here U ( r ) , V ( r ) ∈ T p ( r ) M r U^{(r)}, V^{(r)} \in T_{p^{(r)}}M_r U ( r ) , V ( r ) ∈ T p ( r ) M r are the factor components.
For attention, the sum product metric extends the quadratic form in (4) to all A ∈ A A \in\mathcal{A} A ∈ A :
g A ( B , C ) = ∑ i , j = 1 N B i j C i j A i j , ∥ B ∥ F R , A 2 = g A ( B , B ) , B , C ∈ T A A . (11) g_A(B,C) = \sum_{i,j=1}^{N} \frac{B_{ij}C_{ij}}{A_{ij}}, \qquad\lVert B\rVert_{\mathrm{FR},A}^{2} = g_A(B,B), \qquad B,C \in T_A\mathcal{A}. \tag*{(11)} g A ( B , C ) = i , j = 1 ∑ N A ij B ij C ij , ∥ B ∥ FR , A 2 = g A ( B , B ) , B , C ∈ T A A . (11)
Recall the row-expectation notation E i [ b ] = ∑ j = 1 N A i j b j \mathbb{E}_i[b] = \sum_{j=1}^{N} A_{ij}b_j E i [ b ] = ∑ j = 1 N A ij b j introduced in § 2.2 . Definition 3.11 identifies it with E a ( i ) [ b ] \mathbb{E}_{a^{(i)}}[b] E a ( i ) [ b ] . Along a solution, its weights are A i j ( t ) A_{ij}(t) A ij ( t ) .
Proposition 3.20 (Geometry of the attention space). The manifold A \mathcal{A} A has dimension N ( N − 1 ) N(N-1) N ( N − 1 ) , and
T A A = { B ∈ R N × N : ∑ j = 1 N B i j = 0 for i = 1 , … , N } . T_A\mathcal{A} = \left\{B \in\mathbb{R}^{N \times N} : \sum_{j=1}^{N} B_{ij} = 0 \text{ for } i = 1,\ldots,N\right\}. T A A = { B ∈ R N × N : j = 1 ∑ N B ij = 0 for i = 1 , … , N } .
For a smooth F : A → R F:\mathcal{A}\to\mathbb{R} F : A → R , choose a smooth extension F ‾ \overline{F} F to the positive matrices. Set f i j = ∂ F ‾ ∂ A i j f_{ij} = \frac{\partial\overline{F}}{\partial A_{ij}} f ij = ∂ A ij ∂ F and f i = ( f i 1 , … , f i N ) ⊤ f_i = (f_{i1},\ldots,f_{iN})^\top f i = ( f i 1 , … , f i N ) ⊤ . Then
( grad A F R F ) i j = A i j ( f i j − E i [ f i ] ) , i , j = 1 , … , N . (12) (\operatorname{grad}_{\mathcal{A}}^{\mathrm{FR}} F)_{ij} = A_{ij}(f_{ij} - \mathbb{E}_i[f_i]), \qquad i,j = 1,\ldots,N. \tag*{(12)} ( grad A FR F ) ij = A ij ( f ij − E i [ f i ]) , i , j = 1 , … , N . (12)
This expression is independent of the extension.
Proof. The dimension and tangent space follow by applying Lemma 3.10 to every factor. By the product-metric construction, the i i i th row of the gradient is the simplex gradient of F F F with all other rows held fixed. Applying Proposition 3.13 to this function gives (12) , including independence of the extension.
Attention relaxation and tracking
The evolution law
During a Transformer forward pass, successive layers update the token representations and compute attention from the resulting features [14 ] . We idealise such changes by a prescribed continuous path X : [ 0 , ∞ ) → X X : [0,\infty) \to\mathcal{X} X : [ 0 , ∞ ) → X ; Appendix § 10 discusses the connection with depth. Standard self-attention computes its associated weights directly by softmax. Giving attention its own evolution is the extension studied here: varying X ( t ) X(t) X ( t ) lets us study how closely this evolution tracks the changing softmax target A ∗ ( X ( t ) ) A^*(X(t)) A ∗ ( X ( t )) .
We choose the Fisher–Rao gradient flow of (2) as an evolution law for A A A , using the metric (11) on A \mathcal{A} A (see Definition 3.4). This describes continuous-time steepest descent of the attention free energy. Section 5 will determine X ( t ) X(t) X ( t ) as part of a coupled initial-value problem.
Definition 4.1 (Fisher–Rao attention model). Let X : [ 0 , ∞ ) → X X : [0,\infty) \to\mathcal{X} X : [ 0 , ∞ ) → X be a prescribed C 1 C^1 C 1 feature path. Fix a mobility γ > 0 \gamma> 0 γ > 0 . For A 0 ∈ A A^0 \in\mathcal{A} A 0 ∈ A , we model the attention weights by the Fisher–Rao gradient flow
d A d t = − γ grad A F R F ( A ( t ) ; X ( t ) ) , A ( 0 ) = A 0 . (13) \frac{\mathrm{d}A}{\mathrm{d}t}=-\gamma\operatorname{grad}^{\mathrm{FR}}_{A}\mathcal{F}(A(t);X(t)),\qquad A(0)=A^0. \tag*{(13)} d t d A = − γ grad A FR F ( A ( t ) ; X ( t )) , A ( 0 ) = A 0 . (13) Here F \mathcal{F} F is the free energy defined in (2) .
Proposition 4.2 (The attention gradient). For every A ∈ A A \in\mathcal{A} A ∈ A and X ∈ X X \in\mathcal{X} X ∈ X , and for i , j = 1 , … , N i,j=1,\ldots,N i , j = 1 , … , N ,
( grad A F R F ) i j = A i j [ − s i j ( X ) + E i [ s i ( X ) ] + τ ( log A i j − E i [ log a ( i ) ] ) ] . (\operatorname{grad}^{\mathrm{FR}}_{A}\mathcal{F})_{ij}=A_{ij}\left[-s_{ij}(X)+\mathbb{E}_{i}[s_i(X)]+\tau\left(\log A_{ij}-\mathbb{E}_{i}[\log a^{(i)}]\right)\right]. ( grad A FR F ) ij = A ij [ − s ij ( X ) + E i [ s i ( X )] + τ ( log A ij − E i [ log a ( i ) ] ) ] . Consequently (13) has the component form
A ˙ i j = γ A i j [ s i j ( X ( t ) ) − E i [ s i ( X ( t ) ) ] − τ ( log A i j − E i [ log a ( i ) ] ) ] . (14) \dot{A}_{ij}=\gamma A_{ij}\left[s_{ij}(X(t))-\mathbb{E}_{i}[s_i(X(t))]-\tau\left(\log A_{ij}-\mathbb{E}_{i}[\log a^{(i)}]\right)\right]. \tag*{(14)} A ˙ ij = γ A ij [ s ij ( X ( t )) − E i [ s i ( X ( t ))] − τ ( log A ij − E i [ log a ( i ) ] ) ] . (14)
Proof. For the extension of (2) to positive matrices,
f i j : = ∂ F ‾ ∂ A i j = − s i j ( X ) + τ ( 1 + log A i j ) . (15) f_{ij}:=\frac{\partial\overline{\mathcal{F}}}{\partial A_{ij}}=-s_{ij}(X)+\tau(1+\log A_{ij}). \tag*{(15)} f ij := ∂ A ij ∂ F = − s ij ( X ) + τ ( 1 + log A ij ) . (15)
Its row mean is
∑ k = 1 N A i k f i k = − ∑ k = 1 N A i k s i k ( X ) + τ ( 1 + ∑ k = 1 N A i k log A i k ) . \sum_{k=1}^{N}A_{ik}f_{ik}=-\sum_{k=1}^{N}A_{ik}s_{ik}(X)+\tau\left(1+\sum_{k=1}^{N}A_{ik}\log A_{ik}\right). k = 1 ∑ N A ik f ik = − k = 1 ∑ N A ik s ik ( X ) + τ ( 1 + k = 1 ∑ N A ik log A ik ) .
Subtracting this mean from f i j f_{ij} f ij cancels the constant τ \tau τ . Formula (12) then gives the result. □ \square □
Logit and log-odds formulations
The row softmax map softmax r o w : R N × N → A \operatorname{softmax}_{\mathrm{row}}:\mathbb{R}^{N\times N}\to\mathcal{A} softmax row : R N × N → A applies softmax to each row. For a logit matrix U ∈ R N × N U\in\mathbb{R}^{N\times N} U ∈ R N × N , write
A = softmax r o w ( U ) , A i j = e U i j ∑ k = 1 N e U i k , i , j = 1 , … , N . A=\operatorname{softmax}_{\mathrm{row}}(U),\qquad A_{ij}=\frac{e^{U_{ij}}}{\sum_{k=1}^{N}e^{U_{ik}}},\qquad i,j=1,\ldots,N. A = softmax row ( U ) , A ij = ∑ k = 1 N e U ik e U ij , i , j = 1 , … , N .
ends_mid=1
Proposition 4.4 (Row-wise logit lift). Let X X X and A 0 A^0 A 0 be as in Definition 4.1 . Let U : [ 0 , ∞ ) → R N × N U : [0,\infty) \to\mathbb{R}^{N \times N} U : [ 0 , ∞ ) → R N × N solve
U ˙ = γ ( S ( X ( t ) ) − τ U ) , U ( 0 ) = log A 0 . (16) \dot{U} = \gamma\left(S(X(t))-\tau U\right), \qquad U(0)=\log A^0. \tag*{(16)} U ˙ = γ ( S ( X ( t )) − τ U ) , U ( 0 ) = log A 0 . (16) Then A = softmax r o w ( U ) A=\operatorname{softmax}_{\mathrm{row}}(U) A = softmax row ( U ) solves (13) . For prescribed X X X , equation (16) is linear in U U U .
Proof. Since log A i j = U i j − log ∑ k = 1 N e U i k \log A_{ij}=U_{ij}-\log\sum_{k=1}^{N}e^{U_{ik}} log A ij = U ij − log ∑ k = 1 N e U ik , the derivatives in (15) have the form
f i j = − s i j ( X ( t ) ) + τ U i j + τ ( 1 − log ∑ k = 1 N e U i k ) , i , j = 1 , … , N . f_{ij}=-s_{ij}(X(t))+\tau U_{ij}+\tau\left(1-\log\sum_{k=1}^{N}e^{U_{ik}}\right), \qquad i,j=1,\ldots,N. f ij = − s ij ( X ( t )) + τ U ij + τ ( 1 − log k = 1 ∑ N e U ik ) , i , j = 1 , … , N . Apply Lemma 3.18 row by row with
c i ( t ) = γ τ ( 1 − log ∑ k = 1 N e U i k ( t ) ) , i = 1 , … , N . c_i(t)=\gamma\tau\left(1-\log\sum_{k=1}^{N}e^{U_{ik}(t)}\right), \qquad i=1,\ldots,N. c i ( t ) = γ τ ( 1 − log k = 1 ∑ N e U ik ( t ) ) , i = 1 , … , N . With this choice, we obtain
− γ f i j + c i ( t ) = γ ( s i j ( X ( t ) ) − τ U i j ) , -\gamma f_{ij}+c_i(t)=\gamma\left(s_{ij}(X(t))-\tau U_{ij}\right), − γ f ij + c i ( t ) = γ ( s ij ( X ( t )) − τ U ij ) , which gives (16) . Normalisation gives softmax r o w ( log A 0 ) = A 0 \operatorname{softmax}_{\mathrm{row}}(\log A^0)=A^0 softmax row ( log A 0 ) = A 0 .
For the uniqueness and tracking arguments, it is useful to pass to log-odds, which remove the arbitrary row shifts in U U U . Using Proposition 3.14, set
η i j = log A i j A i N = U i j − U i N , i = 1 , … , N , j = 1 , … , N − 1. \eta_{ij}=\log\frac{A_{ij}}{A_{iN}}=U_{ij}-U_{iN}, \qquad i=1,\ldots,N,\quad j=1,\ldots,N-1. η ij = log A i N A ij = U ij − U i N , i = 1 , … , N , j = 1 , … , N − 1. Differentiating these log-odds using (14) , or subtracting the lifted equations for U i j U_{ij} U ij and U i N U_{iN} U i N , gives
η ˙ i j = γ ( s i j ( X ( t ) ) − s i N ( X ( t ) ) − τ η i j ) , i = 1 , … , N , j = 1 , … , N − 1. (17) \dot{\eta}_{ij}=\gamma\left(s_{ij}(X(t))-s_{iN}(X(t))-\tau\eta_{ij}\right), \qquad i=1,\ldots,N,\quad j=1,\ldots,N-1. \tag*{(17)} η ˙ ij = γ ( s ij ( X ( t )) − s i N ( X ( t )) − τ η ij ) , i = 1 , … , N , j = 1 , … , N − 1. (17)
Existence and filtered scores
Theorem 4.5 (Attention driven by a prescribed feature path). Let X : [ 0 , ∞ ) → X X:[0,\infty)\to\mathcal{X} X : [ 0 , ∞ ) → X be a prescribed C 1 C^1 C 1 feature path. For every A 0 ∈ A A^0\in\mathcal{A} A 0 ∈ A , the initial-value problem (13) has a unique global solution A : [ 0 , ∞ ) → A A:[0,\infty)\to\mathcal{A} A : [ 0 , ∞ ) → A . With κ = γ τ \kappa=\gamma\tau κ = γ τ , define U : [ 0 , ∞ ) → R N × N U:[0,\infty)\to\mathbb{R}^{N\times N} U : [ 0 , ∞ ) → R N × N by the first line below. The attention solution is its row softmax:
U ( t ) = e − κ t log A 0 + γ ∫ 0 t e − κ ( t − r ) S ( X ( r ) ) d r , A ( t ) = softmax r o w ( U ( t ) ) . (18) \begin{aligned}
U(t)&=e^{-\kappa t}\log A^0+\gamma\int_{0}^{t}e^{-\kappa(t-r)}S(X(r))\,\mathrm{d}r,\\
A(t)&=\operatorname{softmax}_{\mathrm{row}}(U(t)). \tag*{(18)}
\end{aligned} U ( t ) A ( t ) = e − κ t log A 0 + γ ∫ 0 t e − κ ( t − r ) S ( X ( r )) d r , = softmax row ( U ( t )) . (18) In particular, all entries stay strictly positive and each row remains normalised at every finite time.
Proof. Multiplying (16) by e κ t e^{\kappa t} e κ t gives
d d t ( e κ t U ( t ) ) = γ e κ t S ( X ( t ) ) . \frac{\mathrm{d}}{\mathrm{d}t}\left(e^{\kappa t}U(t)\right)=\gamma e^{\kappa t}S(X(t)). d t d ( e κ t U ( t ) ) = γ e κ t S ( X ( t )) . Integration gives (18) . The score matrix is continuous and hence bounded on every finite time interval, so every entry of U ( t ) U(t) U ( t ) is finite there. Its row softmax is positive and normalised. At time zero, softmax r o w ( log A 0 ) = A 0 \operatorname{softmax}_{\mathrm{row}}(\log A^0)=A^0 softmax row ( log A 0 ) = A 0 . Proposition 4.4 verifies the differential equation.
For uniqueness, any solution in A \mathcal{A} A has log-odds satisfying (17) , with initial values η i j ( 0 ) = log ( A i j 0 / A i N 0 ) \eta_{ij}(0)=\log(A^0_{ij}/A^0_{iN}) η ij ( 0 ) = log ( A ij 0 / A i N 0 ) for i = 1 , … , N i=1,\ldots,N i = 1 , … , N and j = 1 , … , N − 1 j=1,\ldots,N-1 j = 1 , … , N − 1 . Uniqueness for these scalar linear equations fixes all ratios A i j / A i N A_{ij}/A_{iN} A ij / A i N . Normalisation fixes each remaining entry A i N A_{iN} A i N . Thus every such solution agrees with the one constructed above, which exists for all t ≥ 0 t\geq0 t ≥ 0 .
Remark 4.6 (Exponential filtering). The first line of (4.6) can be written as
U ( t ) = e − κ t log A 0 + ∫ 0 t κ e − κ ( t − r ) S ( X ( r ) ) τ d r . U(t)=e^{-\kappa t}\log A^0+\int_0^t\kappa e^{-\kappa(t-r)}\frac{S(X(r))}{\tau}\,dr. U ( t ) = e − κ t log A 0 + ∫ 0 t κ e − κ ( t − r ) τ S ( X ( r )) d r .
The initial logits and the past target logits S ( X ( r ) ) / τ S(X(r))/\tau S ( X ( r )) / τ carry weights whose sum is e − κ t + 1 − e − κ t = 1 e^{-\kappa t}+1-e^{-\kappa t}=1 e − κ t + 1 − e − κ t = 1 . Thus this model computes the softmax of exponentially filtered scores, with memory time 1 / κ = 1 / ( γ τ ) 1/\kappa=1/(\gamma\tau) 1/ κ = 1/ ( γ τ ) . The averaging takes place in logit space before the softmax is applied.
Tracking changing scores
We now compare the attention solution of Theorem 4.5 with A ∗ ( X ( t ) ) A^*(X(t)) A ∗ ( X ( t )) , the softmax associated with the current features. For each receiving row i = 1 , … , N i=1,\ldots,N i = 1 , … , N , use source N N N as reference. The target and evolving log-odds are
h i j ( t ) : = log A i j ∗ ( X ( t ) ) A i N ∗ ( X ( t ) ) = s i j ( X ( t ) ) − s i N ( X ( t ) ) τ , h_{ij}(t):=\log\frac{A^*_{ij}(X(t))}{A^*_{iN}(X(t))}=\frac{s_{ij}(X(t))-s_{iN}(X(t))}{\tau}, h ij ( t ) := log A i N ∗ ( X ( t )) A ij ∗ ( X ( t )) = τ s ij ( X ( t )) − s i N ( X ( t )) ,
η i j ( t ) = log A i j ( t ) A i N ( t ) , j = 1 , … , N − 1. \eta_{ij}(t)=\log\frac{A_{ij}(t)}{A_{iN}(t)},\qquad j=1,\ldots,N-1. η ij ( t ) = log A i N ( t ) A ij ( t ) , j = 1 , … , N − 1.
Thus h , η : [ 0 , ∞ ) → R N × ( N − 1 ) h,\eta:[0,\infty)\to\mathbb{R}^{N\times(N-1)} h , η : [ 0 , ∞ ) → R N × ( N − 1 ) , and E : = η − h E:=\eta-h E := η − h measures the tracking error in log-odds. For E ∈ R N × ( N − 1 ) E\in\mathbb{R}^{N\times(N-1)} E ∈ R N × ( N − 1 ) and B ∈ R r × s B\in\mathbb{R}^{r\times s} B ∈ R r × s , write
∥ E ∥ ∞ : = max 1 ≤ i ≤ N 1 ≤ j ≤ N − 1 ∣ E i j ∣ , ∥ B ∥ F : = ( ∑ i = 1 r ∑ j = 1 s B i j 2 ) 1 / 2 . \|E\|_\infty:=\max_{\substack{1\le i\le N\\1\le j\le N-1}}|E_{ij}|,\qquad
\|B\|_F:=\left(\sum_{i=1}^{r}\sum_{j=1}^{s}B_{ij}^{2}\right)^{1/2}. ∥ E ∥ ∞ := 1 ≤ i ≤ N 1 ≤ j ≤ N − 1 max ∣ E ij ∣ , ∥ B ∥ F := ( i = 1 ∑ r j = 1 ∑ s B ij 2 ) 1/2 .
We also denote the operator norm induced by the Euclidean norms by ∥ B ∥ o p : = sup v ∈ R s , ∣ v ∣ = 1 ∣ B v ∣ \|B\|_{\mathrm{op}}:=\sup_{v\in\mathbb{R}^{s},\,|v|=1}|Bv| ∥ B ∥ op := sup v ∈ R s , ∣ v ∣ = 1 ∣ B v ∣ .
Theorem 4.7 (Tracking and convergence for changing features). Let X , A X,A X , A be as in Theorem 4.5, and let h , η , E h,\eta,E h , η , E be the paths defined above. Put κ = γ τ \kappa=\gamma\tau κ = γ τ . For every t ≥ 0 t\ge0 t ≥ 0 , the log-odds error satisfies
E ˙ + κ E = − h ˙ , \dot{E}+\kappa E=-\dot{h}, E ˙ + κ E = − h ˙ , ∥ E ( t ) ∥ ∞ ≤ e − κ t ∥ E ( 0 ) ∥ ∞ + ∫ 0 t e − κ ( t − r ) ∥ h ˙ ( r ) ∥ ∞ d r . (19) \|E(t)\|_\infty\le e^{-\kappa t}\|E(0)\|_\infty+\int_0^t e^{-\kappa(t-r)}\|\dot{h}(r)\|_\infty\,dr. \tag*{(19)} ∥ E ( t ) ∥ ∞ ≤ e − κ t ∥ E ( 0 ) ∥ ∞ + ∫ 0 t e − κ ( t − r ) ∥ h ˙ ( r ) ∥ ∞ d r . (19) In particular, if ∥ h ˙ ( t ) ∥ ∞ ≤ L \|\dot{h}(t)\|_\infty\le L ∥ h ˙ ( t ) ∥ ∞ ≤ L for all t ≥ 0 t\ge0 t ≥ 0 , then
∥ E ( t ) ∥ ∞ ≤ e − κ t ∥ E ( 0 ) ∥ ∞ + L κ ( 1 − e − κ t ) . \|E(t)\|_\infty\le e^{-\kappa t}\|E(0)\|_\infty+\frac{L}{\kappa}(1-e^{-\kappa t}). ∥ E ( t ) ∥ ∞ ≤ e − κ t ∥ E ( 0 ) ∥ ∞ + κ L ( 1 − e − κ t ) . There are two useful sufficient conditions for asymptotic agreement:
If h ˙ ( t ) → 0 \dot{h}(t)\to0 h ˙ ( t ) → 0 as t → ∞ t\to\infty t → ∞ , then E ( t ) → 0 E(t)\to0 E ( t ) → 0 and ∥ A ( t ) − A ∗ ( X ( t ) ) ∥ F → 0 \|A(t)-A^*(X(t))\|_F\to0 ∥ A ( t ) − A ∗ ( X ( t )) ∥ F → 0 .
If h ( t ) → h ∞ ∈ R N × ( N − 1 ) h(t)\to h^\infty\in\mathbb{R}^{N\times(N-1)} h ( t ) → h ∞ ∈ R N × ( N − 1 ) , then A ( t ) A(t) A ( t ) and A ∗ ( X ( t ) ) A^*(X(t)) A ∗ ( X ( t )) converge to the same limit A ∞ ∈ A A^\infty\in\mathcal{A} A ∞ ∈ A , given row by row by
A i N ∞ = 1 1 + ∑ k = 1 N − 1 e h i k ∞ , A i j ∞ = e h i j ∞ 1 + ∑ k = 1 N − 1 e h i k ∞ , A^\infty_{iN}=\frac{1}{1+\sum_{k=1}^{N-1}e^{h^\infty_{ik}}},\qquad
A^\infty_{ij}=\frac{e^{h^\infty_{ij}}}{1+\sum_{k=1}^{N-1}e^{h^\infty_{ik}}}, A i N ∞ = 1 + ∑ k = 1 N − 1 e h ik ∞ 1 , A ij ∞ = 1 + ∑ k = 1 N − 1 e h ik ∞ e h ij ∞ , for i = 1 , … , N i=1,\ldots,N i = 1 , … , N and j = 1 , … , N − 1 j=1,\ldots,N-1 j = 1 , … , N − 1 .
The second condition holds in particular if X ( t ) → X ∞ X(t)\to X^\infty X ( t ) → X ∞ , in which case A ∞ = A ∗ ( X ∞ ) A^\infty=A^*(X^\infty) A ∞ = A ∗ ( X ∞ ) .
Proof. Equation (17) is η ˙ ( t ) = κ ( h ( t ) − η ( t ) ) \dot{\eta}(t)=\kappa(h(t)-\eta(t)) η ˙ ( t ) = κ ( h ( t ) − η ( t )) . Since E = η − h E=\eta-h E = η − h , subtracting h ˙ ( t ) \dot{h}(t) h ˙ ( t ) gives E ˙ ( t ) + κ E ( t ) = − h ˙ ( t ) \dot{E}(t)+\kappa E(t)=-\dot{h}(t) E ˙ ( t ) + κ E ( t ) = − h ˙ ( t ) . Multiplying by the integrating factor e κ t e^{\kappa t} e κ t yields
d d t ( e κ t E ( t ) ) = e κ t ( E ˙ ( t ) + κ E ( t ) ) = − e κ t h ˙ ( t ) . \frac{d}{dt}\left(e^{\kappa t}E(t)\right)=e^{\kappa t}\left(\dot{E}(t)+\kappa E(t)\right)=-e^{\kappa t}\dot{h}(t). d t d ( e κ t E ( t ) ) = e κ t ( E ˙ ( t ) + κ E ( t ) ) = − e κ t h ˙ ( t ) .
Integrating from 0 0 0 to t t t and multiplying by e − κ t e^{-\kappa t} e − κ t gives
E ( t ) = e − κ t E ( 0 ) − ∫ 0 t e − κ ( t − r ) h ˙ ( r ) d r . E(t)=e^{-\kappa t}E(0)-\int_{0}^{t}e^{-\kappa(t-r)}\dot{h}(r)\,dr. E ( t ) = e − κ t E ( 0 ) − ∫ 0 t e − κ ( t − r ) h ˙ ( r ) d r .
Taking the maximum norm proves the estimates. If h ˙ ( t ) → 0 \dot{h}(t)\to0 h ˙ ( t ) → 0 , split the integral at a fixed T T T . Its part on [ 0 , T ] [0,T] [ 0 , T ] decays exponentially, while its remaining norm is bounded by sup r ≥ T ∥ h ˙ ( r ) ∥ ∞ / κ \sup_{r\geq T}\|\dot{h}(r)\|_{\infty}/\kappa sup r ≥ T ∥ h ˙ ( r ) ∥ ∞ / κ . Letting first t t t and then T T T tend to infinity gives E ( t ) → 0 E(t)\to0 E ( t ) → 0 . To pass from log-odds to probabilities, use the Jacobian in Proposition 3.17 . The symmetric matrix C ( p ) C(p) C ( p ) has absolute row sums 2 p j ( 1 − p j ) ≤ 1 / 2 2p_j(1-p_j)\leq1/2 2 p j ( 1 − p j ) ≤ 1/2 , so ∥ C ( p ) ∥ o p ≤ 1 / 2 \|C(p)\|_{\mathrm{op}}\leq1/2 ∥ C ( p ) ∥ op ≤ 1/2 for every p ∈ P N p\in\mathcal{P}_N p ∈ P N . Applying the mean-value estimate to each pair ( η i 1 , … , η i , N − 1 , 0 ) (\eta_{i1},\ldots,\eta_{i,N-1},0) ( η i 1 , … , η i , N − 1 , 0 ) and ( h i 1 , … , h i , N − 1 , 0 ) (h_{i1},\ldots,h_{i,N-1},0) ( h i 1 , … , h i , N − 1 , 0 ) gives
∥ A ( t ) − A ∗ ( X ( t ) ) ∥ F ≤ 1 2 ∥ E ( t ) ∥ F ≤ 1 2 N ( N − 1 ) ∥ E ( t ) ∥ ∞ . \|A(t)-A^*(X(t))\|_{\mathrm{F}}\leq\frac{1}{2}\|E(t)\|_{\mathrm{F}}\leq\frac{1}{2}\sqrt{N(N-1)}\|E(t)\|_{\infty}. ∥ A ( t ) − A ∗ ( X ( t )) ∥ F ≤ 2 1 ∥ E ( t ) ∥ F ≤ 2 1 N ( N − 1 ) ∥ E ( t ) ∥ ∞ .
This proves the matrix tracking assertion.
For the second condition, use the equation for η \eta η directly:
η ( t ) − h ∞ = e − κ t ( η ( 0 ) − h ∞ ) + κ ∫ 0 t e − κ ( t − r ) ( h ( r ) − h ∞ ) d r . \eta(t)-h^\infty=e^{-\kappa t}(\eta(0)-h^\infty)+\kappa\int_{0}^{t}e^{-\kappa(t-r)}(h(r)-h^\infty)\,dr. η ( t ) − h ∞ = e − κ t ( η ( 0 ) − h ∞ ) + κ ∫ 0 t e − κ ( t − r ) ( h ( r ) − h ∞ ) d r .
The same splitting argument gives η ( t ) → h ∞ \eta(t)\to h^\infty η ( t ) → h ∞ . The inverse log-odds map in Proposition 3.14 then gives the displayed limit for A A A ; applying it to h ( t ) h(t) h ( t ) gives the same limit for A ∗ ( X ( t ) ) A^*(X(t)) A ∗ ( X ( t )) . Continuity of the bilinear scores proves the assertion for convergent features.
The first condition permits a slowly moving target with no limit. The second gives convergence to a fixed attention matrix without requiring the feature vectors themselves to settle. Persistent score variation can instead leave a nonzero lag.
Corollary 4.8 (Convergence to softmax at fixed features). Fix X ∈ X X\in\mathcal{X} X ∈ X . The unique equilibrium of (13) in A \mathcal{A} A is A ∗ ( X ) A^*(X) A ∗ ( X ) . For every A 0 ∈ A A^0\in\mathcal{A} A 0 ∈ A , its solution satisfies
lim t → ∞ ∥ A ( t ) − A ∗ ( X ) ∥ F = 0. \lim_{t\to\infty}\|A(t)-A^*(X)\|_{\mathrm{F}}=0. t → ∞ lim ∥ A ( t ) − A ∗ ( X ) ∥ F = 0. More precisely, for all i , j , k ∈ { 1 , … , N } i,j,k\in\{1,\ldots,N\} i , j , k ∈ { 1 , … , N } ,
log A i j ( t ) A i k ( t ) − log A i j ∗ ( X ) A i k ∗ ( X ) = e − γ τ t ( log A i j 0 A i k 0 − log A i j ∗ ( X ) A i k ∗ ( X ) ) . (20) \log\frac{A_{ij}(t)}{A_{ik}(t)}-\log\frac{A^*_{ij}(X)}{A^*_{ik}(X)}
=e^{-\gamma\tau t}\left(\log\frac{A^0_{ij}}{A^0_{ik}}-\log\frac{A^*_{ij}(X)}{A^*_{ik}(X)}\right). \tag*{(20)} log A ik ( t ) A ij ( t ) − log A ik ∗ ( X ) A ij ∗ ( X ) = e − γ τ t ( log A ik 0 A ij 0 − log A ik ∗ ( X ) A ij ∗ ( X ) ) . (20)
Proof. For constant scores, h ˙ = 0 \dot{h}=0 h ˙ = 0 and (19) gives E ( t ) = e − γ τ t E ( 0 ) E(t)=e^{-\gamma\tau t}E(0) E ( t ) = e − γ τ t E ( 0 ) . Theorem 4.7 yields matrix convergence. Subtracting two source log-odds gives (20) for arbitrary j , k j,k j , k . At an equilibrium, (17) forces the softmax ratios; normalisation fixes A = A ∗ ( X ) A=A^*(X) A = A ∗ ( X ) , which is indeed stationary.
Example 4.9 (Uniform initial attention). Fix X ∈ X X\in\mathcal{X} X ∈ X and take A i j 0 = 1 / N A^0_{ij}=1/N A ij 0 = 1/ N for i , j = 1 , … , N i,j=1,\ldots,N i , j = 1 , … , N . Since log A i j 0 = − log N \log A^0_{ij}=-\log N log A ij 0 = − log N , Theorem 4.5 gives
U i j ( t ) = − e − γ τ t log N + ( 1 − e − γ τ t ) s i j ( X ) τ , i , j = 1 , … , N . U_{ij}(t)=-e^{-\gamma\tau t}\log N+(1-e^{-\gamma\tau t})\frac{s_{ij}(X)}{\tau},\qquad i,j=1,\ldots,N. U ij ( t ) = − e − γ τ t log N + ( 1 − e − γ τ t ) τ s ij ( X ) , i , j = 1 , … , N . The first term is constant within each row and cancels under softmax. Hence
A ( t ) = softmax r o w ( ( 1 − e − γ τ t ) S ( X ) τ ) . A(t)=\operatorname{softmax}_{\mathrm{row}}\left(\frac{(1-e^{-\gamma\tau t})S(X)}{\tau}\right). A ( t ) = softmax row ( τ ( 1 − e − γ τ t ) S ( X ) ) . This formula exhibits the evolution from uniform attention to the equilibrium weights.
The attention energy balance
We apply the gradient-flow identity in Proposition 3.5 to the attention model (13) , using the gradient computed in Proposition 4.2 . For i = 1 , … , N i=1,\ldots,N i = 1 , … , N , write s ˙ i ( t ) : = d d t s i ( X ( t ) ) \dot{s}_i(t):=\frac{d}{dt}s_i(X(t)) s ˙ i ( t ) := d t d s i ( X ( t )) for the score derivative along the prescribed feature path.
Proposition 4.10 (Free-energy and relative-entropy balances). Let X : [ 0 , ∞ ) → X X:[0,\infty)\to\mathcal{X} X : [ 0 , ∞ ) → X be a prescribed C 1 C^1 C 1 feature path, and let A : [ 0 , ∞ ) → A A:[0,\infty)\to\mathcal{A} A : [ 0 , ∞ ) → A be the corresponding attention trajectory from Theorem 4.5 . For every $t\geq 0,
d d t F ( A ( t ) ; X ( t ) ) = − ∑ i = 1 N E i [ s ˙ i ( t ) ] − γ ∥ grad A F R F ( A ( t ) ; X ( t ) ) ∥ F R , A ( t ) 2 . (21) \frac{d}{dt}\mathcal{F}(A(t);X(t))=-\sum_{i=1}^{N}\mathbb{E}_i[\dot{s}_i(t)]-\gamma\left\|\operatorname{grad}_{A}^{\mathrm{FR}}\mathcal{F}(A(t);X(t))\right\|_{\mathrm{FR},A(t)}^2. \tag*{(21)} d t d F ( A ( t ) ; X ( t )) = − i = 1 ∑ N E i [ s ˙ i ( t )] − γ grad A FR F ( A ( t ) ; X ( t )) FR , A ( t ) 2 . (21) If the feature path is constant, X ( t ) ≡ X 0 ∈ X X(t)\equiv X_0\in\mathcal{X} X ( t ) ≡ X 0 ∈ X , then
d d t F ( A ( t ) ; X 0 ) = − γ ∥ grad A F R F ( A ( t ) ; X 0 ) ∥ F R , A ( t ) 2 . \frac{d}{dt}\mathcal{F}(A(t);X_0)=-\gamma\left\|\operatorname{grad}_{A}^{\mathrm{FR}}\mathcal{F}(A(t);X_0)\right\|_{\mathrm{FR},A(t)}^2. d t d F ( A ( t ) ; X 0 ) = − γ grad A FR F ( A ( t ) ; X 0 ) FR , A ( t ) 2 . In particular, for the equilibrium rows a ∗ ( i ) a^{* (i)} a ∗ ( i ) of A ∗ ( X 0 ) A^*(X_0) A ∗ ( X 0 ) , the free-energy gap identity (3) gives
d d t ∑ i = 1 N D K L ( a ( i ) ( t ) ∥ a ∗ ( i ) ) = − γ τ ∥ grad A F R F ( A ( t ) ; X 0 ) ∥ F R , A ( t ) 2 . (22) \frac{d}{dt}\sum_{i=1}^{N}D_{\mathrm{KL}}(a^{(i)}(t)\mathbin\Vert a^{*(i)})=-\frac{\gamma}{\tau}\left\|\operatorname{grad}_{A}^{\mathrm{FR}}\mathcal{F}(A(t);X_0)\right\|_{\mathrm{FR},A(t)}^2. \tag*{(22)} d t d i = 1 ∑ N D KL ( a ( i ) ( t ) ∥ a ∗ ( i ) ) = − τ γ grad A FR F ( A ( t ) ; X 0 ) FR , A ( t ) 2 . (22) Both the free energy and this total relative entropy have strictly negative derivatives whenever A ( t ) ≠ A ∗ ( X 0 ) A(t)\neq A^*(X_0) A ( t ) = A ∗ ( X 0 ) .
Proof. Apply Proposition 3.5 to ( A , t ) ↦ F ( A ; X ( t ) ) (A,t)\mapsto\mathcal{F}(A;X(t)) ( A , t ) ↦ F ( A ; X ( t )) on A × [ 0 , ∞ ) \mathcal{A}\times[0,\infty) A × [ 0 , ∞ ) . With A A A held fixed, its time derivative is
∂ t F ( A ; X ( t ) ) = − ∑ i , j = 1 N A i j d d t s i j ( X ( t ) ) . \partial_t\mathcal{F}(A;X(t))=-\sum_{i,j=1}^{N}A_{ij}\frac{d}{dt}s_{ij}(X(t)). ∂ t F ( A ; X ( t )) = − i , j = 1 ∑ N A ij d t d s ij ( X ( t )) .
Evaluating at A = A ( t ) A=A(t) A = A ( t ) gives (21) .
To identify the dissipation, let f i j ( t ) f_{ij}(t) f ij ( t ) be the derivatives from (15) evaluated at ( A ( t ) ; X ( t ) ) (A(t);X(t)) ( A ( t ) ; X ( t )) , and write f i ( t ) = ( f i 1 ( t ) , … , f i N ( t ) ) ⊤ f_i(t)=(f_{i1}(t),\ldots,f_{iN}(t))^\top f i ( t ) = ( f i 1 ( t ) , … , f i N ( t ) ) ⊤ . Substitution of (12) into the metric (11) gives
∥ grad A F R F ( A ( t ) ; X ( t ) ) ∥ F R , A ( t ) 2 = ∑ i , j = 1 N A i j ( t ) 2 ( f i j ( t ) − E i [ f i ( t ) ] 2 ) 2 A i j ( t ) = ∑ i , j = 1 N A i j ( t ) ( f i j ( t ) − E i [ f i ( t ) ] 2 ) 2 = ∑ i = 1 N E i [ ( f i ( t ) − E i [ f i ( t ) ] 1 ) 2 ] . (23) \begin{aligned}
\left\|\operatorname{grad}_{A}^{\mathrm{FR}}\mathcal{F}(A(t);X(t))\right\|_{\mathrm{FR},A(t)}^2
=\sum_{i,j=1}^{N}\frac{A_{ij}(t)^2\left(f_{ij}(t)-\mathbb{E}_i[f_i(t)]^2\right)^2}{A_{ij}(t)} \\
&=\sum_{i,j=1}^{N}A_{ij}(t)\left(f_{ij}(t)-\mathbb{E}_i[f_i(t)]^2\right)^2 \\
&=\sum_{i=1}^{N}\mathbb{E}_i\left[\left(f_i(t)-\mathbb{E}_i[f_i(t)]\mathbf{1}\right)^2\right]. \tag*{(23)}
\end{aligned} grad A FR F ( A ( t ) ; X ( t )) FR , A ( t ) 2 = i , j = 1 ∑ N A ij ( t ) A ij ( t ) 2 ( f ij ( t ) − E i [ f i ( t ) ] 2 ) 2 = i , j = 1 ∑ N A ij ( t ) ( f ij ( t ) − E i [ f i ( t ) ] 2 ) 2 = i = 1 ∑ N E i [ ( f i ( t ) − E i [ f i ( t )] 1 ) 2 ] . (23)
Here the square inside the expectation is taken componentwise. Every A i j ( t ) > 0 A_{ij}(t)>0 A ij ( t ) > 0 , so this norm vanishes precisely when f i j ( t ) f_{ij}(t) f ij ( t ) is constant in j j j within each row. By (15) and row normalisation, this is equivalent to A ( t ) = A ∗ ( X ( t ) ) A(t)=A^*(X(t)) A ( t ) = A ∗ ( X ( t )) , as in the proof of Proposition 2.4 . When X ( t ) ≡ X 0 X(t)\equiv X_0 X ( t ) ≡ X 0 , the scores are constant, so the first term in (21) vanishes, giving the stated free-energy dissipation identity. Differentiating (3) and using this identity gives (22) , since F ( A ∗ ( X 0 ) ; X 0 ) \mathcal{F}(A^*(X_0);X_0) F ( A ∗ ( X 0 ) ; X 0 ) is constant. The equality condition above proves the strict decrease of both quantities away from equilibrium.
The derivatives f i j ( t ) f_{ij}(t) f ij ( t ) are marginal free-energy costs. Attention relaxes by shifting weight from above-average to below-average costs, with averages taken under the current attention rows. Equation (23) shows that the dissipation rate is γ \gamma γ times the sum of the corresponding row variances; it vanishes exactly when A ( t ) = A ∗ ( X ( t ) ) A(t)=A^*(X(t)) A ( t ) = A ∗ ( X ( t )) . For fixed features, this decreases the total KL discrepancy from softmax as well as the free energy.
When the features vary, changing the scores changes F \mathcal{F} F even with the weights held fixed. This is the first term in (21) ; it can have either sign and can outweigh the attention dissipation.
Hamiltonian coupling
We now determine the joint evolution of X ( t ) X(t) X ( t ) and A ( t ) A(t) A ( t ) . The feature dynamics are Hamiltonian, and the attention relaxes by Fisher–Rao descent of the same energy.
The canonical model and its derivation
We introduce momenta P = ( p 1 , … , p N ) ∈ R d × N P=(p_1,\ldots,p_N)\in\mathbb{R}^{d\times N} P = ( p 1 , … , p N ) ∈ R d × N , identifying feature covectors with matrices via the Frobenius pairing. We consider the state space
Z = T ∗ X × A ≃ X × R d × N × A , ( X , P , A ) ∈ Z . \mathcal{Z}=T^*\mathcal{X}\times\mathcal{A}\simeq\mathcal{X}\times\mathbb{R}^{d\times N}\times\mathcal{A},\qquad(X,P,A)\in\mathcal{Z}. Z = T ∗ X × A ≃ X × R d × N × A , ( X , P , A ) ∈ Z .
For m > 0 m>0 m > 0 , define the Hamiltonian H : Z → R H:\mathcal{Z}\to\mathbb{R} H : Z → R by
H ( X , P , A ) = 1 2 m ∥ P ∥ F 2 + F ( A ; X ) , (24) H(X,P,A)=\frac{1}{2m}\lVert P\rVert_{\mathrm{F}}^2+\mathcal{F}(A;X), \tag*{(24)} H ( X , P , A ) = 2 m 1 ∥ P ∥ F 2 + F ( A ; X ) , (24)
where ∥ P ∥ F 2 = ∑ i = 1 N ∑ α = 1 d P α i 2 \lVert P\rVert_{\mathrm{F}}^2=\sum_{i=1}^{N}\sum_{\alpha=1}^{d}P_{\alpha i}^2 ∥ P ∥ F 2 = ∑ i = 1 N ∑ α = 1 d P α i 2 is the squared Frobenius norm. For smooth observables f , g : Z → R f,g:\mathcal{Z}\to\mathbb{R} f , g : Z → R , hold A A A fixed and define the canonical Poisson bracket in the feature and momentum variables by
{ f , g } = ⟨ ∇ X f , ∇ P g ⟩ F − ⟨ ∇ P f , ∇ X g ⟩ F . \{f,g\}=\langle\nabla_Xf,\nabla_Pg\rangle_{\mathrm{F}}-\langle\nabla_Pf,\nabla_Xg\rangle_{\mathrm{F}}. { f , g } = ⟨ ∇ X f , ∇ P g ⟩ F − ⟨ ∇ P f , ∇ X g ⟩ F .
We couple its Hamiltonian equations to Fisher–Rao descent in A A A , using the metric (11) . The resulting system is
X ˙ = ∇ P H = P / m , P ˙ = − ∇ X H = − ∇ X F ( A ; X ) , A ˙ = − γ grad A F R H = − γ grad A F R F ( A ; X ) . (25) \begin{aligned}
\dot{X}&=\nabla_PH=P/m,\\
\dot{P}&=-\nabla_XH=-\nabla_X\mathcal{F}(A;X),\\
\dot{A}&=-\gamma\operatorname{grad}^{\mathrm{FR}}_A H=-\gamma\operatorname{grad}^{\mathrm{FR}}_A\mathcal{F}(A;X). \tag*{(25)}
\end{aligned} X ˙ P ˙ A ˙ = ∇ P H = P / m , = − ∇ X H = − ∇ X F ( A ; X ) , = − γ grad A FR H = − γ grad A FR F ( A ; X ) . (25)
All partial gradients hold the other state variables fixed. Note that the attention equation is (13) , since the kinetic energy is independent of A A A .
For a smooth observable f : Z → R f:\mathcal{Z}\to\mathbb{R} f : Z → R , time dependence enters through the solution ( X ( t ) , P ( t ) , A ( t ) ) (X(t),P(t),A(t)) ( X ( t ) , P ( t ) , A ( t )) . The chain rule gives
d d t f ( X ( t ) , P ( t ) , A ( t ) ) = d X f [ X ˙ ] + d P f [ P ˙ ] + d A f [ A ˙ ] = { f , H } − γ g A ( grad A F R f , grad A F R H ) , (26) \begin{aligned}
\frac{d}{dt}f(X(t),P(t),A(t))=d_Xf[\dot{X}]+d_Pf[\dot{P}]+d_Af[\dot{A}] \\
&=\{f,H\}-\gamma g_A(\operatorname{grad}^{\mathrm{FR}}_A f,\operatorname{grad}^{\mathrm{FR}}_A H), \tag*{(26)}
\end{aligned} d t d f ( X ( t ) , P ( t ) , A ( t )) = d X f [ X ˙ ] + d P f [ P ˙ ] + d A f [ A ˙ ] = { f , H } − γ g A ( grad A FR f , grad A FR H ) , (26)
with all terms evaluated along the trajectory.
We prescribe X ( 0 ) = X 0 X(0)=X^0 X ( 0 ) = X 0 , P ( 0 ) = P 0 P(0)=P^0 P ( 0 ) = P 0 , and A ( 0 ) = A 0 A(0)=A^0 A ( 0 ) = A 0 , where X 0 , P 0 ∈ R d × N X^0,P^0\in\mathbb{R}^{d\times N} X 0 , P 0 ∈ R d × N and A 0 ∈ A A^0\in\mathcal{A} A 0 ∈ A . For input features X 0 X^0 X 0 , a simple choice is P 0 = 0 P^0=0 P 0 = 0 with either uniform attention A i j 0 = 1 / N A^0_{ij}=1/N A ij 0 = 1/ N for i , j = 1 , … , N i,j=1,\ldots,N i , j = 1 , … , N , or A 0 = A ∗ ( X 0 ) A^0=A^*(X^0) A 0 = A ∗ ( X 0 ) .
Proposition 5.1 (Force and energy balance). For A ∈ A A\in\mathcal{A} A ∈ A and X ∈ X X\in\mathcal{X} X ∈ X , the feature force in (25) is
− ∇ X F = M X A T + M T X A . (27) -\nabla_X\mathcal{F}=MXA^{\mathsf{T}}+M^{\mathsf{T}}XA. \tag*{(27)} − ∇ X F = M X A T + M T X A . (27) Along any solution of (25) ,
d H d t = − γ ∥ grad A F R F ∥ F R , A 2 = − 1 γ g A ( A ˙ , A ˙ ) ≤ 0. (28) \frac{dH}{dt}=-\gamma\lVert\operatorname{grad}^{\mathrm{FR}}_A\mathcal{F}\rVert_{\mathrm{FR},A}^2=-\frac{1}{\gamma}g_A(\dot{A},\dot{A})\leq0. \tag*{(28)} d t d H = − γ ∥ grad A FR F ∥ FR , A 2 = − γ 1 g A ( A ˙ , A ˙ ) ≤ 0. (28)
Proof. Hold A A A fixed and write the free energy as
F ( A ; X ) = − ∑ k , j = 1 N A k j x k T M x j + τ ∑ k , j = 1 N A k j log A k j . \mathcal{F}(A;X)=-\sum_{k,j=1}^{N}A_{kj}x_k^{\mathsf{T}}Mx_j+\tau\sum_{k,j=1}^{N}A_{kj}\log A_{kj}. F ( A ; X ) = − k , j = 1 ∑ N A k j x k T M x j + τ k , j = 1 ∑ N A k j log A k j .
The entropy term is independent of X X X . Differentiating the two occurrences of the feature vectors in each score gives, for i = 1 , … , N i=1,\ldots,N i = 1 , … , N ,
− ∇ x i F = ∑ k , j = 1 N A k j ( δ i k M x j + δ i j M T x k ) = ∑ j = 1 N A i j M x j + ∑ k = 1 N A k i M T x k . -\nabla_{x_i}\mathcal{F}=\sum_{k,j=1}^{N}A_{kj}\left(\delta_{ik}Mx_j+\delta_{ij}M^{\mathsf{T}}x_k\right)
=\sum_{j=1}^{N}A_{ij}Mx_j+\sum_{k=1}^{N}A_{ki}M^{\mathsf{T}}x_k. − ∇ x i F = k , j = 1 ∑ N A k j ( δ ik M x j + δ ij M T x k ) = j = 1 ∑ N A ij M x j + k = 1 ∑ N A k i M T x k .
These are the columns of (27) . For the energy, the chain rule underlying (26) gives
H ˙ = ⟨ P / m , P ˙ ⟩ F + ⟨ ∇ X F , X ˙ ⟩ F + d A F [ A ˙ ] . \dot{H}=\langle P/m,\dot{P}\rangle_{\mathrm{F}}+\langle\nabla_X\mathcal{F},\dot{X}\rangle_{\mathrm{F}}+d_A\mathcal{F}[\dot{A}]. H ˙ = ⟨ P / m , P ˙ ⟩ F + ⟨ ∇ X F , X ˙ ⟩ F + d A F [ A ˙ ] .
The first two terms cancel by Hamilton’s equations; metric duality and A ˙ = − γ grad A F R F \dot{A}=-\gamma\operatorname{grad}^{\mathrm{FR}}_A\mathcal{F} A ˙ = − γ grad A FR F give (28) .
The two force terms arise because x i x_i x i enters the scores both as a query and as a key. Neither M M M nor A A A needs to be symmetric.
Connection with energy-based attention
For each fixed X ∈ X X \in\mathcal{X} X ∈ X , Proposition 2.4 gives
V ( X ) : = min A ∈ A F ( A ; X ) = − τ ∑ i = 1 N log ( ∑ j = 1 N e s i j ( X ) / τ ) . (29) V(X) := \min_{A \in\mathcal{A}} \mathcal{F}(A;X) = -\tau\sum_{i=1}^{N} \log\left(\sum_{j=1}^{N} e^{s_{ij}(X)/\tau}\right). \tag*{(29)} V ( X ) := A ∈ A min F ( A ; X ) = − τ i = 1 ∑ N log ( j = 1 ∑ N e s ij ( X ) / τ ) . (29)
Combining (24) with (3) gives the exact decomposition
H ( X , P , A ) = 1 2 m ∥ P ∥ F 2 + V ( X ) + τ ∑ i = 1 N D K L ( a ( i ) ∥ a ∗ ( i ) ( X ) ) . (30) H(X,P,A) = \frac{1}{2m}\lVert P\rVert_{F}^{2} + V(X) + \tau\sum_{i=1}^{N} D_{\mathrm{KL}}\left(a^{(i)} \mathbin{\|} a^{*(i)}(X)\right). \tag*{(30)} H ( X , P , A ) = 2 m 1 ∥ P ∥ F 2 + V ( X ) + τ i = 1 ∑ N D KL ( a ( i ) ∥ a ∗ ( i ) ( X ) ) . (30)
Thus the excess over the equilibrated Hamiltonian is exactly the free-energy gap in (3) .
The reduced potential (29) has the same log-sum-exp form as the attention energy in Energy Transformer [8 ] . Our model adds feature inertia and gives A A A its own Fisher–Rao relaxation, rather than computing attention directly from the current scores.
Note that computing attention directly from the current scores, by imposing A = A ∗ ( X ) A = A^{*}(X) A = A ∗ ( X ) , defines a reduced version of our Hamiltonian model with energy ∥ P ∥ F 2 / ( 2 m ) + V ( X ) \lVert P\rVert_{F}^{2}/(2m) + V(X) ∥ P ∥ F 2 / ( 2 m ) + V ( X ) . The envelope identity is
∇ X V ( X ) = ∇ X F ( A ; X ) ∣ A = A ∗ ( X ) . \nabla_{X}V(X) = \left.\nabla_{X}\mathcal{F}(A;X)\right|_{A=A^{*}(X)}. ∇ X V ( X ) = ∇ X F ( A ; X ) ∣ A = A ∗ ( X ) .
where the gradient on the right differentiates X X X with A A A held fixed. Indeed, V ( X ) = F ( A ∗ ( X ) ; X ) V(X) = \mathcal{F}(A^{*}(X);X) V ( X ) = F ( A ∗ ( X ) ; X ) , so for a feature variation Z ∈ R d × N Z \in\mathbb{R}^{d \times N} Z ∈ R d × N the chain rule gives
d V ( X ) [ Z ] = d X F ( A ∗ ( X ) ; X ) [ Z ] + d A F ( A ∗ ( X ) ; X ) [ D A ∗ ( X ) [ Z ] ] . dV(X)[Z] = d_{X}\mathcal{F}(A^{*}(X);X)[Z] + d_{A}\mathcal{F}(A^{*}(X);X)[DA^{*}(X)[Z]]. d V ( X ) [ Z ] = d X F ( A ∗ ( X ) ; X ) [ Z ] + d A F ( A ∗ ( X ) ; X ) [ D A ∗ ( X ) [ Z ]] .
Differentiating row normalisation gives ∑ j = 1 N D A i j ∗ ( X ) [ Z ] = 0 \sum_{j=1}^{N} DA^{*}_{ij}(X)[Z] = 0 ∑ j = 1 N D A ij ∗ ( X ) [ Z ] = 0 for each row i i i , so D A ∗ ( X ) [ Z ] ∈ T A ∗ ( X ) A DA^{*}(X)[Z] \in T_{A^{*}(X)}\mathcal{A} D A ∗ ( X ) [ Z ] ∈ T A ∗ ( X ) A . The second term therefore vanishes by first-order optimality of A ∗ ( X ) A^{*}(X) A ∗ ( X ) . Since Z Z Z is arbitrary, the gradient identity follows.
Momentum Transformer introduces momentum into linear attention [10 ] , while SympFormer develops inertial Hamiltonian dynamics on spaces of token distributions [13 ] . In our formulation, the Fisher–Rao geometry acts on the rows of A A A , and attention has its own evolution equation. A common energy couples this relaxation to the canonical dynamics of ( X , P ) (X,P) ( X , P ) , yielding the dissipation law (28) .
Two alternative Hamiltonian formulations
We present two alternative Hamiltonians incorporating the attention value map: one through an interaction penalty, the other by setting the representation velocity equal to the attention output.
A value-matching energy
The force in (27) depends only on the query and key maps. To let the supplied values also guide the representation dynamics while retaining inertia, we add an attention-weighted squared discrepancy between x i x_i x i and W V x j W_V x_j W V x j to the interaction energy. This favours agreement between each receiving representation and the values to which it assigns weight. For λ > 0 \lambda> 0 λ > 0 , define H λ : Z → R H_{\lambda} : \mathcal{Z} \to\mathbb{R} H λ : Z → R by
H λ ( X , P , A ) = 1 2 λ m ∥ P ∥ F 2 + F λ ( A ; X ) , H_{\lambda}(X,P,A) = \frac{1}{2\lambda m}\lVert P\rVert_{F}^{2} + \mathcal{F}_{\lambda}(A;X), H λ ( X , P , A ) = 2 λm 1 ∥ P ∥ F 2 + F λ ( A ; X ) ,
F λ ( A ; X ) = F ( A ; X ) + λ 2 ∑ i , j = 1 N A i j ∥ x i − W V x j ∥ 2 . (31) \mathcal{F}_{\lambda}(A;X) = \mathcal{F}(A;X) + \frac{\lambda}{2}\sum_{i,j=1}^{N} A_{ij}\lVert x_i - W_Vx_j\rVert^{2}. \tag*{(31)} F λ ( A ; X ) = F ( A ; X ) + 2 λ i , j = 1 ∑ N A ij ∥ x i − W V x j ∥ 2 . (31)
The parameter λ \lambda λ controls the strength of this value-matching penalty. With the same canonical and Fisher–Rao structures, the equations are
X ˙ = P / m , P ˙ = − ∇ X F λ ( A ; X ) , A ˙ = − γ grad A F R F λ ( A ; X ) . \dot{X}=P/m,\qquad\dot{P}=-\nabla_X\mathcal{F}_{\lambda}(A;X),\qquad\dot{A}=-\gamma\operatorname{grad}^{\mathrm{FR}}_{A}\mathcal{F}_{\lambda}(A;X). X ˙ = P / m , P ˙ = − ∇ X F λ ( A ; X ) , A ˙ = − γ grad A FR F λ ( A ; X ) .
Extending F λ \mathcal{F}_{\lambda} F λ in (5.8) to positive matrices and differentiating with respect to A i j A_{ij} A ij gives
∂ F λ ∂ A i j = − s ~ i j ( X ) + τ ( 1 + log A i j ) , i , j = 1 , … , N . \frac{\partial\mathcal{F}_{\lambda}}{\partial A_{ij}}=-\widetilde{s}_{ij}(X)+\tau(1+\log A_{ij}),\qquad i,j=1,\ldots,N. ∂ A ij ∂ F λ = − s ij ( X ) + τ ( 1 + log A ij ) , i , j = 1 , … , N .
where
s ~ i j ( X ) = s i j ( X ) − λ 2 ∥ x i − W V x j ∥ 2 . \widetilde{s}_{ij}(X)=s_{ij}(X)-\frac{\lambda}{2}\lVert x_i-W_Vx_j\rVert^2. s ij ( X ) = s ij ( X ) − 2 λ ∥ x i − W V x j ∥ 2 .
Thus, at fixed X X X , the equilibrium attention is softmax r o w ( S ~ ( X ) / τ ) \operatorname{softmax}_{\mathrm{row}}(\widetilde{S}(X)/\tau) softmax row ( S ( X ) / τ ) , where S ~ = ( s ~ i j ) \widetilde{S}=(\widetilde{s}_{ij}) S = ( s ij ) , and the logit equation is U ˙ = γ ( S ~ ( X ) − τ U ) \dot{U}=\gamma(\widetilde{S}(X)-\tau U) U ˙ = γ ( S ( X ) − τ U ) .
Differentiating the penalty and using row normalisation gives, for i = 1 , … , N i=1,\ldots,N i = 1 , … , N ,
− ∇ x i F λ = − ∇ x i F + λ ( ∑ j = 1 N A i j W V x j − x i ) + λ W V T ∑ k = 1 N A k i ( x k − W V x i ) . -\nabla_{x_i}\mathcal{F}_{\lambda}=-\nabla_{x_i}\mathcal{F}+\lambda\left(\sum_{j=1}^{N}A_{ij}W_Vx_j-x_i\right)+\lambda W_V^{\mathsf{T}}\sum_{k=1}^{N}A_{ki}(x_k-W_Vx_i). − ∇ x i F λ = − ∇ x i F + λ ( j = 1 ∑ N A ij W V x j − x i ) + λ W V T k = 1 ∑ N A k i ( x k − W V x i ) .
The first additional term pulls x i x_i x i towards its attention-weighted value average. The second accounts for how changing x i x_i x i changes the values supplied to other tokens. Applying (5.3) with H λ H_{\lambda} H λ from (5.8) gives the energy law
d H λ d t = − γ ∥ grad A F R F λ ∥ F R , A 2 . \frac{dH_{\lambda}}{dt}=-\gamma\lVert\operatorname{grad}^{\mathrm{FR}}_{A}\mathcal{F}_{\lambda}\rVert_{\mathrm{FR},A}^{2}. d t d H λ = − γ ∥ grad A FR F λ ∥ FR , A 2 .
A cotangent lift of the attention velocity
A different objective is to reproduce the continuous feature velocity associated with residual updates X n + 1 = X n + h W V X n ( A n ) T X^{n+1}=X^n+hW_VX^n(A^n)^{\mathsf{T}} X n + 1 = X n + h W V X n ( A n ) T . To impose the corresponding law X ˙ = W V X A T \dot{X}=W_VXA^{\mathsf{T}} X ˙ = W V X A T , we add the attention free energy to the cotangent-lift Hamiltonian and define H l i f t : Z → R H_{\mathrm{lift}}:Z\to\mathbb{R} H lift : Z → R by
H l i f t ( X , P , A ) = ⟨ P , W V X A T ⟩ F + F ( A ; X ) . (32) H_{\mathrm{lift}}(X,P,A)=\langle P,W_VXA^{\mathsf{T}}\rangle_F+\mathcal{F}(A;X). \tag*{(32)} H lift ( X , P , A ) = ⟨ P , W V X A T ⟩ F + F ( A ; X ) . (32)
Its canonical equations, coupled to Fisher–Rao descent in A A A , are
X ˙ = W V X A T , P ˙ = − W V T P A − ∇ X F , A ˙ = − γ grad A F R H l i f t . \dot{X}=W_VXA^{\mathsf{T}},\qquad\dot{P}=-W_V^{\mathsf{T}}PA-\nabla_X\mathcal{F},\qquad\dot{A}=-\gamma\operatorname{grad}^{\mathrm{FR}}_{A}H_{\mathrm{lift}}. X ˙ = W V X A T , P ˙ = − W V T P A − ∇ X F , A ˙ = − γ grad A FR H lift .
Extending H l i f t H_{\mathrm{lift}} H lift in (5.9) to positive matrices and differentiating with respect to A i j A_{ij} A ij gives
∂ H l i f t ∂ A i j = p i T W V x j − s i j ( X ) + τ ( 1 + log A i j ) , i , j = 1 , … , N . \frac{\partial H_{\mathrm{lift}}}{\partial A_{ij}}=p_i^{\mathsf{T}}W_Vx_j-s_{ij}(X)+\tau(1+\log A_{ij}),\qquad i,j=1,\ldots,N. ∂ A ij ∂ H lift = p i T W V x j − s ij ( X ) + τ ( 1 + log A ij ) , i , j = 1 , … , N .
so the attention equation uses scores s i j ( X ) − p i T W V x j s_{ij}(X)-p_i^{\mathsf{T}}W_Vx_j s ij ( X ) − p i T W V x j . Here H l i f t H_{\mathrm{lift}} H lift is linear in P P P : the momentum is an auxiliary conjugate variable, with no relation P = m X ˙ P=m\dot{X} P = m X ˙ . Applying (5.3) with H l i f t H_{\mathrm{lift}} H lift from (5.9) gives
d H l i f t d t = − γ ∥ grad A F R H l i f t ∥ F R , A 2 ≤ 0. \frac{dH_{\mathrm{lift}}}{dt}=-\gamma\lVert\operatorname{grad}^{\mathrm{FR}}_{A}H_{\mathrm{lift}}\rVert_{\mathrm{FR},A}^{2}\leq0. d t d H lift = − γ ∥ grad A FR H lift ∥ FR , A 2 ≤ 0.
Global well-posedness
We now establish global existence and uniqueness for the three coupled models.
Theorem 5.4 (Global well-posedness of the coupled models). Let H \mathcal{H} H be one of the Hamiltonians H H H , H λ H_{\lambda} H λ , or H l i f t H_{\mathrm{lift}} H lift , with m > 0 m>0 m > 0 for H H H and H λ H_{\lambda} H λ , and λ > 0 \lambda>0 λ > 0 for H λ H_{\lambda} H λ . For every initial state ( X 0 , P 0 , A 0 ) ∈ Z (X^{0},P^{0},A^{0})\in\mathcal{Z} ( X 0 , P 0 , A 0 ) ∈ Z , the system
X ˙ = ∇ P H , P ˙ = − ∇ X H , A ˙ = − γ grad A F R H \dot{X}=\nabla_{P}\mathcal{H},\qquad\dot{P}=-\nabla_{X}\mathcal{H},\qquad\dot{A}=-\gamma\operatorname{grad}_{A}^{\mathrm{FR}}\mathcal{H} X ˙ = ∇ P H , P ˙ = − ∇ X H , A ˙ = − γ grad A FR H admits a unique smooth solution
( X , P , A ) : [ 0 , ∞ ) ⟶ Z , ( X ( 0 ) , P ( 0 ) , A ( 0 ) ) = ( X 0 , P 0 , A 0 ) . (X,P,A):[0,\infty)\longrightarrow\mathcal{Z},\qquad(X(0),P(0),A(0))=(X^{0},P^{0},A^{0}). ( X , P , A ) : [ 0 , ∞ ) ⟶ Z , ( X ( 0 ) , P ( 0 ) , A ( 0 )) = ( X 0 , P 0 , A 0 ) . For each t ≥ 0 t\geq0 t ≥ 0 , the solution depends smoothly on its initial state. Each attention row remains normalised, and for every finite T > 0 T>0 T > 0 ,
min 0 ≤ t ≤ T 1 ≤ i , j ≤ N A i j ( t ) > 0. \min_{\substack{0\leq t\leq T\\1\leq i,j\leq N}}A_{ij}(t)>0. 0 ≤ t ≤ T 1 ≤ i , j ≤ N min A ij ( t ) > 0.
Proof. For each Hamiltonian, define its effective score map Σ : X × R d × N ⟶ R N × N \Sigma:\mathcal{X}\times\mathbb{R}^{d\times N}\longrightarrow\mathbb{R}^{N\times N} Σ : X × R d × N ⟶ R N × N by
Σ ( X , P ) = { S ( X ) , H = H , S ~ ( X ) , H = H λ , S ( X ) − P T W V X , H = H l i f t . \Sigma(X,P)=
\begin{cases}
S(X), & \mathcal{H}=H,\\
\widetilde{S}(X), & \mathcal{H}=H_{\lambda},\\
S(X)-P^{\mathsf{T}}W_{V}X, & \mathcal{H}=H_{\mathrm{lift}}.
\end{cases} Σ ( X , P ) = ⎩ ⎨ ⎧ S ( X ) , S ( X ) , S ( X ) − P T W V X , H = H , H = H λ , H = H lift .
The attention derivatives computed above have the common form ∂ H / ∂ A i j = − Σ i j ( X , P ) + τ ( 1 + log A i j ) \partial\mathcal{H}/\partial A_{ij}=-\Sigma_{ij}(X,P)+\tau(1+\log A_{ij}) ∂ H / ∂ A ij = − Σ ij ( X , P ) + τ ( 1 + log A ij ) . Applying Lemma 3.18 to each row therefore gives
A = softmax r o w ( U ) , U ˙ = γ ( Σ ( X , P ) − τ U ) , U ( 0 ) = log A 0 . A=\operatorname{softmax}_{\mathrm{row}}(U),\qquad\dot{U}=\gamma\bigl(\Sigma(X,P)-\tau U\bigr),\qquad U(0)=\log A^{0}. A = softmax row ( U ) , U ˙ = γ ( Σ ( X , P ) − τ U ) , U ( 0 ) = log A 0 .
Together with the canonical equations, this defines a smooth vector field on X × R d × N × R N × N \mathcal{X}\times\mathbb{R}^{d\times N}\times\mathbb{R}^{N\times N} X × R d × N × R N × N . Local ODE theory gives a unique maximal solution ( X , P , U ) (X,P,U) ( X , P , U ) on [ 0 , T max ) [0,T_{\max}) [ 0 , T m a x ) , with initial state ( X 0 , P 0 , log A 0 ) (X^{0},P^{0},\log A^{0}) ( X 0 , P 0 , log A 0 ) .
For each model, the feature and momentum equations are linear in ( X , P ) (X,P) ( X , P ) when A A A is fixed. Their coefficients extend continuously to the compact set of nonnegative row-normalised matrices, so they are uniformly bounded. Set Q ( t ) = ∥ X ( t ) ∥ F 2 + ∥ P ( t ) ∥ F 2 Q(t)=\lVert X(t)\rVert_{\mathrm{F}}^{2}+\lVert P(t)\rVert_{\mathrm{F}}^{2} Q ( t ) = ∥ X ( t ) ∥ F 2 + ∥ P ( t ) ∥ F 2 . There is a constant C > 0 C>0 C > 0 , depending only on the fixed parameters, such that
Q ˙ ( t ) = 2 ⟨ X , X ˙ ⟩ F + 2 ⟨ P , P ˙ ⟩ F ≤ C Q ( t ) . \dot{Q}(t)=2\langle X,\dot{X}\rangle_{\mathrm{F}}+2\langle P,\dot{P}\rangle_{\mathrm{F}}\leq CQ(t). Q ˙ ( t ) = 2 ⟨ X , X ˙ ⟩ F + 2 ⟨ P , P ˙ ⟩ F ≤ C Q ( t ) .
Grönwall’s inequality gives Q ( t ) ≤ Q ( 0 ) e C t Q(t)\leq Q(0)e^{Ct} Q ( t ) ≤ Q ( 0 ) e C t for t < T max t<T_{\max} t < T m a x . Every entry of Σ \Sigma Σ is quadratic in ( X , P ) (X,P) ( X , P ) , so for some constant K > 0 K>0 K > 0 , ∥ Σ ( X ( t ) , P ( t ) ) ∥ F ≤ K Q ( t ) \lVert\Sigma(X(t),P(t))\rVert_{\mathrm{F}}\leq KQ(t) ∥ Σ ( X ( t ) , P ( t )) ∥ F ≤ K Q ( t ) . Integrating the logit equation, as in (18) , gives
U ( t ) = e − γ τ t log A 0 + γ ∫ 0 t e − γ τ ( t − s ) Σ ( X ( s ) , P ( s ) ) d s , U(t)=e^{-\gamma\tau t}\log A^{0}+\gamma\int_{0}^{t}e^{-\gamma\tau(t-s)}\Sigma(X(s),P(s))\,\mathrm{d}s, U ( t ) = e − γ τ t log A 0 + γ ∫ 0 t e − γ τ ( t − s ) Σ ( X ( s ) , P ( s )) d s ,
∥ U ( t ) ∥ F ≤ ∥ log A 0 ∥ F + K τ Q ( 0 ) e C t . \lVert U(t)\rVert_{\mathrm{F}}\leq\lVert\log A^{0}\rVert_{\mathrm{F}}+\frac{K}{\tau}Q(0)e^{Ct}. ∥ U ( t ) ∥ F ≤ ∥ log A 0 ∥ F + τ K Q ( 0 ) e C t .
If T max < ∞ T_{\max}<\infty T m a x < ∞ , these estimates bound ( X , P , U ) (X,P,U) ( X , P , U ) uniformly up to T max T_{\max} T m a x . Since the lifted vector field is smooth on the whole Euclidean state space, the ODE continuation theorem would extend the solution beyond T max T_{\max} T m a x . Hence T max = ∞ T_{\max}=\infty T m a x = ∞ .
The row softmax of U U U solves the original attention equation and preserves row normalisation. For R T = sup 0 ≤ t ≤ T ∥ U ( t ) ∥ F < ∞ R_{T}=\sup_{0\leq t\leq T}\lVert U(t)\rVert_{\mathrm{F}}<\infty R T = sup 0 ≤ t ≤ T ∥ U ( t ) ∥ F < ∞ ,
A i j ( t ) = e U i j ( t ) ∑ k = 1 N e U i k ( t ) ≥ e − 2 R T N > 0 , 0 ≤ t ≤ T , i , j = 1 , … , N . A_{ij}(t)=\frac{e^{U_{ij}(t)}}{\sum_{k=1}^{N}e^{U_{ik}(t)}}\geq\frac{e^{-2R_{T}}}{N}>0,\qquad0\leq t\leq T,\quad i,j=1,\ldots,N. A ij ( t ) = ∑ k = 1 N e U ik ( t ) e U ij ( t ) ≥ N e − 2 R T > 0 , 0 ≤ t ≤ T , i , j = 1 , … , N .
Finally, the original vector field is smooth on Z \mathcal{Z} Z , so uniqueness and smooth dependence on the initial state follow from smooth ODE theory. □ \square □
Theorem 5.4 concerns the three Hamiltonians specified above. Additional feature potentials as in Remark 5.3 require appropriate growth or coercivity assumptions for global existence. Long-time boundedness and convergence require further analysis.
Summary and Outlook
We developed a variational framework in which softmax minimises an attention free energy. Its Fisher–Rao gradient flow gives the entropy-regularised replicator equation (14) , which preserves row normalisation and positivity for positive initial attention. We proved convergence to the softmax equilibrium at fixed token representations. For prescribed time-dependent representations, we introduced a change to logit variables that linearises the attention dynamics and yields tracking estimates and sufficient conditions for attention to converge to a fixed matrix.
In § 5 , we coupled this attention evolution to Hamiltonian dynamics for the representations and momenta through a common energy, thereby obtaining the closed system (25) . The canonical Poisson bracket governs the conservative evolution of ( X , P ) (X, P) ( X , P ) , while the Fisher–Rao gradient flow in A A A dissipates the total Hamiltonian. We also derived a Lagrange–d’Alembert formulation with Rayleigh dissipation. Finally, we established global well-posedness for this model and the two alternatives involving the value map (Theorem 5.4 ).
For future work, numerical simulations could investigate how feature inertia and finite attention relaxation jointly affect the representations. A useful comparison is with the reduced Hamiltonian model that imposes A ( t ) = A ∗ ( X ( t ) ) A(t) = A^*(X(t)) A ( t ) = A ∗ ( X ( t )) at every time, starting from the same initial representations and momenta. This would help identify how attention memory and lag affect feature evolution.
Stochastic perturbations of the logits provide a way to model fluctuations while maintaining positive, normalised attention. The connection between replicator–mutator dynamics and nonlinear filtering [12 ] suggests studying how such perturbations affect tracking and the coupled energy balance.
A further direction might involve a spatial continuum theory of nonequilibrium attention. One possibility is to treat attention as an internal variable, analogous to an order parameter in complex-fluid models. Such a theory would require specifying the continuum state variables and a suitable Lie–Poisson structure for their conservative evolution, together with a continuum Fisher–Rao gradient flow for attention relaxation.
Acknowledgements
We are grateful for thoughtful discussions during this work with C. J. Cotter, R. Hu, W. Pan, O. D. Street, and C. Tronci. DH is grateful for partial support from the European Research Council (ERC) Synergy grant Stochastic Transport in Upper Ocean Dynamics (STUOD) – DLV-856408. ABDL is grateful for support from the Universidad de La Laguna through its Programa Propio de Apoyo a la Investigación 2026, under the scheme “Proyectos liderados por personal investigador novel”.
Use of artificial intelligence
The authors used large language models to assist with language editing, corrections, optical character recognition (OCR) of handwritten iPad notes and other text notes, \LaTeX coding, and figure preparation. The scientific ideas and results were developed by the authors, who take full responsibility for the content of the manuscript.
Supporting geometric arguments
The definitions needed for the dynamics are given in § 3 . This appendix supplies their supporting arguments and the entropy interpretations of the Fisher–Rao metric.
Induced gradients
Proof of Proposition 3.8. Fix p ∈ M p \in M p ∈ M and v ∈ T p M v \in T_pM v ∈ T p M . By orthogonality and Definition 3.3,
g p ( Π p grad h F ( p ) , v ) = h p ( grad h F ( p ) , v ) = d F ‾ p ( v ) = d F p ( v ) . g_p\left(\Pi_p\operatorname{grad}_h F(p),v\right)=h_p\left(\operatorname{grad}_h F(p),v\right)=d\overline{F}_p(v)=dF_p(v). g p ( Π p grad h F ( p ) , v ) = h p ( grad h F ( p ) , v ) = d F p ( v ) = d F p ( v ) .
The defining metric duality identifies this vector with grad g F ( p ) \operatorname{grad}_g F(p) grad g F ( p ) . It also proves independence of the ambient extension.
The tangent space of the simplex
We prove Lemma 3.10.
Proof. Let h : U n → R h: U_n \to\mathbb{R} h : U n → R be h ( x ) = ∑ j = 1 n x j h(x)=\sum_{j=1}^{n}x_j h ( x ) = ∑ j = 1 n x j . Its differential d h x ( v ) = ∑ j = 1 n v j dh_x(v)=\sum_{j=1}^{n}v_j d h x ( v ) = ∑ j = 1 n v j is surjective at every x x x . Hence P n = h − 1 ( 1 ) \mathcal{P}_n=h^{-1}(1) P n = h − 1 ( 1 ) is a regular level set of dimension n − 1 n-1 n − 1 , with tangent space T p P n = ker d h p T_p\mathcal{P}_n=\ker dh_p T p P n = ker d h p at each p ∈ P n p \in\mathcal{P}_n p ∈ P n . Differentiating ∑ j = 1 n p j ( t ) = 1 \sum_{j=1}^{n}p_j(t)=1 ∑ j = 1 n p j ( t ) = 1 along a curve in P n \mathcal{P}_n P n gives ∑ j = 1 n p ˙ j ( t ) = 0 \sum_{j=1}^{n}\dot{p}_j(t)=0 ∑ j = 1 n p ˙ j ( t ) = 0 . Conversely, for any v ∈ R n v \in\mathbb{R}^n v ∈ R n with zero sum, p + t v p+tv p + t v lies in P n \mathcal{P}_n P n for all sufficiently small ∣ t ∣ |t| ∣ t ∣ .
Relative entropy and the Fisher metric
Proposition A.1 (Nonnegativity and local quadratic form). For p , q ∈ P n p,q \in\mathcal{P}_n p , q ∈ P n , one has D K L ( p ∥ q ) ≥ 0 \mathrm{D}_{\mathrm{KL}}(p\|q) \ge0 D KL ( p ∥ q ) ≥ 0 , with equality if and only if p = q p=q p = q . For fixed p ∈ P n p \in\mathcal{P}_n p ∈ P n and v ∈ T p P n v \in T_p\mathcal{P}_n v ∈ T p P n ,
D K L ( p + ε v ∥ p ) = ε 2 2 g p ( v , v ) + O ( ε 3 ) , \mathrm{D}_{\mathrm{KL}}(p+\varepsilon v\|p)=\frac{\varepsilon^2}{2}g_p(v,v)+O(\varepsilon^3), D KL ( p + ε v ∥ p ) = 2 ε 2 g p ( v , v ) + O ( ε 3 ) , D K L ( p ∥ p + ε v ) = ε 2 2 g p ( v , v ) + O ( ε 3 ) , \mathrm{D}_{\mathrm{KL}}(p\|p+\varepsilon v)=\frac{\varepsilon^2}{2}g_p(v,v)+O(\varepsilon^3), D KL ( p ∥ p + ε v ) = 2 ε 2 g p ( v , v ) + O ( ε 3 ) , ( ε → 0 ) (\varepsilon\to0) ( ε → 0 ) .
Proof. The inequality log x ≤ x − 1 \log x \le x-1 log x ≤ x − 1 gives
− D K L ( p ∥ q ) = ∑ j = 1 n p j log q j p j ≤ ∑ j = 1 n p j ( q j p j − 1 ) = 0. -\mathrm{D}_{\mathrm{KL}}(p\|q)=\sum_{j=1}^{n}p_j\log\frac{q_j}{p_j}\le\sum_{j=1}^{n}p_j\left(\frac{q_j}{p_j}-1\right)=0. − D KL ( p ∥ q ) = j = 1 ∑ n p j log p j q j ≤ j = 1 ∑ n p j ( p j q j − 1 ) = 0.
Equality requires q j / p j = 1 q_j/p_j=1 q j / p j = 1 for every j = 1 , … , n j=1,\ldots,n j = 1 , … , n . For sufficiently small ∣ ε ∣ |\varepsilon| ∣ ε ∣ , the vector p + ε v p+\varepsilon v p + ε v is positive and normalised. Expanding in the first argument gives
D K L ( p + ε v ∥ p ) = ∑ j = 1 n ( p j + ε v j ) log ( 1 + ε v j p j ) = ε ∑ j = 1 n v j + ε 2 2 ∑ j = 1 n v j 2 p j + O ( ε 3 ) . \begin{aligned}
\mathrm{D}_{\mathrm{KL}}(p+\varepsilon v\|p)=\sum_{j=1}^{n}(p_j+\varepsilon v_j)\log\left(1+\varepsilon\frac{v_j}{p_j}\right) \\
&=\varepsilon\sum_{j=1}^{n}v_j+\frac{\varepsilon^2}{2}\sum_{j=1}^{n}\frac{v_j^2}{p_j}+O(\varepsilon^3).
\end{aligned} D KL ( p + ε v ∥ p ) = j = 1 ∑ n ( p j + ε v j ) log ( 1 + ε p j v j ) = ε j = 1 ∑ n v j + 2 ε 2 j = 1 ∑ n p j v j 2 + O ( ε 3 ) .
For the second argument,
D K L ( p ∥ p + ε v ) = − ∑ j = 1 n p j log ( 1 + ε v j p j ) = − ε ∑ j = 1 n v j + ε 2 2 ∑ j = 1 n v j 2 p j + O ( ε 3 ) . \begin{aligned}
\mathrm{D}_{\mathrm{KL}}(p\|p+\varepsilon v)=-\sum_{j=1}^{n}p_j\log\left(1+\varepsilon\frac{v_j}{p_j}\right) \\
&=-\varepsilon\sum_{j=1}^{n}v_j+\frac{\varepsilon^2}{2}\sum_{j=1}^{n}\frac{v_j^2}{p_j}+O(\varepsilon^3).
\end{aligned} D KL ( p ∥ p + ε v ) = − j = 1 ∑ n p j log ( 1 + ε p j v j ) = − ε j = 1 ∑ n v j + 2 ε 2 j = 1 ∑ n p j v j 2 + O ( ε 3 ) .
Both linear terms vanish by (7) , leaving the Fisher–Rao quadratic form (8) .
Entropy, the Fisher metric and log-odds
Proposition A.2 (Entropy, the Fisher metric and log-odds). Let E : P n → R \mathcal{E}:\mathcal{P}_n\to\mathbb{R} E : P n → R be the negative entropy, E ( p ) = ∑ j = 1 n p j log p j \mathcal{E}(p)=\sum_{j=1}^{n}p_j\log p_j E ( p ) = ∑ j = 1 n p j log p j . In affine coordinates z = ( p 1 , … , p n − 1 ) z=(p_1,\ldots,p_{n-1}) z = ( p 1 , … , p n − 1 ) , with p n = 1 − ∑ j = 1 n − 1 z j p_n=1-\sum_{j=1}^{n-1}z_j p n = 1 − ∑ j = 1 n − 1 z j , write E ^ ( z ) = E ( p ( z ) ) \widehat{\mathcal{E}}(z)=\mathcal{E}(p(z)) E ( z ) = E ( p ( z )) . Then
∂ E ^ ∂ z j = log p j p n = η j , j = 1 , … , n − 1 , \frac{\partial\widehat{\mathcal{E}}}{\partial z_j}=\log\frac{p_j}{p_n}=\eta_j,\qquad j=1,\ldots,n-1, ∂ z j ∂ E = log p n p j = η j , j = 1 , … , n − 1 , ∂ 2 E ^ ∂ z j ∂ z k = δ j k p j + 1 p n , j , k = 1 , … , n − 1. \frac{\partial^2\widehat{\mathcal{E}}}{\partial z_j\partial z_k}=\frac{\delta_{jk}}{p_j}+\frac{1}{p_n},\qquad j,k=1,\ldots,n-1. ∂ z j ∂ z k ∂ 2 E = p j δ j k + p n 1 , j , k = 1 , … , n − 1. The Hessian is the coordinate matrix of the Fisher–Rao metric.
Proof. Differentiating p n = 1 − ∑ j = 1 n − 1 z j p_n=1-\sum_{j=1}^{n-1}z_j p n = 1 − ∑ j = 1 n − 1 z j in the entropy expression gives the stated derivatives. For the standard basis e 1 , … , e n e_1,\ldots,e_n e 1 , … , e n of R n \mathbb{R}^n R n , we have ∂ p / ∂ z j = e j − e n \partial p/\partial z_j=e_j-e_n ∂ p / ∂ z j = e j − e n . Formula (8) gives g p ( e j − e n , e k − e n ) = δ j k / p j + 1 / p n g_p(e_j-e_n,e_k-e_n)=\delta_{jk}/p_j+1/p_n g p ( e j − e n , e k − e n ) = δ j k / p j + 1/ p n , identifying the Hessian with the coordinate metric matrix. □ \square □
Residual blocks and continuous depth
We record how scaled residual blocks approximate a differential equation for the token representations.
Residual blocks
Consider L ≥ 1 L\geq1 L ≥ 1 residual blocks, indexed by ℓ = 0 , … , L − 1 \ell=0,\ldots,L-1 ℓ = 0 , … , L − 1 . We retain one unmasked attention head and a feed-forward map, omitting layer normalisation. For block ℓ \ell ℓ , take matrices W Q , ℓ , W K , ℓ ∈ R d k × d W_{Q,\ell},W_{K,\ell}\in\mathbb{R}^{d_k\times d} W Q , ℓ , W K , ℓ ∈ R d k × d and W V , ℓ ∈ R d × d W_{V,\ell}\in\mathbb{R}^{d\times d} W V , ℓ ∈ R d × d . Let A ℓ ∗ ( X ) A_\ell^*(X) A ℓ ∗ ( X ) be the softmax matrix in (1) computed with these matrices. The attention output map is
Att ℓ : X ⟶ X , Att ℓ ( X ) = W V , ℓ X ( A ℓ ∗ ( X ) ) T . \operatorname{Att}_{\ell}:\mathcal{X}\longrightarrow\mathcal{X},\qquad\operatorname{Att}_{\ell}(X)=W_{V,\ell}X\left(A_\ell^*(X)\right)^{\mathsf{T}}. Att ℓ : X ⟶ X , Att ℓ ( X ) = W V , ℓ X ( A ℓ ∗ ( X ) ) T .
Let MLP ℓ : X → X \operatorname{MLP}_{\ell}:\mathcal{X}\to\mathcal{X} MLP ℓ : X → X apply the same smooth feed-forward map R d → R d \mathbb{R}^{d}\to\mathbb{R}^{d} R d → R d to every column. A residual block adds the attention output to its input, then adds the feed-forward output to the resulting representation:
B ℓ = ( Id + MLP ℓ ) ∘ ( Id + Att ℓ ) : X → X . (33) B_{\ell}=(\operatorname{Id}+\operatorname{MLP}_{\ell})\circ(\operatorname{Id}+\operatorname{Att}_{\ell}):\mathcal{X}\to\mathcal{X}. \tag*{(33)} B ℓ = ( Id + MLP ℓ ) ∘ ( Id + Att ℓ ) : X → X . (33)
Starting from an input feature matrix X ( 0 ) = X 0 ∈ X X^{(0)}=X^0\in\mathcal{X} X ( 0 ) = X 0 ∈ X , the forward computation is X ( ℓ + 1 ) = B ℓ ( X ( ℓ ) ) X^{(\ell+1)}=B_{\ell}(X^{(\ell)}) X ( ℓ + 1 ) = B ℓ ( X ( ℓ ) ) for ℓ = 0 , … , L − 1 \ell=0,\ldots,L-1 ℓ = 0 , … , L − 1 .
Continuous-depth scaling
The residual block (B.1) gives a discrete evolution of the feature matrix. To compare it with an ordinary differential equation, introduce a depth spacing h > 0 h>0 h > 0 and scale both residual increments:
B ℓ h = ( Id + h MLP ℓ ) ∘ ( Id + h Att ℓ ) . B_{\ell}^{h}=(\operatorname{Id}+h\operatorname{MLP}_{\ell})\circ(\operatorname{Id}+h\operatorname{Att}_{\ell}). B ℓ h = ( Id + h MLP ℓ ) ∘ ( Id + h Att ℓ ) .
At h = 1 h=1 h = 1 this is the block in (B.1). Writing Y = ( y 1 , … , y N ) Y=(y_1,\ldots,y_N) Y = ( y 1 , … , y N ) , its first stage is Y = X + h Att ℓ ( X ) Y=X+h\operatorname{Att}_{\ell}(X) Y = X + h Att ℓ ( X ) , so
y i − x i h = ∑ j = 1 N A ℓ , i j ∗ ( X ) W V , ℓ x j , i = 1 , … , N . \frac{y_i-x_i}{h}=\sum_{j=1}^{N}A_{\ell,ij}^*(X)W_{V,\ell}x_j,\qquad i=1,\ldots,N. h y i − x i = j = 1 ∑ N A ℓ , ij ∗ ( X ) W V , ℓ x j , i = 1 , … , N .
For the complete block, expanding the composition gives the exact identity
B ℓ h ( X ) = X + h R ℓ h ( X ) , B_{\ell}^{h}(X)=X+hR_{\ell}^{h}(X), B ℓ h ( X ) = X + h R ℓ h ( X ) ,
R ℓ h ( X ) = Att ℓ ( X ) + MLP ℓ ( X + h Att ℓ ( X ) ) . R_{\ell}^{h}(X)=\operatorname{Att}_{\ell}(X)+\operatorname{MLP}_{\ell}\left(X+h\operatorname{Att}_{\ell}(X)\right). R ℓ h ( X ) = Att ℓ ( X ) + MLP ℓ ( X + h Att ℓ ( X ) ) .
The comparison as h → 0 h \to0 h → 0 also requires a choice of how the layer maps vary with depth. Fix T > 0 T > 0 T > 0 and maps U , V : [ 0 , T ] × X → X \mathrm{U}, \mathrm{V} : [0,T] \times\mathcal{X} \to\mathcal{X} U , V : [ 0 , T ] × X → X . For each spacing h = T / L h h = T/L_h h = T / L h , with L h L_h L h a positive integer, use U ( t ℓ , ⋅ ) \mathrm{U}(t_\ell,\cdot) U ( t ℓ , ⋅ ) and V ( t ℓ , ⋅ ) \mathrm{V}(t_\ell,\cdot) V ( t ℓ , ⋅ ) as the attention and feed-forward maps of layer ℓ \ell ℓ , where t ℓ = ℓ h t_\ell= \ell h t ℓ = ℓ h and ℓ = 0 , … , L h − 1 \ell= 0,\ldots,L_h-1 ℓ = 0 , … , L h − 1 . Assume that U ( t , ⋅ ) \mathrm{U}(t,\cdot) U ( t , ⋅ ) and V ( t , ⋅ ) \mathrm{V}(t,\cdot) V ( t , ⋅ ) have the attention and feed-forward forms specified in Section B.1. The maps U , V \mathrm{U}, \mathrm{V} U , V are fixed as h h h changes; the layer maps are obtained by evaluating them at the depth points t ℓ t_\ell t ℓ .
For this family, define B h : [ 0 , T ] × X → X B_h : [0,T] \times\mathcal{X} \to\mathcal{X} B h : [ 0 , T ] × X → X by
B h ( t , X ) = X + h U ( t , X ) + h V ( t , X + h U ( t , X ) ) . B_h(t,X) = X + h\mathrm{U}(t,X) + h\mathrm{V}(t, X + h\mathrm{U}(t,X)). B h ( t , X ) = X + h U ( t , X ) + h V ( t , X + h U ( t , X )) .
Starting from X ( 0 ) = X 0 ∈ X X^{(0)} = X^0 \in\mathcal{X} X ( 0 ) = X 0 ∈ X , the discrete evolution is X ( ℓ + 1 ) = B h ( t ℓ , X ( ℓ ) ) X^{(\ell+1)} = B_h(t_\ell,X^{(\ell)}) X ( ℓ + 1 ) = B h ( t ℓ , X ( ℓ ) ) for ℓ = 0 , … , L h − 1 \ell= 0,\ldots,L_h-1 ℓ = 0 , … , L h − 1 . The corresponding candidate differential equation, with R : [ 0 , T ] × X → X R : [0,T] \times\mathcal{X} \to\mathcal{X} R : [ 0 , T ] × X → X , is
X ˙ ( t ) = R ( t , X ( t ) ) , R ( t , X ) : = U ( t , X ) + V ( t , X ) , X ( 0 ) = X 0 . (34) \dot{X}(t) = R(t,X(t)), \qquad R(t,X) := \mathrm{U}(t,X) + \mathrm{V}(t,X), \qquad X(0) = X^0. \tag*{(34)} X ˙ ( t ) = R ( t , X ( t )) , R ( t , X ) := U ( t , X ) + V ( t , X ) , X ( 0 ) = X 0 . (34)
Assume that U \mathrm{U} U and V \mathrm{V} V extend to locally Lipschitz maps on an open neighbourhood of [ 0 , T ] × X [0,T] \times\mathcal{X} [ 0 , T ] × X . Fix a compact K ⊂ X K \subset\mathcal{X} K ⊂ X . For sufficiently small h > 0 h > 0 h > 0 , the shifted points X + h U ( t , X ) X + h\mathrm{U}(t,X) X + h U ( t , X ) lie in a fixed compact neighbourhood of K K K . Local boundedness of U \mathrm{U} U and the spatial Lipschitz bound for V \mathrm{V} V then give
B h ( t , X ) − X − h R ( t , X ) = h [ V ( t , X + h U ( t , X ) ) − V ( t , X ) ] = O ( h 2 ) , h ↓ 0 , B_h(t,X) - X - hR(t,X) = h\left[\mathrm{V}(t,X + h\mathrm{U}(t,X)) - \mathrm{V}(t,X)\right]
= O(h^2), \qquad h \downarrow0, B h ( t , X ) − X − h R ( t , X ) = h [ V ( t , X + h U ( t , X )) − V ( t , X ) ] = O ( h 2 ) , h ↓ 0 ,
uniformly for t ∈ [ 0 , T ] t \in[0,T] t ∈ [ 0 , T ] and X ∈ K X \in K X ∈ K , in the Frobenius norm. Thus the scaled residual block agrees with a forward Euler step for (B.2) up to an O ( h 2 ) O(h^2) O ( h 2 ) remainder.
For t ∈ [ 0 , T ] t \in[0,T] t ∈ [ 0 , T ] , X ∈ K X \in K X ∈ K , and sufficiently small h > 0 h > 0 h > 0 with t + h ≤ T t + h \le T t + h ≤ T , let X ^ : [ t , t + h ] → X \widehat{X} : [t,t+h] \to\mathcal{X} X : [ t , t + h ] → X solve X ^ ˙ ( s ) = R ( s , X ^ ( s ) ) \dot{\widehat{X}}(s) = R(s,\widehat{X}(s)) X ˙ ( s ) = R ( s , X ( s )) with X ^ ( t ) = X \widehat{X}(t) = X X ( t ) = X . Local boundedness of R R R gives X ^ ( t + r ) − X = O ( r ) \widehat{X}(t+r) - X = O(r) X ( t + r ) − X = O ( r ) for 0 ≤ r ≤ h 0 \le r \le h 0 ≤ r ≤ h ; its local Lipschitz continuity in ( t , X ) (t,X) ( t , X ) therefore gives
X ^ ( t + h ) − X − h R ( t , X ) = ∫ 0 h [ R ( t + r , X ^ ( t + r ) ) − R ( t , X ) ] d r = O ( h 2 ) . \widehat{X}(t+h) - X - hR(t,X) = \int_0^h \left[R(t+r,\widehat{X}(t+r)) - R(t,X)\right] \,\mathrm{d}r = O(h^2). X ( t + h ) − X − h R ( t , X ) = ∫ 0 h [ R ( t + r , X ( t + r )) − R ( t , X ) ] d r = O ( h 2 ) .
Combining the two estimates yields the one-step error ∥ B h ( t , X ) − X ^ ( t + h ) ∥ F = O ( h 2 ) \lVert B_h(t,X) - \widehat{X}(t+h)\rVert_{\mathrm{F}} = O(h^2) ∥ B h ( t , X ) − X ( t + h ) ∥ F = O ( h 2 ) , with the same uniformity.