Introduction
This paper develops foundations of infinite-dimensional Nesterov acceleration and connects it to the mathematical theory of deep transformers—two fields that at first glance appear largely unrelated. To motivate this connection, we first consider the role of attention in transformers.
The transformer architecture introduced in Attention Is All You Need [77 ] forms the backbone of modern large language models. Its attention layers update token representations by allowing tokens to exchange information across a sequence. The widespread use and societal importance of these systems make a mathematical understanding of their behavior important. Although attention has become an active subject of mathematical research, with contributions to approximation theory, stability, learning, and the dynamics of token interactions [17 , 24 , 30 , 41 , 45 , 81 , 89 ] , even the behavior of the original transformer architecture remains only partially understood.
Attention layers can be viewed as discrete dynamical systems for token representations [24 , 81 ] and, in a continuous-depth limit, as interacting particle systems [35 , 51 , 68 ] . Under suitable assumptions, a subsequent mean-field limit describes the evolution of the token distribution through a non-linear transport equation, allowing PDE methods to be used to study collective behavior [13 , 16 ] . We provide further details in Section 2 and illustrate this passage in Figure 2 . In their recent work, Castin–Ablin–Carrillo–Peyré [16 ] establish well-posedness of the mean-field PDE for softmax attention for compactly supported initial data and study Gaussian initial data separately. General non-Gaussian initial densities with unbounded support remain outside their analysis. As in their work, we consider probability densities on R d \mathbb{R}^{d} R d , allowing unbounded support, which lets us study growth in token magnitudes, which sphere-constrained models exclude [13 , 33 , 34 , 35 , 56 ] .
Figure 2. From attention layers to transport dynamics: with shared weights, the formal continuous-depth limit turns residual updates into an interacting particle system. A suitably normalized mean-field limit then describes the token distribution through a continuity equation.
The transformer PDE arising in the mean-field limit can also be interpreted formally as a gradient flow of an interaction energy on a space of probability densities equipped with a suitable Riemannian metric. For softmax attention, Castin et al. [16, Sec. 5] develop this interpretation, but explicitly leave its rigorous geometric justification open. Before moving on, we highlight several obstacles to making this gradient-flow interpretation rigorous. The first concerns differentiation itself: the admissible density classes on R d \mathbb{R}^{d} R d are not assumed to carry a smooth manifold structure. Therefore the relevant tangent and cotangent spaces and notions of differentiation must be specified explicitly. A further difficulty comes from weak Riemannian metrics, for which even smooth energies need not admit a gradient vector field; see § 3.1.1 . To define the Riemannian metrics considered here, we follow a guiding principle: associate a differential operator, called the mobility operator , with the attention kernel and use its inverse to define the metric. Making this construction rigorous requires identifying suitable function spaces on which this operator is invertible. Figure 3 illustrates this passage from attention kernels to Riemannian metrics.
Figure 3. From attention to geometry: an attention kernel gives rise to a mobility operator. The key step is to identify suitable function spaces on which this operator is invertible. Its inverse then defines the metric underlying a rigorous analog of Otto calculus.
A natural extension of this viewpoint is inspired by Nesterov acceleration [60 ] in finite-dimensional optimization, which, roughly speaking, introduces momentum to accelerate gradient descent; see § 2.2 . On spaces of probability densities, this leads us to study infinite-dimensional damped Newton systems, a particular class of damped Hamiltonian systems. Newton dynamics on such spaces also arise in geometric hydrodynamics; see Khesin–Misiołek–Modin [40 ] . Wang and Li [84 ] study such damped Newton dynamics as accelerated gradient flows on spaces of probability densities. However, their geometric construction has serious analytic gaps even in the Otto–Wasserstein and Stein settings; see § 3.1 and Appendix § 7.4 . The authors of [71 ] use this framework to derive inertial PDEs for linear and softmax attention, but leave the underlying geometric construction at a formal level.
These connections motivate the following question: *Can PDE methods, functional analysis, and geometry applied to the mean-field limit yield new insights into the dynamics of transformers with finitely many tokens and layers?
Main results. Our paper makes the following main contributions, whose connections we summarize in Figure 1 .
Figure 1. From attention geometry to analysis and architecture. The local-mobility and quotient Stein constructions provide function spaces and a differential calculus for attention dynamics on R d \mathbb{R}^{d} R d . For softmax attention, they identify when the equations admit regular gradient and Hamiltonian formulations. For inertial linear attention under the stated assumptions, continuous blow-up predicts large token norms in discrete layers, even at a fixed number of tokens. A direct finite-token geometric construction motivates corrections to token and momentum updates. The reported experiments evaluate their predictive gains and additional computational cost.
Foundations for infinite-dimensional Nesterov acceleration. For linear and softmax attention, we develop the geometric foundations needed to make the formal gradient-flow and damped Hamiltonian formulations rigorous on specified domains (Corollary 6.4 , Theorem 6.7 , Proposition 7.4 , and Lemma 7.16 ). This provides the rigorous justification left open in [16 Sec. 5.4] . For softmax attention, we introduce a local-mobility geometry , which includes both Otto–Wasserstein and softmax–Wasserstein geometry on smooth probability densities over R d \mathbb{R}^{d} R d (Theorem 4.3 ). For linear attention, we introduce a quotient Stein geometry adapted to the bilinear attention kernel (Theorem 5.3 ). Since a differentiable structure on the spaces of probability densities involved is not given a priori, we develop the associated differential calculus directly on these spaces (Appendices § 7.5 and § 7.6 ).
Well-posedness and blow-up for the linear-attention transformer PDE and its accelerated counterpart. By choosing coordinates adapted to the evolving density, we reduce the infinite-dimensional dynamics of both equations to finite-dimensional matrix ODEs. The resulting two-way PDE–ODE correspondences yield existence and uniqueness of strong solutions for admissible non-Gaussian initial data with potentially unbounded support (Theorems 7.6 and 7.18 ). In the accelerated case, we require a quadratic initial momentum potential. These reductions also give conditions for global-in-time existence and finite-time second-moment blow-up (Proposition 7.13 and Theorem 7.23 ).
From PDE blow-up to growth of token norms across layers. For accelerated linear attention at fixed token count, we use continuous moment growth to predict the first explicit-Euler layer crossing a prescribed directional second-moment threshold. The prediction has a bounded error in layer count as the step size tends to zero (Theorem 7.26 ). As a corollary, continuum moment blow-up implies that every prescribed token-norm bound is exceeded in the discrete layers for sufficiently small step sizes (Corollary 7.28 ). To the best of our knowledge, these are the first results linking continuum moment blow-up to quantitative threshold-depth predictions and arbitrarily large token norms in discrete attention networks at fixed token count.
Numerical evaluation of geometric attention updates. We derive corrections to token and momentum updates from the finite-token softmax metric (Lemma 6.11 ). On TinyStories and OpenWebText, we compare practical versions of these updates with matched momentum-only controls. The corrections give small reductions in mean validation loss in some settings, at increased computational cost (§ 7.2 ).
We discuss the relation to previous work in § 7.3 .
Organization. Section 2 recalls the interpretation of attention layers as interacting particle systems and introduces their continuum descriptions. § 2.2 reviews Nesterov acceleration and motivates its extension to attention through geometric formulations of density dynamics. The obstruction to using the full formal tangent space in Stein geometry is established in Appendix § 7.4 . § 3.1 and § 3.3 develop the local-mobility and quotient Stein frameworks, respectively. They identify suitable function spaces, establish invertibility of the corresponding mobility operators,
and define the induced metrics and gradients. The notions of differentiability needed for the gradient and Hamiltonian formulations are developed in Appendices § 7.5 and § 7.6 . § 4.1 treats softmax attention. It distinguishes pointwise well-definedness from regular gradient and Hamiltonian formulations, identifies their admissibility conditions, and derives the finite-token geometric correction used in the numerical section. § 4.2 applies the quotient Stein framework to linear attention. It introduces covariance coordinates, establishes the ODE–PDE correspondence and strong well-posedness, and studies global existence and finite-time moment blow-up. It then relates continuous growth to activation thresholds and effective depth in discrete attention layers. § 7.1 introduces the discretizations and resulting architectures, and § 7.2 compares their linear and softmax variants in language-modeling experiments. We discuss the relation to previous work in § 7.3 .
I Attention Models and Acceleration
In this section, we recall the mean-field description of transformer dynamics and discuss possible geometric interpretations of the resulting PDE; see, e.g., [16 , 35 ] . We first specify the attention weights and residual layers, then describe the formal passage to continuous-depth particle dynamics and their mean-field continuity equation, as illustrated in Figure 2 . Finally, we discuss the geometric requirements for a gradient-flow interpretation and the obstacles to making it rigorous.
Token configurations and attention weights. Transformers act on token configurations
X : = ( X 1 , … , X N ) ∈ ( R d ) N . \mathbf{X} := (X_1,\ldots,X_N) \in(\mathbb{R}^{d})^N. X := ( X 1 , … , X N ) ∈ ( R d ) N .
The literature also considers embeddings in ( S d ) N (\mathbb{S}^{d})^N ( S d ) N ; see, e.g., [13 , 18 , 21 , 34 ] , which this paper does not handle. Self-attention combines information across tokens, whereas MLP layers apply tokenwise non-linear transformations. Let Q , K ∈ R m × d Q, K \in\mathbb{R}^{m \times d} Q , K ∈ R m × d and V ∈ R d × d V \in\mathbb{R}^{d \times d} V ∈ R d × d , m ≤ d m \le d m ≤ d , denote the query, key, and value matrices, and set A : = K ⊤ Q ∈ R d × d A := K^{\top}Q \in\mathbb{R}^{d \times d} A := K ⊤ Q ∈ R d × d . For the geometric constructions in § 4.2 and § 4.1 , we assume that m = d m = d m = d and that A A A is a symmetric matrix. In the following, we ignore the causal masking, layer normalization, the standard attention scaling, the output projection, and the multi-head structure usually used in transformer architectures.
Recall that the softmax function is defined for w ∈ R d w \in\mathbb{R}^d w ∈ R d by
softmax ( w ) : = ( exp ( w i ) ∑ j = 1 d exp ( w j ) ) i = 1 d . \operatorname{softmax}(w) := \left(\frac{\exp(w_i)}{\sum_{j=1}^{d}\exp(w_j)}\right)_{i=1}^{d}. softmax ( w ) := ( ∑ j = 1 d exp ( w j ) exp ( w i ) ) i = 1 d .
We consider the following two choices of attention weights κ = κ X : R d × R d → R \kappa= \kappa_X : \mathbb{R}^d \times\mathbb{R}^d \to\mathbb{R} κ = κ X : R d × R d → R :
κ X ( X i , X j ) = { exp ( ( X j ) ⊤ A X i ) ∑ ℓ = 1 N exp ( ( X ℓ ) ⊤ A X i ) = softmax j ( ( X ℓ ) ⊤ A X i ) ℓ = 1 N softmax self-attention , ( X j ) ⊤ A X i linear self-attention . (1) \kappa_X(X_i,X_j) =
\begin{cases}
\frac{\exp\left((X_j)^\top A X_i\right)}{\sum_{\ell=1}^{N}\exp\left((X_\ell)^\top A X_i\right)}
= \operatorname{softmax}_j\left((X_\ell)^\top A X_i\right)_{\ell=1}^{N} & \text{softmax self-attention},\\
(X_j)^\top A X_i & \text{linear self-attention}.
\end{cases} \tag*{(1)} κ X ( X i , X j ) = ⎩ ⎨ ⎧ ∑ ℓ = 1 N e x p ( ( X ℓ ) ⊤ A X i ) e x p ( ( X j ) ⊤ A X i ) = softmax j ( ( X ℓ ) ⊤ A X i ) ℓ = 1 N ( X j ) ⊤ A X i softmax self-attention , linear self-attention . (1)
We use the term “linear self-attention” for unnormalized dot-product attention, to distinguish it from other linear self-attention mechanisms [39 ] . Note that the evaluation cost of transformers with softmax attention scales quadratically with sequence length. Kernel attention addresses this by replacing softmax with a more general kernel function. For example, [66 ] aims to identify kernels that retain the expressivity of attention while enabling quasi-linear computation.
Residual attention and MLP layers. Let ( X 1 0 , … , X N 0 ) (X_1^0,\ldots,X_N^0) ( X 1 0 , … , X N 0 ) be the input tokens, let h > 0 h > 0 h > 0 be the step size, and let k ∈ N k \in\mathbb{N} k ∈ N denote the layer index. The attention matrices usually differ between layers. For the continuous-depth description below, we instead keep Q Q Q , K K K , and V V V fixed across layers (“weight sharing”), as in architectures such as ALBERT [43 ] . The residual attention sublayer is
X i ( k + 1 ) = X i ( k ) + h ∑ j = 1 N κ X ( X i ( k ) , X j ( k ) ) V X j ( k ) , (2) X_i^{(k+1)} = X_i^{(k)} + h \sum_{j=1}^{N}\kappa_X\left(X_i^{(k)},X_j^{(k)}\right)VX_j^{(k)}, \tag*{(2)} X i ( k + 1 ) = X i ( k ) + h j = 1 ∑ N κ X ( X i ( k ) , X j ( k ) ) V X j ( k ) , (2)
where we have to replace h h h by h / N h/N h / N for linear attention. For weight matrices W , W ~ ∈ R d × d W,\widetilde{W} \in\mathbb{R}^{d \times d} W , W ∈ R d × d , a bias b ∈ R d b \in\mathbb{R}^d b ∈ R d , and a non-linear function σ : R → R \sigma: \mathbb{R} \to\mathbb{R} σ : R → R applied componentwise, the residual MLP sublayer is
X i ( k + 1 ) = X i ( k ) + h W σ ( W ~ X i ( k ) + b ) , (3) X_i^{(k+1)} = X_i^{(k)} + hW\sigma\left(\widetilde{W}X_i^{(k)} + b\right), \tag*{(3)} X i ( k + 1 ) = X i ( k ) + hW σ ( W X i ( k ) + b ) , (3)
where the MLP parameters may differ between layers. The updates (2) and (3) describe the two sublayers separately. In the following passage to particle and density dynamics, we consider only the attention update (2).
Particle dynamics. Given input tokens X ( 0 ) = ( X 1 ( 0 ) , … , X N ( 0 ) ) ∈ ( R d ) N X(0) = (X_1(0),\ldots,X_N(0)) \in(\mathbb{R}^d)^N X ( 0 ) = ( X 1 ( 0 ) , … , X N ( 0 )) ∈ ( R d ) N , we interpret the attention iteration as a discretization in depth. Formally, as the step size h → 0 h \to0 h → 0 , this gives
X ˙ i ( t ) = Γ X ( t ) ( X i ( t ) ) , 1 ≤ i ≤ N , (4) \dot{X}_i(t) = \Gamma_{X(t)}(X_i(t)), \qquad1 \leq i \leq N, \tag*{(4)} X ˙ i ( t ) = Γ X ( t ) ( X i ( t )) , 1 ≤ i ≤ N , (4)
where Γ X : R d → R d \Gamma_X : \mathbb{R}^d \to\mathbb{R}^d Γ X : R d → R d is the attention vector field determined by Q Q Q , K K K , V V V , and the current token configuration X ( t ) X(t) X ( t ) . For softmax attention, it is given by
Γ X ( x ) : = ∑ j = 1 N exp ( Q x ⋅ K X j ) ∑ ℓ = 1 N exp ( Q x ⋅ K X ℓ ) V X j , for x ∈ R d . \Gamma_X(x) := \sum_{j=1}^{N}\frac{\exp\left(Qx \cdot KX_j\right)}{\sum_{\ell=1}^{N}\exp\left(Qx \cdot KX_\ell\right)}VX_j, \qquad\text{for } x \in\mathbb{R}^d. Γ X ( x ) := j = 1 ∑ N ∑ ℓ = 1 N exp ( Q x ⋅ K X ℓ ) exp ( Q x ⋅ K X j ) V X j , for x ∈ R d .
The velocity of each token in (4) is a similarity-weighted average of the value vectors. The resulting dynamics form a non-linear interacting particle system.
Density continuity equation. For a probability density ρ \rho ρ , the corresponding attention field is
Γ ρ : R d → R d , x ↦ ∫ V y κ ρ ( x , y ) ρ ( y ) d y , \Gamma_\rho:\mathbb{R}^d\to\mathbb{R}^d,\qquad x\mapsto\int_V y\kappa_\rho(x,y)\rho(y)\,\mathrm{d}y, Γ ρ : R d → R d , x ↦ ∫ V y κ ρ ( x , y ) ρ ( y ) d y ,
for all ρ \rho ρ such that this integral is finite. The two choices in (1) lead to the density-dependent weights
κ ρ ( x , y ) : = { exp ( y ⊤ A x ) ∫ exp ( z ⊤ A x ) ρ ( z ) d z , softmax self-attention , y ⊤ A x , linear self-attention . \kappa_\rho(x,y):=
\begin{cases}
\dfrac{\exp(y^\top A x)}{\int\exp(z^\top A x)\rho(z)\,\mathrm{d}z}, & \text{softmax self-attention},\\
y^\top A x, & \text{linear self-attention}.
\end{cases} κ ρ ( x , y ) := ⎩ ⎨ ⎧ ∫ exp ( z ⊤ A x ) ρ ( z ) d z exp ( y ⊤ A x ) , y ⊤ A x , softmax self-attention , linear self-attention .
In the formal mean-field limit, the self-attention dynamics (4) are described on R d \mathbb{R}^d R d by the continuity equation
∂ t ρ t = − ∇ ⋅ ( ρ t Γ ρ t ) , t > 0 , (5) \partial_t\rho_t=-\nabla\cdot(\rho_t\Gamma_{\rho_t}),\qquad t>0, \tag*{(5)} ∂ t ρ t = − ∇ ⋅ ( ρ t Γ ρ t ) , t > 0 , (5)
see [16 , 35 ] and also [13 , 35 , 68 , 90 ] .
Geometry of the mean-field limit. Under suitable assumptions on the attention matrices, the mean-field equation (5) admits a formal gradient-flow interpretation. Recall that a Riemannian gradient flow on a manifold M \mathcal{M} M is specified by a pair ( F , G ) (\mathcal{F},G) ( F , G ) : a smooth functional F : M → R \mathcal{F}:\mathcal{M}\to\mathbb{R} F : M → R and a Riemannian metric G G G . Thus, identifying an energy is only part of the construction; the geometry determines the corresponding direction of steepest descent.
Making this interpretation rigorous requires specifying the underlying space of probability densities. A central distinction is whether these densities have compact support or extend over all of R d \mathbb{R}^d R d . In the latter case, additional integrability and coercivity conditions are needed to justify the formal geometric operators. For softmax attention, a general rigorous justification of this geometry for non-compactly supported densities was left open in [16 Sec. 5] .
This geometric viewpoint provides a way to introduce momentum into attention dynamics, following the principle of Nesterov acceleration, as discussed in the next section. Discretizing the resulting inertial equations yields attention layers that propagate both token positions and a momentum variable across network depth.
We close this subsection by recalling the guiding question of this paper:
Can PDE methods, functional analysis, and geometry applied to the mean-field limit yield new insights into the dynamics of transformers with finitely many tokens and layers?
Nesterov's Acceleration Method and Attention Models
The aim of this section is to explain how Nesterov acceleration provides a way to introduce momentum into attention dynamics and to identify the geometric constructions needed to justify this approach.
We first recall Nesterov acceleration in Euclidean space and its continuous-time formulation as the damped Hamiltonian system (10) . We then describe the corresponding construction for smooth probability densities on compact manifolds, leading to the accelerated Otto–Wasserstein equations (13) . This construction relies on the mobility operator being invertible between suitable cotangent and tangent spaces. We explain why this requirement presents an obstruction in Stein geometry.
Finally, we connect this viewpoint to the linear and softmax attention models introduced in Section 2. Their kernels give rise to different mobility operators and hence different formal geometric descriptions. We state the resulting accelerated systems in (17) and (19), and identify the function-space and differentiability questions that must be resolved to make them rigorous. These questions motivate the local-mobility calculus developed in § 3.1 and the quotient Stein geometry developed in § 3.3 .
Nesterov Acceleration on R d \mathbb{R}^{d} R d
Nesterov acceleration [60 ] is a well-known technique for improving the convergence rate of gradient descent methods. For an objective function F ∈ C 1 ( R d ; R ) F \in C^{1}(\mathbb{R}^{d};\mathbb{R}) F ∈ C 1 ( R d ; R ) , we consider the optimization problem
min x ∈ R d F ( x ) . (6) \min_{x \in\mathbb{R}^{d}} F(x). \tag*{(6)} x ∈ R d min F ( x ) . (6)
The gradient descent sequence ( x ( k ) ) k ∈ N (x^{(k)})_{k \in\mathbb{N}} ( x ( k ) ) k ∈ N starting at x 0 ∈ R d x_{0} \in\mathbb{R}^{d} x 0 ∈ R d with step size τ \tau τ is given by
{ x ( k + 1 ) = x ( k ) − τ ∇ F ( x ( k ) ) , x ( 0 ) = x 0 . \begin{cases}
x^{(k+1)} = x^{(k)} - \tau\nabla F(x^{(k)}),\\
x^{(0)} = x_{0}.
\end{cases} { x ( k + 1 ) = x ( k ) − τ ∇ F ( x ( k ) ) , x ( 0 ) = x 0 .
Nesterov’s accelerated gradient descent is obtained by choosing an extrapolation coefficient as
{ x ( k + 1 ) = y ( k ) − τ ∇ F ( y ( k ) ) , y ( k + 1 ) = x ( k + 1 ) + ( 1 − 3 k + 3 ) ( x ( k + 1 ) − x ( k ) ) , x ( 0 ) = x 0 , y ( 0 ) = x 0 . (7) \begin{cases}
x^{(k+1)} = y^{(k)} - \tau\nabla F(y^{(k)}),\\
y^{(k+1)} = x^{(k+1)} + \left(1 - \frac{3}{k+3}\right)(x^{(k+1)} - x^{(k)}),\\
x^{(0)} = x_{0},\ y^{(0)} = x_{0}. \tag*{(7)}
\end{cases} ⎩ ⎨ ⎧ x ( k + 1 ) = y ( k ) − τ ∇ F ( y ( k ) ) , y ( k + 1 ) = x ( k + 1 ) + ( 1 − k + 3 3 ) ( x ( k + 1 ) − x ( k ) ) , x ( 0 ) = x 0 , y ( 0 ) = x 0 . (7)
Substituting t = k τ t = k\sqrt{\tau} t = k τ and letting τ → 0 \tau\to0 τ → 0 , Nesterov’s method can be interpreted through the following continuous-time dynamical system; see [72 , 85 ] :
{ x ¨ t + 3 t x ˙ t + ∇ F ( x t ) = 0 , x t ∣ t = 0 = x 0 , x ˙ t ∣ t = 0 = 0. (8) \begin{cases}
\ddot{x}_{t} + \frac{3}{t}\dot{x}_{t} + \nabla F(x_{t}) = 0,\\
x_{t}\big|_{t=0} = x_{0},\ \dot{x}_{t}\big|_{t=0} = 0. \tag*{(8)}
\end{cases} { x ¨ t + t 3 x ˙ t + ∇ F ( x t ) = 0 , x t t = 0 = x 0 , x ˙ t t = 0 = 0. (8)
For a convex function F F F possessing a minimizer, and any step size τ ∈ ( 0 , 1 L ] \tau\in\left(0,\frac{1}{L}\right] τ ∈ ( 0 , L 1 ] , where L L L is the Lipschitz constant of ∇ F \nabla F ∇ F , the discrete scheme (7) exhibits a convergence rate F ( x ( k ) ) − F ∗ ≤ O ( ∥ x 0 − x ∗ ∥ 2 τ k 2 ) F(x^{(k)}) - F^{*} \le O\left(\frac{\lVert x_{0}-x^{*}\rVert^{2}}{\tau k^{2}}\right) F ( x ( k ) ) − F ∗ ≤ O ( τ k 2 ∥ x 0 − x ∗ ∥ 2 ) , where x ∗ x^{*} x ∗ is a minimizer of F F F and F ∗ = F ( x ∗ ) F^{*} = F(x^{*}) F ∗ = F ( x ∗ ) [72 p. 2] . This rate is optimal among all methods having only information about the gradient of F F F at consecutive iterates [61 ] . For the continuous dynamics (8), we have similarly F ( x ( t ) ) − F ∗ ≤ O ( ∥ x 0 − x ∗ ∥ 2 t 2 ) F(x(t)) - F^{*} \le O\left(\frac{\lVert x_{0}-x^{*}\rVert^{2}}{t^{2}}\right) F ( x ( t )) − F ∗ ≤ O ( t 2 ∥ x 0 − x ∗ ∥ 2 ) .
Introducing a momentum variable p t p_{t} p t and α t : = 3 t \alpha_{t} := \frac{3}{t} α t := t 3 , the second-order ODE (8) can be rewritten as the first-order system
{ x ˙ t = p t , p ˙ t + α t p t = − ∇ F ( x t ) , x t ∣ t = 0 = x 0 , p 0 = 0. (9) \begin{cases}
\dot{x}_{t} = p_{t},\\
\dot{p}_{t} + \alpha_{t}p_{t} = -\nabla F(x_{t}),\\
x_{t}\big|_{t=0} = x_{0},\ p_{0} = 0. \tag*{(9)}
\end{cases} ⎩ ⎨ ⎧ x ˙ t = p t , p ˙ t + α t p t = − ∇ F ( x t ) , x t t = 0 = x 0 , p 0 = 0. (9)
It was observed, e.g., in [73 ] , that with the Hamiltonian and its derivatives
H ( x , p ) = 1 2 ∥ p ∥ 2 + F ( x ) , ∇ p H ( x , p ) = p , ∇ x H ( x , p ) = ∇ F ( x ) , H(x,p) = \frac{1}{2}\lVert p\rVert^{2} + F(x),\quad\nabla_{p}H(x,p) = p,\quad\nabla_{x}H(x,p) = \nabla F(x), H ( x , p ) = 2 1 ∥ p ∥ 2 + F ( x ) , ∇ p H ( x , p ) = p , ∇ x H ( x , p ) = ∇ F ( x ) ,
the first-order system (9) becomes a damped Hamiltonian flow
x ˙ t = ∇ p H ( x t , p t ) , x t ∣ t = 0 = x 0 , p ˙ t + α t p t = − ∇ x H ( x t , p t ) , p 0 = 0. (10) \begin{aligned}
\dot{x}_t &= \nabla_p H(x_t,p_t), \qquad x_t|_{t=0}=x_0,\\
\dot{p}_t+\alpha_t p_t &= -\nabla_x H(x_t,p_t), \qquad p_0=0. \tag*{(10)}
\end{aligned} x ˙ t p ˙ t + α t p t = ∇ p H ( x t , p t ) , x t ∣ t = 0 = x 0 , = − ∇ x H ( x t , p t ) , p 0 = 0. (10)
Nesterov Acceleration on Probability Measures on Compact Sets
In [84 ] (see [20 , 75 ] for alternative approaches), Nesterov acceleration is formulated on the space of smooth, strictly positive probability densities on a closed, connected manifold Ω \Omega Ω . In this subsection, we briefly recall this approach to get a sense of the path we must follow for our transformer setting on the space of smooth probability densities on R d \mathbb{R}^d R d . We focus on Otto–Wasserstein geometry from optimal transport [65 ] and Stein geometry associated with Stein variational gradient descent [25 , 49 ] ; see also the references therein.
We begin by fixing notation for the space of smooth, strictly positive probability densities on Ω \Omega Ω :
P ∞ ( Ω ) : = { ρ ∈ C ∞ ( Ω ) : ρ > 0 , ∫ Ω ρ ( x ) d x = 1 } . \mathcal{P}^{\infty}(\Omega):=\left\{\rho\in C^{\infty}(\Omega):\rho>0,\ \int_{\Omega}\rho(x)\,\mathrm{d}x=1\right\}. P ∞ ( Ω ) := { ρ ∈ C ∞ ( Ω ) : ρ > 0 , ∫ Ω ρ ( x ) d x = 1 } .
This space carries a smooth Fréchet manifold structure [36 , 42 ] . The tangent space of P ∞ ( Ω ) \mathcal{P}^{\infty}(\Omega) P ∞ ( Ω ) at a smooth probability density ρ \rho ρ is given by
T ρ P ∞ ( Ω ) = C 0 ∞ ( Ω ) : = { ρ ˙ ∈ C ∞ ( Ω ) : ∫ Ω ρ ˙ d x = 0 } . T_{\rho}\mathcal{P}^{\infty}(\Omega)=C_{0}^{\infty}(\Omega):=\left\{\dot{\rho}\in C^{\infty}(\Omega):\int_{\Omega}\dot{\rho}\,\mathrm{d}x=0\right\}. T ρ P ∞ ( Ω ) = C 0 ∞ ( Ω ) := { ρ ˙ ∈ C ∞ ( Ω ) : ∫ Ω ρ ˙ d x = 0 } .
The cotangent space of P ∞ ( Ω ) \mathcal{P}^{\infty}(\Omega) P ∞ ( Ω ) at a smooth probability density ρ \rho ρ , to be precise its smooth dual, reads as T ρ ∗ P ∞ ( Ω ) : = C ∞ ( Ω ) / R T_{\rho}^{*}\mathcal{P}^{\infty}(\Omega):=C^{\infty}(\Omega)/\mathbb{R} T ρ ∗ P ∞ ( Ω ) := C ∞ ( Ω ) / R . The class of a representative Φ \Phi Φ modulo constants is denoted by [ Φ ] [\Phi] [ Φ ] .
Nesterov acceleration in Otto–Wasserstein geometry. For all ρ ∈ P ∞ ( Ω ) \rho\in\mathcal{P}^{\infty}(\Omega) ρ ∈ P ∞ ( Ω ) , the Otto–Wasserstein mobility operator
L ρ W : T ρ ∗ P ∞ ( Ω ) ⟶ T ρ P ∞ ( Ω ) , [ Φ ] ↦ − ∇ ⋅ ( ρ ∇ Φ ) , L_{\rho}^{W}:T_{\rho}^{*}\mathcal{P}^{\infty}(\Omega)\longrightarrow T_{\rho}\mathcal{P}^{\infty}(\Omega),\qquad[\Phi]\mapsto-\nabla\cdot(\rho\nabla\Phi), L ρ W : T ρ ∗ P ∞ ( Ω ) ⟶ T ρ P ∞ ( Ω ) , [ Φ ] ↦ − ∇ ⋅ ( ρ ∇Φ ) ,
is a linear isomorphism, and the corresponding bilinear form induces a well-defined Riemannian metric on T ρ P ∞ ( Ω ) T_{\rho}\mathcal{P}^{\infty}(\Omega) T ρ P ∞ ( Ω ) , given by G ρ W : T ρ P ∞ ( Ω ) × T ρ P ∞ ( Ω ) → R G_{\rho}^{W}:T_{\rho}\mathcal{P}^{\infty}(\Omega)\times T_{\rho}\mathcal{P}^{\infty}(\Omega)\to\mathbb{R} G ρ W : T ρ P ∞ ( Ω ) × T ρ P ∞ ( Ω ) → R with
G ρ W ( ρ ˙ , η ˙ ) : = ∫ Ω Φ ρ ˙ η ˙ d x , G_{\rho}^{W}(\dot{\rho},\dot{\eta}):=\int_{\Omega}\Phi_{\dot{\rho}}\dot{\eta}\,\mathrm{d}x, G ρ W ( ρ ˙ , η ˙ ) := ∫ Ω Φ ρ ˙ η ˙ d x ,
where Φ ρ ˙ \Phi_{\dot{\rho}} Φ ρ ˙ is uniquely defined for any given ρ ˙ \dot{\rho} ρ ˙ via
ρ ˙ = L ρ W ( [ Φ ρ ˙ ] ) = − ∇ ⋅ ( ρ ∇ Φ ρ ˙ ) . (11) \dot{\rho}=L_{\rho}^{W}([\Phi_{\dot{\rho}}])=-\nabla\cdot(\rho\nabla\Phi_{\dot{\rho}}). \tag*{(11)} ρ ˙ = L ρ W ([ Φ ρ ˙ ]) = − ∇ ⋅ ( ρ ∇ Φ ρ ˙ ) . (11)
see [42 , 65 , 79 ] . Note that the invertibility of L ρ W L_{\rho}^{W} L ρ W is ensured as long as we work with measures on compact, connected spaces. An important ingredient is that on these spaces the so-called Poincaré inequality holds. In our transformer setting, we have to establish such a relation for a more general operator related to the softmax function as well as for the whole space R d \mathbb{R}^{d} R d .
Let 0 ≤ t 0 < T ≤ ∞ 0 \le t_0 < T \le\infty 0 ≤ t 0 < T ≤ ∞ . For every accelerated evolution considered on [ t 0 , T ) [t_0,T) [ t 0 , T ) , we assume that the damping coefficient satisfies
α ∈ C ( [ t 0 , T ) ; [ 0 , ∞ ) ) . \alpha\in C([t_0,T);[0,\infty)). α ∈ C ([ t 0 , T ) ; [ 0 , ∞ )) .
For schedules singular at the origin, such as α ( t ) = r / t \alpha(t)=r/t α ( t ) = r / t [6 , 7 ] or α ( t ) = r / t + m \alpha(t)=r/t+m α ( t ) = r / t + m , we always choose t 0 > 0 t_0>0 t 0 > 0 or translate time and use α ( t 0 + t ) \alpha(t_0+t) α ( t 0 + t ) with t 0 > 0 t_0>0 t 0 > 0 . Then, instead of (6) , we are looking for a minimizer of F : P ∞ ( Ω ) → R \mathcal{F}:\mathcal{P}^{\infty}(\Omega)\to\mathbb{R} F : P ∞ ( Ω ) → R , i.e.,
min ρ ∈ P ∞ ( Ω ) F ( ρ ) . \min_{\rho\in\mathcal{P}^{\infty}(\Omega)}\mathcal{F}(\rho). ρ ∈ P ∞ ( Ω ) min F ( ρ ) .
The gradient flow of F ( ρ ) \mathcal{F}(\rho) F ( ρ ) in the probability density space ( P ∞ ( Ω ) , G W ) (\mathcal{P}^{\infty}(\Omega),G^{W}) ( P ∞ ( Ω ) , G W ) satisfies
∂ t ρ t = − L ρ t W [ δ F ( ρ t ) δ ρ t ] . \partial_t\rho_t=-L_{\rho_t}^{W}\left[\frac{\delta\mathcal{F}(\rho_t)}{\delta\rho_t}\right]. ∂ t ρ t = − L ρ t W [ δ ρ t δ F ( ρ t ) ] .
where δ F δ ρ \frac{\delta\mathcal{F}}{\delta\rho} δ ρ δ F denotes the first variation of the functional F \mathcal{F} F at the density ρ \rho ρ . With the Hamiltonian defined by
H ( ρ , Φ ) : = 1 2 ∫ Ω Φ L ρ W Φ d x + F ( ρ ) , ρ ∈ P ∞ ( Ω ) , Φ ∈ T ρ ∗ P ∞ ( Ω ) \mathcal{H}(\rho,\Phi):=\frac{1}{2}\int_{\Omega}\Phi L_{\rho}^{W}\Phi\,\mathrm{d}x+\mathcal{F}(\rho),\qquad\rho\in\mathcal{P}^{\infty}(\Omega),\ \Phi\in T_{\rho}^{*}\mathcal{P}^{\infty}(\Omega) H ( ρ , Φ ) := 2 1 ∫ Ω Φ L ρ W Φ d x + F ( ρ ) , ρ ∈ P ∞ ( Ω ) , Φ ∈ T ρ ∗ P ∞ ( Ω )
the accelerated dynamics by the damped Hamiltonian equations suggested in [84 ] satisfy
∂ t ρ t = δ δ Φ t H ( ρ t , Φ t ) , ρ t ∣ t = 0 = ρ 0 , ∂ t Φ t + α t Φ t = − δ δ ρ t H ( ρ t , Φ t ) , Φ 0 = 0 , (12) \begin{aligned}
\partial_t\rho_t&=\frac{\delta}{\delta\Phi_t}\mathcal{H}(\rho_t,\Phi_t),\qquad&&\rho_t|_{t=0}=\rho_0,\\
\partial_t\Phi_t+\alpha_t\Phi_t&=-\frac{\delta}{\delta\rho_t}\mathcal{H}(\rho_t,\Phi_t),\qquad&&\Phi_0=0, \tag*{(12)}
\end{aligned} ∂ t ρ t ∂ t Φ t + α t Φ t = δ Φ t δ H ( ρ t , Φ t ) , = − δ ρ t δ H ( ρ t , Φ t ) , ρ t ∣ t = 0 = ρ 0 , Φ 0 = 0 , (12)
where ρ 0 ∈ P ∞ ( Ω ) \rho_0\in\mathcal{P}^{\infty}(\Omega) ρ 0 ∈ P ∞ ( Ω ) is a given initial density function, see also [40 , 80 ] . The system (12) can be rewritten as
∂ t ρ t = L ρ t W ( Φ t ) , ρ t ∣ t = 0 = ρ 0 , ∂ t Φ t + α t Φ t = − 1 2 δ δ ρ t ( ∫ Ω Φ t L ρ t W ( Φ t ) d x ) − δ F δ ρ t , Φ 0 = 0 , \begin{aligned}
\partial_t\rho_t&=L_{\rho_t}^{W}(\Phi_t),\qquad&&\rho_t|_{t=0}=\rho_0,\\
\partial_t\Phi_t+\alpha_t\Phi_t&=-\frac{1}{2}\frac{\delta}{\delta\rho_t}\left(\int_{\Omega}\Phi_tL_{\rho_t}^{W}(\Phi_t)\,\mathrm{d}x\right)-\frac{\delta\mathcal{F}}{\delta\rho_t},\qquad&&\Phi_0=0,
\end{aligned} ∂ t ρ t ∂ t Φ t + α t Φ t = L ρ t W ( Φ t ) , = − 2 1 δ ρ t δ ( ∫ Ω Φ t L ρ t W ( Φ t ) d x ) − δ ρ t δ F , ρ t ∣ t = 0 = ρ 0 , Φ 0 = 0 ,
and plugging in (11) , we obtain the Nesterov acceleration in the Otto–Wasserstein geometry
∂ t ρ t = − ∇ ⋅ ( ρ t ∇ Φ t ) , ρ t ∣ t = 0 = ρ 0 , ∂ t Φ t + α t Φ t = − 1 2 ∥ ∇ Φ t ∥ 2 − δ F ( ρ t ) δ ρ t , Φ 0 = 0. (13) \begin{aligned}
\partial_t\rho_t&=-\nabla\cdot(\rho_t\nabla\Phi_t),\qquad&&\rho_t|_{t=0}=\rho_0,\\
\partial_t\Phi_t+\alpha_t\Phi_t&=-\frac{1}{2}\lVert\nabla\Phi_t\rVert^2-\frac{\delta\mathcal{F}(\rho_t)}{\delta\rho_t},\qquad&&\Phi_0=0. \tag*{(13)}
\end{aligned} ∂ t ρ t ∂ t Φ t + α t Φ t = − ∇ ⋅ ( ρ t ∇ Φ t ) , = − 2 1 ∥ ∇ Φ t ∥ 2 − δ ρ t δ F ( ρ t ) , ρ t ∣ t = 0 = ρ 0 , Φ 0 = 0. (13)
This PDE is related to damped Euler equations, whose long-time behavior under constant damping was studied in [14 ] for suitable free energies, assuming the existence of global energy-dissipating solutions. That work does not establish general well-posedness. For a related kinetic formulation, see [59 ] .
To connect this construction with attention dynamics, we next consider the mobility operators associated with the attention kernels. For softmax attention, the operator in (15) gives a modification of Otto–Wasserstein geometry that remains closely related to it on compact domains. Linear attention instead suggests a Stein-type geometry through its bilinear kernel. Here, an additional obstruction arises: even for symmetric, bounded, integrally strictly positive definite kernels, the Stein mobility operator need not be invertible between the full formal cotangent and tangent spaces. The following paragraph explains why the preceding Otto–Wasserstein construction therefore requires further modifications in the Stein setting, even on compact domains.
Nesterov acceleration in Stein geometry and its failure. In [84 ] and follow-up papers, it was assumed that the Stein mobility operator
L ρ S : T ρ ∗ P ∞ ( Ω ) → T ρ P ∞ ( Ω ) , [ Φ ] ↦ − ∇ ⋅ ( ρ ∫ Ω K ( ⋅ , y ) ∇ Φ ( y ) ρ ( y ) d y ) (14) L_{\rho}^{S}:T_{\rho}^{*}\mathcal{P}^{\infty}(\Omega)\to T_{\rho}\mathcal{P}^{\infty}(\Omega),\qquad[\Phi]\mapsto-\nabla\cdot\left(\rho\int_{\Omega}K(\cdot,y)\nabla\Phi(y)\rho(y)\,dy\right) \tag*{(14)} L ρ S : T ρ ∗ P ∞ ( Ω ) → T ρ P ∞ ( Ω ) , [ Φ ] ↦ − ∇ ⋅ ( ρ ∫ Ω K ( ⋅ , y ) ∇Φ ( y ) ρ ( y ) d y ) (14)
with a symmetric, bounded, integrally strictly positive definite kernel K K K is also a linear isomorphism, so that G ρ S : T ρ P ∞ ( Ω ) × T ρ P ∞ ( Ω ) → R G_{\rho}^{S}:T_{\rho}\mathcal{P}^{\infty}(\Omega)\times T_{\rho}\mathcal{P}^{\infty}(\Omega)\to\mathbb{R} G ρ S : T ρ P ∞ ( Ω ) × T ρ P ∞ ( Ω ) → R given by
G ρ S ( ρ ˙ , η ˙ ) : = ∫ Ω Φ ρ ˙ ( x ) η ˙ ( x ) d x = ∫ Ω ∫ Ω K ( x , y ) ⟨ ∇ Φ ρ ˙ ( x ) , ∇ Φ η ˙ ( y ) ⟩ ρ ( x ) ρ ( y ) d y d x G_{\rho}^{S}(\dot{\rho},\dot{\eta}):=\int_{\Omega}\Phi_{\dot{\rho}}(x)\dot{\eta}(x)\,dx=\int_{\Omega}\int_{\Omega}K(x,y)\langle\nabla\Phi_{\dot{\rho}}(x),\nabla\Phi_{\dot{\eta}}(y)\rangle\rho(x)\rho(y)\,dy\,dx G ρ S ( ρ ˙ , η ˙ ) := ∫ Ω Φ ρ ˙ ( x ) η ˙ ( x ) d x = ∫ Ω ∫ Ω K ( x , y ) ⟨ ∇ Φ ρ ˙ ( x ) , ∇ Φ η ˙ ( y )⟩ ρ ( x ) ρ ( y ) d y d x
with
ρ ˙ = L ρ S ( [ Φ ρ ˙ ] ) , η ˙ = L ρ S ( [ Φ η ˙ ] ) . \dot{\rho}=L_{\rho}^{S}([\Phi_{\dot{\rho}}]),\qquad\dot{\eta}=L_{\rho}^{S}([\Phi_{\dot{\eta}}]). ρ ˙ = L ρ S ([ Φ ρ ˙ ]) , η ˙ = L ρ S ([ Φ η ˙ ]) .
imposes a well-defined Riemannian metric on T ρ P ∞ ( Ω ) T_{\rho}\mathcal{P}^{\infty}(\Omega) T ρ P ∞ ( Ω ) . Unfortunately, this is not true, since L ρ S L_{\rho}^{S} L ρ S fails in general to be surjective, and a restriction of the formal tangent space is necessary to ensure surjectivity; see Appendix § 7.4 .
We close this paragraph by noting an additional geometric obstruction on general compact manifolds Ω \Omega Ω . The kernel in (14) has to do more than just weighting the gradients: it must map tangent vectors at the integration point to tangent vectors at the evaluation point, so that they can be averaged in a common tangent space. Thus, an intrinsic formulation requires an operator-valued kernel acting between the respective tangent spaces. A scalar kernel alone does not provide this identification, which is implicit in the Euclidean setting. For one possible way to overcome this obstruction, we refer to [48 ] .
Nesterov Acceleration and Attention Models on R d \mathbb{R}^{d} R d
This subsection connects the previously discussed interpretation of transformers as dynamical systems with Nesterov acceleration on R d \mathbb{R}^{d} R d , motivating the theory developed later in this paper. At a high level, each choice of a self-attention kernel in (1) corresponds to a choice of geometry. More precisely, the softmax attention kernel corresponds to softmax Otto–Wasserstein geometry, while the linear attention kernel corresponds to a geometry reminiscent of Stein geometry. We begin by stating at this point formally the analog of (13) in these geometries. At a formal level, the self-attention kernel determines a mobility operator, whose inverse on suitable tangent and cotangent spaces induces the corresponding metric. This is the mechanism through which the choice of an attention kernel determines a geometry.
Accelerated softmax attention. Let A = A T A=A^{\mathsf{T}} A = A T be invertible and let B = V A − 1 = B T ≻ 0 B=VA^{-1}=B^{\mathsf{T}}\succ0 B = V A − 1 = B T ≻ 0 . The softmax attention kernel induces the formal local-mobility operator
L ρ S M [ Φ ] : = − ∇ ⋅ ( ρ Z ρ B ∇ Φ ) , Z ρ ( x ) : = ∫ R d exp ( y T A x ) ρ ( y ) d y . (15) L_{\rho}^{\mathrm{SM}}[\Phi]:=-\nabla\cdot\left(\frac{\rho}{Z_{\rho}}B\nabla\Phi\right),\qquad Z_{\rho}(x):=\int_{\mathbb{R}^{d}}\exp(y^{\mathsf{T}}Ax)\rho(y)\,\mathrm{d}y. \tag*{(15)} L ρ SM [ Φ ] := − ∇ ⋅ ( Z ρ ρ B ∇Φ ) , Z ρ ( x ) := ∫ R d exp ( y T A x ) ρ ( y ) d y . (15)
As in the Otto–Wasserstein case, this formally induces a Riemannian metric through
G ρ S M ( L ρ S M [ Φ ] , L ρ S M [ Ψ ] ) : = ∫ R d Φ L ρ S M [ Ψ ] d x = ∫ R d ρ Z ρ ∇ Φ T B ∇ Ψ d x . G_{\rho}^{\mathrm{SM}}\left(L_{\rho}^{\mathrm{SM}}[\Phi],L_{\rho}^{\mathrm{SM}}[\Psi]\right):=\int_{\mathbb{R}^{d}}\Phi L_{\rho}^{\mathrm{SM}}[\Psi]\,\mathrm{d}x=\int_{\mathbb{R}^{d}}\frac{\rho}{Z_{\rho}}\nabla\Phi^{\mathsf{T}}B\nabla\Psi\,\mathrm{d}x. G ρ SM ( L ρ SM [ Φ ] , L ρ SM [ Ψ ] ) := ∫ R d Φ L ρ SM [ Ψ ] d x = ∫ R d Z ρ ρ ∇ Φ T B ∇Ψ d x .
For a functional F \mathcal{F} F , the associated Hamiltonian is
H S M ( ρ , Φ ) : = 1 2 G ρ S M ( L ρ S M [ Φ ] , L ρ S M [ Φ ] ) + F ( ρ ) . (16) \mathcal{H}^{\mathrm{SM}}(\rho,\Phi):=\frac{1}{2}G_{\rho}^{\mathrm{SM}}\left(L_{\rho}^{\mathrm{SM}}[\Phi],L_{\rho}^{\mathrm{SM}}[\Phi]\right)+\mathcal{F}(\rho). \tag*{(16)} H SM ( ρ , Φ ) := 2 1 G ρ SM ( L ρ SM [ Φ ] , L ρ SM [ Φ ] ) + F ( ρ ) . (16)
The damped Hamiltonian equations associated with (16) formally read
{ ∂ t ρ t = − ∇ ⋅ ( ρ t Z ρ t B ∇ Φ t ) , ∂ t Φ t ( x ) + α t Φ t ( x ) = − ∇ Φ t ( x ) T B ∇ Φ t ( x ) 2 Z ρ t ( x ) − δ F δ ρ ( ρ t ) ( x ) + 1 2 ∫ R d exp ( y T A x ) ∇ Φ t ( y ) T B ∇ Φ t ( y ) Z ρ t ( y ) 2 ρ t ( y ) d y . (17) \begin{cases}
\partial_t\rho_t=-\nabla\cdot\left(\dfrac{\rho_t}{Z_{\rho_t}}B\nabla\Phi_t\right),\\
\partial_t\Phi_t(x)+\alpha_t\Phi_t(x)=-\dfrac{\nabla\Phi_t(x)^{\mathsf{T}}B\nabla\Phi_t(x)}{2Z_{\rho_t}(x)}-\dfrac{\delta\mathcal{F}}{\delta\rho}(\rho_t)(x)+\dfrac{1}{2}\int_{\mathbb{R}^{d}}\exp(y^{\mathsf{T}}Ax)\dfrac{\nabla\Phi_t(y)^{\mathsf{T}}B\nabla\Phi_t(y)}{Z_{\rho_t}(y)^2}\rho_t(y)\,\mathrm{d}y. \tag*{(17)}
\end{cases} ⎩ ⎨ ⎧ ∂ t ρ t = − ∇ ⋅ ( Z ρ t ρ t B ∇ Φ t ) , ∂ t Φ t ( x ) + α t Φ t ( x ) = − 2 Z ρ t ( x ) ∇ Φ t ( x ) T B ∇ Φ t ( x ) − δ ρ δ F ( ρ t ) ( x ) + 2 1 ∫ R d exp ( y T A x ) Z ρ t ( y ) 2 ∇ Φ t ( y ) T B ∇ Φ t ( y ) ρ t ( y ) d y . (17)
The second equation is understood modulo spatial constants. The integral term accounts for the dependence of Z ρ Z_{\rho} Z ρ on the density.
To make this construction rigorous, we must identify suitable spaces of probability densities and corresponding tangent and cotangent spaces on which the mobility operator is well-defined and invertible. On R d \mathbb{R}^{d} R d , smooth positivity and finite second moments alone do not suffice: the normalization Z ρ Z_{\rho} Z ρ may be infinite, and the required coercivity of the mobility operator is not automatic. Section 4 develops the corresponding functional framework using weighted Sobolev spaces and a weighted Poincaré inequality for the mobility ρ / Z ρ \rho/Z_{\rho} ρ / Z ρ .
Accelerated linear attention. Let A = A T ≻ 0 A=A^{\mathsf{T}}\succ0 A = A T ≻ 0 . The linear attention kernel K A ( x , y ) = x T A y K_A(x,y)=x^{\mathsf{T}}Ay K A ( x , y ) = x T A y induces the formal mobility operator
L ρ K A [ Φ ] : = − ∇ x ⋅ ( ρ ( x ) ∫ R d K A ( x , y ) ∇ Φ ( y ) ρ ( y ) d y ) . L_{\rho}^{K_A}[\Phi]:=-\nabla_x\cdot\left(\rho(x)\int_{\mathbb{R}^{d}}K_A(x,y)\nabla\Phi(y)\rho(y)\,\mathrm{d}y\right). L ρ K A [ Φ ] := − ∇ x ⋅ ( ρ ( x ) ∫ R d K A ( x , y ) ∇Φ ( y ) ρ ( y ) d y ) .
As in the Otto–Wasserstein case, L ρ K A L_{\rho}^{K_A} L ρ K A formally gives rise to a Riemannian metric on the directions generated by the mobility operator:
G ρ K A ( L ρ K A [ Φ ] , L ρ K A [ Ψ ] ) : = ∫ R d Φ L ρ K A [ Ψ ] d x = ∫ R d ∫ R d K A ( x , y ) ∇ Φ ( x ) T ∇ Ψ ( y ) ρ ( x ) ρ ( y ) d y d x . \begin{aligned}
G_{\rho}^{K_A}\left(L_{\rho}^{K_A}[\Phi],L_{\rho}^{K_A}[\Psi]\right):=\int_{\mathbb{R}^{d}}\Phi L_{\rho}^{K_A}[\Psi]\,\mathrm{d}x \\
&=\int_{\mathbb{R}^{d}}\int_{\mathbb{R}^{d}}K_A(x,y)\nabla\Phi(x)^{\mathsf{T}}\nabla\Psi(y)\rho(x)\rho(y)\,\mathrm{d}y\,\mathrm{d}x.
\end{aligned} G ρ K A ( L ρ K A [ Φ ] , L ρ K A [ Ψ ] ) := ∫ R d Φ L ρ K A [ Ψ ] d x = ∫ R d ∫ R d K A ( x , y ) ∇Φ ( x ) T ∇Ψ ( y ) ρ ( x ) ρ ( y ) d y d x .
For a functional F \mathcal{F} F , the associated Hamiltonian is
H K A ( ρ , Φ ) : = 1 2 G ρ K A ( L ρ K A [ Φ ] , L ρ K A [ Φ ] ) + F ( ρ ) . (18) \mathcal{H}^{K_A}(\rho,\Phi):=\frac{1}{2}G_{\rho}^{K_A}\left(L_{\rho}^{K_A}[\Phi],L_{\rho}^{K_A}[\Phi]\right)+\mathcal{F}(\rho). \tag*{(18)} H K A ( ρ , Φ ) := 2 1 G ρ K A ( L ρ K A [ Φ ] , L ρ K A [ Φ ] ) + F ( ρ ) . (18)
The damped Hamiltonian equations associated with (18) formally read
∂ t ρ t ( x ) = − ∇ x ⋅ ( ρ t ( x ) ∫ R d K A ( x , y ) ∇ Φ t ( y ) ρ t ( y ) d y ) , ∂ t Φ t ( x ) + α t Φ t ( x ) = − ∫ R d K A ( x , y ) ∇ Φ t ( x ) ⊤ ∇ Φ t ( y ) ρ t ( y ) d y − δ F δ ρ ( ρ t ) ( x ) , (19) \begin{aligned}
\partial_t \rho_t(x) &= -\nabla_x \cdot\left(\rho_t(x) \int_{\mathbb{R}^d} K_A(x,y)\nabla\Phi_t(y)\rho_t(y)\,\mathrm{d}y\right),\\
\partial_t\Phi_t(x) + \alpha_t\Phi_t(x) &= -\int_{\mathbb{R}^d} K_A(x,y)\nabla\Phi_t(x)^{\top}\nabla\Phi_t(y)\rho_t(y)\,\mathrm{d}y - \frac{\delta F}{\delta\rho}(\rho_t)(x), \tag*{(19)}
\end{aligned} ∂ t ρ t ( x ) ∂ t Φ t ( x ) + α t Φ t ( x ) = − ∇ x ⋅ ( ρ t ( x ) ∫ R d K A ( x , y ) ∇ Φ t ( y ) ρ t ( y ) d y ) , = − ∫ R d K A ( x , y ) ∇ Φ t ( x ) ⊤ ∇ Φ t ( y ) ρ t ( y ) d y − δ ρ δ F ( ρ t ) ( x ) , (19)
where the second equation is understood modulo spatial constants.
To make this construction rigorous, we must identify suitable spaces of probability densities and corresponding tangent and cotangent spaces on which the mobility operator is well-defined. Several obstacles arise: in particular, the bilinear kernel is not integrally strictly positive definite, so the usual Stein framework based on this assumption does not apply directly. The mobility operator has finite-dimensional range and a nontrivial kernel beyond constants. Recovering an isomorphism therefore requires restricting the tangent space to its range and quotienting the cotangent space by its kernel. We develop this construction in § 3.3 .
In summary, making the geometric interpretation of accelerated linear and softmax attention rigorous involves several fundamental obstacles, including:
Function spaces and mobility operators. Choosing appropriate function spaces, including the necessary restrictions and quotients, so that the mobility operators L ρ K A L_{\rho}^{K_A} L ρ K A and L ρ S M L_{\rho}^{\mathrm{SM}} L ρ SM become isomorphisms between suitable cotangent and tangent spaces and induce well-defined metrics.
II Geometric foundations
A Non-Compact Functional Framework for Otto Calculus
In this section, we develop a rigorous local-mobility calculus for smooth probability densities on R d \mathbb{R}^d R d , encompassing both Otto–Wasserstein geometry from optimal transport [78 , 79 ] , and softmax–Wasserstein geometry. Even for the classical Otto calculus, smooth positivity and finite second moments do not by themselves justify the formal geometric construction on a non-compact domain. In particular, the operator
Φ ↦ − ∇ ⋅ ( ρ ∇ Φ ) \Phi\mapsto-\nabla\cdot(\rho\nabla\Phi) Φ ↦ − ∇ ⋅ ( ρ ∇Φ )
requires suitable domains before it can be inverted modulo constants and used to define a metric or a gradient.
On a closed connected Riemannian manifold, smooth strictly positive densities are bounded above and away from zero. Weighted Poincaré inequalities [8 ] and elliptic regularity therefore provide the analytic tools needed for this construction. On R d \mathbb{R}^d R d , these properties require additional assumptions. Although the Wasserstein distance is well defined on P 2 ( R d ) \mathcal{P}_2(\mathbb{R}^d) P 2 ( R d ) , membership in this space does not supply the regular tangent and cotangent spaces [3 ] , operator invertibility, or notion of differentiation needed for the density calculus used here.
We address these issues by specifying the function spaces and analytic assumptions explicitly. After recalling the relevant distinctions from infinite-dimensional Riemannian geometry in Subsection 4.1, we introduce the local-mobility framework in Subsection 4.2. Its starting point is a weighted Sobolev space of potentials and a weighted Poincaré inequality. This inequality makes the centered potential space complete, as established in Lemma 4.2 , and provides the coercivity needed to solve the potential equation.
The main result in Theorem 4.3 shows that the local-mobility operator is a linear isomorphism between the chosen regular cotangent and tangent spaces. Its inverse defines the metric in Corollary 4.4 . Then Corollary 4.6 identifies the gradient of a functional whenever its first variation belongs to the regular cotangent space. The required notion of differentiation is defined through regular first variations in Appendix § 7.5 .
Finally, we specialize this construction to the Otto–Wasserstein and softmax–Wasserstein geometry, which correspond to different choices of mobility. This gives explicit admissible density classes and regular tangent and cotangent spaces for both geometries. The resulting calculus provides the analytic framework for the gradient and Hamiltonian formulations used later in the paper. All proofs are given in Appendix § 7.7 .
Background on Infinite Dimensional Geometry
We start by recalling the basic notation and definitions from infinite-dimensional Riemannian geometry used throughout the paper. For a more comprehensive introduction, see [44 , 55 , 69 ] .
Let M \mathcal{M} M be a smooth infinite-dimensional manifold. A Riemannian metric on M \mathcal{M} M is a smooth map
G : T M × M T M → R G : T\mathcal{M} \times_{\mathcal{M}} T\mathcal{M} \to\mathbb{R} G : T M × M T M → R
such that, for every x ∈ M x \in\mathcal{M} x ∈ M , the map
G x : T x M × T x M → R G_x : T_x\mathcal{M} \times T_x\mathcal{M} \to\mathbb{R} G x : T x M × T x M → R
is an inner product on T x M T_x\mathcal{M} T x M . The metric G G G is called strong if, for every x ∈ M x \in\mathcal{M} x ∈ M , the topology induced by G x G_x G x agrees with the given manifold topology on T x M T_x\mathcal{M} T x M . It is called weak if, for at least one x ∈ M x \in\mathcal{M} x ∈ M , the topology induced by G x G_x G x is strictly weaker than the given manifold topology. We recall that the existence of a strong Riemannian metric is a very restrictive condition in infinite dimensions. By [10 Thm. 7.2.] , a smooth infinite-dimensional manifold M \mathcal{M} M admits a strong Riemannian metric if and only if M \mathcal{M} M is a Hilbert manifold and the induced musical map
G ♭ : T M → T ∗ M , G x ♭ ( v ) ( w ) = G x ( v , w ) , (20) G^\flat: T\mathcal{M} \to T^*\mathcal{M}, \qquad G_x^\flat(v)(w) = G_x(v,w), \tag*{(20)} G ♭ : T M → T ∗ M , G x ♭ ( v ) ( w ) = G x ( v , w ) , (20)
is a bundle isomorphism. Consequently, Fréchet manifolds, such as the space of smooth probability densities P ∞ ( R d ) \mathcal{P}^\infty(\mathbb{R}^d) P ∞ ( R d ) , do not admit strong Riemannian metrics in this sense.
The metrics considered in this paper therefore are weak Riemannian metrics. Prominent examples include, besides the Otto–Wasserstein metric on spaces of probability densities, the L 2 L^2 L 2 -metric appearing in Arnold’s geometric formulation of hydrodynamics [4 , 65 ] . Weak Riemannian metrics that are not strong are a genuinely infinite-dimensional phenomenon and lead to several difficulties that have no finite-dimensional analog. We highlight the distinction that is most important in our context.
We conclude this subsection by noting striking infinite-dimensional pathologies that have no finite-dimensional counterparts. One classical manifestation is the failure of the Hopf–Rinow theorem, first observed by Atkin [5 ] and later extended by Ekeland [27 ] . A second striking phenomenon is that the geodesic distance induced by a weak Riemannian or weak Finsler metric may vanish identically, even though the infinitesimal metric is pointwise non-degenerate. In symplectic geometry, Eliashberg and Polterovich [29 ] exhibited this phenomenon for L p L^p L p -analogs of Hofer’s metric on the Hamiltonian diffeomorphism group. Michor and Mumford [57 ] studied related vanishing-distance phenomena for weak Riemannian metrics. Finally, while a Levi–Civita connection exists for strong Riemannian metrics, this may fail in the weak Riemannian setting [9 ] .
Weighted Local-Mobility Geometry on P ∞ ( R d ) \mathcal{P}^{\infty}(\mathbb{R}^d) P ∞ ( R d )
This subsection develops a unified, rigorous framework that encompasses both Otto–Wasserstein and softmax–Wasserstein geometries for smooth probability densities on R d \mathbb{R}^d R d . On closed manifolds, these two geometries are norm-equivalent, whereas on non-compact domains such as R d \mathbb{R}^d R d , they differ significantly. Nevertheless, both belong to a more general class of geometries [15 , 23 ] , which we call local mobility geometries. In the following, we introduce our general framework and then show that both settings fit within it.
General setting. For the rest of the paper, we set
P ∞ ( R d ) : = { ρ ∈ C 0 ∞ ( R d ) : ∫ R d ρ ( x ) d x = 1 } . \mathcal{P}^{\infty}(\mathbb{R}^d):=\left\{\rho\in C_0^{\infty}(\mathbb{R}^d):\int_{\mathbb{R}^d}\rho(x)\,\mathrm{d}x=1\right\}. P ∞ ( R d ) := { ρ ∈ C 0 ∞ ( R d ) : ∫ R d ρ ( x ) d x = 1 } .
Further, let m ρ = m ρ ( x ) d x m_\rho=m_\rho(x)\,\mathrm{d}x m ρ = m ρ ( x ) d x , and B ∈ R d × d B\in\mathbb{R}^{d\times d} B ∈ R d × d a symmetric, positive definite matrix. Here are our main model cases.
Otto–Wasserstein: m ρ = ρ m_\rho=\rho m ρ = ρ and B = I B=I B = I ,
Equation not extracted
(21)
Softmax–Wasserstein: m ρ = ρ / Z ρ m_\rho=\rho/Z_\rho m ρ = ρ / Z ρ , Z ρ ( x ) : = ∫ R d exp ( z T A x ) ρ ( z ) d z Z_\rho(x):=\int_{\mathbb{R}^d}\exp(z^\mathsf{T}Ax)\rho(z)\,\mathrm{d}z Z ρ ( x ) := ∫ R d exp ( z T A x ) ρ ( z ) d z and B : = V A − 1 B:=VA^{-1} B := V A − 1
Equation not extracted
(22)
Before moving on, we note that sigmoid–Wasserstein and native L 2 L^2 L 2 geometry also fit within this framework. Now let
L 2 ( m ρ ) : = { Φ : R d → R measurable : ∥ Φ ∥ L 2 ( m ρ ) 2 : = ∫ R d ∣ Φ ∣ 2 d m ρ < ∞ } , L^2(m_\rho) := \left\{ \Phi: \mathbb{R}^d \to\mathbb{R}\ \text{measurable} : \|\Phi\|^2_{L^2(m_\rho)} := \int_{\mathbb{R}^d} |\Phi|^2 \,\mathrm{d}m_\rho< \infty\right\}, L 2 ( m ρ ) := { Φ : R d → R measurable : ∥Φ ∥ L 2 ( m ρ ) 2 := ∫ R d ∣Φ ∣ 2 d m ρ < ∞ } ,
L 2 ( m ρ , B ; R d ) : = { v : R d → R d measurable : ∥ v ∥ L 2 ( m ρ , B ; R d ) 2 : = ∫ R d v T B v d m ρ < ∞ } . L^2(m_\rho,B;\mathbb{R}^d) := \left\{ v : \mathbb{R}^d \to\mathbb{R}^d\ \text{measurable} : \|v\|^2_{L^2(m_\rho,B;\mathbb{R}^d)} := \int_{\mathbb{R}^d} v^\mathsf{T}Bv \,\mathrm{d}m_\rho< \infty\right\}. L 2 ( m ρ , B ; R d ) := { v : R d → R d measurable : ∥ v ∥ L 2 ( m ρ , B ; R d ) 2 := ∫ R d v T B v d m ρ < ∞ } .
We consider the Hilbert space
H 1 ( m ρ , B ) : = { Φ ∈ L 2 ( m ρ ) : ∇ Φ ∈ L 2 ( m ρ , B ; R d ) } H^1(m_\rho,B) := \left\{ \Phi\in L^2(m_\rho) : \nabla\Phi\in L^2(m_\rho,B;\mathbb{R}^d) \right\} H 1 ( m ρ , B ) := { Φ ∈ L 2 ( m ρ ) : ∇Φ ∈ L 2 ( m ρ , B ; R d ) }
with distributional gradient and equipped with the norm
∥ Φ ∥ H 1 ( m ρ , B ) 2 : = ∥ Φ ∥ L 2 ( m ρ ) 2 + ∥ ∇ Φ ∥ L 2 ( m ρ , B ; R d ) 2 = ∫ R d ∣ Φ ∣ 2 d m ρ + ∫ R d ∇ Φ T B ∇ Φ d m ρ . (23) \|\Phi\|^2_{H^1(m_\rho,B)} := \|\Phi\|^2_{L^2(m_\rho)} + \|\nabla\Phi\|^2_{L^2(m_\rho,B;\mathbb{R}^d)} = \int_{\mathbb{R}^d} |\Phi|^2 \,\mathrm{d}m_\rho+ \int_{\mathbb{R}^d} \nabla\Phi^\mathsf{T}B\nabla\Phi\,\mathrm{d}m_\rho. \tag*{(23)} ∥Φ ∥ H 1 ( m ρ , B ) 2 := ∥Φ ∥ L 2 ( m ρ ) 2 + ∥∇Φ ∥ L 2 ( m ρ , B ; R d ) 2 = ∫ R d ∣Φ ∣ 2 d m ρ + ∫ R d ∇ Φ T B ∇Φ d m ρ . (23)
We further define the centered potential space by
H 0 1 ( m ρ , B ) : = { Φ ∈ H 1 ( m ρ , B ) : ∫ R d Φ d m ρ = 0 } . (24) H^1_0(m_\rho,B) := \left\{ \Phi\in H^1(m_\rho,B) : \int_{\mathbb{R}^d} \Phi\,\mathrm{d}m_\rho= 0 \right\}. \tag*{(24)} H 0 1 ( m ρ , B ) := { Φ ∈ H 1 ( m ρ , B ) : ∫ R d Φ d m ρ = 0 } . (24)
Then
∥ Φ ∥ m ρ , B : = ( ∫ R d ∇ Φ T B ∇ Φ d m ρ ) 1 / 2 (25) \|\Phi\|_{m_\rho,B} := \left(\int_{\mathbb{R}^d} \nabla\Phi^\mathsf{T}B\nabla\Phi\,\mathrm{d}m_\rho\right)^{1/2} \tag*{(25)} ∥Φ ∥ m ρ , B := ( ∫ R d ∇ Φ T B ∇Φ d m ρ ) 1/2 (25)
is a norm on H 0 1 ( m ρ , B ) H^1_0(m_\rho,B) H 0 1 ( m ρ , B ) , but the space is in general not complete with this norm. The missing analytic condition is a weighted Poincaré inequality. We say that m ρ m_\rho m ρ satisfies the local-mobility Poincaré inequality (LM-PI), if there exists C ρ > 0 C_\rho> 0 C ρ > 0 such that
∫ R d ∣ Φ ∣ 2 d m ρ ≤ C ρ ∫ R d ∇ Φ T B ∇ Φ d m ρ for all Φ ∈ H 0 1 ( m ρ , B ) . (26) \int_{\mathbb{R}^d} |\Phi|^2 \,\mathrm{d}m_\rho\le C_\rho\int_{\mathbb{R}^d} \nabla\Phi^\mathsf{T}B\nabla\Phi\,\mathrm{d}m_\rho\quad\text{for all } \Phi\in H^1_0(m_\rho,B). \tag*{(26)} ∫ R d ∣Φ ∣ 2 d m ρ ≤ C ρ ∫ R d ∇ Φ T B ∇Φ d m ρ for all Φ ∈ H 0 1 ( m ρ , B ) . (26)
With this notation at hand, we define the space of smooth local mobility probability densities by
P m ρ , B : = { ρ ∈ P ∞ ( R d ) : m ρ ( R d ) ∈ ( 0 , ∞ ) , m ρ ∈ C 0 ∞ ( R d ) and satisfies LM-PI (26) } . (27) \mathcal{P}_{m_\rho,B} := \left\{ \rho\in\mathcal{P}^\infty(\mathbb{R}^d) : m_\rho(\mathbb{R}^d) \in(0,\infty), m_\rho\in C^\infty_0(\mathbb{R}^d)\ \text{and satisfies LM-PI (26)} \right\}. \tag*{(27)} P m ρ , B := { ρ ∈ P ∞ ( R d ) : m ρ ( R d ) ∈ ( 0 , ∞ ) , m ρ ∈ C 0 ∞ ( R d ) and satisfies LM-PI (26) } . (27)
Before proceeding, we note that the admissible densities in both (21) and (22) have finite second moments. Thus, each defines a probability measure in P 2 ( R d ) \mathcal{P}_2(\mathbb{R}^d) P 2 ( R d ) . In the Otto–Wasserstein case, this follows by applying the Poincaré inequality (26) to centered truncations of the coordinate functions and passing to the limit. In the softmax–Wasserstein case, invertibility of A A A and finiteness of Z ρ ( x ) Z_\rho(x) Z ρ ( x ) for every x ∈ R d x \in\mathbb{R}^d x ∈ R d imply
∫ R d exp ( w T y ) ρ ( y ) d y < ∞ for every w ∈ R d . \int_{\mathbb{R}^d} \exp(w^\mathsf{T}y)\rho(y)\,\mathrm{d}y < \infty\quad\text{for every } w \in\mathbb{R}^d. ∫ R d exp ( w T y ) ρ ( y ) d y < ∞ for every w ∈ R d .
Consequently, all polynomial moments are finite.
We now establish the properties of the potential spaces needed to invert the mobility operator. The following lemma shows that the Poincaré inequality (26) makes the centered space (24) complete under the gradient norm (25) , and that its elements can be approximated by centered smooth test functions.
Lemma 4.2 (Centered local-mobility energy space). For ρ ∈ P m ρ , B \rho\in\mathcal{P}_{m_\rho,B} ρ ∈ P m ρ , B , it holds:
H 0 1 ( m ρ , B ) H^1_0(m_\rho,B) H 0 1 ( m ρ , B ) with norm (25) is a Hilbert space; in particular, this norm is equivalent to the induced norm (23).
C c ∞ ( R d ) C^\infty_c(\mathbb{R}^d) C c ∞ ( R d ) is dense in H 1 ( m ρ , B ) H^1(m_\rho,B) H 1 ( m ρ , B ) , and the centered functions
ψ − 1 m ρ ( R d ) ∫ R d ψ d m ρ , ψ ∈ C c ∞ ( R d ) , (28) \psi-\frac{1}{\mathbf{m}_\rho(\mathbb{R}^d)}\int_{\mathbb{R}^d}\psi\,\mathrm{d}m_\rho,\qquad\psi\in C^\infty_c(\mathbb{R}^d), \tag*{(28)} ψ − m ρ ( R d ) 1 ∫ R d ψ d m ρ , ψ ∈ C c ∞ ( R d ) , (28) are dense in H 0 1 ( m ρ , B ) H^1_0(m_\rho,B) H 0 1 ( m ρ , B ) .
Regular tangent and cotangent spaces. For ρ ∈ P m ρ , B \rho\in\mathcal{P}_{m_\rho,B} ρ ∈ P m ρ , B , define the regular formal tangent space by
T ρ P m ρ , B : = { ρ ˙ ∈ C ∞ ( R d ) : ρ ˙ m ρ ∈ L 2 ( m ρ ) , ∫ R d ρ ˙ d x = 0 } . (29) T_\rho\mathcal{P}_{m_\rho,B}:=\left\{\dot{\rho}\in C^\infty(\mathbb{R}^d):\frac{\dot{\rho}}{m_\rho}\in L^2(m_\rho),\ \int_{\mathbb{R}^d}\dot{\rho}\,\mathrm{d}x=0\right\}. \tag*{(29)} T ρ P m ρ , B := { ρ ˙ ∈ C ∞ ( R d ) : m ρ ρ ˙ ∈ L 2 ( m ρ ) , ∫ R d ρ ˙ d x = 0 } . (29)
In the Otto–Wasserstein setting, the condition ρ ˙ m ρ ∈ L 2 ( m ρ ) \frac{\dot{\rho}}{m_\rho}\in L^2(m_\rho) m ρ ρ ˙ ∈ L 2 ( m ρ ) simplifies to ρ ˙ ρ ∈ L 2 ( ρ ) \frac{\dot{\rho}}{\rho}\in L^2(\rho) ρ ρ ˙ ∈ L 2 ( ρ ) , which can be thought of as a natural decay condition. It also arises in Fisher–Rao geometry on the space of probability densities on R d \mathbb{R}^d R d , as recently investigated in [52 , 53 ] . Next, for a smooth potential Φ \Phi Φ , we define the local-mobility operator by
L m ρ , B Φ : = − ∇ ⋅ ( m ρ B ∇ Φ ) . (30) L_{m_\rho,B}\Phi:=-\nabla\cdot(m_\rho B\nabla\Phi). \tag*{(30)} L m ρ , B Φ := − ∇ ⋅ ( m ρ B ∇Φ ) . (30)
In Otto’s calculus, we need this operator to be well-defined and invertible. Next, we introduce the regular formal cotangent space as a domain of this operator, so that it will be invertible, namely
T ρ ∗ P m ρ , B : = { [ Φ ] ∈ C ∞ ( R d ) ∩ H 1 ( m ρ , B ) R : L m ρ , B Φ m ρ ∈ L 2 ( m ρ ) } . (31) T_\rho^*\mathcal{P}_{m_\rho,B}:=\left\{[\Phi]\in\frac{C^\infty(\mathbb{R}^d)\cap H^1(m_\rho,B)}{\mathbb{R}}:\frac{L_{m_\rho,B}\Phi}{m_\rho}\in L^2(m_\rho)\right\}. \tag*{(31)} T ρ ∗ P m ρ , B := { [ Φ ] ∈ R C ∞ ( R d ) ∩ H 1 ( m ρ , B ) : m ρ L m ρ , B Φ ∈ L 2 ( m ρ ) } . (31)
The usual tangent–cotangent pairing is
⟨ [ Φ ] , ρ ˙ ⟩ : = ∫ R d Φ ρ ˙ d x . \langle[\Phi],\dot{\rho}\rangle:=\int_{\mathbb{R}^d}\Phi\dot{\rho}\,\mathrm{d}x. ⟨[ Φ ] , ρ ˙ ⟩ := ∫ R d Φ ρ ˙ d x .
Indeed, by the Cauchy–Schwarz inequality, the pairing is well-defined because
∣ ∫ R d Φ ρ ˙ d x ∣ = ∣ ∫ R d Φ ρ ˙ m ρ d m ρ ∣ ≤ ∥ Φ ∥ L 2 ( m ρ ) ∥ ρ ˙ m ρ ∥ L 2 ( m ρ ) . \left|\int_{\mathbb{R}^d}\Phi\dot{\rho}\,\mathrm{d}x\right|=\left|\int_{\mathbb{R}^d}\Phi\frac{\dot{\rho}}{m_\rho}\,\mathrm{d}m_\rho\right|\leq\|\Phi\|_{L^2(m_\rho)}\left\|\frac{\dot{\rho}}{m_\rho}\right\|_{L^2(m_\rho)}. ∫ R d Φ ρ ˙ d x = ∫ R d Φ m ρ ρ ˙ d m ρ ≤ ∥Φ ∥ L 2 ( m ρ ) m ρ ρ ˙ L 2 ( m ρ ) .
Moreover, the pairing is independent of the representative of [ Φ ] [\Phi] [ Φ ] , since ∫ R d ρ ˙ d x = 0 \int_{\mathbb{R}^d}\dot{\rho}\,\mathrm{d}x=0 ∫ R d ρ ˙ d x = 0 .
Invertibility of the local mobility operator and mobility metric. Next, we check that the local mobility operator (30) is a bijection from the regular cotangent space (31) to the regular tangent space (29).
Theorem 4.3 (Invertibility of the local-mobility operator). Let ρ ∈ P m ρ , B \rho\in\mathcal{P}_{m_\rho,B} ρ ∈ P m ρ , B . Then it holds:
For every [ Φ ] ∈ T ρ ∗ P m ρ , B [\Phi]\in T_\rho^*\mathcal{P}_{m_\rho,B} [ Φ ] ∈ T ρ ∗ P m ρ , B and every Ψ ∈ H 1 ( m ρ , B ) \Psi\in H^1(m_\rho,B) Ψ ∈ H 1 ( m ρ , B ) , we have
∫ R d ( L m ρ , B Φ ) Ψ d x = ∫ R d ∇ Φ ⊤ B ∇ Ψ d m ρ . (32) \int_{\mathbb{R}^d}(L_{m_\rho,B}\Phi)\Psi\,\mathrm{d}x=\int_{\mathbb{R}^d}\nabla\Phi^\top B\nabla\Psi\,\mathrm{d}m_\rho. \tag*{(32)} ∫ R d ( L m ρ , B Φ ) Ψ d x = ∫ R d ∇ Φ ⊤ B ∇Ψ d m ρ . (32)
The local-mobility operator
L m ρ , B : T ρ ∗ P m ρ , B → T ρ P m ρ , B , [ Φ ] ↦ − ∇ ⋅ ( m ρ B ∇ Φ ) , (33) L_{m_\rho,B}:T_\rho^*\mathcal{P}_{m_\rho,B}\to T_\rho\mathcal{P}_{m_\rho,B},\qquad[\Phi]\mapsto-\nabla\cdot(m_\rho B\nabla\Phi), \tag*{(33)} L m ρ , B : T ρ ∗ P m ρ , B → T ρ P m ρ , B , [ Φ ] ↦ − ∇ ⋅ ( m ρ B ∇Φ ) , (33)
is a linear isomorphism.
As a direct consequence of Theorem 4.3 we can define the local-mobility metric.
Corollary 4.4 (Well-definedness of the local-mobility metric). For every ρ ∈ P m ρ , B \rho\in\mathcal{P}_{m_\rho,B} ρ ∈ P m ρ , B , the following local-mobility metric is well-defined:
G m ρ , B : T ρ P m ρ , B × T ρ P m ρ , B → R , G m ρ , B ( ρ ˙ , ρ ˙ ) : = ∫ R d Φ ρ ˙ ρ ˙ d x , G_{m_\rho,B}:T_\rho\mathcal{P}_{m_\rho,B}\times T_\rho\mathcal{P}_{m_\rho,B}\to\mathbb{R},\qquad G_{m_\rho,B}(\dot{\rho},\dot{\rho}):=\int_{\mathbb{R}^d}\Phi_{\dot{\rho}}\dot{\rho}\,\mathrm{d}x, G m ρ , B : T ρ P m ρ , B × T ρ P m ρ , B → R , G m ρ , B ( ρ ˙ , ρ ˙ ) := ∫ R d Φ ρ ˙ ρ ˙ d x , where [ Φ ρ ˙ ] ∈ T ρ ∗ P m ρ , B [\Phi_{\dot{\rho}}]\in T_\rho^*\mathcal{P}_{m_\rho,B} [ Φ ρ ˙ ] ∈ T ρ ∗ P m ρ , B is determined by
ρ ˙ + ∇ ⋅ ( m ρ B ∇ Φ ρ ˙ ) = 0. (34) \dot{\rho}+\nabla\cdot(m_\rho B\nabla\Phi_{\dot{\rho}})=0. \tag*{(34)} ρ ˙ + ∇ ⋅ ( m ρ B ∇ Φ ρ ˙ ) = 0. (34)
First variations and the local-mobility gradient. Before stating the following corollary, we want to stress that P m ρ , B \mathcal{P}_{m_\rho,B} P m ρ , B has not been equipped with a differentiable structure, so the notion of differentiability requires a separate definition. In Appendix B, we provide such a definition in terms of local regular first variations. It agrees with ordinary Fréchet differentiation in the Otto–Wasserstein geometry on smooth probability densities on a closed manifold, whenever the differential has a smooth representative. The reader may therefore think of ordinary differentiation with a smooth first-variation representative in what follows, while D F ( ρ ) DF(\rho) D F ( ρ ) denotes the associated regular extension defined in Appendix B. Finally, we provide a criterion for the corresponding local-mobility gradient to be a regular tangent vector.
Corollary 4.6 (Local-mobility gradient). Let F : P m ρ , B → R F:\mathcal{P}_{m_\rho,B}\to\mathbb{R} F : P m ρ , B → R have a first variation satisfying
D F ( ρ ) [ η ˙ ] = ∫ R d δ F δ ρ η ˙ d x for all η ˙ ∈ T ρ P m ρ , B , DF(\rho)[\dot{\eta}]=\int_{\mathbb{R}^d}\frac{\delta F}{\delta\rho}\dot{\eta}\,\mathrm{d}x\quad\text{for all }\dot{\eta}\in T_\rho\mathcal{P}_{m_\rho,B}, D F ( ρ ) [ η ˙ ] = ∫ R d δ ρ δ F η ˙ d x for all η ˙ ∈ T ρ P m ρ , B , and so that [ δ F δ ρ ] ∈ T ρ ∗ P m ρ , B \left[\frac{\delta F}{\delta\rho}\right]\in T_\rho^*\mathcal{P}_{m_\rho,B} [ δ ρ δ F ] ∈ T ρ ∗ P m ρ , B . Then the gradient of F F F with respect to the local-mobility metric G m ρ , B G_{m_\rho,B} G m ρ , B is
grad G m ρ , B F ( ρ ) = L m ρ , B [ δ F δ ρ ] = − ∇ ⋅ ( m ρ B ∇ δ F δ ρ ) ∈ T ρ P m ρ , B . (36) \operatorname{grad}_{G_{m_\rho,B}}F(\rho)=L_{m_\rho,B}\left[\frac{\delta F}{\delta\rho}\right]=-\nabla\cdot\left(m_\rho B\nabla\frac{\delta F}{\delta\rho}\right)\in T_\rho\mathcal{P}_{m_\rho,B}. \tag*{(36)} grad G m ρ , B F ( ρ ) = L m ρ , B [ δ ρ δ F ] = − ∇ ⋅ ( m ρ B ∇ δ ρ δ F ) ∈ T ρ P m ρ , B . (36)
Since every tangent vector satisfies ∫ R d ρ ˙ d x = 0 \int_{\mathbb{R}^d}\dot{\rho}\,\mathrm{d}x=0 ∫ R d ρ ˙ d x = 0 , the first variation δ F δ ρ \frac{\delta F}{\delta\rho} δ ρ δ F is defined only modulo constants.
Finally, we consider the model cases from (21) and (22) .
Otto–Wasserstein. For d m ρ = ρ d x \mathrm{d}m_\rho=\rho\,\mathrm{d}x d m ρ = ρ d x and B = I d B=I_d B = I d , we obtain
H 1 ( ρ ) : = H 1 ( m ρ , I d ) = { Φ ∈ L 2 ( ρ ) : ∇ Φ ∈ L 2 ( ρ ; R d ) } . H^1(\rho):=H^1(m_\rho,I_d)=\left\{\Phi\in L^2(\rho):\nabla\Phi\in L^2(\rho;\mathbb{R}^d)\right\}. H 1 ( ρ ) := H 1 ( m ρ , I d ) = { Φ ∈ L 2 ( ρ ) : ∇Φ ∈ L 2 ( ρ ; R d ) } .
Then LM-PI (26) becomes the usual ρ \rho ρ -weighted Poincaré inequality: there exists C ρ > 0 C_\rho>0 C ρ > 0 such that
∫ R d ∣ Φ ∣ 2 ρ d x ≤ C ρ ∫ R d ∣ ∇ Φ ∣ 2 ρ d x for all Φ ∈ H 0 1 ( ρ ) , (37) \int_{\mathbb{R}^d}|\Phi|^2\rho\,\mathrm{d}x\le C_\rho\int_{\mathbb{R}^d}|\nabla\Phi|^2\rho\,\mathrm{d}x\qquad\text{for all }\Phi\in H^1_0(\rho), \tag*{(37)} ∫ R d ∣Φ ∣ 2 ρ d x ≤ C ρ ∫ R d ∣∇Φ ∣ 2 ρ d x for all Φ ∈ H 0 1 ( ρ ) , (37)
and the space (27) reads as
P W : = { ρ ∈ P ∞ ( R d ) : ρ satisfies (37) } . \mathcal{P}_{W}:=\left\{\rho\in\mathcal{P}^{\infty}(\mathbb{R}^d):\rho\text{ satisfies (37)}\right\}. P W := { ρ ∈ P ∞ ( R d ) : ρ satisfies (37) } .
Typical measures in this space are Gaussians and more general strongly log-concave measures. Moreover, P W \mathcal{P}_{W} P W is convex and hence path-connected in the topology induced by C ∞ ( R d ) C^\infty(\mathbb{R}^d) C ∞ ( R d ) . Its regular tangent space is given by
T ρ P W = { ρ ˙ ∈ C ∞ ( R d ) : ρ ˙ ρ ∈ L 2 ( ρ ) , ∫ R d ρ ˙ d x = 0 } . T_\rho\mathcal{P}_{W}=\left\{\dot{\rho}\in C^\infty(\mathbb{R}^d):\frac{\dot{\rho}}{\rho}\in L^2(\rho),\ \int_{\mathbb{R}^d}\dot{\rho}\,\mathrm{d}x=0\right\}. T ρ P W = { ρ ˙ ∈ C ∞ ( R d ) : ρ ρ ˙ ∈ L 2 ( ρ ) , ∫ R d ρ ˙ d x = 0 } .
The local mobility operator (30) becomes
L ρ W Φ = − ∇ ⋅ ( ρ ∇ Φ ) , L_\rho^W\Phi=-\nabla\cdot(\rho\nabla\Phi), L ρ W Φ = − ∇ ⋅ ( ρ ∇Φ ) ,
and the cotangent space is
T ρ ∗ P W = { [ Φ ] ∈ C ∞ ( R d ) ∩ H 1 ( ρ ) R : L ρ W Φ ρ ∈ L 2 ( ρ ) } . T_\rho^*\mathcal{P}_{W}=\left\{[\Phi]\in\frac{C^\infty(\mathbb{R}^d)\cap H^1(\rho)}{\mathbb{R}}:\frac{L_\rho^W\Phi}{\rho}\in L^2(\rho)\right\}. T ρ ∗ P W = { [ Φ ] ∈ R C ∞ ( R d ) ∩ H 1 ( ρ ) : ρ L ρ W Φ ∈ L 2 ( ρ ) } .
Finally, the local mobility metric reads as
G ρ W : T ρ P W × T ρ P W → R , G ρ W ( ρ ˙ , η ˙ ) : = ∫ R d ρ ∇ Φ ρ ˙ ⋅ ∇ Φ η ˙ d x , G_\rho^W:T_\rho\mathcal{P}_{W}\times T_\rho\mathcal{P}_{W}\to\mathbb{R},\qquad G_\rho^W(\dot{\rho},\dot{\eta}):=\int_{\mathbb{R}^d}\rho\nabla\Phi_{\dot{\rho}}\cdot\nabla\Phi_{\dot{\eta}}\,\mathrm{d}x, G ρ W : T ρ P W × T ρ P W → R , G ρ W ( ρ ˙ , η ˙ ) := ∫ R d ρ ∇ Φ ρ ˙ ⋅ ∇ Φ η ˙ d x ,
where Φ ρ ˙ , Φ η ˙ \Phi_{\dot{\rho}},\Phi_{\dot{\eta}} Φ ρ ˙ , Φ η ˙ are defined with the potential equation
ρ ˙ + ∇ ⋅ ( ρ ∇ Φ ρ ˙ ) = 0. \dot{\rho}+\nabla\cdot(\rho\nabla\Phi_{\dot{\rho}})=0. ρ ˙ + ∇ ⋅ ( ρ ∇ Φ ρ ˙ ) = 0.
These regular tangent and cotangent spaces make explicit the regularity assumptions tacitly made in [58 Subsec. 4.2] and [1 Subsec. 4.1] .
Softmax–Wasserstein. Softmax gradient flows have recently gained interest because they have applications in the mathematical foundations of transformers. Let
A = A ⊤ ∈ R d × d be invertible and B : = V A − 1 = ( V A − 1 ) ⊤ ≻ 0. (38) A=A^\top\in\mathbb{R}^{d\times d}\text{ be invertible and }B:=VA^{-1}=(VA^{-1})^\top\succ0. \tag*{(38)} A = A ⊤ ∈ R d × d be invertible and B := V A − 1 = ( V A − 1 ) ⊤ ≻ 0. (38)
For ρ ∈ P 2 ( R d ) ∩ P ∞ ( R d ) \rho\in\mathcal{P}_2(\mathbb{R}^d)\cap\mathcal{P}^{\infty}(\mathbb{R}^d) ρ ∈ P 2 ( R d ) ∩ P ∞ ( R d ) , we define the softmax mobility measure by
d m ρ ( x ) : = ρ ( x ) Z ρ ( x ) d x , Z ρ ( x ) : = ∫ R d exp ( z ⊤ A x ) ρ ( z ) d z . (39) \mathrm{d}m_\rho(x):=\frac{\rho(x)}{Z_\rho(x)}\,\mathrm{d}x,\qquad Z_\rho(x):=\int_{\mathbb{R}^d}\exp(z^\top Ax)\rho(z)\,\mathrm{d}z. \tag*{(39)} d m ρ ( x ) := Z ρ ( x ) ρ ( x ) d x , Z ρ ( x ) := ∫ R d exp ( z ⊤ A x ) ρ ( z ) d z . (39)
This gives rise to the weighted Sobolev space
H 1 ( m ρ , B ) = { Φ ∈ L 2 ( m ρ ) : ∇ Φ ∈ L 2 ( m ρ , V A − 1 ; R d ) } . H^{1}(m_{\rho},B)=\left\{\Phi\in L^{2}(m_{\rho}):\nabla\Phi\in L^{2}(m_{\rho},VA^{-1};\mathbb{R}^{d})\right\}. H 1 ( m ρ , B ) = { Φ ∈ L 2 ( m ρ ) : ∇Φ ∈ L 2 ( m ρ , V A − 1 ; R d ) } .
We say that m ρ : = ρ ( x ) Z ρ ( x ) m_{\rho}:=\frac{\rho(x)}{Z_{\rho}(x)} m ρ := Z ρ ( x ) ρ ( x ) satisfies the local-mobility Poincaré inequality , if there exists C ρ > 0 C_{\rho}>0 C ρ > 0 such that
∥ Φ ∥ L ( ρ ( x ) Z ρ ( x ) ) 2 2 ≤ C ρ ∥ ∇ Φ ∥ L ( ρ ( x ) Z ρ ( x ) ) , V A − 1 ; R d 2 2 for all Φ ∈ H 0 1 ( m ρ , B ) . (40) \|\Phi\|_{L^{2}_{\left(\frac{\rho(x)}{Z_{\rho}(x)}\right)}}^{2}\leq C_{\rho}\|\nabla\Phi\|_{L^{2}_{\left(\frac{\rho(x)}{Z_{\rho}(x)}\right),VA^{-1};\mathbb{R}^{d}}}^{2}\quad\text{for all }\Phi\in H_{0}^{1}(m_{\rho},B). \tag*{(40)} ∥Φ ∥ L ( Z ρ ( x ) ρ ( x ) ) 2 2 ≤ C ρ ∥∇Φ ∥ L ( Z ρ ( x ) ρ ( x ) ) , V A − 1 ; R d 2 2 for all Φ ∈ H 0 1 ( m ρ , B ) . (40)
With this notation, we can define, as a special case of (27) , the corresponding smooth softmax–Wasserstein class by
P S M : = { ρ ∈ P ∞ ( R d ) : Z ρ ∈ C > 0 ∞ ( R d ) , ∫ R d ρ ( x ) Z ρ ( x ) d x ∈ ( 0 , ∞ ) , m ρ satisfies (40) } . (41) \mathcal{P}_{\mathrm{SM}}:=\left\{\rho\in\mathcal{P}^{\infty}(\mathbb{R}^{d}):Z_{\rho}\in C_{>0}^{\infty}(\mathbb{R}^{d}),\int_{\mathbb{R}^{d}}\frac{\rho(x)}{Z_{\rho}(x)}\,\mathrm{d}x\in(0,\infty),m_{\rho}\text{ satisfies (40)}\right\}. \tag*{(41)} P SM := { ρ ∈ P ∞ ( R d ) : Z ρ ∈ C > 0 ∞ ( R d ) , ∫ R d Z ρ ( x ) ρ ( x ) d x ∈ ( 0 , ∞ ) , m ρ satisfies (40) } . (41)
Before we define the associated regular tangent space, we observe that
∥ ρ ˙ m ρ ∥ L 2 ( m ρ ) 2 = ∫ R d ∣ Z ρ ρ ˙ ρ ∣ 2 ρ Z ρ d x = ∫ R d Z ρ ∣ ρ ˙ ∣ 2 ρ d x . \left\|\frac{\dot{\rho}}{m_{\rho}}\right\|_{L^{2}(m_{\rho})}^{2}=\int_{\mathbb{R}^{d}}\left|\frac{Z_{\rho}\dot{\rho}}{\rho}\right|^{2}\frac{\rho}{Z_{\rho}}\,\mathrm{d}x=\int_{\mathbb{R}^{d}}Z_{\rho}\frac{|\dot{\rho}|^{2}}{\rho}\,\mathrm{d}x. m ρ ρ ˙ L 2 ( m ρ ) 2 = ∫ R d ρ Z ρ ρ ˙ 2 Z ρ ρ d x = ∫ R d Z ρ ρ ∣ ρ ˙ ∣ 2 d x .
So the regular local mobility tangent space simplifies in the softmax setting to
T ρ P S M = { ρ ˙ ∈ C ∞ ( R d ) : ∫ R d ρ ˙ d x = 0 , ∫ R d Z ρ ∣ ρ ˙ ∣ 2 ρ d x < ∞ } . (42) T_{\rho}\mathcal{P}_{\mathrm{SM}}=\left\{\dot{\rho}\in C^{\infty}(\mathbb{R}^{d}):\int_{\mathbb{R}^{d}}\dot{\rho}\,\mathrm{d}x=0,\ \int_{\mathbb{R}^{d}}Z_{\rho}\frac{|\dot{\rho}|^{2}}{\rho}\,\mathrm{d}x<\infty\right\}. \tag*{(42)} T ρ P SM = { ρ ˙ ∈ C ∞ ( R d ) : ∫ R d ρ ˙ d x = 0 , ∫ R d Z ρ ρ ∣ ρ ˙ ∣ 2 d x < ∞ } . (42)
Next, we define the softmax operator as
L ρ S M Φ = − ∇ ⋅ ( ρ Z ρ V A − 1 ∇ Φ ) . (43) L_{\rho}^{\mathrm{SM}}\Phi=-\nabla\cdot\left(\frac{\rho}{Z_{\rho}}VA^{-1}\nabla\Phi\right). \tag*{(43)} L ρ SM Φ = − ∇ ⋅ ( Z ρ ρ V A − 1 ∇Φ ) . (43)
and the regular softmax cotangent space is
T ρ ∗ P S M = { [ Φ ] ∈ C ∞ ( R d ) ∩ H 1 ( m ρ , B ) R : ∫ R d Z ρ ρ ∣ L ρ S M Φ ∣ 2 d x < ∞ } . (44) T_{\rho}^{*}\mathcal{P}_{\mathrm{SM}}=\left\{[\Phi]\in\frac{C^{\infty}(\mathbb{R}^{d})\cap H^{1}(m_{\rho},B)}{\mathbb{R}}:\int_{\mathbb{R}^{d}}\frac{Z_{\rho}}{\rho}|L_{\rho}^{\mathrm{SM}}\Phi|^{2}\,\mathrm{d}x<\infty\right\}. \tag*{(44)} T ρ ∗ P SM = { [ Φ ] ∈ R C ∞ ( R d ) ∩ H 1 ( m ρ , B ) : ∫ R d ρ Z ρ ∣ L ρ SM Φ ∣ 2 d x < ∞ } . (44)
Finally, the local-mobility metric at ρ ∈ P S M \rho\in\mathcal{P}_{\mathrm{SM}} ρ ∈ P SM is given by
G ρ S M : T ρ P S M × T ρ P S M → R , G ρ S M ( ρ ˙ , ρ ˙ ) : = ∫ R d Φ ρ ˙ ρ ˙ d x , (45) G_{\rho}^{\mathrm{SM}}:T_{\rho}\mathcal{P}_{\mathrm{SM}}\times T_{\rho}\mathcal{P}_{\mathrm{SM}}\to\mathbb{R},\qquad G_{\rho}^{\mathrm{SM}}(\dot{\rho},\dot{\rho}):=\int_{\mathbb{R}^{d}}\Phi_{\dot{\rho}}\dot{\rho}\,\mathrm{d}x, \tag*{(45)} G ρ SM : T ρ P SM × T ρ P SM → R , G ρ SM ( ρ ˙ , ρ ˙ ) := ∫ R d Φ ρ ˙ ρ ˙ d x , (45)
where [ Φ ρ ˙ ] ∈ T ρ ∗ P S M [\Phi_{\dot{\rho}}]\in T_{\rho}^{*}\mathcal{P}_{\mathrm{SM}} [ Φ ρ ˙ ] ∈ T ρ ∗ P SM is determined by
ρ ˙ + ∇ ⋅ ( ρ Z ρ V A − 1 ∇ Φ ρ ˙ ) = 0. \dot{\rho}+\nabla\cdot\left(\frac{\rho}{Z_{\rho}}VA^{-1}\nabla\Phi_{\dot{\rho}}\right)=0. ρ ˙ + ∇ ⋅ ( Z ρ ρ V A − 1 ∇ Φ ρ ˙ ) = 0.
We close this subsection by highlighting a fundamental difference between softmax–Wasserstein and Otto–Wasserstein geometry on probability densities on R d \mathbb{R}^{d} R d , in contrast to the more commonly considered setting of compact domains.
Stein Calculus for Bilinear Kernels
To interpret linear-attention dynamics geometrically, we would like to associate a Stein-type geometry with the bilinear kernel K A ( x , y ) = x T A y K_A(x,y)=x^\mathsf{T}Ay K A ( x , y ) = x T A y , where A = A T ≻ 0 A=A^\mathsf{T}\succ0 A = A T ≻ 0 . However, this kernel is not integrally strictly positive definite, so the usual Stein framework based on this assumption does not apply directly. Its associated Stein–Onsager operator has finite-dimensional range and an infinite-dimensional kernel, even after identifying potentials that differ by a constant, see Lemma 5.2 . Consequently, it cannot be inverted on the full regular tangent and cotangent spaces.
We overcome this obstruction by introducing a quotient Stein geometry . We restrict the tangent space to the density variations generated by the mobility operator and identify cotangent elements that generate the same variation. On these effective spaces, Theorem 5.3 establishes an induced Stein–Onsager isomorphism. Its inverse defines a positive definite metric on the effective tangent space, as shown in Corollary 5.4 .
We begin by specifying an admissible density class and the regular tangent–cotangent pairing. Then we establish the well-definedness of the Stein–Onsager operator and its Green identity in Lemma 5.1 , identify its kernel, and carry out the quotient construction. Finally, Corollary 5.5 identifies the gradient of a functional with an admissible regular first variation. The corresponding notion of differentiation is introduced in Appendix § 7.6 . This calculus provides the geometric framework for the linear-attention dynamics studied in § 4.2 . All proofs are given in Appendix § 7.8 .
General setting. Throughout this section, let A = A T ≻ 0 A=A^\mathsf{T}\succ0 A = A T ≻ 0 and K A ( x , y ) ≔ x T A y K_A(x,y)\coloneqq x^\mathsf{T}Ay K A ( x , y ) : = x T A y . We define the space of smooth bilinear Stein probability densities by
P K A ≔ { ρ ∈ P ∞ ( R d ) : ∫ R d ∣ x ∣ 2 ρ d x < ∞ , ∫ R d ∣ x ∣ 2 ∣ ∇ log ρ ∣ 2 ρ d x < ∞ } . (46) \mathcal{P}_{K_A}\coloneqq\left\{\rho\in\mathcal{P}^{\infty}(\mathbb{R}^d):\ \int_{\mathbb{R}^d}|x|^2\rho\,\mathrm{d}x<\infty,\quad\int_{\mathbb{R}^d}|x|^2\lvert\nabla\log\rho\rvert^2\rho\,\mathrm{d}x<\infty\right\}. \tag*{(46)} P K A : = { ρ ∈ P ∞ ( R d ) : ∫ R d ∣ x ∣ 2 ρ d x < ∞ , ∫ R d ∣ x ∣ 2 ∣ ∇ log ρ ∣ 2 ρ d x < ∞ } . (46)
First, we observe that, as P ∞ ( R d ) \mathcal{P}^{\infty}(\mathbb{R}^d) P ∞ ( R d ) , the set P K A \mathcal{P}_{K_A} P K A is convex. As we shall see, a rigorous extension of the local-mobility calculus developed in § 3.1 to the Stein setting requires two additional assumptions on the probability density: a finite second moment and the logarithmic condition.
Regular tangent and cotangent spaces. For ρ ∈ P K A \rho\in\mathcal{P}_{K_A} ρ ∈ P K A , we define the regular formal tangent space by
T ρ P K A : = { ρ ˙ ∈ C ∞ ( R d ) : ρ ˙ ρ ∈ L 2 ( ρ ) , ∫ R d ρ ˙ d x = 0 } . (47) T_\rho\mathcal{P}_{K_A}:=\left\{\dot{\rho}\in C^\infty(\mathbb{R}^d):\frac{\dot{\rho}}{\rho}\in L^2(\rho),\ \int_{\mathbb{R}^d}\dot{\rho}\,\mathrm{d}x=0\right\}. \tag*{(47)} T ρ P K A := { ρ ˙ ∈ C ∞ ( R d ) : ρ ρ ˙ ∈ L 2 ( ρ ) , ∫ R d ρ ˙ d x = 0 } . (47)
Next, we define the corresponding regular formal cotangent space by
T ρ ∗ P K A : = C ∞ ( R d ) ∩ H 1 ( ρ ) R . (48) T_\rho^*\mathcal{P}_{K_A}:=\frac{C^\infty(\mathbb{R}^d)\cap H^1(\rho)}{\mathbb{R}}. \tag*{(48)} T ρ ∗ P K A := R C ∞ ( R d ) ∩ H 1 ( ρ ) . (48)
For [ Φ ] ∈ T ρ ∗ P K A [\Phi]\in T_\rho^*\mathcal{P}_{K_A} [ Φ ] ∈ T ρ ∗ P K A and ρ ˙ ∈ T ρ P K A \dot{\rho}\in T_\rho\mathcal{P}_{K_A} ρ ˙ ∈ T ρ P K A , the usual tangent–cotangent pairing is
⟨ [ Φ ] , ρ ˙ ⟩ ρ : = ∫ R d Φ ρ ˙ d x = ∫ R d Φ ρ ˙ ρ ρ d x . \langle[\Phi],\dot{\rho}\rangle_\rho:=\int_{\mathbb{R}^d}\Phi\dot{\rho}\,\mathrm{d}x=\int_{\mathbb{R}^d}\Phi\frac{\dot{\rho}}{\rho}\,\rho\,\mathrm{d}x. ⟨[ Φ ] , ρ ˙ ⟩ ρ := ∫ R d Φ ρ ˙ d x = ∫ R d Φ ρ ρ ˙ ρ d x .
This pairing is well-defined by Cauchy–Schwarz in combination with the defining conditions of the tangent and cotangent spaces in (47) and (48) :
∣ ⟨ [ Φ ] , ρ ˙ ⟩ ρ ∣ ≤ ∥ Φ ∥ L 2 ( ρ ) ∥ ρ ˙ ρ ∥ L 2 ( ρ ) . \left|\langle[\Phi],\dot{\rho}\rangle_\rho\right|\leq\|\Phi\|_{L^2(\rho)}\left\|\frac{\dot{\rho}}{\rho}\right\|_{L^2(\rho)}. ∣ ⟨[ Φ ] , ρ ˙ ⟩ ρ ∣ ≤ ∥Φ ∥ L 2 ( ρ ) ρ ρ ˙ L 2 ( ρ ) .
Moreover, replacing Φ \Phi Φ by Φ + c \Phi+c Φ + c changes the pairing by c ∫ R d ρ ˙ d x = 0 c\int_{\mathbb{R}^d}\dot{\rho}\,\mathrm{d}x=0 c ∫ R d ρ ˙ d x = 0 .
Stein–Onsager operator. To define the analog of the local-mobility operator in this framework, we first introduce the matrix-moment operator on representatives by
M ρ : C ∞ ( R d ) ∩ H 1 ( ρ ) ⟶ R d × d , M_\rho:C^\infty(\mathbb{R}^d)\cap H^1(\rho)\longrightarrow\mathbb{R}^{d\times d}, M ρ : C ∞ ( R d ) ∩ H 1 ( ρ ) ⟶ R d × d ,
Φ ⟼ M ρ ( Φ ) : = ∫ R d x ∇ Φ ( x ) ⊤ d ρ ( x ) . (49) \Phi\longmapsto M_\rho(\Phi):=\int_{\mathbb{R}^d}x\nabla\Phi(x)^\top\,\mathrm{d}\rho(x). \tag*{(49)} Φ ⟼ M ρ ( Φ ) := ∫ R d x ∇Φ ( x ) ⊤ d ρ ( x ) . (49)
By the Cauchy–Schwarz inequality, we can check that M ρ ( Φ ) M_\rho(\Phi) M ρ ( Φ ) is well-defined for every element of its domain, and using M ρ ( Φ + c ) = M ρ ( Φ ) M_\rho(\Phi+c)=M_\rho(\Phi) M ρ ( Φ + c ) = M ρ ( Φ ) for every c ∈ R c\in\mathbb{R} c ∈ R , we see that the induced operator
M ‾ ρ : T ρ ∗ P K A ⟶ R d × d , M ‾ ρ [ Φ ] : = M ρ ( Φ ) . \overline{M}_\rho:T_\rho^*\mathcal{P}_{K_A}\longrightarrow\mathbb{R}^{d\times d},\qquad\overline{M}_\rho[\Phi]:=M_\rho(\Phi). M ρ : T ρ ∗ P K A ⟶ R d × d , M ρ [ Φ ] := M ρ ( Φ ) .
is independent of the representative. For the bilinear kernel K A K_A K A , the kernel velocity factors through exactly this matrix moment, that is,
T K A , ρ ( ∇ Φ ) ( x ) : = ∫ R d K A ( x , y ) ∇ Φ ( y ) ρ d y = M ρ ( Φ ) ⊤ A x . T_{K_A,\rho}(\nabla\Phi)(x):=\int_{\mathbb{R}^d}K_A(x,y)\nabla\Phi(y)\,\rho\,\mathrm{d}y=M_\rho(\Phi)^\top Ax. T K A , ρ ( ∇Φ ) ( x ) := ∫ R d K A ( x , y ) ∇Φ ( y ) ρ d y = M ρ ( Φ ) ⊤ A x .
Now we define, for ρ ∈ P K A \rho\in\mathcal{P}_{K_A} ρ ∈ P K A , the Stein–Onsager operator by
L ρ K A : T ρ ∗ P K A ⟶ T ρ P K A , L ρ K A [ Φ ] : = − ∇ ⋅ ( ρ M ρ ( Φ ) ⊤ A x ) . (50) L_\rho^{K_A}:T_\rho^*\mathcal{P}_{K_A}\longrightarrow T_\rho\mathcal{P}_{K_A},\qquad L_\rho^{K_A}[\Phi]:=-\nabla\cdot\left(\rho M_\rho(\Phi)^\top Ax\right). \tag*{(50)} L ρ K A : T ρ ∗ P K A ⟶ T ρ P K A , L ρ K A [ Φ ] := − ∇ ⋅ ( ρ M ρ ( Φ ) ⊤ A x ) . (50)
Next, we check that this operator is indeed well-defined and provide a useful correspondence between the matrix-moment operator and the Stein–Onsager operator.
Lemma 5.1 (Well-definedness and Green identity). For each ρ ∈ P K A \rho\in\mathcal{P}_{K_A} ρ ∈ P K A , the following holds true:
Well-definedness: The Stein–Onsager operator L ρ K A L_\rho^{K_A} L ρ K A in (50) is a well-defined linear operator.
Green identity: For all [ Φ ] , [ Ψ ] ∈ T ρ ∗ P K A [\Phi],[\Psi] \in T_\rho^* \mathcal{P}_{K_A} [ Φ ] , [ Ψ ] ∈ T ρ ∗ P K A :
⟨ [ Φ ] , L ρ K A [ Ψ ] ⟩ ρ = tr ( M ρ ( Φ ) T A M ρ ( Ψ ) ) . (51) \langle[\Phi],L_\rho^{K_A}[\Psi]\rangle_\rho=\operatorname{tr}\left(M_\rho(\Phi)^{\mathsf{T}}A M_\rho(\Psi)\right). \tag*{(51)} ⟨[ Φ ] , L ρ K A [ Ψ ] ⟩ ρ = tr ( M ρ ( Φ ) T A M ρ ( Ψ ) ) . (51)
Next, we observe a feature absent in the local-mobility geometry framework in § 3.1 : a nonzero kernel of the operator L ρ K A L_\rho^{K_A} L ρ K A .
Lemma 5.2 (Kernel of the Stein–Onsager operator). For every ρ ∈ P K A \rho\in\mathcal{P}_{K_A} ρ ∈ P K A , the kernel of the Stein–Onsager operator L ρ K A L_\rho^{K_A} L ρ K A in (50) is precisely the space of zero-moment cotangent classes, that is,
ker L ρ K A = { [ Φ ] ∈ T ρ ∗ P K A : M ρ ( Φ ) = 0 } . \operatorname{ker}L_\rho^{K_A}=\{[\Phi]\in T_\rho^*\mathcal{P}_{K_A}:M_\rho(\Phi)=0\}. ker L ρ K A = {[ Φ ] ∈ T ρ ∗ P K A : M ρ ( Φ ) = 0 } . In particular, ker L ρ K A \operatorname{ker}L_\rho^{K_A} ker L ρ K A is infinite-dimensional.
We have seen in § 3.1 that an essential ingredient for making the local-mobility calculus rigorous is ensuring that the local-mobility operator is an isomorphism from the cotangent space to the tangent space. By Lemma 5.2, this cannot hold for the Stein geometry corresponding to the bilinear kernel. To work around this issue, we restrict ourselves to smaller subspaces of the tangent and cotangent spaces:
T ρ e f f P K A : = ran L ρ K A ⊂ T ρ P K A , T ρ ∗ , e f f P K A : = T ρ ∗ P K A ker L ρ K A . (52) T_\rho^{\mathrm{eff}}\mathcal{P}_{K_A}:=\operatorname{ran}L_\rho^{K_A}\subset T_\rho\mathcal{P}_{K_A},\qquad T_\rho^{*,\mathrm{eff}}\mathcal{P}_{K_A}:=\frac{T_\rho^*\mathcal{P}_{K_A}}{\operatorname{ker}L_\rho^{K_A}}. \tag*{(52)} T ρ eff P K A := ran L ρ K A ⊂ T ρ P K A , T ρ ∗ , eff P K A := ker L ρ K A T ρ ∗ P K A . (52)
Before moving on, we first want to point out that the dimension of T ρ e f f T_\rho^{\mathrm{eff}} T ρ eff varies with ρ \rho ρ . Furthermore, we comment on (52) through the lens of geometry: T ρ e f f T_\rho^{\mathrm{eff}} T ρ eff contains exactly the directions generated by the bilinear Stein kernel, while T ρ ∗ , e f f T_\rho^{*,\mathrm{eff}} T ρ ∗ , eff identifies precisely those regular cotangent classes that generate the same direction. Next, we see that on these spaces the Stein–Onsager operator induces an isomorphism.
Theorem 5.3 (Effective Stein–Onsager isomorphism). For all ρ ∈ P K A \rho\in\mathcal{P}_{K_A} ρ ∈ P K A , the operator L ρ K A L_\rho^{K_A} L ρ K A in (50) induces a linear isomorphism
L ‾ ρ K A : T ρ ∗ , e f f P K A ⟶ T ρ e f f P K A , L ‾ ρ K A ( [ [ Φ ] ] ) : = L ρ K A [ Φ ] . (53) \overline{L}_\rho^{K_A}:T_\rho^{*,\mathrm{eff}}\mathcal{P}_{K_A}\longrightarrow T_\rho^{\mathrm{eff}}\mathcal{P}_{K_A},\qquad\overline{L}_\rho^{K_A}([[ \Phi]]):=L_\rho^{K_A}[\Phi]. \tag*{(53)} L ρ K A : T ρ ∗ , eff P K A ⟶ T ρ eff P K A , L ρ K A ([[ Φ ]]) := L ρ K A [ Φ ] . (53) Moreover, T ρ ∗ , e f f ≅ ran M ρ T_\rho^{*,\mathrm{eff}}\cong\operatorname{ran}M_\rho T ρ ∗ , eff ≅ ran M ρ via [ [ Φ ] ] ↦ M ρ ( Φ ) [[\Phi]]\mapsto M_\rho(\Phi) [[ Φ ]] ↦ M ρ ( Φ ) .
Stein metric for the bilinear kernel and Stein gradients. Next, as in the local-mobility framework, we use Theorem 5.3 to define the Stein metric associated with the bilinear kernel and show it is well-defined. First, for each ρ ∈ P K A \rho\in\mathcal{P}_{K_A} ρ ∈ P K A , we define the effective Stein metric by
G ρ K A : T ρ e f f P K A × T ρ e f f P K A ⟶ R , G ρ K A ( ρ ˙ , ρ ˙ ) : = ∫ R d Φ ρ ˙ ρ ˙ d x . (54) G_\rho^{K_A}:T_\rho^{\mathrm{eff}}\mathcal{P}_{K_A}\times T_\rho^{\mathrm{eff}}\mathcal{P}_{K_A}\longrightarrow\mathbb{R},\qquad G_\rho^{K_A}(\dot{\rho},\dot{\rho}):=\int_{\mathbb{R}^d}\Phi_{\dot{\rho}}\dot{\rho}\,\mathrm{d}x. \tag*{(54)} G ρ K A : T ρ eff P K A × T ρ eff P K A ⟶ R , G ρ K A ( ρ ˙ , ρ ˙ ) := ∫ R d Φ ρ ˙ ρ ˙ d x . (54)
where the unique effective cotangent class [ [ Φ ρ ˙ ] ] ∈ T ρ ∗ , e f f P K A [[\Phi_{\dot{\rho}}]]\in T_\rho^{*,\mathrm{eff}}\mathcal{P}_{K_A} [[ Φ ρ ˙ ]] ∈ T ρ ∗ , eff P K A is determined by
ρ ˙ + ∇ ⋅ ( ρ M ρ ( Φ ρ ˙ ) T A x ) = 0. \dot{\rho}+\nabla\cdot\left(\rho M_\rho(\Phi_{\dot{\rho}})^{\mathsf{T}}Ax\right)=0. ρ ˙ + ∇ ⋅ ( ρ M ρ ( Φ ρ ˙ ) T A x ) = 0.
Next, we obtain the following corollary as a direct consequence of Lemma 5.1, Lemma 5.2 and Theorem 5.3.
Corollary 5.4 (Expression for the Stein metric). For all ρ ∈ P K A \rho\in\mathcal{P}_{K_A} ρ ∈ P K A , the following holds true:
The metric G ρ K A G_{\rho}^{K_A} G ρ K A in (54) is a well-defined, positive-definite quadratic form on T ρ e f f P K A T_{\rho}^{\mathrm{eff}}\mathcal{P}_{K_A} T ρ eff P K A .
For all [ Φ ] ∈ T ρ ∗ P K A [\Phi] \in T_{\rho}^{*}\mathcal{P}_{K_A} [ Φ ] ∈ T ρ ∗ P K A , we have
G ρ K A ( L ρ K A [ Φ ] , L ρ K A [ Φ ] ) = tr ( M ρ ( Φ ) T A M ρ ( Φ ) ) . (55) G_{\rho}^{K_A}\left(L_{\rho}^{K_A}[\Phi],L_{\rho}^{K_A}[\Phi]\right)=\operatorname{tr}\left(M_{\rho}(\Phi)^{\mathsf{T}} A M_{\rho}(\Phi)\right). \tag*{(55)} G ρ K A ( L ρ K A [ Φ ] , L ρ K A [ Φ ] ) = tr ( M ρ ( Φ ) T A M ρ ( Φ ) ) . (55)
We want to stress that P K A \mathcal{P}_{K_A} P K A has not been equipped with a differentiable structure, so the notion of differentiability requires a separate definition. In Appendix C, we provide such a definition in terms of local regular first variations and their associated regular extensions. The reader may think of ordinary differentiation with a regular first-variation representative in what follows, while D F ( ρ ) D F(\rho) D F ( ρ ) denotes the associated regular extension defined in the appendix.
Finally, following the argument of Corollary 4.6, we provide a criterion for the gradient of a functional with respect to the metric in (55) to be a regular tangent vector.
Corollary 5.5 (The Stein gradient). Let F : P K A ⟶ R \mathcal{F}:\mathcal{P}_{K_A}\longrightarrow\mathbb{R} F : P K A ⟶ R have a first variation satisfying
D F ( ρ ) [ η ˙ ] = ∫ R d δ F δ ρ η ˙ d x for all η ˙ ∈ T ρ e f f P K A , D\mathcal{F}(\rho)[\dot{\eta}]=\int_{\mathbb{R}^{d}}\frac{\delta\mathcal{F}}{\delta\rho}\dot{\eta}\,\mathrm{d}x \qquad\text{for all }\dot{\eta}\in T_{\rho}^{\mathrm{eff}}\mathcal{P}_{K_A}, D F ( ρ ) [ η ˙ ] = ∫ R d δ ρ δ F η ˙ d x for all η ˙ ∈ T ρ eff P K A , and so that [ [ δ F δ ρ ] ] ∈ T ρ ∗ , e f f P K A \left[\left[\frac{\delta\mathcal{F}}{\delta\rho}\right]\right]\in T_{\rho}^{*,\mathrm{eff}}\mathcal{P}_{K_A} [ [ δ ρ δ F ] ] ∈ T ρ ∗ , eff P K A . Then, the gradient with respect to the Stein metric G K A G^{K_A} G K A in (55) is given by
grad G K A F ( ρ ) = L ρ K A [ δ F δ ρ ] = x ⟼ − ∇ ⋅ ( ρ M ρ ( δ F δ ρ ) T A x ) ∈ T ρ e f f P K A . \operatorname{grad}_{G^{K_A}}\mathcal{F}(\rho)=L_{\rho}^{K_A}\left[\frac{\delta\mathcal{F}}{\delta\rho}\right]=x\longmapsto-\nabla\cdot\left(\rho M_{\rho}\left(\frac{\delta\mathcal{F}}{\delta\rho}\right)^{\mathsf{T}}Ax\right)\in T_{\rho}^{\mathrm{eff}}\mathcal{P}_{K_A}. grad G K A F ( ρ ) = L ρ K A [ δ ρ δ F ] = x ⟼ − ∇ ⋅ ( ρ M ρ ( δ ρ δ F ) T A x ) ∈ T ρ eff P K A .
III Geometry and Dynamics of Attention
Softmax Attention: Geometry and Particle Dynamics
This section aims to provide a rigorous approach to softmax attention.
In Subsection 6.1, we give a gradient-flow interpretation of the transformer PDE. We characterize the densities at which the interaction energy admits a regular first variation in Proposition 6.2. Corollary 6.4 then identifies the PDE as the corresponding regular gradient equation. Example 6.1 illustrates why pointwise finiteness of the attention velocity alone does not suffice for this interpretation. The resulting gradient identity holds at each admissible density.
Then, in Subsection 6.2 we identify conditions under which the accelerated softmax system is pointwise well-defined. We then restrict the domain further to ensure that its right-hand sides arise from regular derivatives of the Hamiltonian, giving the PDE a rigorous interpretation as a damped Hamiltonian system. We also provide explicit examples of states satisfying the conditions for the pointwise formulation and the stronger conditions for the regular Hamiltonian formulation.
Finally, Subsection 6.3 uses the softmax geometry to construct the attention updates considered in Section 9. Proofs are given in Appendix F.
Figure 4 summarizes the geometric interpretations studied in this section and their relation to the density and particle dynamics.
Figure 4. From pointwise softmax equations to regular geometric formulations. The arrows indicate additional conditions on the density and potential under which the equations admit a gradient or Hamiltonian interpretation. These are statements at admissible states; existence of trajectories and preservation of admissibility require a separate analysis. The finite-token geometry is constructed directly and provides explicit corrections for attention layers.
Geometry and First-Order Softmax Dynamics
Let A = A T A = A^{\mathsf{T}} A = A T be invertible and B = V A − 1 = B T ≻ 0 B = VA^{-1} = B^{\mathsf{T}} \succ0 B = V A − 1 = B T ≻ 0 as in (38) . We use the density class P S M \mathcal{P}_{\mathrm{SM}} P SM in (41) , the regular tangent and cotangent spaces in (42) and (44) , and the metric G ρ S M \mathcal{G}^{\mathrm{SM}}_{\rho} G ρ SM in (45) . The normalizer Z ρ Z_{\rho} Z ρ and mobility m ρ = ρ / Z ρ m_{\rho} = \rho/Z_{\rho} m ρ = ρ / Z ρ are defined in (39) .
We begin by recalling the transformer PDE (5) for softmax attention
∂ t ρ t + ∇ ⋅ ( ρ t Γ ρ t ) = 0 , Γ ρ ( x ) : = 1 Z ρ ( x ) ∫ R d V y exp ( y T A x ) ρ ( y ) d y (56) \partial_t \rho_t + \nabla\cdot\left(\rho_t \Gamma_{\rho_t}\right) = 0, \qquad\Gamma_{\rho}(x) := \frac{1}{Z_{\rho}(x)} \int_{\mathbb{R}^{d}} Vy \exp\left(y^{\mathsf{T}}Ax\right)\rho(y)\,\mathrm{d}y \tag*{(56)} ∂ t ρ t + ∇ ⋅ ( ρ t Γ ρ t ) = 0 , Γ ρ ( x ) := Z ρ ( x ) 1 ∫ R d V y exp ( y T A x ) ρ ( y ) d y (56)
with normalizing factor
Z ρ ( x ) : = ∫ R d exp ( y T A x ) ρ ( y ) d y . (57) Z_{\rho}(x) := \int_{\mathbb{R}^{d}} \exp\left(y^{\mathsf{T}}Ax\right)\rho(y)\,\mathrm{d}y. \tag*{(57)} Z ρ ( x ) := ∫ R d exp ( y T A x ) ρ ( y ) d y . (57)
In [16 Sec. 5.4] , the authors give a formal gradient-flow interpretation of this PDE and leave its rigorous justification on non-compact domains open. This subsection aims to address this issue within the local-mobility framework of § 3.1 .
Gradient-flow structure of the softmax transformer PDE. Recall that a Riemannian gradient flow is specified by a metric G G G and a functional F \mathcal{F} F whose gradient with respect to G G G exists.
Here, we use the softmax–Wasserstein metric G ρ S M G^{\mathrm{SM}}_{\rho} G ρ SM in (45) , together with the interaction energy
F S M ( ρ ) : = − 1 2 ∫ R d ρ ( x ) Z ρ ( x ) d x . (58) \mathcal{F}^{\mathrm{SM}}(\rho) := -\frac{1}{2}\int_{\mathbb{R}^{d}} \rho(x)Z_{\rho}(x)\,\mathrm{d}x. \tag*{(58)} F SM ( ρ ) := − 2 1 ∫ R d ρ ( x ) Z ρ ( x ) d x . (58)
A formal computation as in [16 Sec. 5.4] identifies (56) as the gradient-flow equation of F \mathcal{F} F with respect to G ρ S M G^{\mathrm{SM}}_{\rho} G ρ SM . However, F S M \mathcal{F}^{\mathrm{SM}} F SM is not finite on all of P S M \mathcal{P}_{\mathrm{SM}} P SM , defined in (41) , as the following example shows.
Example 6.1 (Infinite energy at an admissible density). Let d = 1 d=1 d = 1 , A = B = 1 A=B=1 A = B = 1 , and ρ = N ( 0 , 2 ) \rho=\mathcal{N}(0,2) ρ = N ( 0 , 2 ) . Then
Z ρ ( x ) = exp ( x 2 ) , m ρ ( x ) = ρ ( x ) Z ρ ( x ) = 1 4 π exp ( − 5 x 2 / 4 ) . Z_{\rho}(x)=\exp(x^{2}), \qquad m_{\rho}(x)=\frac{\rho(x)}{Z_{\rho}(x)}=\frac{1}{\sqrt{4\pi}}\exp(-5x^{2}/4). Z ρ ( x ) = exp ( x 2 ) , m ρ ( x ) = Z ρ ( x ) ρ ( x ) = 4 π 1 exp ( − 5 x 2 /4 ) . Since m ρ m_{\rho} m ρ is a finite Gaussian weight, it satisfies the weighted Poincaré inequality (40) . Thus ρ ∈ P S M \rho\in\mathcal{P}_{\mathrm{SM}} ρ ∈ P SM . Nevertheless, we have
∫ R ρ ( x ) Z ρ ( x ) d x = 1 4 π ∫ R exp ( 3 x 2 / 4 ) d x = ∞ , \int_{\mathbb{R}}\rho(x)Z_{\rho}(x)\,\mathrm{d}x=\frac{1}{\sqrt{4\pi}}\int_{\mathbb{R}}\exp(3x^{2}/4)\,\mathrm{d}x=\infty, ∫ R ρ ( x ) Z ρ ( x ) d x = 4 π 1 ∫ R exp ( 3 x 2 /4 ) d x = ∞ , so that the interaction energy is not finite at $\rho.
To resolve this obstruction, we restrict F S M \mathcal{F}^{\mathrm{SM}} F SM to its finite-energy domain. We must then identify its domain of differentiability, that is, the densities at which F S M \mathcal{F}^{\mathrm{SM}} F SM admits a local regular first variation in the sense of (117) . The following proposition characterizes this domain.
Proposition 6.2 (Regular variation criterion). Let ρ ∈ P S M \rho\in\mathcal{P}_{\mathrm{SM}} ρ ∈ P SM satisfy F S M ( ρ ) ∈ R \mathcal{F}^{\mathrm{SM}}(\rho)\in\mathbb{R} F SM ( ρ ) ∈ R , where F S M \mathcal{F}^{\mathrm{SM}} F SM is defined in (58) . Then F S M \mathcal{F}^{\mathrm{SM}} F SM admits a local regular first variation at ρ \rho ρ in the sense of (117) if and only if
∫ R d Z ρ ρ ∣ L ρ S M Z ρ ∣ 2 d x < ∞ . (59) \int_{\mathbb{R}^{d}}\frac{Z_{\rho}}{\rho}\left|L^{\mathrm{SM}}_{\rho}Z_{\rho}\right|^{2}\,\mathrm{d}x<\infty. \tag*{(59)} ∫ R d ρ Z ρ L ρ SM Z ρ 2 d x < ∞. (59)
Before moving on, we comment briefly on (59) from three perspectives.
Proposition 6.2 allows us to justify the formal gradient computation in the following corollary.
Corollary 6.4 (Regular softmax gradient). Let ρ \rho ρ be as in Proposition 6.2 and satisfy (59). Then the following hold true:
(Gradient identity) The negative gradient of F S M \mathcal{F}^{\mathrm{SM}} F SM at ρ \rho ρ is
− grad ρ S M F S M ( ρ ) = L ρ S M [ Z ρ ] = − ∇ ⋅ ( ρ Γ ρ ) in T ρ P S M . (60) -\operatorname{grad}_{\rho}^{\mathrm{SM}}\mathcal{F}^{\mathrm{SM}}(\rho)=L_{\rho}^{\mathrm{SM}}[Z_{\rho}]=-\nabla\cdot(\rho\Gamma_{\rho})\qquad\text{in }T_{\rho}\mathcal{P}_{\mathrm{SM}}. \tag*{(60)} − grad ρ SM F SM ( ρ ) = L ρ SM [ Z ρ ] = − ∇ ⋅ ( ρ Γ ρ ) in T ρ P SM . (60) (Gradient-flow formulation) On the domain of densities satisfying the above assumptions, the transformer PDE (56) is the regular gradient-flow equation associated with ( G S M , F S M ) (G^{\mathrm{SM}},\mathcal{F}^{\mathrm{SM}}) ( G SM , F SM ) :
∂ t ρ t = − grad ρ t S M F S M ( ρ t ) . \partial_t\rho_t=-\operatorname{grad}_{\rho_t}^{\mathrm{SM}}\mathcal{F}^{\mathrm{SM}}(\rho_t). ∂ t ρ t = − grad ρ t SM F SM ( ρ t ) .
Corollary 6.4 identifies the gradient at each admissible density. In this sense, it provides a rigorous justification of the formal gradient-flow interpretation in [16 Sec. 5.4] on the domain specified above.
We close this subsection by emphasizing that constructing gradient-flow trajectories additionally requires local existence and short-time preservation of the domain on which (60) holds. A solution theory for general admissible initial densities is left for future research. For more restricted classes of initial densities, such as perturbations of Gaussian densities, partial results can be obtained concerning local well-posedness, preservation of the gradient domain, and loss of finite energy. We leave these developments to separate work.
Accelerated Softmax Dynamics and Hamiltonian Structure
We begin with the accelerated softmax system introduced in (17) ,
{ ∂ t ρ t = − ∇ ⋅ ( ρ t Z ρ t B ∇ Φ t ) , ∂ t Φ t ( x ) + α t Φ t ( x ) = Z ρ t ( x ) − ∇ Φ t ( x ) T B ∇ Φ t ( x ) 2 Z ρ t ( x ) + 1 2 ∫ R d exp ( y T A x ) ∇ Φ t ( y ) T B ∇ Φ t ( y ) Z ρ t ( y ) 2 ρ t ( y ) d y . (61) \begin{cases}
\partial_t\rho_t=-\nabla\cdot\left(\dfrac{\rho_t}{Z_{\rho_t}}B\nabla\Phi_t\right),\\
\partial_t\Phi_t(x)+\alpha_t\Phi_t(x)=Z_{\rho_t}(x)-\dfrac{\nabla\Phi_t(x)^{\mathsf{T}}B\nabla\Phi_t(x)}{2Z_{\rho_t}(x)}+\dfrac{1}{2}\int_{\mathbb{R}^d}\exp(y^{\mathsf{T}}Ax)\dfrac{\nabla\Phi_t(y)^{\mathsf{T}}B\nabla\Phi_t(y)}{Z_{\rho_t}(y)^2}\rho_t(y)\,\mathrm{d}y. \tag*{(61)}
\end{cases} ⎩ ⎨ ⎧ ∂ t ρ t = − ∇ ⋅ ( Z ρ t ρ t B ∇ Φ t ) , ∂ t Φ t ( x ) + α t Φ t ( x ) = Z ρ t ( x ) − 2 Z ρ t ( x ) ∇ Φ t ( x ) T B ∇ Φ t ( x ) + 2 1 ∫ R d exp ( y T A x ) Z ρ t ( y ) 2 ∇ Φ t ( y ) T B ∇ Φ t ( y ) ρ t ( y ) d y . (61)
where we used, if the differential of F \mathcal{F} F is defined, that δ F δ ρ = − Z ρ \dfrac{\delta\mathcal{F}}{\delta\rho}=-Z_{\rho} δ ρ δ F = − Z ρ for the interaction energy F \mathcal{F} F in (58) , see Remark 6.8. Before moving on, we point out that the second equation in (61) is understood modulo spatial constants. This means that its two sides may differ by a term depending only on time. This term can be removed by choosing a suitable additive function of time in Φ t \Phi_t Φ t , which leaves ∇ Φ t \nabla\Phi_t ∇ Φ t and hence the density evolution unchanged.
The aim of this subsection is first to identify conditions under which (61) is pointwise well-defined. We then restrict the domain further to ensure that its right-hand sides arise from regular derivatives of the Hamiltonian (63) , giving the PDE a rigorous interpretation as a damped Hamiltonian system. We also provide explicit examples of states satisfying the conditions for the pointwise formulation and the stronger conditions for the regular Hamiltonian formulation.
Pointwise well-definedness. We first point out that the defining conditions for P S M \mathcal{P}_{\mathrm{SM}} P SM in (41) and the regular cotangent space T ρ ∗ P S M T_{\rho}^{*}\mathcal{P}_{\mathrm{SM}} T ρ ∗ P SM in (44) ensure that (61) is pointwise well-defined at every pair ( ρ , [ Φ ] ) (\rho,[\Phi]) ( ρ , [ Φ ]) with ρ ∈ P S M \rho\in\mathcal{P}_{\mathrm{SM}} ρ ∈ P SM and [ Φ ] ∈ T ρ ∗ P S M [\Phi]\in T_{\rho}^{*}\mathcal{P}_{\mathrm{SM}} [ Φ ] ∈ T ρ ∗ P SM .
Lemma 6.5 (Pointwise formulation). Let ρ ∈ P S M \rho\in\mathcal{P}_{\mathrm{SM}} ρ ∈ P SM and [ Φ ] ∈ T ρ ∗ P S M [\Phi] \in T_\rho^*\mathcal{P}_{\mathrm{SM}} [ Φ ] ∈ T ρ ∗ P SM with the cotangent space defined in (44) . Then, all spatial quantities in (61) are pointwise well-defined at ( ρ , Φ ) (\rho,\Phi) ( ρ , Φ ) , whenever α ( t ) \alpha(t) α ( t ) is finite.
Note that finiteness of the interaction energy F S M ( ρ ) \mathcal{F}^{\mathrm{SM}}(\rho) F SM ( ρ ) is not required for this pointwise formulation. Further, it is straightforward to verify that (61) is also pointwise well-defined for the following family of densities and potentials.
Example 6.6 (Perturbations of Gaussian densities). Let γ = N ( μ , Σ ) \gamma= \mathcal{N}(\mu,\Sigma) γ = N ( μ , Σ ) with Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 and
ρ 0 = γ ( 1 + u ) , u ∈ C ∞ ( R d ) , ∫ R d γ u d x = 0 , ∥ u ∥ ∞ < 1. (62) \rho_0 = \gamma(1+u), \qquad u \in C^\infty(\mathbb{R}^d), \qquad\int_{\mathbb{R}^d} \gamma u\,\mathrm{d}x = 0, \qquad\lVert u\rVert_\infty< 1. \tag*{(62)} ρ 0 = γ ( 1 + u ) , u ∈ C ∞ ( R d ) , ∫ R d γ u d x = 0 , ∥ u ∥ ∞ < 1. (62) Let Φ 0 ∈ C ∞ ( R d ) \Phi_0 \in C^\infty(\mathbb{R}^d) Φ 0 ∈ C ∞ ( R d ) have a gradient of at most polynomial growth, that is,
∥ ∇ Φ 0 ( x ) ∥ ≤ C ( 1 + ∣ x ∣ ) k ( x ∈ R d ) \lVert\nabla\Phi_0(x)\rVert\le C(1+\lvert x\rvert)^k \qquad(x \in\mathbb{R}^d) ∥ ∇ Φ 0 ( x )∥ ≤ C ( 1 + ∣ x ∣ ) k ( x ∈ R d ) for some finite C , k ≥ 0 C,k \ge0 C , k ≥ 0 . Then all spatial quantities in (61) are pointwise well-defined at ( ρ 0 , Φ 0 ) (\rho_0,\Phi_0) ( ρ 0 , Φ 0 ) whenever α ( 0 ) \alpha(0) α ( 0 ) is finite.
Hamiltonian and formal equations. As for gradient flows on a manifold M \mathcal{M} M , the construction of a damped Hamiltonian system starts from a Riemannian metric and a functional F S M \mathcal{F}^{\mathrm{SM}} F SM , together with a damping coefficient. Here, we use the softmax–Wasserstein metric G ρ S M G_\rho^{\mathrm{SM}} G ρ SM in (45) and the interaction energy F S M \mathcal{F}^{\mathrm{SM}} F SM in (58) . For ρ ∈ P S M \rho\in\mathcal{P}_{\mathrm{SM}} ρ ∈ P SM and [ Φ ] ∈ T ρ ∗ P S M [\Phi] \in T_\rho^*\mathcal{P}_{\mathrm{SM}} [ Φ ] ∈ T ρ ∗ P SM , we define
H ( ρ , [ Φ ] ) : = 1 2 G ρ S M ( L ρ S M [ Φ ] , L ρ S M [ Φ ] ) + F S M ( ρ ) = 1 2 ∫ R d ρ Z ρ ∇ Φ T B ∇ Φ d x + F S M ( ρ ) . (63) \mathcal{H}(\rho,[\Phi]) := \frac{1}{2}G_\rho^{\mathrm{SM}}\left(L_\rho^{\mathrm{SM}}[\Phi],L_\rho^{\mathrm{SM}}[\Phi]\right)+\mathcal{F}^{\mathrm{SM}}(\rho)=\frac{1}{2}\int_{\mathbb{R}^d}\frac{\rho}{Z_\rho}\nabla\Phi^{\mathsf{T}}B\nabla\Phi\,\mathrm{d}x+\mathcal{F}^{\mathrm{SM}}(\rho). \tag*{(63)} H ( ρ , [ Φ ]) := 2 1 G ρ SM ( L ρ SM [ Φ ] , L ρ SM [ Φ ] ) + F SM ( ρ ) = 2 1 ∫ R d Z ρ ρ ∇ Φ T B ∇Φ d x + F SM ( ρ ) . (63)
The kinetic term in (63) is finite by the definition of T ρ ∗ P S M T_\rho^*\mathcal{P}_{\mathrm{SM}} T ρ ∗ P SM in (44) . Consequently, H ( ρ , [ Φ ] ) \mathcal{H}(\rho,[\Phi]) H ( ρ , [ Φ ]) is finite if and only if F S M ( ρ ) \mathcal{F}^{\mathrm{SM}}(\rho) F SM ( ρ ) is finite. As Example 6.1 shows, this is not guaranteed by ρ ∈ P S M \rho\in\mathcal{P}_{\mathrm{SM}} ρ ∈ P SM alone.
For finite-energy densities, Proposition 6.2 already characterizes when F S M \mathcal{F}^{\mathrm{SM}} F SM admits a local regular first variation. Next, we give sufficient conditions for the regular variations of H \mathcal{H} H with respect to Φ \Phi Φ and ρ \rho ρ , understood in the sense of (122) and (123) , respectively.
Theorem 6.7 (Regular Hamiltonian formulation). Let ρ \rho ρ satisfy the assumptions of Corollary 6.4, and let [ Φ ] ∈ T ρ ∗ P S M [\Phi] \in T_\rho^*\mathcal{P}_{\mathrm{SM}} [ Φ ] ∈ T ρ ∗ P SM . Further, assume the following:
The following cotangent condition holds:
[ ∇ Φ ( x ) T B ∇ Φ ( x ) 2 Z ρ ( x ) − 1 2 ∫ R d exp ( y T A x ) ∇ Φ ( y ) T B ∇ Φ ( y ) Z ρ ( y ) 2 ρ ( y ) d y ] ∈ T ρ ∗ P S M . (64) \left[
\frac{\nabla\Phi(x)^{\mathsf{T}}B\nabla\Phi(x)}{2Z_\rho(x)}
-\frac{1}{2}\int_{\mathbb{R}^d}\exp(y^{\mathsf{T}}Ax)\frac{\nabla\Phi(y)^{\mathsf{T}}B\nabla\Phi(y)}{Z_\rho(y)^2}\rho(y)\,\mathrm{d}y
\right] \in T_\rho^*\mathcal{P}_{\mathrm{SM}}. \tag*{(64)} [ 2 Z ρ ( x ) ∇Φ ( x ) T B ∇Φ ( x ) − 2 1 ∫ R d exp ( y T A x ) Z ρ ( y ) 2 ∇Φ ( y ) T B ∇Φ ( y ) ρ ( y ) d y ] ∈ T ρ ∗ P SM . (64) For every η ∈ C c ∞ ( R d ) \eta\in C^\infty_{\mathrm{c}}(\mathbb{R}^d) η ∈ C c ∞ ( R d ) with ∫ R d η d x = 0 \int_{\mathbb{R}^d}\eta\,\mathrm{d}x=0 ∫ R d η d x = 0 , we have
( ∇ Z η Z ρ ) T B ∇ Φ ∈ L 2 ( m ρ ) . (65) \left(\frac{\nabla Z_\eta}{Z_\rho}\right)^{\mathsf{T}}B\nabla\Phi\in L^2(m_\rho). \tag*{(65)} ( Z ρ ∇ Z η ) T B ∇Φ ∈ L 2 ( m ρ ) . (65)
Then (61) can be written for the Hamiltonian H \mathcal{H} H in (63) as a regular damped Hamiltonian system
{ ∂ t ρ t = δ r e g H δ Φ ( ρ t , [ Φ t ] ) , [ ∂ t Φ t ] + α t [ Φ t ] = − δ r e g H δ ρ ( ρ t , [ Φ t ] ) . (66) \begin{cases}
\partial_t \rho_t = \dfrac{\delta^{\mathrm{reg}}\mathcal{H}}{\delta\Phi}(\rho_t,[\Phi_t]),\\
[\partial_t\Phi_t] + \alpha_t[\Phi_t] = -\dfrac{\delta^{\mathrm{reg}}\mathcal{H}}{\delta\rho}(\rho_t,[\Phi_t]). \tag*{(66)}
\end{cases} ⎩ ⎨ ⎧ ∂ t ρ t = δ Φ δ reg H ( ρ t , [ Φ t ]) , [ ∂ t Φ t ] + α t [ Φ t ] = − δ ρ δ reg H ( ρ t , [ Φ t ]) . (66)
Let us examine the above assumptions.
Next, we further illustrate the assumptions of Theorem 6.7 and how they go beyond those of Corollary 6.4. The first example provides density–potential pairs satisfying the regular Hamiltonian conditions. The second shows that a density can satisfy the first-order gradient criterion while an associated regular cotangent potential fails the additional Hamiltonian conditions. We leave the computations verifying these examples to the reader.
Example 6.9 (Gaussian densities and their perturbations). Let γ = N ( μ , Σ ) \gamma= \mathcal{N}(\mu,\Sigma) γ = N ( μ , Σ ) with Σ ≻ 0 \Sigma\succ0 Σ ≻ 0 satisfy
Σ − 1 − A Σ A ≻ 0 , \Sigma^{-1} - A\Sigma A \succ0, Σ − 1 − A Σ A ≻ 0 , and let Φ 0 ∈ C ∞ ( R d ) \Phi_0 \in C^\infty(\mathbb{R}^d) Φ 0 ∈ C ∞ ( R d ) have compactly supported gradient. Then the following pairs satisfy the assumptions of Theorem 6.7:
(Gaussian densities) The pair ( γ , [ Φ 0 ] ) (\gamma,[\Phi_0]) ( γ , [ Φ 0 ]) .
(Non-Gaussian perturbations) The pair ( ρ 0 , [ Φ 0 ] ) (\rho_0,[\Phi_0]) ( ρ 0 , [ Φ 0 ]) , where ρ 0 = γ ( 1 + u ) \rho_0 = \gamma(1+u) ρ 0 = γ ( 1 + u ) satisfies (62) and ∥ ∇ u ∥ ∞ < ∞ \lVert\nabla u\rVert_\infty< \infty ∥ ∇ u ∥ ∞ < ∞ .
The choice Φ 0 = 0 \Phi_0 = 0 Φ 0 = 0 is included, and no amplitude restriction on Φ 0 \Phi_0 Φ 0 is required.
Finally, we give an example where Corollary 6.4 applies to the density, but the density–potential pair fails the additional conditions of Theorem 6.7.
Example 6.10 (Failure of regular Hamiltonian admissibility). Let d = 1 d = 1 d = 1 , A = B = 1 A = B = 1 A = B = 1 , and choose
ρ = N ( 0 , 1 2 ) , Φ ( x ) = exp ( x 2 / 2 ) . \rho= \mathcal{N}\left(0,\frac{1}{2}\right), \qquad\Phi(x) = \exp\left(x^2/2\right). ρ = N ( 0 , 2 1 ) , Φ ( x ) = exp ( x 2 /2 ) . :::`
Then
Z ρ ( x ) = exp ( x 2 / 4 ) , m ρ ( x ) = 1 π exp ( − 5 x 2 / 4 ) . Z_{\rho}(x)=\exp(x^{2}/4),\qquad m_{\rho}(x)=\frac{1}{\sqrt{\pi}}\exp(-5x^{2}/4). Z ρ ( x ) = exp ( x 2 /4 ) , m ρ ( x ) = π 1 exp ( − 5 x 2 /4 ) .
One can check that ρ \rho ρ satisfies the assumptions of Corollary 6.4 , that [ Φ ] ∈ T ρ ∗ P S M [\Phi]\in T_{\rho}^{*}\mathrm{PSM} [ Φ ] ∈ T ρ ∗ PSM , and that (65) holds.
However, a direct computation shows that the expression inside the brackets in (64) equals
x 2 2 exp ( 3 x 2 / 4 ) − 1 + x 2 2 exp ( x 2 / 2 ) . \frac{x^{2}}{2}\exp(3x^{2}/4)-\frac{1+x^{2}}{\sqrt{2}}\exp(x^{2}/2). 2 x 2 exp ( 3 x 2 /4 ) − 2 1 + x 2 exp ( x 2 /2 ) .
Its square is not integrable against m ρ m_{\rho} m ρ . Thus, (64) fails, although Corollary 6.4 applies to ρ \rho ρ and the accelerated PDE is pointwise well-defined at ( ρ , Φ ) (\rho,\Phi) ( ρ , Φ ) .
From Softmax Geometry to Attention Layers
Now we use the softmax geometry to construct the attention updates considered in Section 9. We first write the inertial particle dynamics in matrix coordinates. Then we compute the geometric correction and explain how it enters the position and velocity updates.
Inertial particle dynamics. We keep A = A T A=A^{\mathsf T} A = A T invertible and B = V A − 1 = B T ≻ 0 B=VA^{-1}=B^{\mathsf T}\succ0 B = V A − 1 = B T ≻ 0 fixed. Let X ∈ R N × d X\in\mathbb{R}^{N\times d} X ∈ R N × d contain the token positions, with rows X i T X_i^{\mathsf T} X i T , and define
M i j ( X ) : = exp ( X i T A X j ) , z i ( X ) : = ∑ j = 1 N M i j ( X ) . (67) M_{ij}(X):=\exp(X_i^{\mathsf T}AX_j),\qquad z_i(X):=\sum_{j=1}^{N}M_{ij}(X). \tag*{(67)} M ij ( X ) := exp ( X i T A X j ) , z i ( X ) := j = 1 ∑ N M ij ( X ) . (67)
We write D z = diag ( z 1 , … , z N ) D_z=\operatorname{diag}(z_1,\ldots,z_N) D z = diag ( z 1 , … , z N ) and denote by 1 ∈ R N \mathbf{1}\in\mathbb{R}^{N} 1 ∈ R N the vector whose entries are all one. On the token configuration space R N × d \mathbb{R}^{N\times d} R N × d , we introduce the finite-token analog of the softmax–Wasserstein metric G ρ S M G_{\rho}^{\mathrm{SM}} G ρ SM in (45) :
g X ( U , H ) : = 1 N tr ( D z ( X ) U B − 1 H T ) , U , H ∈ R N × d , (68) g_X(U,H):=\frac{1}{N}\operatorname{tr}\left(D_z(X)UB^{-1}H^{\mathsf T}\right),\qquad U,H\in\mathbb{R}^{N\times d}, \tag*{(68)} g X ( U , H ) := N 1 tr ( D z ( X ) U B − 1 H T ) , U , H ∈ R N × d , (68)
where the weights z i ( X ) z_i(X) z i ( X ) are defined in (67) . Since z i ( X ) > 0 z_i(X)>0 z i ( X ) > 0 and B ≻ 0 B\succ0 B ≻ 0 , expression (68) defines a smooth Riemannian metric. The interaction energy F \mathcal{F} F in (58) has the corresponding finite-token form
F S M ( X ) : = − 1 2 N 1 T M ( X ) 1. (69) \mathcal{F}^{\mathrm{SM}}(X):=-\frac{1}{2N}\mathbf{1}^{\mathsf T}M(X)\mathbf{1}. \tag*{(69)} F SM ( X ) := − 2 N 1 1 T M ( X ) 1 . (69)
Differentiating the energy (69) and using the metric (68) , the symmetry of A A A and M ( X ) M(X) M ( X ) , and the identity A B = V T AB=V^{\mathsf T} A B = V T gives
− grad g F S M ( X ) = D z − 1 ( X ) M ( X ) X V T = : Attn ( X ) . (70) -\operatorname{grad}_{g}\mathcal{F}^{\mathrm{SM}}(X)=D_z^{-1}(X)M(X)XV^{\mathsf T}=:\operatorname{Attn}(X). \tag*{(70)} − grad g F SM ( X ) = D z − 1 ( X ) M ( X ) X V T =: Attn ( X ) . (70)
The right-hand side is precisely the softmax attention map introduced in Section 2, which we denote by Attn ( X ) \operatorname{Attn}(X) Attn ( X ) . The gradient identity (70) suggests an inertial extension of softmax attention: we replace the first-order equation X ˙ t = Attn ( X t ) \dot{X}_t=\operatorname{Attn}(X_t) X ˙ t = Attn ( X t ) by the damped Newton equation
∇ X ˙ t g X ˙ t + α t X ˙ t + grad g F S M ( X t ) = 0 , \nabla^{g}_{\dot{X}_t}\dot{X}_t+\alpha_t\dot{X}_t+\operatorname{grad}_{g}\mathcal{F}^{\mathrm{SM}}(X_t)=0, ∇ X ˙ t g X ˙ t + α t X ˙ t + grad g F SM ( X t ) = 0 ,
where ∇ g \nabla^{g} ∇ g is the Levi–Civita connection of (68) and α ∈ C ( [ 0 , ∞ ) ; [ 0 , ∞ ) ) \alpha\in C([0,\infty);[0,\infty)) α ∈ C ([ 0 , ∞ ) ; [ 0 , ∞ )) . In matrix coordinates, this equation reads
X ¨ t + α t X ˙ t = Attn ( X t ) − Γ X t g ( X ˙ t , X ˙ t ) , (71) \ddot{X}_{t}+\alpha_{t}\dot{X}_{t}=\operatorname{Attn}(X_{t})-\Gamma^{g}_{X_{t}}(\dot{X}_{t},\dot{X}_{t}), \tag*{(71)} X ¨ t + α t X ˙ t = Attn ( X t ) − Γ X t g ( X ˙ t , X ˙ t ) , (71)
where Γ X t g \Gamma^{g}_{X_{t}} Γ X t g denotes the Christoffel symbol of the metric (68). Momentum-based attention updates already appear in YuriiFormer [91 Sec. 3] . The geometric formulation (71) additionally includes the connection term − Γ X t g ( X ˙ t , X ˙ t ) -\Gamma^{g}_{X_{t}}(\dot{X}_{t},\dot{X}_{t}) − Γ X t g ( X ˙ t , X ˙ t ) , which is determined by the position-dependent metric (68). Thus, this geometry provides a specific correction to momentum attention. We next compute this term explicitly in Lemma 6.11, providing the basis for the corrected updates introduced in subsection 8.1.
Computing the geometric correction. To use (71) in a layer update, we need an explicit expression for Γ X g ( U , U ) \Gamma^{g}_{X}(U,U) Γ X g ( U , U ) . We first record the directional derivative of the normalizing vector z z z given by
δ z X [ U ] : = D z ( X ) [ U ] = [ M ( X ) ⊙ ( U A X T + X A U T ) ] 1 , (72) \delta z_{X}[U] := Dz(X)[U] = [M(X) \odot(UAX^{\mathsf{T}}+XAU^{\mathsf{T}})]\mathbf{1}, \tag*{(72)} δ z X [ U ] := D z ( X ) [ U ] = [ M ( X ) ⊙ ( U A X T + X A U T )] 1 , (72)
where ⊙ \odot ⊙ denotes entrywise multiplication.
The following lemma shows that, since we consider densities on R d \mathbb{R}^{d} R d rather than the sphere, the softmax metric can be expressed in ordinary coordinates, yielding an explicit form of the Christoffel symbols.
Lemma 6.11 (Explicit connection correction). Let U ∈ R N × d U \in\mathbb{R}^{N \times d} U ∈ R N × d have rows U i T U_{i}^{\mathsf{T}} U i T , and set s i : = U i T B − 1 U i s_i := U_i^{\mathsf{T}}B^{-1}U_i s i := U i T B − 1 U i and D s : = diag ( s 1 , … , s N ) D_s := \operatorname{diag}(s_1,\ldots,s_N) D s := diag ( s 1 , … , s N ) . Then
Γ X g ( U , U ) = D z ( X ) − 1 ( D δ z X [ U ] U − 1 2 ( D s M ( X ) + M ( X ) D s ) X ) . (73) \Gamma^{g}_{X}(U,U)=D_{z(X)}^{-1}\left(D_{\delta z_X[U]}U-\frac{1}{2}(D_sM(X)+M(X)D_s)X\right). \tag*{(73)} Γ X g ( U , U ) = D z ( X ) − 1 ( D δ z X [ U ] U − 2 1 ( D s M ( X ) + M ( X ) D s ) X ) . (73)
The proof of Lemma 6.11 is given in Appendix F. Formula (73) makes the connection correction directly computable using matrix products and row rescalings, so that it can be incorporated into attention layers. For fixed dense A A A , B B B and a precomputed B − 1 B^{-1} B − 1 , evaluating the correction requires O ( N 2 d + N d 2 ) O(N^{2}d+Nd^{2}) O ( N 2 d + N d 2 ) arithmetic operations. The practical computational overhead is assessed in Section 9.
Position and velocity updates. Next, we use the connection correction (73) to construct discrete position and velocity updates for (71) . Let U U U denote a proposed displacement from X X X , obtained by combining the current velocity with the attention output Attn ( X ) \operatorname{Attn}(X) Attn ( X ) defined in (70). To update the token positions, we consider the geodesic on R N × d \mathbb{R}^{N \times d} R N × d with respect to the Levi–Civita connection ∇ g \nabla^{g} ∇ g of the metric (68), starting at X X X with initial velocity U U U . Its second-order Taylor expansion gives the position update
R X ( 2 ) ( U ) : = X + U − 1 2 Γ X g ( U , U ) . (74) R_{X}^{(2)}(U):=X+U-\frac{1}{2}\Gamma^{g}_{X}(U,U). \tag*{(74)} R X ( 2 ) ( U ) := X + U − 2 1 Γ X g ( U , U ) . (74)
To carry the displacement into the next layer, we use the differential of (74) with respect to U U U , keeping X X X fixed. By bilinearity of Γ X g \Gamma^{g}_{X} Γ X g , this gives
T U ( 2 ) [ U ] : = D R X ( 2 ) ( U ) [ U ] = U − Γ X g ( U , U ) . (75) \mathcal{T}_{U}^{(2)}[U]:=DR_{X}^{(2)}(U)[U]=U-\Gamma^{g}_{X}(U,U). \tag*{(75)} T U ( 2 ) [ U ] := D R X ( 2 ) ( U ) [ U ] = U − Γ X g ( U , U ) . (75)
Both updates reuse the same connection correction, computed explicitly by (73) . They can therefore be implemented directly as layer updates, without numerically solving geodesic or parallel-transport equations.
Section 9 adapts this construction to causal multi-head attention with learned weights. There, the connection correction is replaced by the proxy Γ ^ \widehat{\Gamma} Γ in (113) , which uses B = I d B=I_d B = I d and the native attention weights. Combining (74) and (75) with the LayerNorm modifications gives the layer update (111) . The coordinate variant (110) instead inserts the correction directly into the velocity update, following (71) . These adaptations provide the architectures whose predictive performance and computational cost we compare in the numerical experiments.
Linear Attention: Geometry, Inertial Dynamics and Effective Depth
In this section, we give a rigorous interpretation of the accelerated linear-attention system (19) as damped Hamiltonian dynamics within the quotient Stein geometry developed in § 3.3 , specialized to the linear-attention energy. Then we use this formulation to study the corresponding PDE and determine how its finite-time blow-up appears in discrete attention layers.
In Subsection 7.1, we study first-order linear attention. We introduce covariance coordinates on families of elliptically contoured densities and identify the induced Stein metric in Lemma 7.2 . These coordinates give a finite-dimensional gradient-flow formulation and specify the metric and energy used to introduce acceleration. Theorem 7.6 also establishes an exact correspondence between the first-order PDE and its second-moment equation for general C 1 C^1 C 1 initial probability densities with finite second moment. Thus, the solution theory extends beyond the elliptical families used to describe the geometry.
Subsection 7.2 uses this metric and energy to construct the accelerated dynamics. We derive their matrix Hamiltonian formulation and identify the corresponding damped Newton equation in Proposition 7.17 . The main analytical result, Theorem 7.18 , establishes an exact ODE–PDE correspondence for admissible initial densities and quadratic initial potentials. This yields local strong well-posedness and continuous dependence on the initial data. We then use the matrix equations to characterize continuation: Proposition 7.22 shows that finite-time breakdown requires divergence of the second moment and that V ⪯ 0 V \preceq0 V ⪯ 0 gives global existence. For commuting data and zero initial momentum, Theorem 7.23 proves that a positive eigenvalue of V V V forces finite-time moment blow-up under bounded nonnegative damping.
Finally, Subsection 7.3 connects these continuous dynamics to finite-token attention layers. Proposition 7.24 shows that the same matrix system determines the continuous token dynamics, so moment blow-up also forces unbounded token norms at a fixed number of tokens. After discretization, Theorem 7.26 predicts the number of layers needed to reach a prescribed second-moment threshold from the corresponding continuous crossing time. Corollaries 7.31 and 7.30 then identify the continuous blow-up time as a critical effective depth: as the discretization is refined at fixed effective depth, token norms remain bounded below this time and become unbounded at or above it. All proofs of this section are given in Appendix G.
Figure 5 summarizes the structure of this section and shows how the continuous analysis leads to predictions for discrete attention layers.
Figure 5. Understanding transformer dynamics through the continuum limit. Analytical information about the growth of PDE solutions yields quantitative predictions for the depth at which large token norms occur in discrete attention layers, as established in Theorem 7.26.
First-Order Linear Attention: Geometry and Analysis
Let V = V ⊤ V=V^{\top} V = V ⊤ and consider K A ( x , y ) = x ⊤ A y K_A(x,y)=x^{\top}Ay K A ( x , y ) = x ⊤ A y . To a given probability density ρ \rho ρ with finite second moment, we associate the second-moment matrix
S ( ρ ) : = ∫ R d y y ⊤ d ρ ( y ) . (76) S(\rho):=\int_{\mathbb{R}^d}yy^{\top}\,\mathrm{d}\rho(y). \tag*{(76)} S ( ρ ) := ∫ R d y y ⊤ d ρ ( y ) . (76)
If ρ \rho ρ is centered, the expression in (76) is its covariance. We consider the quadratic energy
F l i n ( ρ ) : = − 1 2 ∫ R d x ⊤ V x d ρ ( x ) = − 1 2 tr ( V S ( ρ ) ) . \mathcal{F}^{\mathrm{lin}}(\rho):=-\frac{1}{2}\int_{\mathbb{R}^d}x^{\top}Vx\,\mathrm{d}\rho(x)=-\frac{1}{2}\operatorname{tr}(VS(\rho)). F lin ( ρ ) := − 2 1 ∫ R d x ⊤ V x d ρ ( x ) = − 2 1 tr ( V S ( ρ )) .
Averaging the values V y Vy V y with weights K A ( x , y ) K_A(x,y) K A ( x , y ) gives the attention field from Section 2,
Γ ρ ( x ) = ∫ R d K A ( x , y ) V y d ρ ( y ) = V S ( ρ ) A x . \Gamma_\rho(x)=\int_{\mathbb{R}^d}K_A(x,y)Vy\,\mathrm{d}\rho(y)=VS(\rho)Ax. Γ ρ ( x ) = ∫ R d K A ( x , y ) V y d ρ ( y ) = V S ( ρ ) A x .
Thus, the associated transformer PDE reads
∂ t ρ t + ∇ ⋅ ( ρ t V S ( ρ t ) A x ) = 0 , ρ t = 0 = ρ 0 . (77) \partial_t\rho_t+\nabla\cdot\left(\rho_tVS(\rho_t)Ax\right)=0,\qquad\rho_{t=0}=\rho_0. \tag*{(77)} ∂ t ρ t + ∇ ⋅ ( ρ t V S ( ρ t ) A x ) = 0 , ρ t = 0 = ρ 0 . (77)
Next, we comment on how this relates to Gaussian softmax dynamics studied in [16 ] .
Comparison with Gaussian softmax dynamics. As observed by Castin, Ablin, Carrillo, and Peyré [16 Sec. 2.2 and Lemma 4.1] , first-order linear and softmax attention coincide for centered Gaussian densities. Here, we study linear attention for non-Gaussian initial densities as well, where the two dynamics need not coincide.
The quotient Stein geometry also admits matrix choices excluded by the softmax–Wasserstein framework. Indeed, for A = A T ≻ 0 A=A^{\mathsf{T}}\succ0 A = A T ≻ 0 , the construction in § 3.3 allows any symmetric V V V , whereas the softmax geometry in § 3.1 requires B = V A − 1 B=VA^{-1} B = V A − 1 to be symmetric positive definite. For example, V = − A V=-A V = − A is admissible in the quotient Stein framework but gives B = − I d B=-I_d B = − I d , which is excluded by the softmax metric.
Next, we introduce covariance coordinates to describe the dynamics on families generated by radial reference densities.
Covariance coordinates. Recall the notation P K A \mathcal{P}_{K_A} P K A , M ρ M_\rho M ρ , and L ρ K A L_\rho^{K_A} L ρ K A from § 3.3 , the regular effective tangent and cotangent spaces (52), and their metric G ρ K A G_\rho^{K_A} G ρ K A from Corollary 5.4 . Next, we choose a radial, standardized reference density η ∈ P K A \eta\in\mathcal{P}_{K_A} η ∈ P K A , so that
η ( O x ) = η ( x ) ( O ∈ O ( d ) ) , ∫ R d x x T η ( x ) d x = I d . \eta(Ox)=\eta(x)\quad(O\in O(d)),\qquad\int_{\mathbb{R}^d}xx^{\mathsf{T}}\eta(x)\,\mathrm{d}x=I_d. η ( O x ) = η ( x ) ( O ∈ O ( d )) , ∫ R d x x T η ( x ) d x = I d .
Then, we introduce a family of probability densities parametrized by their covariance matrices
ι : Sym + + ( d ) ⟶ E η : = { ρ S : S ∈ Sym + + ( d ) } , \iota:\operatorname{Sym}_{++}(d)\longrightarrow\mathcal{E}_\eta:=\{\rho_S:S\in\operatorname{Sym}_{++}(d)\}, ι : Sym ++ ( d ) ⟶ E η := { ρ S : S ∈ Sym ++ ( d )} ,
S ⟼ ρ S , ρ S ( x ) : = η ( S − 1 / 2 x ) det S . (78) S\longmapsto\rho_S,\qquad\rho_S(x):=\frac{\eta(S^{-1/2}x)}{\sqrt{\det S}}. \tag*{(78)} S ⟼ ρ S , ρ S ( x ) := det S η ( S − 1/2 x ) . (78)
Since ρ S \rho_S ρ S has covariance S S S , the map ι \iota ι is injective. This parametrization equips E η \mathcal{E}_\eta E η with a finite-dimensional smooth structure. It is straightforward to check that for every S ∈ Sym + + ( d ) S\in\operatorname{Sym}_{++}(d) S ∈ Sym ++ ( d ) , the density ρ S \rho_S ρ S belongs to P K A \mathcal{P}_{K_A} P K A and is centered. For each S ∈ Sym + + ( d ) S\in\operatorname{Sym}_{++}(d) S ∈ Sym ++ ( d ) , we introduce the Lyapunov operator associated with the matrix S A S SAS S A S by
Λ S : Sym ( d ) ⟶ Sym ( d ) , Λ S ( P ) = P S A S + S A S P . (79) \Lambda_S:\operatorname{Sym}(d)\longrightarrow\operatorname{Sym}(d),\qquad\Lambda_S(P)=PSAS+SASP. \tag*{(79)} Λ S : Sym ( d ) ⟶ Sym ( d ) , Λ S ( P ) = P S A S + S A S P . (79)
With this notation, we can express the quotient Stein metric in covariance coordinates and reduce the associated gradient flows from infinite to finite dimensions. The first step is the following observation.
Lemma 7.1 (Properties of the Lyapunov operator). For each S ∈ Sym + + ( d ) S\in\operatorname{Sym}_{++}(d) S ∈ Sym ++ ( d ) , the map Λ S \Lambda_S Λ S in (79) is an isomorphism, and D ι ( S ) D\iota(S) D ι ( S ) , with ι \iota ι defined in (78), is injective.
Now we can use Λ S − 1 \Lambda_S^{-1} Λ S − 1 to define a smooth Riemannian metric on Sym + + ( d ) \operatorname{Sym}_{++}(d) Sym ++ ( d ) . For H , K ∈ Sym ( d ) H,K\in\operatorname{Sym}(d) H , K ∈ Sym ( d ) , set
g S l i n ( H , K ) : = 1 2 tr ( H Λ S − 1 ( K ) ) = tr ( P S A S Q ) , g_S^{\mathrm{lin}}(H,K):=\frac{1}{2}\operatorname{tr}\left(H\Lambda_S^{-1}(K)\right)=\operatorname{tr}(PSASQ), g S lin ( H , K ) := 2 1 tr ( H Λ S − 1 ( K ) ) = tr ( P S A S Q ) ,
P = Λ S − 1 ( H ) , Q = Λ S − 1 ( K ) . (80) P=\Lambda_S^{-1}(H),\qquad Q=\Lambda_S^{-1}(K). \tag*{(80)} P = Λ S − 1 ( H ) , Q = Λ S − 1 ( K ) . (80)
This metric can be interpreted as the pullback of G ρ K A G_\rho^{K_A} G ρ K A under the parametrization ι \iota ι , as the following lemma shows.
Lemma 7.2 (Relating g l i n g^{\mathrm{lin}} g lin and G K A G^{K_A} G K A ). For each S ∈ Sym + + ( d ) S\in\operatorname{Sym}_{++}(d) S ∈ Sym ++ ( d ) and all H , K ∈ Sym ( d ) H,K\in\operatorname{Sym}(d) H , K ∈ Sym ( d ) ,
g S l i n ( H , K ) = G ρ S K A ( D ι ( S ) [ H ] , D ι ( S ) [ K ] ) . (81) g_S^{\mathrm{lin}}(H,K)=G_{\rho_S}^{K_A}\left(D\iota(S)[H],D\iota(S)[K]\right). \tag*{(81)} g S lin ( H , K ) = G ρ S K A ( D ι ( S ) [ H ] , D ι ( S ) [ K ] ) . (81)
There is the following relation to the Bures–Wasserstein metric.
We close this paragraph by reducing (77) on E η \mathcal{E}_{\eta} E η to a finite-dimensional gradient flow with respect to this metric. For the functional
F l i n ( S ) : = F l i n ( ρ S ) = − 1 2 tr ( V S ) , \mathcal{F}^{\mathrm{lin}}(S) := \mathcal{F}^{\mathrm{lin}}(\rho_S) = -\frac{1}{2}\operatorname{tr}(VS), F lin ( S ) := F lin ( ρ S ) = − 2 1 tr ( V S ) ,
we first observe that
grad g l i n F l i n ( S ) = − Λ S ( V ) . \operatorname{grad}_{g^{\mathrm{lin}}}\mathcal{F}^{\mathrm{lin}}(S) = -\Lambda_S(V). grad g lin F lin ( S ) = − Λ S ( V ) .
Moreover, the following proposition establishes the equivalence between the density evolution and this covariance gradient flow.
Proposition 7.4 (Covariance metric and gradient-flow formulation). A C 1 C^1 C 1 curve S t ∈ Sym + + ( d ) S_t \in\operatorname{Sym}_{++}(d) S t ∈ Sym ++ ( d ) gives a solution ρ t = ρ S t \rho_t = \rho_{S_t} ρ t = ρ S t of (77) if and only if
S ˙ t = V S t A S t + S t A S t V = Λ S t ( V ) = − grad g l i n F l i n ( S t ) , (82) \dot{S}_t = VS_tAS_t + S_tAS_tV = \Lambda_{S_t}(V) = -\operatorname{grad}_{g^{\mathrm{lin}}}\mathcal{F}^{\mathrm{lin}}(S_t), \tag*{(82)} S ˙ t = V S t A S t + S t A S t V = Λ S t ( V ) = − grad g lin F lin ( S t ) , (82) with the corresponding initial data.
Next, we use this finite-dimensional parametrization to establish local well-posedness of the associated PDE and give precise criteria for finite-time blow-up.
Strong solutions and ODE–PDE correspondence. We begin by defining a notion of a (local) strong solution to the transformer PDE (77) for C 1 C^1 C 1 initial probability densities with finite second moment. We omit the proofs of this paragraph, since they follow by minor adaptations of the arguments developed for the more involved damped Hamiltonian case in § 4.2.2 .
Let I = [ 0 , T ) I = [0,T) I = [ 0 , T ) , and let ρ i n ∈ C 1 ( R d ) \rho_{\mathrm{in}} \in C^1(\mathbb{R}^d) ρ in ∈ C 1 ( R d ) be a probability density with finite second moment. A function ρ \rho ρ is called a strong solution of (77) with initial datum ρ i n \rho_{\mathrm{in}} ρ in , if the following conditions hold:
The map t ↦ ρ t ∈ C ( I ; C l o c 1 ( R d ) ) ∩ W l o c 1 , 1 ( I ; C l o c 0 ( R d ) ) t \mapsto\rho_t \in C(I;C^1_{\mathrm{loc}}(\mathbb{R}^d)) \cap W^{1,1}_{\mathrm{loc}}(I;C^0_{\mathrm{loc}}(\mathbb{R}^d)) t ↦ ρ t ∈ C ( I ; C loc 1 ( R d )) ∩ W loc 1 , 1 ( I ; C loc 0 ( R d )) .
The initial condition holds pointwise ρ 0 ( x ) = ρ i n ( x ) \rho_0(x)=\rho_{\mathrm{in}}(x) ρ 0 ( x ) = ρ in ( x ) for x ∈ R d x\in\mathbb{R}^d x ∈ R d .
For almost every t ∈ I t\in I t ∈ I , the matrix integral S t : = S ( ρ t ) S_t:=S(\rho_t) S t := S ( ρ t ) with S ( ρ t ) S(\rho_t) S ( ρ t ) as in (76) is absolutely convergent, and S ∈ L l o c 1 ( I ; Sym ( d ) ) S\in L^1_{\mathrm{loc}}(I;\operatorname{Sym}(d)) S ∈ L loc 1 ( I ; Sym ( d )) .
For every compact set K ⊂ R d K\subset\mathbb{R}^d K ⊂ R d and almost every t ∈ I t\in I t ∈ I ,
∂ t ρ t + ∇ ⋅ ( ρ t V S t A x ) = 0 in C ( K ) . \partial_t\rho_t+\nabla\cdot(\rho_t V S_t A x)=0\qquad\text{in }C(K). ∂ t ρ t + ∇ ⋅ ( ρ t V S t A x ) = 0 in C ( K ) .
The third condition ensures that the moment-dependent velocity is well-defined and locally integrable in time, while condition four requires the density to satisfy the resulting continuity equation on every compact set, hence throughout R d \mathbb{R}^d R d , without imposing global uniform bounds on the density or its derivatives.
With this definition, we can establish local well-posedness of (77) by reducing it to the matrix equation (82) . Moreover, this correspondence shows that the evolution preserves the class P K A \mathcal{P}_{K_A} P K A , i.e., if ρ i n ∈ P K A \rho_{\mathrm{in}}\in\mathcal{P}_{K_A} ρ in ∈ P K A , then ρ t ∈ P K A \rho_t\in\mathcal{P}_{K_A} ρ t ∈ P K A throughout its interval of existence.
Theorem 7.6 (Gradient-flow ODE–PDE correspondence). Let ρ 0 ∈ C 1 ( R d ) \rho_0\in C^1(\mathbb{R}^d) ρ 0 ∈ C 1 ( R d ) be a probability density with finite second moment, and set S 0 : = S ( ρ 0 ) S_0:=S(\rho_0) S 0 := S ( ρ 0 ) . Then the following statements hold:
The matrix equation (82) has a unique maximal solution S t ∈ C 1 ( [ 0 , T max ) ; Sym + ( d ) ) S_t\in C^1([0,T_{\max});\operatorname{Sym}_+(d)) S t ∈ C 1 ([ 0 , T m a x ) ; Sym + ( d )) with initial value S 0 S_0 S 0 .
Let G G G solve
G ˙ t = V S t A G t , G 0 = I d . (83) \dot{G}_t=V S_t A G_t,\qquad G_0=I_d. \tag*{(83)} G ˙ t = V S t A G t , G 0 = I d . (83) Then
ρ t = ( G t ) # ρ 0 (84) \rho_t=(G_t)_\#\rho_0 \tag*{(84)} ρ t = ( G t ) # ρ 0 (84) defines a strong solution of (77) with second-moment matrix S t = G t S 0 G t ⊤ S_t=G_tS_0G_t^\top S t = G t S 0 G t ⊤ . Conversely, every strong solution with initial datum ρ 0 \rho_0 ρ 0 has the form (84) , and its second moment solves (82) .
If additionally ρ 0 ∈ P K A \rho_0\in\mathcal{P}_{K_A} ρ 0 ∈ P K A , then we have throughout the evolution that
ρ t ∈ P K A , ∂ t ρ t = − ∇ ⋅ ( ρ t V S t A x ) ∈ T ρ t e f f P K A . \rho_t\in\mathcal{P}_{K_A},\qquad\partial_t\rho_t=-\nabla\cdot(\rho_t V S_t A x)\in T^{\mathrm{eff}}_{\rho_t}\mathcal{P}_{K_A}. ρ t ∈ P K A , ∂ t ρ t = − ∇ ⋅ ( ρ t V S t A x ) ∈ T ρ t eff P K A . If ρ 0 = ρ S 0 ∈ E η \rho_0=\rho_{S_0}\in\mathcal{E}_\eta ρ 0 = ρ S 0 ∈ E η , then ρ t = ρ S t \rho_t=\rho_{S_t} ρ t = ρ S t remains in E η \mathcal{E}_\eta E η and is the gradient-flow curve represented in covariance coordinates by (82) .
We omit the proof of the theorem as well as of its corollaries, since we will prove the corresponding result in the more involved setting of damped Hamiltonian dynamics in Theorem 7.18.
Remark 7.7 (Relation to Gaussian softmax dynamics). For ρ 0 = N ( 0 , S 0 ) \rho_0=\mathcal{N}(0,S_0) ρ 0 = N ( 0 , S 0 ) , the solution is ρ t = N ( 0 , S t ) \rho_t=\mathcal{N}(0,S_t) ρ t = N ( 0 , S t ) . Since linear and softmax attention agree on centered Gaussians, this recovers the centered Gaussian case of [16 Proposition 4.2] for the constant symmetric matrices considered here. The theorem extends the linear-attention correspondence to non-Gaussian initial densities.
Local and global well-posedness. We first derive local well-posedness of (77) by applying the Picard–Lindelöf theorem to (83) and using the ODE–PDE correspondence in Theorem 7.6 , item 2.
Corollary 7.8 (Local strong well-posedness). For every C 1 C^{1} C 1 probability density ρ 0 \rho_{0} ρ 0 on R d \mathbb{R}^{d} R d with finite second moment, (77) has a unique maximal strong solution with initial value ρ 0 \rho_{0} ρ 0 . Its maximal interval equals that of (82) *with initial value S 0 = S ( ρ 0 ) S_{0}=S(\rho_{0}) S 0 = S ( ρ 0 ) .
It is natural to ask how the solution of (77) depends on the initial density. The ODE–PDE correspondence in Theorem 7.6 transfers continuous dependence from the matrix equation (82) to the PDE, yielding the following corollary. We omit its proof, which follows the same argument as the corresponding result for the damped system in Corollary 7.21 .
Corollary 7.9 (Wasserstein convergence and continuous dependence). Let ρ 0 ( n ) , ρ 0 \rho_{0}^{(n)},\rho_{0} ρ 0 ( n ) , ρ 0 be C 1 C^{1} C 1 probability densities satisfying
ρ 0 ( n ) ⟶ ρ 0 in W 2 and in C l o c 1 ( R d ) . \rho_{0}^{(n)} \longrightarrow\rho_{0} \quad\text{in } W_{2} \text{ and in } C^{1}_{\mathrm{loc}}(\mathbb{R}^{d}). ρ 0 ( n ) ⟶ ρ 0 in W 2 and in C loc 1 ( R d ) . Denote by ρ ( n ) \rho^{(n)} ρ ( n ) and ρ \rho ρ the corresponding maximal strong solutions of (77) . Let [ 0 , T max ) [0,T_{\max}) [ 0 , T m a x ) be the maximal interval of existence of ρ \rho ρ . Then, for every 0 < T < T max 0<T<T_{\max} 0 < T < T m a x , the solutions ρ ( n ) \rho^{(n)} ρ ( n ) exist on [ 0 , T ] [0,T] [ 0 , T ] for all sufficiently large n n n , and
ρ ( n ) ⟶ ρ in C ( [ 0 , T ] ; W 2 ) ∩ C ( [ 0 , T ] ; C l o c 1 ) ∩ C 1 ( [ 0 , T ] ; C l o c 0 ) . \rho^{(n)} \longrightarrow\rho\quad\text{in } C([0,T];W_{2})\cap C([0,T];C^{1}_{\mathrm{loc}})\cap C^{1}([0,T];C^{0}_{\mathrm{loc}}). ρ ( n ) ⟶ ρ in C ([ 0 , T ] ; W 2 ) ∩ C ([ 0 , T ] ; C loc 1 ) ∩ C 1 ([ 0 , T ] ; C loc 0 ) .
Continuation and moment blow-up. Like many transport equations, (77) can exhibit finite-time breakdown. The following proposition characterizes this behavior through the second moment.
Proposition 7.10 (Continuation criterion). Let ρ 0 ∈ C 1 ( R d ) \rho_{0}\in C^{1}(\mathbb{R}^{d}) ρ 0 ∈ C 1 ( R d ) be a probability density with finite second moment, and let ρ \rho ρ be the maximal strong solution of (77) with initial datum ρ 0 \rho_{0} ρ 0 . Denote its maximal existence time by T max T_{\max} T m a x and set S t : = S ( ρ t ) S_{t}:=S(\rho_{t}) S t := S ( ρ t ) . If T max < ∞ T_{\max}<\infty T m a x < ∞ , then the second moments of ρ t \rho_{t} ρ t become unbounded as t ↑ T max t\uparrow T_{\max} t ↑ T m a x , meaning that
lim sup t ↑ T max ∫ R d ∣ x ∣ 2 d ρ t ( x ) = ∞ . \limsup_{t\uparrow T_{\max}}\int_{\mathbb{R}^{d}}|x|^{2}\,\mathrm{d}\rho_{t}(x)=\infty. t ↑ T m a x lim sup ∫ R d ∣ x ∣ 2 d ρ t ( x ) = ∞.
We again omit the proof, as it follows by minor adaptations of the proof of the more involved continuation criterion for the damped system in Proposition 7.22 .
Remark 7.11. The second-moment blow-up is equivalent to lim sup t ↑ T max ∥ S t ∥ F = ∞ \limsup_{t\uparrow T_{\max}}\|S_{t}\|_{\mathrm{F}}=\infty lim sup t ↑ T m a x ∥ S t ∥ F = ∞ . Indeed, since S t S_{t} S t is positive semidefinite,
∥ S t ∥ F ≤ tr ( S t ) = ∫ R d ∣ x ∣ 2 d ρ t ( x ) ≤ d ∥ S t ∥ F . \|S_{t}\|_{\mathrm{F}}\leq\operatorname{tr}(S_{t})=\int_{\mathbb{R}^{d}}|x|^{2}\,\mathrm{d}\rho_{t}(x)\leq\sqrt{d}\,\|S_{t}\|_{\mathrm{F}}. ∥ S t ∥ F ≤ tr ( S t ) = ∫ R d ∣ x ∣ 2 d ρ t ( x ) ≤ d ∥ S t ∥ F .
Next, we use Proposition 7.10 to obtain global well-posedness under a sign condition on V V V .
Corollary 7.12 (Well-posedness under sign condition). Under the assumptions of Proposition 7.10 , if V ⪯ 0 V\preceq0 V ⪯ 0 , then the maximal existence time of ρ \rho ρ is T max = ∞ T_{\max}=\infty T m a x = ∞ .
More interestingly, when A A A , S 0 S_{0} S 0 , and V V V commute, we can solve the matrix equation explicitly. Together with Proposition 7.10 , this allows us to determine directly from the initial data whether finite-time blow-up occurs and, if so, when.
Proposition 7.13 (Explicit solution in the commuting case). Under the assumptions of Proposition 7.10 , suppose that A A A , S 0 = S ( ρ 0 ) S_0 = S(\rho_0) S 0 = S ( ρ 0 ) , and V V V commute. In a common orthonormal eigenbasis, write
A = diag ( a i ) , S 0 = diag ( s i , 0 ) , V = diag ( v i ) , a i , s i , 0 > 0. A = \operatorname{diag}(a_i), \qquad S_0 = \operatorname{diag}(s_{i,0}), \qquad V = \operatorname{diag}(v_i), \qquad a_i, s_{i,0} > 0. A = diag ( a i ) , S 0 = diag ( s i , 0 ) , V = diag ( v i ) , a i , s i , 0 > 0. Then the following statements hold:
(Representation in the eigenbasis) For 0 ≤ t < T max 0 \le t < T_{\max} 0 ≤ t < T m a x ,
S t = diag ( s i ( t ) ) , s i ( t ) = s i , 0 1 − 2 a i v i s i , 0 t , S_t = \operatorname{diag}(s_i(t)), \qquad s_i(t) = \frac{s_{i,0}}{1 - 2a_i v_i s_{i,0} t}, S t = diag ( s i ( t )) , s i ( t ) = 1 − 2 a i v i s i , 0 t s i , 0 , and the maximal existence time is
T max = min i : v i > 0 1 2 a i v i s i , 0 , T_{\max} = \min_{i:v_i>0} \frac{1}{2a_i v_i s_{i,0}}, T m a x = i : v i > 0 min 2 a i v i s i , 0 1 , where the minimum over the empty set is understood as infinity.
(Moment blow-up) If some v i > 0 v_i > 0 v i > 0 , then T max < ∞ T_{\max} < \infty T m a x < ∞ and
lim t ↑ T max ∫ R d ∣ x ∣ 2 d ρ t ( x ) = ∞ . \lim_{t \uparrow T_{\max}} \int_{\mathbb{R}^d} |x|^2 \, \mathrm{d}\rho_t(x) = \infty. t ↑ T m a x lim ∫ R d ∣ x ∣ 2 d ρ t ( x ) = ∞.
We omit the proof, which follows by diagonalizing (82) and solving the resulting scalar equations. The more involved commuting case of the damped system is treated in Theorem 7.23 .
For centered Gaussian initial data, the explicit formulas and blow-up criterion recover the corresponding results of [16 Sec. 4.1] . For linear attention, these formulas also hold for every C 1 C^1 C 1 initial probability density with finite second moment, provided the commutativity assumption of Proposition 7.13 holds. Indeed, Theorem 7.6 shows that, for fixed A A A , V V V , initial densities with the same second-moment matrix S 0 S_0 S 0 have the same moment evolution and maximal existence time, regardless of whether they are Gaussian.
Accelerated Linear Attention: Geometry and Hamiltonian Dynamics
The aim of this subsection is to formulate accelerated linear attention as a damped Hamiltonian system and use this formulation to study the associated PDE. We first identify admissible densities and quadratic potentials, then derive the matrix equations in covariance coordinates.
The exact ODE–PDE correspondence in Theorem 7.18 allows us to establish strong well-posedness beyond the elliptical family E η \mathcal{E}_\eta E η and to study global existence and finite-time moment blow-up. These results provide the basis for the analysis of finite-token layers in Subsection 7.3.
We first fix the notation and assumptions used throughout this subsection. Let A = A ⊤ ≻ 0 A = A^\top\succ0 A = A ⊤ ≻ 0 and V = V ⊤ V = V^\top V = V ⊤ be fixed, and let α ∈ C ( [ 0 , ∞ ) ; [ 0 , ∞ ) ) \alpha\in C([0,\infty); [0,\infty)) α ∈ C ([ 0 , ∞ ) ; [ 0 , ∞ )) . For damping schedules singular at the origin, we translate time and use α ( t 0 + t ) \alpha(t_0 + t) α ( t 0 + t ) with t 0 > 0 t_0 > 0 t 0 > 0 . Here, acceleration refers to the introduction of inertia; a convergence-rate improvement would require a separate analysis.
Quadratic potentials. We begin by identifying a class of probability densities for which quadratic potentials belong to the regular cotangent space. For each P ∈ Sym ( d ) P \in\operatorname{Sym}(d) P ∈ Sym ( d ) , define
Φ P ( x ) : = 1 2 x T P x . (85) \Phi_P(x) := \frac{1}{2}x^{\mathsf T}Px. \tag*{(85)} Φ P ( x ) := 2 1 x T P x . (85)
We observe that
∣ Φ P ( x ) ∣ 2 ≤ 1 4 ∥ P ∥ o p 2 ∣ x ∣ 4 , ∣ ∇ Φ P ( x ) ∣ 2 ≤ ∥ P ∥ o p 2 ∣ x ∣ 2 . \lvert\Phi_P(x)\rvert^2 \le\frac{1}{4}\lVert P\rVert_{\mathrm{op}}^2\lvert x\rvert^4,\qquad\lvert\nabla\Phi_P(x)\rvert^2 \le\lVert P\rVert_{\mathrm{op}}^2\lvert x\rvert^2. ∣ Φ P ( x ) ∣ 2 ≤ 4 1 ∥ P ∥ op 2 ∣ x ∣ 4 , ∣ ∇ Φ P ( x ) ∣ 2 ≤ ∥ P ∥ op 2 ∣ x ∣ 2 .
Thus, Φ P ∈ H 1 ( ρ ) \Phi_P \in H^1(\rho) Φ P ∈ H 1 ( ρ ) whenever ρ ∈ P K A \rho\in\mathcal{P}_{K_A} ρ ∈ P K A has finite fourth moment. The choice P = I d P=I_d P = I d also shows that Φ I d ∈ L 2 ( ρ ) \Phi_{I_d} \in L^2(\rho) Φ I d ∈ L 2 ( ρ ) requires a finite fourth moment. We therefore introduce
P K A 4 : = { ρ ∈ P K A : ∫ R d ∣ x ∣ 4 d ρ ( x ) < ∞ } . (86) \mathcal{P}_{K_A}^{4} := \left\{\rho\in\mathcal{P}_{K_A} : \int_{\mathbb{R}^d}\lvert x\rvert^4\,\mathrm{d}\rho(x) < \infty\right\}. \tag*{(86)} P K A 4 := { ρ ∈ P K A : ∫ R d ∣ x ∣ 4 d ρ ( x ) < ∞ } . (86)
For the family E η \mathcal{E}_{\eta} E η , we henceforth choose the radial, standardized reference density η ∈ P K A 4 \eta\in\mathcal{P}_{K_A}^{4} η ∈ P K A 4 . The covariance metric g l i n g^{\mathrm{lin}} g lin remains the one defined in (80) . Under this additional moment assumption, a short computation, which we leave to the reader, shows that, for every ρ S ∈ E η \rho_S \in\mathcal{E}_{\eta} ρ S ∈ E η ,
ran M ρ S = S Sym ( d ) , T ρ S e f f P K A = T ρ S E η . \operatorname{ran} M_{\rho_S}=S\operatorname{Sym}(d),\qquad T_{\rho_S}^{\mathrm{eff}}\mathcal{P}_{K_A}=T_{\rho_S}\mathcal{E}_{\eta}. ran M ρ S = S Sym ( d ) , T ρ S eff P K A = T ρ S E η .
Moreover, every effective cotangent class has a unique homogeneous quadratic representative
Φ P ( x ) = 1 2 x T P x , P = S − 1 M ρ S ( Φ ) ∈ Sym ( d ) , \Phi_P(x)=\frac{1}{2}x^{\mathsf T}Px,\qquad P=S^{-1}M_{\rho_S}(\Phi)\in\operatorname{Sym}(d), Φ P ( x ) = 2 1 x T P x , P = S − 1 M ρ S ( Φ ) ∈ Sym ( d ) ,
where Φ \Phi Φ is any representative of the class.
Hamiltonian formulation. For ρ ∈ P K A 4 \rho\in\mathcal{P}_{K_A}^{4} ρ ∈ P K A 4 and [ Φ ] ∈ T ρ ∗ P K A [\Phi]\in T_\rho^*\mathcal{P}_{K_A} [ Φ ] ∈ T ρ ∗ P K A , we consider the Hamiltonian associated with the energy F l i n \mathcal{F}^{\mathrm{lin}} F lin :
H ( ρ , [ Φ ] ) : = 1 2 ⟨ [ Φ ] , L ρ K A [ Φ ] ⟩ ρ + F l i n ( ρ ) = 1 2 tr ( M ρ ( Φ ) T A M ρ ( Φ ) ) − 1 2 tr ( V S ( ρ ) ) . \mathcal{H}(\rho,[\Phi]) := \frac{1}{2}\langle[\Phi],L_{\rho}^{K_A}[\Phi]\rangle_\rho+\mathcal{F}^{\mathrm{lin}}(\rho)
=\frac{1}{2}\operatorname{tr}\bigl(M_\rho(\Phi)^{\mathsf T}AM_\rho(\Phi)\bigr)-\frac{1}{2}\operatorname{tr}(VS(\rho)). H ( ρ , [ Φ ]) := 2 1 ⟨[ Φ ] , L ρ K A [ Φ ] ⟩ ρ + F lin ( ρ ) = 2 1 tr ( M ρ ( Φ ) T A M ρ ( Φ ) ) − 2 1 tr ( V S ( ρ )) .
This is the Hamiltonian in (127) with F = F l i n \mathcal{F}=\mathcal{F}^{\mathrm{lin}} F = F lin . The notion of differentiability used to compute its variations is introduced in Appendix § 7.6 . To describe its variation with respect to the density, we set
Q ρ ( Φ ) ( x ) : = ∇ Φ ( x ) T M ρ ( Φ ) T A x . Q_\rho(\Phi)(x):=\nabla\Phi(x)^{\mathsf T}M_\rho(\Phi)^{\mathsf T}Ax. Q ρ ( Φ ) ( x ) := ∇Φ ( x ) T M ρ ( Φ ) T A x .
A short computation confirms that the first variation of F l i n \mathcal{F}^{\mathrm{lin}} F lin is − Φ V -\Phi_V − Φ V . So for a quadratic potential Φ P \Phi_P Φ P , we have
M ρ ( Φ P ) = S ( ρ ) P , Q ρ ( Φ P ) = Φ P 2 S ( ρ ) A + A S ( ρ ) P 2 . M_\rho(\Phi_P)=S(\rho)P,\qquad Q_\rho(\Phi_P)=\Phi_{P^2S(\rho)A+AS(\rho)P^2}. M ρ ( Φ P ) = S ( ρ ) P , Q ρ ( Φ P ) = Φ P 2 S ( ρ ) A + A S ( ρ ) P 2 .
Here the assumption ρ ∈ P K A 4 \rho\in\mathcal{P}_{K_A}^{4} ρ ∈ P K A 4 , that is ρ \rho ρ has a finite fourth moment, ensures that both Q ρ ( Φ P ) Q_\rho(\Phi_P) Q ρ ( Φ P ) and Φ V \Phi_V Φ V belong to H 1 ( ρ ) H^1(\rho) H 1 ( ρ ) . Moreover, compactly supported admissible density variations preserve finiteness of the fourth moment. For Φ = Φ P \Phi=\Phi_P Φ = Φ P , we can therefore apply Lemma C.4 to obtain
δ r e g H δ Φ = L ρ K A [ Φ ] , δ r e g H δ ρ = [ Q ρ ( Φ ) − Φ V ] . \frac{\delta^{\mathrm{reg}}\mathcal{H}}{\delta\Phi}=L_\rho^{K_A}[\Phi],\qquad
\frac{\delta^{\mathrm{reg}}\mathcal{H}}{\delta\rho}=[Q_\rho(\Phi)-\Phi_V]. δ Φ δ reg H = L ρ K A [ Φ ] , δ ρ δ reg H = [ Q ρ ( Φ ) − Φ V ] .
It would be interesting to extend this Hamiltonian formulation to non-quadratic potentials in future work. This would require, in particular, identifying conditions that ensure Q ρ ( Φ ) ∈ H 1 ( ρ ) Q_\rho(\Phi)\in H^1(\rho) Q ρ ( Φ ) ∈ H 1 ( ρ ) .
With these variations at hand, the damped Hamiltonian equations read
∂ t ρ t + ∇ ⋅ ( ρ t M ρ t ( Φ t ) ⊤ A x ) = 0 , ρ t = 0 = ρ 0 , ∂ t Φ t + α t Φ t + ∇ Φ t ⊤ M ρ t ( Φ t ) ⊤ A x − 1 2 x ⊤ V x = 0 , Φ t = 0 = Φ 0 . (87) \begin{aligned}
\partial_t\rho_t+\nabla\cdot\left(\rho_tM_{\rho_t}(\Phi_t)^{\top}Ax\right)&=0,\qquad\rho_{t=0}=\rho_0,\\
\partial_t\Phi_t+\alpha_t\Phi_t+\nabla\Phi_t^{\top}M_{\rho_t}(\Phi_t)^{\top}Ax-\frac{1}{2}x^{\top}Vx&=0,\qquad\Phi_{t=0}=\Phi_0. \tag*{(87)}
\end{aligned} ∂ t ρ t + ∇ ⋅ ( ρ t M ρ t ( Φ t ) ⊤ A x ) ∂ t Φ t + α t Φ t + ∇ Φ t ⊤ M ρ t ( Φ t ) ⊤ A x − 2 1 x ⊤ V x = 0 , ρ t = 0 = ρ 0 , = 0 , Φ t = 0 = Φ 0 . (87)
Before moving on, we emphasize the following.
Covariance Hamiltonian and Newton equation. We now reduce (87) on E η \mathcal{E}_{\eta} E η to a finite-dimensional system for the covariance matrix S t S_t S t in (78) and the quadratic coefficient P t P_t P t in (85).
Lemma 7.15 (Quadratic reduction). Let I = [ 0 , T ) I=[0,T) I = [ 0 , T ) , S ∈ C 1 ( I ; Sym + + ( d ) ) S\in C^1(I;\operatorname{Sym}_{++}(d)) S ∈ C 1 ( I ; Sym ++ ( d )) , and P ∈ C 1 ( I ; Sym ( d ) ) P\in C^1(I;\operatorname{Sym}(d)) P ∈ C 1 ( I ; Sym ( d )) . Set ρ t = ρ S t \rho_t=\rho_{S_t} ρ t = ρ S t and Φ t = Φ P t \Phi_t=\Phi_{P_t} Φ t = Φ P t , as defined in (78) and (85), respectively. Then ( ρ , Φ ) (\rho,\Phi) ( ρ , Φ ) solves (87) with initial data ρ 0 = ρ S 0 \rho_0=\rho_{S_0} ρ 0 = ρ S 0 and Φ 0 = Φ P 0 \Phi_0=\Phi_{P_0} Φ 0 = Φ P 0 if and only if
S ˙ t = P t S t A S t + S t A S t P t , P ˙ t = − α t P t − A S t P t 2 − P t 2 S t A + V . (88) \begin{aligned}
\dot{S}_t&=P_tS_tAS_t+S_tAS_tP_t,\\
\dot{P}_t&=-\alpha_tP_t-AS_tP_t^2-P_t^2S_tA+V. \tag*{(88)}
\end{aligned} S ˙ t P ˙ t = P t S t A S t + S t A S t P t , = − α t P t − A S t P t 2 − P t 2 S t A + V . (88)
The proof of Lemma 7.15 is given in Appendix G.2. Next, we formulate the matrix dynamics on the cotangent bundle T ∗ Sym + + ( d ) T^*\operatorname{Sym}_{++}(d) T ∗ Sym ++ ( d ) of the covariance manifold. Since Sym + + ( d ) \operatorname{Sym}_{++}(d) Sym ++ ( d ) is an open subset of Sym ( d ) \operatorname{Sym}(d) Sym ( d ) , its tangent space at each S S S is Sym ( d ) \operatorname{Sym}(d) Sym ( d ) . Every cotangent vector can therefore be represented uniquely by a symmetric matrix Π \Pi Π through the Frobenius pairing
H ⟼ ⟨ Π , H ⟩ F : = tr ( Π H ) , H ∈ Sym ( d ) . H\longmapsto\langle\Pi,H\rangle_{\mathrm{F}}:=\operatorname{tr}(\Pi H),\qquad H\in\operatorname{Sym}(d). H ⟼ ⟨ Π , H ⟩ F := tr ( Π H ) , H ∈ Sym ( d ) .
This identifies T ∗ Sym + + ( d ) T^*\operatorname{Sym}_{++}(d) T ∗ Sym ++ ( d ) with Sym + + ( d ) × Sym ( d ) \operatorname{Sym}_{++}(d)\times\operatorname{Sym}(d) Sym ++ ( d ) × Sym ( d ) .
The quadratic potential Φ P \Phi_P Φ P pairs with a covariance variation H H H as 1 2 tr ( P H ) \frac{1}{2}\operatorname{tr}(PH) 2 1 tr ( P H ) . Its corresponding momentum coordinate is therefore Π = P / 2 \Pi=P/2 Π = P /2 . We define the Hamiltonian
H l i n : T ∗ Sym + + ( d ) ⟶ R , H l i n ( S , Π ) : = 2 tr ( Π S A S Π ) − 1 2 tr ( V S ) . (89) \mathcal{H}^{\mathrm{lin}}:T^*\operatorname{Sym}_{++}(d)\longrightarrow\mathbb{R},\qquad\mathcal{H}^{\mathrm{lin}}(S,\Pi):=2\operatorname{tr}(\Pi SAS\Pi)-\frac{1}{2}\operatorname{tr}(VS). \tag*{(89)} H lin : T ∗ Sym ++ ( d ) ⟶ R , H lin ( S , Π ) := 2 tr ( Π S A S Π ) − 2 1 tr ( V S ) . (89)
With this notation, we can confirm that (88) can be written as a damped Hamiltonian system.
Lemma 7.16 (Covariance Hamiltonian). The change of variables Π = P / 2 \Pi=P/2 Π = P /2 transforms (88) into the damped Hamiltonian system
S ˙ t = ∂ Π H l i n ( S t , Π t ) , Π ˙ t = − ∂ S H l i n ( S t , Π t ) − α t Π t . \dot{S}_t=\partial_{\Pi}\mathcal{H}^{\mathrm{lin}}(S_t,\Pi_t),\qquad\dot{\Pi}_t=-\partial_{S}\mathcal{H}^{\mathrm{lin}}(S_t,\Pi_t)-\alpha_t\Pi_t. S ˙ t = ∂ Π H lin ( S t , Π t ) , Π ˙ t = − ∂ S H lin ( S t , Π t ) − α t Π t .
The proof of Lemma 7.16 is given in Appendix G.2. We close this paragraph by expressing (88) as a damped Newton equation for the metric g l i n g^{\mathrm{lin}} g lin in (80). We denote its Levi–Civita connection by ∇ g l i n \nabla^{g^{\mathrm{lin}}} ∇ g lin .
Proposition 7.17 (Newton formulation). Let ( S , P ) (S,P) ( S , P ) be a solution of (88). Then S ∈ C 2 S \in C^2 S ∈ C 2 and
∇ S ˙ t g l i n S ˙ t + α t S ˙ t + grad g l i n F l i n ( S t ) = 0. (90) \nabla^{g_{\mathrm{lin}}}_{\dot{S}_t}\dot{S}_t+\alpha_t\dot{S}_t+\operatorname{grad}_{g_{\mathrm{lin}}}F^{\mathrm{lin}}(S_t)=0. \tag*{(90)} ∇ S ˙ t g lin S ˙ t + α t S ˙ t + grad g lin F lin ( S t ) = 0. (90) Conversely, every solution S S S of (90) in Sym + + ( d ) \operatorname{Sym}_{++}(d) Sym ++ ( d ) induces a solution of (88) upon setting P t = Λ S t − 1 ( S ˙ t ) P_t=\Lambda_{S_t}^{-1}(\dot{S}_t) P t = Λ S t − 1 ( S ˙ t ) .
The proof of Proposition 7.17 is given in Appendix G.2. Next, we use the viewpoint developed here to gain further insight into the dynamics of (87).
Strong solutions and exact correspondence
First, we define a suitable notion of a strong solution of (87). Then, we use the covariance formulation to establish local well-posedness for initial densities in P K A 4 \mathcal{P}_{K_A}^4 P K A 4 with quadratic initial potentials.
Let I = [ 0 , T ) I=[0,T) I = [ 0 , T ) , and let ρ 0 , Φ 0 ∈ C 1 ( R d ) \rho_0,\Phi_0\in C^1(\mathbb{R}^d) ρ 0 , Φ 0 ∈ C 1 ( R d ) , where ρ 0 \rho_0 ρ 0 is a probability density with finite second moment. A pair ( ρ , Φ ) (\rho,\Phi) ( ρ , Φ ) is called a strong solution of (87) with initial data ( ρ 0 , Φ 0 ) (\rho_0,\Phi_0) ( ρ 0 , Φ 0 ) , if the following conditions are fulfilled:
The maps t ↦ ρ t t\mapsto\rho_t t ↦ ρ t and t ↦ Φ t t\mapsto\Phi_t t ↦ Φ t belong to C ( I ; C l o c 1 ( R d ) ) ∩ W l o c 1 , 1 ( I ; C l o c 0 ( R d ) ) C(I;C^1_{\mathrm{loc}}(\mathbb{R}^d))\cap W^{1,1}_{\mathrm{loc}}(I;C^0_{\mathrm{loc}}(\mathbb{R}^d)) C ( I ; C loc 1 ( R d )) ∩ W loc 1 , 1 ( I ; C loc 0 ( R d )) .
The initial conditions hold pointwise: ρ ( 0 , x ) = ρ 0 ( x ) \rho(0,x)=\rho_0(x) ρ ( 0 , x ) = ρ 0 ( x ) and Φ ( 0 , x ) = Φ 0 ( x ) \Phi(0,x)=\Phi_0(x) Φ ( 0 , x ) = Φ 0 ( x ) for all x ∈ R d x\in\mathbb{R}^d x ∈ R d .
For almost every t ∈ I t\in I t ∈ I , the integral defining M ρ t ( Φ t ) M_{\rho_t}(\Phi_t) M ρ t ( Φ t ) in (49) is absolutely convergent, and
t ↦ M ρ t ( Φ t ) ∈ L l o c 1 ( I ; R d × d ) . t\mapsto M_{\rho_t}(\Phi_t)\in L^1_{\mathrm{loc}}(I;\mathbb{R}^{d\times d}). t ↦ M ρ t ( Φ t ) ∈ L loc 1 ( I ; R d × d ) .
For every compact set K ⊂ R d K\subset\mathbb{R}^d K ⊂ R d and almost every t ∈ I t\in I t ∈ I ,
( ∂ t ρ t + ∇ ⋅ ( ρ t M ρ t ( Φ t ) T A x ) ∂ t Φ t + α t Φ t + ∇ Φ t T M ρ t ( Φ t ) T A x ) = ( 0 1 2 x T V x ) in C ( K ) × C ( K ) . \left(
\begin{aligned}
\partial_t\rho_t+\nabla\cdot\left(\rho_tM_{\rho_t}(\Phi_t)^{\mathsf{T}}Ax\right)\\
\partial_t\Phi_t+\alpha_t\Phi_t+\nabla\Phi_t^{\mathsf{T}}M_{\rho_t}(\Phi_t)^{\mathsf{T}}Ax
\end{aligned}
\right)
=
\binom{0}{\frac{1}{2}x^{\mathsf{T}}Vx}
\quad\text{in }C(K)\times C(K). ( ∂ t ρ t + ∇ ⋅ ( ρ t M ρ t ( Φ t ) T A x ) ∂ t Φ t + α t Φ t + ∇ Φ t T M ρ t ( Φ t ) T A x ) = ( 2 1 x T V x 0 ) in C ( K ) × C ( K ) .
With this definition, we can now establish the equivalence between an ODE and PDE system, which will be essential to establish well-posedness and show that the flow preserves P K A 4 \mathcal{P}_{K_A}^4 P K A 4 .
Theorem 7.18 (Strong correspondence and maximal well-posedness). Let ρ 0 ∈ P K A 4 \rho_0\in\mathcal{P}_{K_A}^4 ρ 0 ∈ P K A 4 , with P K A 4 \mathcal{P}_{K_A}^4 P K A 4 defined in (86). Set S 0 = S ( ρ 0 ) S_0=S(\rho_0) S 0 = S ( ρ 0 ) as in (76), and choose P 0 ∈ Sym ( d ) P_0\in\operatorname{Sym}(d) P 0 ∈ Sym ( d ) with Φ 0 = Φ P 0 \Phi_0=\Phi_{P_0} Φ 0 = Φ P 0 as in (85). Assume that α ∈ C ( [ 0 , ∞ ) ; [ 0 , ∞ ) ) \alpha\in C([0,\infty);[0,\infty)) α ∈ C ([ 0 , ∞ ) ; [ 0 , ∞ )) . Then the following statements hold:
(Matrix well-posedness) The matrix system (88) has a unique maximal solution
( S , P ) ∈ C 1 ( [ 0 , T max ) ; Sym + + ( d ) × Sym ( d ) ) (S,P)\in C^1([0,T_{\max});\operatorname{Sym}_{++}(d)\times\operatorname{Sym}(d)) ( S , P ) ∈ C 1 ([ 0 , T m a x ) ; Sym ++ ( d ) × Sym ( d )) with initial values ( S 0 , P 0 ) (S_0,P_0) ( S 0 , P 0 ) .
(Construction of strong solutions) Let G G G solve
G ˙ t = P t S t A G t , G 0 = I d . (91) \dot{G}_t=P_tS_tAG_t,\qquad G_0=I_d. \tag*{(91)} G ˙ t = P t S t A G t , G 0 = I d . (91) Then
ρ t = ( G t ) # ρ 0 , Φ t ( x ) = 1 2 x T P t x , S t = G t S 0 G t T (92) \rho_t=(G_t)_{\#}\rho_0,\qquad\Phi_t(x)=\frac{1}{2}x^{\mathsf{T}}P_tx,\qquad S_t=G_tS_0G_t^{\mathsf{T}} \tag*{(92)} ρ t = ( G t ) # ρ 0 , Φ t ( x ) = 2 1 x T P t x , S t = G t S 0 G t T (92) defines a strong solution of (87) with initial data ( ρ 0 , Φ 0 ) (\rho_0,\Phi_0) ( ρ 0 , Φ 0 ) .
(Converse correspondence) Every strong solution ( ρ , Φ ) (\rho,\Phi) ( ρ , Φ ) of (87) with initial values ( ρ 0 , Φ 0 ) (\rho_0,\Phi_0) ( ρ 0 , Φ 0 ) has the form (92), and its coefficients solve (88).
(Preservation of geometric admissibility) For every t ∈ [ 0 , T max ) t \in[0,T_{\max}) t ∈ [ 0 , T m a x ) , we have
ρ t ∈ P K A 4 , ∂ t ρ t ∈ T ρ t e f f P K A 4 . \rho_t \in\mathcal{P}_{K_A}^{4}, \qquad\partial_t\rho_t \in T_{\rho_t}^{\mathrm{eff}}\mathcal{P}_{K_A}^{4}. ρ t ∈ P K A 4 , ∂ t ρ t ∈ T ρ t eff P K A 4 .
The proof of Theorem 7.18 is given in Appendix G.2.
Remark 7.19 (Relation to accelerated Stein flows). 1. For K A K_A K A and F l i n F^{\mathrm{lin}} F lin , (87) agrees algebraically with [70 Example 4.1] . Our quotient Stein construction makes this specialization rigorous, while Theorem 7.18 establishes strong well-posedness beyond Gaussian densities and allows arbitrary quadratic initial potentials.
The Gaussian kinetic terms in [50 Theorem 3.5] and [70 Theorem 5.1(4)] are recovered for A = I d A=I_d A = I d and general A A A , respectively. Their full dynamics differ because they use KL energy rather than F l i n F^{\mathrm{lin}} F lin .
Local well-posedness and continuous dependence on initial values
We start by recording the following direct consequence of Theorem 7.18.
Corollary 7.20 (Local strong well-posedness). Under the assumptions of Theorem 7.18, equation (87) has a unique maximal strong solution with initial data ( ρ 0 , Φ 0 ) (\rho_0,\Phi_0) ( ρ 0 , Φ 0 ) . Its maximal interval equals that of the matrix system (88) with initial values ( S 0 , P 0 ) (S_0,P_0) ( S 0 , P 0 ) .
Next, we study how solutions of (87) depend on their initial values in the Wasserstein metric and the local strong topologies. The following corollary is proved in Appendix G.2.
Corollary 7.21 (Wasserstein convergence and continuous dependence). Suppose ρ 0 ( n ) , ρ 0 ∈ P K A 4 \rho_0^{(n)},\rho_0 \in\mathcal{P}_{K_A}^{4} ρ 0 ( n ) , ρ 0 ∈ P K A 4 satisfy
ρ 0 ( n ) ⟶ ρ 0 in W 2 and in C l o c 1 ( R d ) , P 0 ( n ) ⟶ P 0 in ( Sym ( d ) , ∥ ⋅ ∥ F ) . \rho_0^{(n)} \longrightarrow\rho_0 \quad\text{in } W_2 \text{ and in } C_{\mathrm{loc}}^{1}(\mathbb{R}^{d}), \qquad P_0^{(n)} \longrightarrow P_0 \quad\text{in } (\operatorname{Sym}(d),\|\cdot\|_F). ρ 0 ( n ) ⟶ ρ 0 in W 2 and in C loc 1 ( R d ) , P 0 ( n ) ⟶ P 0 in ( Sym ( d ) , ∥ ⋅ ∥ F ) . For fixed α \alpha α , let ( ρ ( n ) , Φ ( n ) ) (\rho^{(n)},\Phi^{(n)}) ( ρ ( n ) , Φ ( n ) ) and ( ρ , Φ ) (\rho,\Phi) ( ρ , Φ ) be the unique maximal strong solutions of (87) with initial data ( ρ 0 ( n ) , Φ P 0 ( n ) ) (\rho_0^{(n)},\Phi_{P_0^{(n)}}) ( ρ 0 ( n ) , Φ P 0 ( n ) ) and ( ρ 0 , Φ P 0 ) (\rho_0,\Phi_{P_0}) ( ρ 0 , Φ P 0 ) , respectively. Let T max T_{\max} T m a x denote the maximal existence time of ( ρ , Φ ) (\rho,\Phi) ( ρ , Φ ) . Then, for every 0 < T < T max 0<T<T_{\max} 0 < T < T m a x , the solutions ( ρ ( n ) , Φ ( n ) ) (\rho^{(n)},\Phi^{(n)}) ( ρ ( n ) , Φ ( n ) ) exist on [ 0 , T ] [0,T] [ 0 , T ] for sufficiently large n n n , and
ρ ( n ) ⟶ ρ in C ( [ 0 , T ] ; W 2 ) ∩ C ( [ 0 , T ] ; C l o c 1 ) ∩ C 1 ( [ 0 , T ] ; C l o c 0 ) , \rho^{(n)} \longrightarrow\rho\quad\text{in } C([0,T];W_2)\cap C([0,T];C_{\mathrm{loc}}^{1})\cap C^{1}([0,T];C_{\mathrm{loc}}^{0}), ρ ( n ) ⟶ ρ in C ([ 0 , T ] ; W 2 ) ∩ C ([ 0 , T ] ; C loc 1 ) ∩ C 1 ([ 0 , T ] ; C loc 0 ) , Φ ( n ) ⟶ Φ in C ( [ 0 , T ] ; C l o c 1 ) ∩ C 1 ( [ 0 , T ] ; C l o c 0 ) . (93) \Phi^{(n)} \longrightarrow\Phi\quad\text{in } C([0,T];C_{\mathrm{loc}}^{1})\cap C^{1}([0,T];C_{\mathrm{loc}}^{0}). \tag*{(93)} Φ ( n ) ⟶ Φ in C ([ 0 , T ] ; C loc 1 ) ∩ C 1 ([ 0 , T ] ; C loc 0 ) . (93)
Next, we investigate when strong solutions of (87) can be continued. Proposition 7.22 shows that, if the maximal existence time is finite, the second moment of ρ t \rho_t ρ t must become unbounded.
Continuation, global existence, and blow-up
Next, we give a continuation criterion for (87) and identify conditions for global existence and finite-time blow-up.
Proposition 7.22 (Continuation and global existence). Under the assumptions of Theorem 7.18, let ( S , P ) (S,P) ( S , P ) be the maximal solution of (88) and ( ρ , Φ ) (\rho,\Phi) ( ρ , Φ ) the corresponding maximal strong solution of (87). Denote their common maximal existence interval by [ 0 , T max ) [0,T_{\max}) [ 0 , T m a x ) . Then the following statements hold:
If T max < ∞ T_{\max}<\infty T m a x < ∞ , then
lim sup t ↑ T max ∥ S t ∥ F = ∞ , lim sup t ↑ T max ∫ R d ∣ x ∣ 2 d ρ t ( x ) = ∞ . (94) \limsup_{t\uparrow T_{\max}}\lVert S_t\rVert_{\mathrm{F}}=\infty,\qquad\limsup_{t\uparrow T_{\max}}\int_{\mathbb{R}^d}|x|^2\,\mathrm{d}\rho_t(x)=\infty. \tag*{(94)} t ↑ T m a x lim sup ∥ S t ∥ F = ∞ , t ↑ T m a x lim sup ∫ R d ∣ x ∣ 2 d ρ t ( x ) = ∞. (94)
If V ⪯ 0 V\preceq0 V ⪯ 0 , then T max = ∞ T_{\max}=\infty T m a x = ∞ .
The proof of Proposition 7.22 is given in Appendix G.2. To identify conditions for finite-time blow-up, we now assume that P 0 = 0 P_0=0 P 0 = 0 and that A , S 0 = S ( ρ 0 ) , V A,S_0=S(\rho_0),V A , S 0 = S ( ρ 0 ) , V commute. In a common orthonormal eigenbasis, write
A = diag ( a i ) , S 0 = diag ( s i , 0 ) , V = diag ( v i ) , a i , s i , 0 > 0. A=\operatorname{diag}(a_i),\qquad S_0=\operatorname{diag}(s_{i,0}),\qquad V=\operatorname{diag}(v_i),\qquad a_i,s_{i,0}>0. A = diag ( a i ) , S 0 = diag ( s i , 0 ) , V = diag ( v i ) , a i , s i , 0 > 0.
Uniqueness for (88) implies that S t S_t S t and P t P_t P t remain diagonal in this basis. Writing S t = diag ( s i ( t ) ) S_t=\operatorname{diag}(s_i(t)) S t = diag ( s i ( t )) and P t = diag ( p i ( t ) ) P_t=\operatorname{diag}(p_i(t)) P t = diag ( p i ( t )) , the matrix system reduces to
{ s ˙ i = 2 a i p i s i 2 , s i ( 0 ) = s i , 0 , p ˙ i = − α t p i − 2 a i s i p i 2 + v i , p i ( 0 ) = 0. (95) \begin{cases}
\dot{s}_i=2a_ip_is_i^2, & s_i(0)=s_{i,0},\\
\dot{p}_i=-\alpha_tp_i-2a_is_ip_i^2+v_i, & p_i(0)=0. \tag*{(95)}
\end{cases} { s ˙ i = 2 a i p i s i 2 , p ˙ i = − α t p i − 2 a i s i p i 2 + v i , s i ( 0 ) = s i , 0 , p i ( 0 ) = 0. (95)
This allows us to state the following theorem.
Theorem 7.23 (Finite-time moment blow-up under bounded damping). Under the assumptions of Proposition 7.22, suppose additionally that P 0 = 0 P_0=0 P 0 = 0 , that A , S 0 , V A,S_0,V A , S 0 , V commute, and that α t ≤ α ˉ < ∞ \alpha_t\leq\bar{\alpha}<\infty α t ≤ α ˉ < ∞ for all t ≥ 0 t\geq0 t ≥ 0 . Let s i , p i s_i,p_i s i , p i be the solutions of (95). Then we have the following:
If some v i > 0 v_i>0 v i > 0 , then T max < ∞ T_{\max}<\infty T m a x < ∞ . At least one index i i i with v i > 0 v_i>0 v i > 0 satisfies
lim t ↑ T max s i ( t ) = ∞ , lim t ↑ T max ∫ R d ∣ x ∣ 2 d ρ t ( x ) = ∞ . \lim_{t\uparrow T_{\max}}s_i(t)=\infty,\qquad\lim_{t\uparrow T_{\max}}\int_{\mathbb{R}^d}|x|^2\,\mathrm{d}\rho_t(x)=\infty. t ↑ T m a x lim s i ( t ) = ∞ , t ↑ T m a x lim ∫ R d ∣ x ∣ 2 d ρ t ( x ) = ∞. For every index i i i with v i > 0 v_i>0 v i > 0 , the function s i s_i s i is strictly increasing on ( 0 , T max ) (0,T_{\max}) ( 0 , T m a x ) .
If all v i ≤ 0 v_i\leq0 v i ≤ 0 , then T max = ∞ T_{\max}=\infty T m a x = ∞ .
The proof of Theorem 7.23 is given in Appendix G.2. Its key step is to establish that, for v i > 0 v_i>0 v i > 0 ,
s ˙ i ≥ k s i 3 / 2 \dot{s}_i\geq ks_i^{3/2} s ˙ i ≥ k s i 3/2
for some k > 0 k>0 k > 0 once s i s_i s i is sufficiently large. This superlinear growth forces finite-time blow-up despite bounded damping.
For illustration, in the undamped case α ≡ 0 \alpha\equiv0 α ≡ 0 , the solutions of (95) can be computed explicitly. For each v i > 0 v_i>0 v i > 0 , one obtains
s i ( t ) = s i , 0 sec 2 ( a i v i s i , 0 t ) . s_i(t)=s_{i,0}\sec^2\left(\sqrt{a_iv_is_{i,0}}\,t\right). s i ( t ) = s i , 0 sec 2 ( a i v i s i , 0 t ) .
and hence
T max = min i : v i > 0 π 2 a i v i s i , 0 . T_{\max}=\min_{i:v_i>0}\frac{\pi}{2\sqrt{a_iv_is_{i,0}}}. T m a x = i : v i > 0 min 2 a i v i s i , 0 π .
Thus, the initial second moments and the eigenvalues of A A A and V V V determine the blow-up time explicitly.
Finite-Token Layers and Effective Depth
The aim of this subsection is to use continuous moment growth to predict when large token norms occur in discrete attention layers. We first relate the continuous token dynamics to the matrix system (88) , then discretize them using the explicit Euler method.
Theorem 7.26 uses this correspondence to predict the number of layers needed to reach a prescribed second-moment threshold. We then examine how the continuous blow-up time separates bounded and unbounded token growth as the discretization is refined, keeping the number of tokens fixed.
Finite-token dynamics and attention layers. We begin by fixing N ≥ d N \ge d N ≥ d , A = A T ≻ 0 A = A^{\mathsf{T}} \succ0 A = A T ≻ 0 , V = V T V = V^{\mathsf{T}} V = V T , and X 0 ∈ R N × d X_0 \in\mathbb{R}^{N \times d} X 0 ∈ R N × d with full column rank. The rows of X 0 X_0 X 0 are the initial token vectors, and
S 0 : = 1 N X 0 T X 0 ≻ 0 S_0 := \frac{1}{N}X_0^{\mathsf{T}}X_0 \succ0 S 0 := N 1 X 0 T X 0 ≻ 0
is their second-moment matrix. Note that, as is well known, for centered tokens the second-moment matrix X T X / N X^{\mathsf{T}}X/N X T X / N is also their empirical covariance matrix, analogous to the covariance parameter S S S of the centered densities in (78) . Next, we introduce a coupled system of ODEs for the token matrix X ( t ) ∈ R N × d X(t) \in\mathbb{R}^{N \times d} X ( t ) ∈ R N × d and its momentum matrix Y ( t ) ∈ R N × d Y(t) \in\mathbb{R}^{N \times d} Y ( t ) ∈ R N × d :
X ˙ = 1 N X A X T Y , X ( 0 ) = X 0 , Y ˙ = − α ( t ) Y − 1 N Y Y T X A + X V , Y ( 0 ) = 0. (96) \begin{aligned}
\dot{X} &= \frac{1}{N}XAX^{\mathsf{T}}Y, \qquad X(0)=X_0,\\
\dot{Y} &= -\alpha(t)Y-\frac{1}{N}YY^{\mathsf{T}}XA+XV, \qquad Y(0)=0. \tag*{(96)}
\end{aligned} X ˙ Y ˙ = N 1 X A X T Y , X ( 0 ) = X 0 , = − α ( t ) Y − N 1 Y Y T X A + X V , Y ( 0 ) = 0. (96)
Next, we observe that the matrix system (88) also determines the token positions and momenta in (96) . This is part of the following proposition, which follows directly from a computation, uniqueness for (96) , and the fact that an unbounded second-moment matrix forces unbounded token positions.
Proposition 7.24 (Exact finite-token representation). Let α ∈ C ( [ 0 , ∞ ) ; [ 0 , ∞ ) ) \alpha\in C([0,\infty);[0,\infty)) α ∈ C ([ 0 , ∞ ) ; [ 0 , ∞ )) , and let ( S , P ) (S,P) ( S , P ) be the maximal solution of (88) with initial values ( X 0 T X 0 / N , 0 ) (X_0^{\mathsf{T}}X_0/N,0) ( X 0 T X 0 / N , 0 ) on [ 0 , T max ) [0,T_{\max}) [ 0 , T m a x ) . Let G G G solve (91) . Then the following statements hold:
(Exact finite-token representation) The unique solution of (96) on [ 0 , T max ) [0,T_{\max}) [ 0 , T m a x ) is given by
X ( t ) = X 0 G t T , Y ( t ) = X ( t ) P t , 1 N X ( t ) T X ( t ) = S t . (97) X(t)=X_0G_t^{\mathsf{T}}, \qquad Y(t)=X(t)P_t, \qquad\frac{1}{N}X(t)^{\mathsf{T}}X(t)=S_t. \tag*{(97)} X ( t ) = X 0 G t T , Y ( t ) = X ( t ) P t , N 1 X ( t ) T X ( t ) = S t . (97) (Equality of maximal existence times) If T max < ∞ T_{\max}<\infty T m a x < ∞ and S t S_t S t is unbounded as t ↑ T max t \uparrow T_{\max} t ↑ T m a x , then T max T_{\max} T m a x is also the maximal existence time of (96) .
The proof of Proposition 7.24 is given in Appendix G.§ 2.2 . By (97) , blow-up of the second-moment matrix S t S_t S t forces unbounded token norms in (96) , even for a fixed number of tokens N N N .
Having described the continuous token dynamics, we next construct the corresponding discrete attention layers. Applying the explicit Euler method to (96) with step size h > 0 h>0 h > 0 gives
X ℓ + 1 h = X ℓ h + h N X ℓ h A ( X ℓ h ) T Y ℓ h , Y ℓ + 1 h = ( 1 − h α ( ℓ h ) ) Y ℓ h − h N Y ℓ h ( Y ℓ h ) T X ℓ h A + h X ℓ h V . (98) \begin{aligned}
X_{\ell+1}^{h} &= X_{\ell}^{h}+\frac{h}{N}X_{\ell}^{h}A(X_{\ell}^{h})^{\mathsf{T}}Y_{\ell}^{h},\\
Y_{\ell+1}^{h} &= (1-h\alpha(\ell h))Y_{\ell}^{h}-\frac{h}{N}Y_{\ell}^{h}(Y_{\ell}^{h})^{\mathsf{T}}X_{\ell}^{h}A+hX_{\ell}^{h}V. \tag*{(98)}
\end{aligned} X ℓ + 1 h Y ℓ + 1 h = X ℓ h + N h X ℓ h A ( X ℓ h ) T Y ℓ h , = ( 1 − h α ( ℓ h )) Y ℓ h − N h Y ℓ h ( Y ℓ h ) T X ℓ h A + h X ℓ h V . (98)
with initial values ( X 0 h , Y 0 h ) = ( X 0 , 0 ) (X_0^h,Y_0^h)=(X_0,0) ( X 0 h , Y 0 h ) = ( X 0 , 0 ) . Following the residual attention formulation in Section 2, (98) defines layers with shared matrices A , V A,V A , V and an additional momentum variable Y Y Y . Our analysis concerns these updates without normalization or MLP blocks.
Next, to measure how well the layers (98) approximate the solution of (96) , we introduce the notation
Z ( t ) : = ( X ( t ) , Y ( t ) ) , Z ℓ h : = ( X ℓ h , Y ℓ h ) , ∥ ( X , Y ) ∥ 2 : = ∥ X ∥ F 2 + ∥ Y ∥ F 2 . Z(t):=(X(t),Y(t)),\qquad Z_\ell^h:=(X_\ell^h,Y_\ell^h),\qquad\|(X,Y)\|^2:=\|X\|_{\mathrm{F}}^2+\|Y\|_{\mathrm{F}}^2. Z ( t ) := ( X ( t ) , Y ( t )) , Z ℓ h := ( X ℓ h , Y ℓ h ) , ∥ ( X , Y ) ∥ 2 := ∥ X ∥ F 2 + ∥ Y ∥ F 2 .
To compare their second moments, we also set
S ℓ h : = 1 N ( X ℓ h ) T X ℓ h . (99) S_\ell^h:=\frac{1}{N}(X_\ell^h)^{\mathsf{T}}X_\ell^h. \tag*{(99)} S ℓ h := N 1 ( X ℓ h ) T X ℓ h . (99)
This matrix will also allow us to identify the layers at which a prescribed moment threshold is reached. With this notation at hand, we can state the following lemma.
Lemma 7.25 (Euler approximation before blow-up). Under the assumptions of Proposition 7.24 , suppose additionally that α ∈ C 1 ( [ 0 , ∞ ) ) \alpha\in C^1([0,\infty)) α ∈ C 1 ([ 0 , ∞ )) . Let ( X , Y ) (X,Y) ( X , Y ) be the solution given by (97) , and let T max T_{\max} T m a x be the maximal existence time of ( S , P ) (S,P) ( S , P ) . Let ( X ℓ h , Y ℓ h ) (X_\ell^h,Y_\ell^h) ( X ℓ h , Y ℓ h ) be the iterates of (98) with initial values ( X 0 h , Y 0 h ) = ( X 0 , 0 ) (X_0^h,Y_0^h)=(X_0,0) ( X 0 h , Y 0 h ) = ( X 0 , 0 ) . Then, for every 0 < T < T max 0<T<T_{\max} 0 < T < T m a x , there are constants C T < ∞ C_T<\infty C T < ∞ and h T > 0 h_T>0 h T > 0 such that
max 0 ≤ ℓ h ≤ T ( ∥ Z ℓ h − Z ( ℓ h ) ∥ + ∥ S ℓ h − S ( ℓ h ) ∥ F ) ≤ C T h ( 0 < h < h T ) . (100) \max_{0\leq\ell h\leq T}\left(\|Z_\ell^h-Z(\ell h)\|+\|S_\ell^h-S(\ell h)\|_{\mathrm{F}}\right)\leq C_T h\qquad(0<h<h_T). \tag*{(100)} 0 ≤ ℓ h ≤ T max ( ∥ Z ℓ h − Z ( ℓ h ) ∥ + ∥ S ℓ h − S ( ℓ h ) ∥ F ) ≤ C T h ( 0 < h < h T ) . (100)
The proof of Lemma 7.25 is given in Appendix G.3. Next, we use the Euler error estimate (100) to relate the continuous blow-up time T max T_{\max} T m a x to the depth at which large token norms occur in the discrete attention layers.
Activation thresholds and effective depth. The aim of this paragraph is to use the continuous dynamics (96) to predict when large token norms occur in the discrete attention layers (98) . Using the Euler estimate in Lemma 7.25, we first estimate the number of layers needed to reach a prescribed second-moment threshold in Theorem 7.26. We then derive the consequences for token growth and compare networks whose effective depth lies below or above the continuous blow-up time. To this end, we first fix the assumptions and notation.
Let ( S , P ) (S,P) ( S , P ) , ( X , Y ) (X,Y) ( X , Y ) , and ( X ℓ h , Y ℓ h ) (X_\ell^h,Y_\ell^h) ( X ℓ h , Y ℓ h ) be the solutions and iterates from Lemma 7.25. Denote the maximal existence time of ( S , P ) (S,P) ( S , P ) by T max T_{\max} T m a x . Throughout this paragraph, assume additionally that A , V , S 0 A,V,S_0 A , V , S 0 commute, that V V V has a positive eigenvalue, and that 0 ≤ α t ≤ α ˉ < ∞ 0\leq\alpha_t\leq\bar{\alpha}<\infty 0 ≤ α t ≤ α ˉ < ∞ .
Since V V V has a positive eigenvalue, Theorem 7.23 gives T max < ∞ T_{\max}<\infty T m a x < ∞ . We choose the common unit eigenvector e i ∗ e_{i_*} e i ∗ provided by this theorem and, using Proposition 7.24, write its directional second moment as
s ( t ) : = e i ∗ T S t e i ∗ = 1 N ∥ X ( t ) e i ∗ ∥ 2 2 . (101) s(t):=e_{i_*}^{\mathsf{T}}S_te_{i_*}=\frac{1}{N}\|X(t)e_{i_*}\|_2^2. \tag*{(101)} s ( t ) := e i ∗ T S t e i ∗ = N 1 ∥ X ( t ) e i ∗ ∥ 2 2 . (101)
The function s s s is continuous because S S S is a C 1 C^1 C 1 solution of (88) . Its strict increase and divergence as t ↑ T max t\uparrow T_{\max} t ↑ T m a x follow from Theorem 7.23. Setting s ∗ : = s ( 0 ) s_*:=s(0) s ∗ := s ( 0 ) , we therefore obtain, for every R > s ∗ R>s_* R > s ∗ , a unique continuous crossing time t R t_R t R satisfying
s ( t R ) = R , 0 < t R < T max . (102) s(t_R)=R,\qquad0<t_R<T_{\max}. \tag*{(102)} s ( t R ) = R , 0 < t R < T m a x . (102)
For the discrete layers (98) , the directional second moment corresponding to (101) is
1 N ∥ X ℓ h e i ∗ ∥ 2 2 . \frac{1}{N}\|X_\ell^h e_{i_*}\|_2^2. N 1 ∥ X ℓ h e i ∗ ∥ 2 2 .
To determine how many layers are needed to reach the same threshold R R R , we define the first threshold layer by
L R ( h ) : = min { ℓ ∈ N 0 : 1 N ∑ i ∥ X ℓ h e i ∗ ∥ 2 2 ≥ R } , (103) L_R(h) := \min\left\{ \ell\in\mathbb{N}_0 : \frac{1}{N} \sum_i \left\|X_{\ell}^{h} e_{i_*}\right\|_2^2 \ge R \right\}, \tag*{(103)} L R ( h ) := min { ℓ ∈ N 0 : N 1 i ∑ X ℓ h e i ∗ 2 2 ≥ R } , (103)
with L R ( h ) = ∞ L_R(h)=\infty L R ( h ) = ∞ if no layer reaches R R R . Since each layer represents a time step h h h , the corresponding accumulated time is h L R ( h ) hL_R(h) h L R ( h ) . Comparing it with t R t_R t R tells us how accurately the continuous dynamics predict the first threshold layer.
Theorem 7.26 (Activation thresholds). Under the preceding assumptions, let t R t_R t R and L R ( h ) L_R(h) L R ( h ) be defined by (102) and (103).
Then for every fixed R > s ∗ R>s_* R > s ∗ , the layer index L R ( h ) L_R(h) L R ( h ) is finite for all sufficiently small h > 0 h>0 h > 0 , and
h L R ( h ) = t R + O R ( h ) . hL_R(h) = t_R + O_R(h). h L R ( h ) = t R + O R ( h ) . Thus, in particular,
lim R → ∞ lim h ↓ 0 h L R ( h ) = T max . \lim_{R\to\infty}\lim_{h\downarrow0} hL_R(h) = T_{\max}. R → ∞ lim h ↓ 0 lim h L R ( h ) = T m a x .
The proof of Theorem 7.26 is given in Appendix G.§ 2.2 . Before moving on, we explain how this result connects continuous moment growth to the depth of the discrete network.
Next, we express this conclusion in terms of individual tokens. Write ( x j , ℓ h ) ⊤ (x_{j,\ell}^{h})^{\top} ( x j , ℓ h ) ⊤ for row j j j of X ℓ h X_{\ell}^{h} X ℓ h . Reaching a sufficiently large second-moment threshold forces at least one of these tokens to have a large norm.
Corollary 7.28 (Arbitrarily large token norms). Under the assumptions of Theorem 7.26, for every M > 0 M>0 M > 0 , all sufficiently small h > 0 h>0 h > 0 admit a layer ℓ \ell ℓ with ℓ h < T max \ell h<T_{\max} ℓ h < T m a x such that
max 1 ≤ j ≤ N ∥ x j , ℓ h ∥ 2 > M . \max_{1\le j\le N} \left\|x_{j,\ell}^{h}\right\|_2 > M. 1 ≤ j ≤ N max x j , ℓ h 2 > M .
The proof of Corollary 7.28 is given in Appendix G.§ 2.2 . The following remark explains its implication for token growth in the discrete network.
We now compare networks with different numbers of layers. For L ∈ N L\in\mathbb{N} L ∈ N layers and step size h > 0 h>0 h > 0 , we call T = L h T=Lh T = L h the effective depth . Keeping T T T fixed while increasing L L L therefore means choosing h = T / L h=T/L h = T / L . To record the largest token norm throughout such a network, set
B L ( T ) : = max 0 ≤ ℓ ≤ L max 1 ≤ j ≤ N ∥ x j , ℓ T / L ∥ 2 . B_L(T) := \max_{0\le\ell\le L}\max_{1\le j\le N}\left\|x_{j,\ell}^{T/L}\right\|_2. B L ( T ) := 0 ≤ ℓ ≤ L max 1 ≤ j ≤ N max x j , ℓ T / L 2 .
Corollary 7.28 implies that, for fixed T ≥ T max T \ge T_{\max} T ≥ T m a x , these maxima diverge as the number of layers tends to infinity.
Corollary 7.30 (Critical and supercritical effective depth). Under the assumptions of Theorem 7.26 , for every fixed T ≥ T max T \ge T_{\max} T ≥ T m a x ,
lim L → ∞ B L ( T ) = ∞ . \lim_{L\to\infty} B_L(T) = \infty. L → ∞ lim B L ( T ) = ∞.
The proof of Corollary 7.30 is given in Appendix G.3. We now turn to networks whose fixed effective depth lies below T max T_{\max} T m a x . In this regime, the Euler estimate in Lemma 7.25 shows that increasing the number of layers improves the approximation while token norms remain bounded, as the following corollary states.
Corollary 7.31 (Subcritical effective depth). Under the assumptions of Theorem 7.26 , let Z ( t ) = ( X ( t ) , Y ( t ) ) Z(t) = (X(t),Y(t)) Z ( t ) = ( X ( t ) , Y ( t )) and Z ℓ h = ( X ℓ h , Y ℓ h ) Z_\ell^h = (X_\ell^h,Y_\ell^h) Z ℓ h = ( X ℓ h , Y ℓ h ) , equipped with the norm
∥ ( X , Y ) ∥ 2 = ∥ X ∥ F 2 + ∥ Y ∥ F 2 . \lVert(X,Y)\rVert^2 = \lVert X\rVert_F^2 + \lVert Y\rVert_F^2. ∥( X , Y ) ∥ 2 = ∥ X ∥ F 2 + ∥ Y ∥ F 2 . For every fixed 0 < T < T max 0 < T < T_{\max} 0 < T < T m a x , choosing h = T / L h = T/L h = T / L gives
max 0 ≤ ℓ ≤ L ∥ Z ℓ T / L − Z ( ℓ T / L ) ∥ = O ( L − 1 ) , lim sup L → ∞ B L ( T ) < ∞ . \max_{0\le\ell\le L}\left\lVert Z_\ell^{T/L} - Z(\ell T/L)\right\rVert= O(L^{-1}), \qquad\limsup_{L\to\infty} B_L(T) < \infty. 0 ≤ ℓ ≤ L max Z ℓ T / L − Z ( ℓ T / L ) = O ( L − 1 ) , L → ∞ lim sup B L ( T ) < ∞.
The proof of Corollary 7.31 is given in Appendix G.3. Before moving on, we explain what this result means for refining the discrete network.
These results lead us to ask how normalization and residual scaling affect the relation between effective depth and token growth.
Relation to normalization and depth control. Under the assumptions of Corollary 7.30, bounded nonnegative damping does not remove the critical effective-depth threshold for the unnormalized updates (98) .
Other approaches modify the architecture to control growth. DeepNorm combines modified residual connections with depth-dependent initialization to control model updates during training [83 ] . The spherical attention model in [35 Sec. 2.2] instead constrains token positions to the unit sphere, excluding token-norm blow-up by construction. Extending the present analysis to such modifications would clarify how normalization and residual scaling affect accelerated attention dynamics.
We close this section by summarizing how the continuous analysis yields information about discrete attention layers. Figure 5 shows the successive reductions. Theorem 7.26 connects the first and last stages: continuous moment growth predicts the layer at which a prescribed threshold is reached.
Part IV: Discretizations and Numerical Results
Discrete SympFormer architectures
The aim of this section is to turn the accelerated evolutions derived in Subsection 6.3 into explicit attention layers and to evaluate the performance of the resulting architectures. First, we specify the discrete updates. Then, we explain their implementation in a decoder-only transformer and compare predictive performance, computational cost, and the effect of the correction. We observe that the geometrically corrected discretizations outperform the nanoGPT2-style baseline and a different accelerated transformer architecture without geometric correction in terms of mean NLL at a fixed training-token budget in most settings, most prominently in looped architectures. However, the additional geometric correction increases computational time compared with the other methods.
Time Discretizations of Riemannian Accelerated Gradient Flows
We begin by discretizing (71). The purpose is to obtain explicit updates for the token positions and an auxiliary variable carrying information between consecutive layers.
Let h > 0 h > 0 h > 0 , set t k = k h t_k = kh t k = k h , and let U ( k ) \mathbf{U}^{(k)} U ( k ) approximate the physical velocity X ˙ ( t k ) \dot{\mathbf{X}}(t_k) X ˙ ( t k ) . We use the displacement variable
V ( k ) : = h U ( k ) . (104) \mathbf{V}^{(k)} := h\mathbf{U}^{(k)}. \tag*{(104)} V ( k ) := h U ( k ) . (104)
For a fixed-step discretization, the damping and force coefficients are β k : = 1 − h α ( t k ) \beta_k := 1 - h\alpha(t_k) β k := 1 − h α ( t k ) and γ k : = h 2 \gamma_k := h^2 γ k := h 2 .
Following the sequential velocity-discretization approach in [2 ] , we first combine the previous displacement with the attention force. We then update the position and carry the displacement to the new position. Using (74) and (75) gives
V ~ ( k + 1 ) = β k V ( k ) + γ k Attn ( X ( k ) ) , C ( k ) = Γ X ( k ) g ( V ~ ( k + 1 ) , V ~ ( k + 1 ) ) , X ( k + 1 ) = X ( k ) + V ~ ( k + 1 ) − 1 2 C ( k ) , V ( k + 1 ) = V ~ ( k + 1 ) − C ( k ) . (105) \begin{aligned}
\widetilde{\mathbf{V}}^{(k+1)} &= \beta_k \mathbf{V}^{(k)} + \gamma_k \operatorname{Attn}(\mathbf{X}^{(k)}),\\
\mathbf{C}^{(k)} &= \Gamma^{g}_{\mathbf{X}^{(k)}}\left(\widetilde{\mathbf{V}}^{(k+1)},\widetilde{\mathbf{V}}^{(k+1)}\right),\\
\mathbf{X}^{(k+1)} &= \mathbf{X}^{(k)} + \widetilde{\mathbf{V}}^{(k+1)} - \frac{1}{2}\mathbf{C}^{(k)},\\
\mathbf{V}^{(k+1)} &= \widetilde{\mathbf{V}}^{(k+1)} - \mathbf{C}^{(k)}. \tag*{(105)}
\end{aligned} V ( k + 1 ) C ( k ) X ( k + 1 ) V ( k + 1 ) = β k V ( k ) + γ k Attn ( X ( k ) ) , = Γ X ( k ) g ( V ( k + 1 ) , V ( k + 1 ) ) , = X ( k ) + V ( k + 1 ) − 2 1 C ( k ) , = V ( k + 1 ) − C ( k ) . (105)
Alternatively, we can discretize the coordinate equation (71) directly. Explicit Euler for the physical velocity, followed by a position update using the new velocity, gives
V ( k + 1 ) = β k V ( k ) + γ k Attn ( X ( k ) ) − Γ X ( k ) g ( V ( k ) , V ( k ) ) , X ( k + 1 ) = X ( k ) + V ( k + 1 ) . (106) \begin{aligned}
\mathbf{V}^{(k+1)} &= \beta_k \mathbf{V}^{(k)} + \gamma_k \operatorname{Attn}(\mathbf{X}^{(k)}) - \Gamma^{g}_{\mathbf{X}^{(k)}}\left(\mathbf{V}^{(k)},\mathbf{V}^{(k)}\right),\\
\mathbf{X}^{(k+1)} &= \mathbf{X}^{(k)} + \mathbf{V}^{(k+1)}. \tag*{(106)}
\end{aligned} V ( k + 1 ) X ( k + 1 ) = β k V ( k ) + γ k Attn ( X ( k ) ) − Γ X ( k ) g ( V ( k ) , V ( k ) ) , = X ( k ) + V ( k + 1 ) . (106)
The connection is evaluated at the previous displacement in (106) , whereas (105) uses the proposed displacement V ~ ( k + 1 ) \widetilde{\mathbf{V}}^{(k+1)} V ( k + 1 ) .
Omitting the connection correction in either scheme gives
V ( k + 1 ) = β k V ( k ) + γ k Attn ( X ( k ) ) , X ( k + 1 ) = X ( k ) + V ( k + 1 ) . \begin{aligned}
\mathbf{V}^{(k+1)} &= \beta_k \mathbf{V}^{(k)} + \gamma_k \operatorname{Attn}(\mathbf{X}^{(k)}),\\
\mathbf{X}^{(k+1)} &= \mathbf{X}^{(k)} + \mathbf{V}^{(k+1)}.
\end{aligned} V ( k + 1 ) X ( k + 1 ) = β k V ( k ) + γ k Attn ( X ( k ) ) , = X ( k ) + V ( k + 1 ) .
SympFormer Architecture
Next, we adapt (105) and (106) to a decoder-only transformer for next-token prediction. The implementation follows a causal, multi-head nanoGPT-style architecture.1 Our construction modifies the attention update and retains the MLP update of the momentum baseline.
Let X ( k ) ∈ R N × d \mathbf{X}^{(k)} \in\mathbb{R}^{N \times d} X ( k ) ∈ R N × d contain the token states entering layer k k k , and let V ( k ) ∈ R N × d \mathbf{V}^{(k)} \in\mathbb{R}^{N \times d} V ( k ) ∈ R N × d denote the auxiliary velocity stream. Additional state variables also appear in augmented neural ODEs and second-order latent models, where they serve expressivity and dynamical-modeling purposes, respectively [26 , 88 ] . The intermediate index k + 1 2 k+\frac{1}{2} k + 2 1 denotes the output of the attention update, before applying the MLP. We write Attn k \operatorname{Attn}_{k} Attn k and MLP k \operatorname{MLP}_{k} MLP k for the attention and MLP maps in layer k k k . The maps LN k A \operatorname{LN}^{A}_{k} LN k A and LN k M \operatorname{LN}^{M}_{k} LN k M are the corresponding position LayerNorms [12 ] , while LN k V \operatorname{LN}^{V}_{k} LN k V is the velocity LayerNorm shared by the two updates within that layer.
We compare our approaches with the following architectures from the literature:
Ordinary transformer baseline B. It has the layer update
X ( k + 1 2 ) = X ( k ) + Attn k ( LN k A ( X ( k ) ) ) , X ( k + 1 ) = X ( k + 1 2 ) + MLP k ( LN k M ( X ( k + 1 2 ) ) ) . (107) \begin{aligned}
\mathbf{X}^{(k+\frac{1}{2})} &= \mathbf{X}^{(k)}+\operatorname{Attn}_{k}\left(\operatorname{LN}^{A}_{k}\left(\mathbf{X}^{(k)}\right)\right),\\
\mathbf{X}^{(k+1)} &= \mathbf{X}^{(k+\frac{1}{2})}+\operatorname{MLP}_{k}\left(\operatorname{LN}^{M}_{k}\left(\mathbf{X}^{(k+\frac{1}{2})}\right)\right). \tag*{(107)}
\end{aligned} X ( k + 2 1 ) X ( k + 1 ) = X ( k ) + Attn k ( LN k A ( X ( k ) ) ) , = X ( k + 2 1 ) + MLP k ( LN k M ( X ( k + 2 1 ) ) ) . (107)
Momentum baseline with YuriiFormer Y [91 ] . Its coefficients β k A , β k M \beta^{A}_{k},\beta^{M}_{k} β k A , β k M control the previous velocity contribution, while γ k A , γ k M \gamma^{A}_{k},\gamma^{M}_{k} γ k A , γ k M scale the attention and MLP outputs. The coefficients μ k A , μ k M \mu^{A}_{k},\mu^{M}_{k} μ k A , μ k M specify the look-ahead points
Q k A : = X ( k ) + μ k A V ( k ) , Q k M : = X ( k + 1 2 ) + μ k M V ( k + 1 2 ) . \mathbf{Q}^{A}_{k}:=\mathbf{X}^{(k)}+\mu^{A}_{k}\mathbf{V}^{(k)},\qquad
\mathbf{Q}^{M}_{k}:=\mathbf{X}^{(k+\frac{1}{2})}+\mu^{M}_{k}\mathbf{V}^{(k+\frac{1}{2})}. Q k A := X ( k ) + μ k A V ( k ) , Q k M := X ( k + 2 1 ) + μ k M V ( k + 2 1 ) .
The layer update is
V ( k + 1 2 ) = LN k V ( β k A V ( k ) + γ k A Attn k ( LN k A ( Q k A ) ) ) , X ( k + 1 2 ) = X ( k ) + V ( k + 1 2 ) , V ( k + 1 ) = LN k V ( β k M V ( k + 1 2 ) + γ k M MLP k ( LN k M ( Q k M ) ) ) , X ( k + 1 ) = X ( k + 1 2 ) + V ( k + 1 ) . (108) \begin{aligned}
\mathbf{V}^{(k+\frac{1}{2})} &= \operatorname{LN}^{V}_{k}\left(\beta^{A}_{k}\mathbf{V}^{(k)}+\gamma^{A}_{k}\operatorname{Attn}_{k}\left(\operatorname{LN}^{A}_{k}\left(\mathbf{Q}^{A}_{k}\right)\right)\right),\\
\mathbf{X}^{(k+\frac{1}{2})} &= \mathbf{X}^{(k)}+\mathbf{V}^{(k+\frac{1}{2})},\\
\mathbf{V}^{(k+1)} &= \operatorname{LN}^{V}_{k}\left(\beta^{M}_{k}\mathbf{V}^{(k+\frac{1}{2})}+\gamma^{M}_{k}\operatorname{MLP}_{k}\left(\operatorname{LN}^{M}_{k}\left(\mathbf{Q}^{M}_{k}\right)\right)\right),\\
\mathbf{X}^{(k+1)} &= \mathbf{X}^{(k+\frac{1}{2})}+\mathbf{V}^{(k+1)}. \tag*{(108)}
\end{aligned} V ( k + 2 1 ) X ( k + 2 1 ) V ( k + 1 ) X ( k + 1 ) = LN k V ( β k A V ( k ) + γ k A Attn k ( LN k A ( Q k A ) ) ) , = X ( k ) + V ( k + 2 1 ) , = LN k V ( β k M V ( k + 2 1 ) + γ k M MLP k ( LN k M ( Q k M ) ) ) , = X ( k + 2 1 ) + V ( k + 1 ) . (108)
The attention and MLP updates have independently learned coefficients.
YuriiFormer variant H0 _0 0 . Setting μ k A = 0 \mu^{A}_{k}=0 μ k A = 0 yields
V ( k + 1 2 ) = LN k V ( β k A V ( k ) + γ k A Attn k ( LN k A ( X ( k ) ) ) ) , X ( k + 1 2 ) = X ( k ) + V ( k + 1 2 ) , V ( k + 1 ) = LN k V ( β k M V ( k + 1 2 ) + γ k M MLP k ( LN k M ( Q k M ) ) ) , X ( k + 1 ) = X ( k + 1 2 ) + V ( k + 1 ) . (109) \begin{aligned}
\mathbf{V}^{(k+\frac{1}{2})} &= \operatorname{LN}^{V}_{k}\left(\beta^{A}_{k}\mathbf{V}^{(k)}+\gamma^{A}_{k}\operatorname{Attn}_{k}\left(\operatorname{LN}^{A}_{k}\left(\mathbf{X}^{(k)}\right)\right)\right),\\
\mathbf{X}^{(k+\frac{1}{2})} &= \mathbf{X}^{(k)}+\mathbf{V}^{(k+\frac{1}{2})},\\
\mathbf{V}^{(k+1)} &= \operatorname{LN}^{V}_{k}\left(\beta^{M}_{k}\mathbf{V}^{(k+\frac{1}{2})}+\gamma^{M}_{k}\operatorname{MLP}_{k}\left(\operatorname{LN}^{M}_{k}\left(\mathbf{Q}^{M}_{k}\right)\right)\right),\\
\mathbf{X}^{(k+1)} &= \mathbf{X}^{(k+\frac{1}{2})}+\mathbf{V}^{(k+1)}. \tag*{(109)}
\end{aligned} V ( k + 2 1 ) X ( k + 2 1 ) V ( k + 1 ) X ( k + 1 ) = LN k V ( β k A V ( k ) + γ k A Attn k ( LN k A ( X ( k ) ) ) ) , = X ( k ) + V ( k + 2 1 ) , = LN k V ( β k M V ( k + 2 1 ) + γ k M MLP k ( LN k M ( Q k M ) ) ) , = X ( k + 2 1 ) + V ( k + 1 ) . (109)
Now, we introduce the two SympFormer architectures based on the previous discussion. Both replace the exact connection in (105) by (113) , as detailed in Subsection 8.3.1 : The implementation is available at https://github.com/ViktorAJStein/SympFormer.
4. Coordinate SympFormer H1 _1 1 . Adapting the coordinate variant (106) yields the following H1 _1 1 -architecture:
V ( k + 1 2 ) = LN k V ( β k A V ( k ) + γ k A Attn k ( LN k A ( X ( k ) ) ) − Γ ^ LN k A ( X ( k ) ) ( V ( k ) , V ( k ) ) ) , X ( k + 1 2 ) = X ( k ) + V ( k + 1 2 ) . (110) \begin{aligned}
\mathbf{V}^{(k+\frac{1}{2})} &= \operatorname{LN}_{k}^{V}\left(\beta_{k}^{A}\mathbf{V}^{(k)}+\gamma_{k}^{A}\operatorname{Attn}_{k}\left(\operatorname{LN}_{k}^{A}(\mathbf{X}^{(k)})\right)-\widehat{\Gamma}_{\operatorname{LN}_{k}^{A}(\mathbf{X}^{(k)})}\left(\mathbf{V}^{(k)},\mathbf{V}^{(k)}\right)\right),\\
\mathbf{X}^{(k+\frac{1}{2})} &= \mathbf{X}^{(k)}+\mathbf{V}^{(k+\frac{1}{2})}. \tag*{(110)}
\end{aligned} V ( k + 2 1 ) X ( k + 2 1 ) = LN k V ( β k A V ( k ) + γ k A Attn k ( LN k A ( X ( k ) ) ) − Γ LN k A ( X ( k ) ) ( V ( k ) , V ( k ) ) ) , = X ( k ) + V ( k + 2 1 ) . (110)
5. R2 _2 2 SympFormer R2 _2 2 . For the R2 _2 2 variant, we apply velocity LayerNorm to both corrected displacements:
V ~ ( k + 1 2 ) = β k A V ( k ) + γ k A Attn k ( LN k A ( X ( k ) ) ) , C ( k ) = Γ ^ LN k A ( X ( k ) ) ( V ~ ( k + 1 2 ) , V ~ ( k + 1 2 ) ) , X ( k + 1 2 ) = X ( k ) + LN k V ( V ~ ( k + 1 2 ) − 1 2 C ( k ) ) , V ( k + 1 2 ) = LN k V ( V ~ ( k + 1 2 ) − C ( k ) ) . (111) \begin{aligned}
\widetilde{\mathbf{V}}^{(k+\frac{1}{2})} &= \beta_{k}^{A}\mathbf{V}^{(k)}+\gamma_{k}^{A}\operatorname{Attn}_{k}\left(\operatorname{LN}_{k}^{A}(\mathbf{X}^{(k)})\right),\\
\mathbf{C}^{(k)} &= \widehat{\Gamma}_{\operatorname{LN}_{k}^{A}(\mathbf{X}^{(k)})}\left(\widetilde{\mathbf{V}}^{(k+\frac{1}{2})},\widetilde{\mathbf{V}}^{(k+\frac{1}{2})}\right),\\
\mathbf{X}^{(k+\frac{1}{2})} &= \mathbf{X}^{(k)}+\operatorname{LN}_{k}^{V}\left(\widetilde{\mathbf{V}}^{(k+\frac{1}{2})}-\frac{1}{2}\mathbf{C}^{(k)}\right),\\
\mathbf{V}^{(k+\frac{1}{2})} &= \operatorname{LN}_{k}^{V}\left(\widetilde{\mathbf{V}}^{(k+\frac{1}{2})}-\mathbf{C}^{(k)}\right). \tag*{(111)}
\end{aligned} V ( k + 2 1 ) C ( k ) X ( k + 2 1 ) V ( k + 2 1 ) = β k A V ( k ) + γ k A Attn k ( LN k A ( X ( k ) ) ) , = Γ LN k A ( X ( k ) ) ( V ( k + 2 1 ) , V ( k + 2 1 ) ) , = X ( k ) + LN k V ( V ( k + 2 1 ) − 2 1 C ( k ) ) , = LN k V ( V ( k + 2 1 ) − C ( k ) ) . (111)
6. Look-ahead SympFormers H1 L A _1^{\mathrm{LA}} 1 LA and R2 L A _2^{\mathrm{LA}} 2 LA . These architectures evaluate the attention force and connection proxy at
Z = LN k A ( X ( k ) + μ k A V ( k ) ) . \mathbf{Z}=\operatorname{LN}_{k}^{A}\left(\mathbf{X}^{(k)}+\mu_{k}^{A}\mathbf{V}^{(k)}\right). Z = LN k A ( X ( k ) + μ k A V ( k ) ) .
All SympFormer variants use the MLP update from (108). Setting the connection correction to zero in H0 _0 0 or R2 _2 2 recovers the attention update of H0 _0 0 . Table 1 summarizes the key differences between the architectures we consider.
Table 1. Architectures for comparison.
The H1 _1 1 and R2 _2 2 architectures differ from the fixed-step discretizations in subsection 8.1 in several respects. First, both replace the exact Levi–Civita connection by the computable proxy Γ ^ \widehat{\Gamma} Γ defined below. Second, LayerNorm alters the updates and the displacement scaling in (104). In particular, the proxy does not include derivatives of LayerNorm. Finally, causal multi-head attention, learned layer-dependent update coefficients, and an additional MLP momentum step replace the fixed-force, fixed-step model analyzed above. Thus, the experiments test the usefulness of a geometrically motivated correction within a trained architecture, not the numerical accuracy of a solver for the Newton equation. Training of the parameters of the architectures minimizes next-token negative log-likelihood (NLL), not the interaction energy appearing in the dynamical analysis.
Causal Multi-Head Realization and Connection Proxy
To introduce the geometric correction Γ \Gamma Γ in (106) and (105) , we adapt the true expression (73) to the attention weights used by the model. We start with the softmax geometry.
The exact formula concerns the symmetric particle metric developed above. For causal multi-head attention, we retain its algebraic form, set the quadratic form to B = I d B=I_d B = I d , and obtain a connection proxy Γ ^ \widehat{\Gamma} Γ .
We define this proxy for one attention block, suppressing the layer index. Let Z Z Z have rows z i T z_i^{\mathsf{T}} z i T , and let U U U have rows u i T u_i^{\mathsf{T}} u i T . Here Z Z Z is the normalized token matrix at which attention is evaluated, and U U U is the direction supplied by the update. For head r ∈ { 1 , … , H } r\in\{1,\ldots,H\} r ∈ { 1 , … , H } , define
q i ( r ) = W Q ( r ) z i , k j ( r ) = W K ( r ) z j , w j ( r ) = W V ( r ) z j . q_i^{(r)}=W_Q^{(r)}z_i,\qquad k_j^{(r)}=W_K^{(r)}z_j,\qquad w_j^{(r)}=W_V^{(r)}z_j. q i ( r ) = W Q ( r ) z i , k j ( r ) = W K ( r ) z j , w j ( r ) = W V ( r ) z j .
Writing d h d_h d h for the head dimension, the scores and causal attention weights are
ℓ i j ( r ) = ⟨ q i ( r ) , k j ( r ) ⟩ d h , P i j ( r ) = exp ( ℓ i j ( r ) ) ∑ m ≤ i exp ( ℓ i m ( r ) ) , j ≤ i . (112) \ell_{ij}^{(r)}=\frac{\left\langle q_i^{(r)},k_j^{(r)}\right\rangle}{\sqrt{d_h}},\qquad P_{ij}^{(r)}=\frac{\exp\left(\ell_{ij}^{(r)}\right)}{\sum_{m\leq i}\exp\left(\ell_{im}^{(r)}\right)},\qquad j\leq i. \tag*{(112)} ℓ ij ( r ) = d h ⟨ q i ( r ) , k j ( r ) ⟩ , P ij ( r ) = ∑ m ≤ i exp ( ℓ im ( r ) ) exp ( ℓ ij ( r ) ) , j ≤ i . (112)
The directional score derivative is
ℓ ˙ i j ( r ) [ U ] = ⟨ W Q ( r ) u i , k j ( r ) ⟩ + ⟨ q i ( r ) , W K ( r ) u j ⟩ d h . \dot{\ell}_{ij}^{(r)}[U]=\frac{\left\langle W_Q^{(r)}u_i,k_j^{(r)}\right\rangle+\left\langle q_i^{(r)},W_K^{(r)}u_j\right\rangle}{\sqrt{d_h}}. ℓ ˙ ij ( r ) [ U ] = d h ⟨ W Q ( r ) u i , k j ( r ) ⟩ + ⟨ q i ( r ) , W K ( r ) u j ⟩ .
We then set
a i ( U ) : = 1 H ∑ r = 1 H ∑ j ≤ i P i j ( r ) ℓ ˙ i j ( r ) [ U ] , s i ( U ) : = ∥ u i ∥ 2 2 . a_i(U):=\frac{1}{H}\sum_{r=1}^{H}\sum_{j\leq i}P_{ij}^{(r)}\dot{\ell}_{ij}^{(r)}[U],\qquad s_i(U):=\lVert u_i\rVert_2^2. a i ( U ) := H 1 r = 1 ∑ H j ≤ i ∑ P ij ( r ) ℓ ˙ ij ( r ) [ U ] , s i ( U ) := ∥ u i ∥ 2 2 .
For the output projection W O W_O W O , the ordinary attention output and the additional weighted value aggregation are
Attn i ( Z ) = W O Concat r = 1 H ( ∑ j ≤ i P i j ( r ) w j ( r ) ) , \operatorname{Attn}_i(Z)=W_O\operatorname{Concat}_{r=1}^{H}\left(\sum_{j\leq i}P_{ij}^{(r)}w_j^{(r)}\right), Attn i ( Z ) = W O Concat r = 1 H ( j ≤ i ∑ P ij ( r ) w j ( r ) ) ,
Attn i ( s ) ( Z , U ) : = W O Concat r = 1 H ( ∑ j ≤ i P i j ( r ) s j ( U ) w j ( r ) ) . \operatorname{Attn}_i^{(s)}(Z,U):=W_O\operatorname{Concat}_{r=1}^{H}\left(\sum_{j\leq i}P_{ij}^{(r)}s_j(U)w_j^{(r)}\right). Attn i ( s ) ( Z , U ) := W O Concat r = 1 H ( j ≤ i ∑ P ij ( r ) s j ( U ) w j ( r ) ) .
With this notation, the proxy is
Γ ^ Z ( U , U ) i = a i ( U ) u i − 1 2 ( s i ( U ) Attn i ( Z ) + Attn i ( s ) ( Z , U ) ) . (113) \widehat{\Gamma}_Z(U,U)_i=a_i(U)u_i-\frac{1}{2}\left(s_i(U)\operatorname{Attn}_i(Z)+\operatorname{Attn}_i^{(s)}(Z,U)\right). \tag*{(113)} Γ Z ( U , U ) i = a i ( U ) u i − 2 1 ( s i ( U ) Attn i ( Z ) + Attn i ( s ) ( Z , U ) ) . (113)
All terms use the weights in (112) . The additional computation includes the directional projections and score contractions, together with the weighted value aggregation. Although no further softmax evaluation or matrix inverse is required, these operations incur additional training cost, which we report below.
The proxy for the connection for the other types of attention evaluated below is described on the implementation’s GitHub page https://github.com/ViktorAJStein/SympFormer.
Numerical Results
We evaluate the models on TinyStories (TS) and OpenWebText (OWT) to address whether the connection-inspired correction improves next-token prediction compared to the baseline and an otherwise matched momentum architecture without this correction. These experiments probe the practical relevance of the geometric derivation. We report validation NLL (lower is better), together with computational cost. Throughout, bold marks the lowest observed mean within each dataset and layout.
Softmax Attention
The following softmax comparison uses four layers, four heads, width 64, context length 128, and 99,991,552 training tokens. We report five paired seeds. Table 2 gives the final validation NLL and computational cost. Implementation details are given in Appendix H.
Table 2. Final validation NLL for softmax attention, together with parameter counts, reported GPU-hours per run, and peak allocated GPU memory. NLL values are means and sample standard deviations over five paired seeds.
The R 2 R_2 R 2 variant has the lowest mean NLL on both datasets in this comparison. Comparing H 1 H_1 H 1 and R 2 R_2 R 2 with the correction-off control H 0 H_0 H 0 examines the added value of the connection proxy. The differences in mean NLL are small relative to the reported seed-to-seed variability, and Table 2 alone does not establish a statistically significant advantage.
Table 2 shows that the geometry-corrected models have the same parameter count as H 0 H_0 H 0 , but require more computation. The higher parameter count comes primarily from the additional learned velocity embeddings, which add d ( T + V ) d(T + V) d ( T + V ) parameters, where d d d is the embedding dimension, V V V is the vocabulary size, and T T T is the maximal context length.
Parameter Sharing Across Depth
In Table 3 , we compare H 1 H_1 H 1 and R 2 R_2 R 2 with their matched correction-off control H 0 H_0 H 0 and the native residual baseline B when parameters are shared across depth, as in looped transformers [22 , 32 , 46 , 87 ] . The untied model uses eight independently parameterized blocks. The tied model reuses two distinct trainable blocks four times. Both execute eight logical block applications. The token-state stream and the velocity stream, when present, continue across repetitions without resets.
Table 3. Final validation NLL for untied and tied softmax models. Values are means (sample SD) over ten paired seeds, after 99,991,552 tokens, with eight heads, width 256, and context length 512.
Both H 1 H_1 H 1 and R 2 R_2 R 2 have lower mean NLL than the matched control H 0 H_0 H 0 in each layout and on both datasets. Thus the correction-control improvement is also observed when the same trainable blocks are applied repeatedly. Native-baseline utility is more conditional: H 1 H_1 H 1 has the lowest mean NLL in both tied comparisons, whereas the untied minima are B on TinyStories and R 2 R_2 R 2 on OpenWebText. Both corrected variants require more GPU time than H 0 H_0 H 0 .
Linear Attention & Other Attention Mechanisms
Next, we investigate linear attention and normalized-sigmoid and native-L 2 L^2 L 2 attention. The native-L 2 L^2 L 2 results use an SPD tensor proxy. Table 4 reports prediction quality and computational cost using three paired seeds, four layers, four heads, width 64, and context length 512, and 99,991,552 training tokens.
Table 4. Final validation NLL for the linear, normalized-sigmoid, and native-L2 SPD-proxy attention comparisons, together with parameter counts, mean allocated GPU-hours per run, and maximum peak allocated memory over seeds.
The results depend on the attention mechanism and dataset. R 2 R_2 R 2 has the lowest mean NLL for linear attention on both datasets and for native-L 2 L^2 L 2 attention on OpenWebText, whereas other methods have the lowest means in the remaining comparisons. In particular, the geometric correction is not uniformly beneficial. Its additional GPU-hour and memory costs must be considered alongside the observed NLL differences.
Related results
From continuum blow-up to discrete depth. The authors of [16 ] establish finite-time blow-up for certain Gaussian solutions of continuum attention equations. For the unnormalized, shared-weight inertial linear-attention model considered here, Theorem 7.26 predicts the first layer reaching a prescribed second-moment threshold, up to a bounded error in the layer index as the step size decreases. To the best of our knowledge, this is the first quantitative connection between continuum moment blow-up and threshold-crossing depth in discrete attention networks with a fixed number of tokens. Corollary 7.28 translates this prediction into arbitrarily large token norms. Moreover, the continuous blow-up time determines a critical effective depth, defined as the number of layers multiplied by the step size. Under refinement at fixed effective depth, the maximum token norm over all layers remains bounded below this critical value and diverges at or above it (Corollaries 7.31 and 7.30 ).
Geometry and differential calculus on density spaces. In [65 ] , Otto introduced a formal Riemannian calculus for density evolutions on R d \mathbb{R}^{d} R d , interpreting diffusion equations as gradient flows in Wasserstein geometry; see also [38 ] . Generalizations based on non-linear mobilities were developed in [15 , 23 , 47 ] . Hamiltonian forms on Wasserstein space are introduced in [31 ] . Our local-mobility framework specifies the analytic domains required by Otto–Wasserstein and softmax–Wasserstein geometry for smooth densities with potentially unbounded support. Theorem 4.3 identifies weighted function spaces on which the mobility operator is invertible, and Corollary 4.4 establishes the induced metric.
For the geometry associated with Stein variational gradient descent [49 ] , [25 ] and [64 ] construct restricted tangent and cotangent spaces assuming the kernel is integrally strictly positive definite. The bilinear kernel of linear attention falls outside that setting. Lemma 5.2 characterizes the infinite-dimensional kernel of its mobility operator, leading to the quotient construction in Theorem 5.3 . Proposition A.1 addresses a separate issue: even an integrally strictly positive definite kernel need not yield surjectivity onto the full smooth tangent space. This obstruction concerns that larger space, rather than the restricted spaces constructed in [25 ] .
In both settings, we develop a differential calculus without assuming that the admissible density classes form smooth manifolds. Lemmas B.6 and C.4 identify admissible representatives of Hamiltonian variations; the calculus is developed in Appendices B and C.
Inertial dynamics of probability distributions. Acceleration on spaces of probability measures has been studied through optimal-control formulations [73 ] , formal Hamiltonian equations on density spaces [84 ] , and phase-space and Hilbert-space approaches [20 , 75 ] . The formal density-space construction requires an invertible metric operator and meaningful variations; the frameworks developed here supply precise conditions for these operations in the attention setting.
For bilinear Stein kernels, Liu et al. [50 ] and Stein and Li [70 ] derive and analyze Gaussian covariance equations, including inertial dynamics, for sampling problems governed by relative entropy. Our inertial linear-attention equations follow the same formal Hamiltonian construction with the attention energy in place of relative entropy. Lemma 7.16 and Proposition 7.17 establish their Hamiltonian and Newton formulations. Theorem 7.18 proves strong well-posedness for general admissible initial densities and quadratic initial potentials. The resulting matrix correspondence enables the continuation and global-existence analysis in Proposition 7.22 and the finite-time moment blow-up result under bounded damping in Theorem 7.23 .
For softmax attention, Theorem 6.7 identifies the additional domain and compatibility conditions required for regular Hamiltonian derivatives. Both this result and the first-order gradient identity are statements at admissible states; constructing trajectories and proving preservation of their domains require a separate analysis.
Momentum and the construction of attention layers. Dynamical interpretations have guided neural architectures through numerical discretizations [51 ] and modified attention normalizations [68 ] . Momentum has been incorporated into residual networks [67 ] , heavy-ball and Nesterov neural ODEs [62 , 86 ] , and linear transformers [63 , 82 ] . Zimin et al. [91 ] construct YuriiFormer from a variational interpretation of attention and discrete Nesterov-style updates. Our contribution concerns the additional corrections prescribed by the token-dependent attention metric. Lemma 6.11 identifies the connection correction used in the position and momentum updates (74) and (75). Comparisons with matched momentum controls in Table 2 and Table 4 assess their predictive benefit and computational cost.
Declaration of generative AI use During the research and preparation of this manuscript, the authors used OpenAI’s ChatGPT and Codex to assist with language and LaTeX editing, software implementation and debugging, and exploratory discussions of mathematical formulations and numerical methods. AI tools also assisted with the organization and presentation of computational results. All AI-assisted material incorporated into the manuscript or the associated code was reviewed and, where necessary, revised by the authors. The authors are responsible for the mathematical statements and arguments, the validation of the numerical experiments, the interpretation of the results, and the conclusions of the work.
Acknowledgments. The paper was started during a visit of W. Li to the TU Berlin, supported by the SPP 2298 “Theoretical Foundations of Deep Learning” in Summer 2025. W. Li is supported by the AFOSR YIP award No. FA9550-23-1-0087, NSF RTG: 2038080, NSF FRG: 2245097, NSF DMS: 2601952, and the McCausland Faculty Fellowship at the University of South Carolina. GS acknowledges funding from the DFG STE 571/23-1 and as a CNRS fellow ambassador. V.S. thanks T. Laux for inviting him for a research stay in Heidelberg in May 2026 and acknowledges fruitful conversations with J. Chemseddine and R. Duong. L.M. gratefully acknowledges support from the Institute for Advanced Study and an anonymous donor during the 2026–2027 academic year, as well as funding from the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 281869850 (RTG 2229), 390900948 (EXC-2181/1), and 281071066 (TRR 191).
V.S. acknowledges funding from the European Research Council (ERC) under the European Union’s Horizon Europe research and innovation programme (grant agreement No. 101198055, project acronym NEITALG).
Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor any of the aforementioned granting authorities can be held responsible for them.
Non-Surjectivity of Stein’s Operator on Compact Domains
This subsection aims to prove that, even on the torus, the Stein operator is not surjective. Thus, in general, there is no hope that the Stein calculus performed, for example, in [84 ] can be made rigorous on the full smooth tangent space. For more general closed manifolds, additional issues arise, as already mentioned. In the following, let T d : = R d / Z d \mathbb{T}^{d} := \mathbb{R}^{d}/\mathbb{Z}^{d} T d := R d / Z d be the flat torus, equipped with normalized Lebesgue measure d x dx d x . Recall that the space of probability densities P ∞ ( T d ) \mathcal{P}^{\infty}(\mathbb{T}^{d}) P ∞ ( T d ) on T d \mathbb{T}^{d} T d is a smooth Fréchet manifold modeled on C ∞ ( T d ) C^{\infty}(\mathbb{T}^{d}) C ∞ ( T d ) , with corresponding tangent space T ρ P ∞ ( T d ) = C 0 ∞ ( T d ) T_{\rho}\mathcal{P}^{\infty}(\mathbb{T}^{d}) = C^{\infty}_{0}(\mathbb{T}^{d}) T ρ P ∞ ( T d ) = C 0 ∞ ( T d ) and smooth dual C ∞ ( T d ) / R C^{\infty}(\mathbb{T}^{d})/\mathbb{R} C ∞ ( T d ) / R , which is the cotangent space T ρ ∗ P ∞ ( T d ) T_{\rho}^{*}\mathcal{P}^{\infty}(\mathbb{T}^{d}) T ρ ∗ P ∞ ( T d ) . Let k : T d × T d → R k:\mathbb{T}^{d} \times\mathbb{T}^{d} \to\mathbb{R} k : T d × T d → R be a smooth, symmetric, positive-definite kernel, and let H k \mathcal{H}_{k} H k denote its reproducing kernel Hilbert space. Assume that k k k is integrally strictly positive definite , i.e.,
∫ T d ∫ T d k ( x , y ) d μ ( x ) d μ ( y ) > 0 \int_{\mathbb{T}^{d}}\int_{\mathbb{T}^{d}} k(x,y)\,d\mu(x)\,d\mu(y) > 0 ∫ T d ∫ T d k ( x , y ) d μ ( x ) d μ ( y ) > 0
for every nonzero finite signed Borel measure μ \mu μ on T d \mathbb{T}^{d} T d . This gives rise to the Stein operator
L ρ S : T ρ ∗ P ∞ ( T d ) ⟶ T ρ P ∞ ( T d ) , L ρ S φ : = − ∇ x ⋅ ( ρ ( x ) ∫ T d k ( x , y ) ∇ y φ ( y ) ρ ( y ) d y ) . (114) L_{\rho}^{S}:T_{\rho}^{*}\mathcal{P}^{\infty}(\mathbb{T}^{d}) \longrightarrow T_{\rho}\mathcal{P}^{\infty}(\mathbb{T}^{d}), \qquad L_{\rho}^{S}\varphi:= -\nabla_{x}\cdot\left(\rho(x)\int_{\mathbb{T}^{d}} k(x,y)\nabla_{y}\varphi(y)\rho(y)\,dy\right). \tag*{(114)} L ρ S : T ρ ∗ P ∞ ( T d ) ⟶ T ρ P ∞ ( T d ) , L ρ S φ := − ∇ x ⋅ ( ρ ( x ) ∫ T d k ( x , y ) ∇ y φ ( y ) ρ ( y ) d y ) . (114)
Its well-definedness follows from Stokes’ theorem and the fact that T d \mathbb{T}^{d} T d has no boundary.
Proposition A.1 (Non-surjectivity of the smooth Stein operator). Let k ∈ C ∞ ( T d × T d ) k \in C^{\infty}(\mathbb{T}^{d}\times\mathbb{T}^{d}) k ∈ C ∞ ( T d × T d ) be symmetric and integrally strictly positive definite, and let ρ ∈ P ∞ ( T d ) \rho\in\mathcal{P}^{\infty}(\mathbb{T}^{d}) ρ ∈ P ∞ ( T d ) . Then, the Stein operator L ρ S L_{\rho}^{S} L ρ S in (114) is injective, but not surjective.
Before delving into the proof, we highlight two implications of Proposition A.1:
Proof of Proposition A.1. We use in the following identifications T ρ P ∞ ( T d ) ≃ C 0 ∞ ( T d ) T_{\rho}\mathcal{P}^{\infty}(\mathbb{T}^{d}) \simeq C^{\infty}_{0}(\mathbb{T}^{d}) T ρ P ∞ ( T d ) ≃ C 0 ∞ ( T d ) and T ρ ∗ P ∞ ( T d ) ≃ C ∞ ( T d ) / R T_{\rho}^{*}\mathcal{P}^{\infty}(\mathbb{T}^{d}) \simeq C^{\infty}(\mathbb{T}^{d})/\mathbb{R} T ρ ∗ P ∞ ( T d ) ≃ C ∞ ( T d ) / R . Endow C 0 ∞ ( T d ) C^{\infty}_{0}(\mathbb{T}^{d}) C 0 ∞ ( T d ) with its standard Fréchet topology. For φ ∈ C ∞ ( T d ) \varphi\in C^{\infty}(\mathbb{T}^{d}) φ ∈ C ∞ ( T d ) , set
P 0 φ : = φ − 1 vol ( T d ) ∫ T d φ d x . P_{0}\varphi:= \varphi- \frac{1}{\operatorname{vol}(\mathbb{T}^{d})}\int_{\mathbb{T}^{d}} \varphi\,dx. P 0 φ := φ − vol ( T d ) 1 ∫ T d φ d x .
Every class [ φ ] ∈ C ∞ ( T d ) / R [\varphi] \in C^{\infty}(\mathbb{T}^{d})/\mathbb{R} [ φ ] ∈ C ∞ ( T d ) / R has the unique zero-mean representative P 0 φ P_{0}\varphi P 0 φ . The maps [ φ ] ↦ P 0 φ [\varphi] \mapsto P_{0}\varphi [ φ ] ↦ P 0 φ and ψ ↦ [ ψ ] \psi\mapsto[\psi] ψ ↦ [ ψ ] are continuous inverses, so they identify the quotient with C 0 ∞ ( T d ) C^{\infty}_{0}(\mathbb{T}^{d}) C 0 ∞ ( T d ) as Fréchet spaces. Since L ρ S ( φ + c ) = L ρ S φ L_\rho^S(\varphi+c)=L_\rho^S\varphi L ρ S ( φ + c ) = L ρ S φ , the quotient operator is surjective (respectively, injective) exactly when L ρ S : C 0 ∞ ( T d ) → C 0 ∞ ( T d ) L_\rho^S:C_0^\infty(\mathbb{T}^d)\to C_0^\infty(\mathbb{T}^d) L ρ S : C 0 ∞ ( T d ) → C 0 ∞ ( T d ) is. We use this identification below.
Injectivity. As k k k is integrally strictly positive definite and ρ > 0 \rho>0 ρ > 0 , it holds that L ρ S φ = 0 L_\rho^S\varphi=0 L ρ S φ = 0 if and only if φ ∈ R \varphi\in\mathbb{R} φ ∈ R . Thus, L ρ S L_\rho^S L ρ S is injective.
Non Surjectivity. Periodic integration by parts gives
L ρ S φ ( x ) = ∫ T d a ρ ( x , y ) φ ( y ) d y , a ρ ( x , y ) : = ∇ x ⋅ [ ρ ( x ) ∇ y ( k ( x , y ) ρ ( y ) ) ] . L_\rho^S\varphi(x)=\int_{\mathbb{T}^d}a_\rho(x,y)\varphi(y)\,\mathrm{d}y,\qquad a_\rho(x,y):=\nabla_x\cdot\left[\rho(x)\nabla_y\bigl(k(x,y)\rho(y)\bigr)\right]. L ρ S φ ( x ) = ∫ T d a ρ ( x , y ) φ ( y ) d y , a ρ ( x , y ) := ∇ x ⋅ [ ρ ( x ) ∇ y ( k ( x , y ) ρ ( y ) ) ] .
The kernel a ρ a_\rho a ρ is smooth; that is, a ρ ∈ C ∞ ( T d × T d ) a_\rho\in C^\infty(\mathbb{T}^d\times\mathbb{T}^d) a ρ ∈ C ∞ ( T d × T d ) . Every mixed derivative ∂ x α ∂ y β a ρ \partial_x^\alpha\partial_y^\beta a_\rho ∂ x α ∂ y β a ρ is continuous. This follows because k k k and ρ \rho ρ are smooth, and the displayed formula for a ρ a_\rho a ρ uses only products and finitely many derivatives. Since T d × T d \mathbb{T}^d\times\mathbb{T}^d T d × T d is compact, all these derivatives are bounded. For every multi-index α \alpha α and φ ∈ C ∞ ( T d ) \varphi\in C^\infty(\mathbb{T}^d) φ ∈ C ∞ ( T d ) , differentiation under the integral gives
∂ x α L ρ S φ ( x ) = ∫ T d ∂ x α a ρ ( x , y ) φ ( y ) d y . \partial_x^\alpha L_\rho^S\varphi(x)=\int_{\mathbb{T}^d}\partial_x^\alpha a_\rho(x,y)\varphi(y)\,\mathrm{d}y. ∂ x α L ρ S φ ( x ) = ∫ T d ∂ x α a ρ ( x , y ) φ ( y ) d y .
Cauchy–Schwarz in y y y , followed by integration in x x x , yields
∥ ∂ x α L ρ S φ ∥ L 2 ≤ ∥ ∂ x α a ρ ∥ L 2 ( T d × T d ) ∥ φ ∥ L 2 . \left\|\partial_x^\alpha L_\rho^S\varphi\right\|_{L^2}\leq\left\|\partial_x^\alpha a_\rho\right\|_{L^2(\mathbb{T}^d\times\mathbb{T}^d)}\|\varphi\|_{L^2}. ∂ x α L ρ S φ L 2 ≤ ∥ ∂ x α a ρ ∥ L 2 ( T d × T d ) ∥ φ ∥ L 2 .
The coefficient on the right is finite because a ρ a_\rho a ρ is smooth. Summing this estimate over ∣ α ∣ ≤ r + 1 |\alpha|\leq r+1 ∣ α ∣ ≤ r + 1 gives a constant C r C_r C r such that
∥ L ρ S φ ∥ H r + 1 ≤ C r ∥ φ ∥ L 2 . \left\|L_\rho^S\varphi\right\|_{H^{r+1}}\leq C_r\|\varphi\|_{L^2}. L ρ S φ H r + 1 ≤ C r ∥ φ ∥ L 2 .
Since L ρ S φ L_\rho^S\varphi L ρ S φ has zero mean (it is an x x x -divergence), this proves, by density, that L ρ S L_\rho^S L ρ S extends to a bounded map
L 0 2 ( T d ) ⟶ H 0 r + 1 ( T d ) , L_0^2(\mathbb{T}^d)\longrightarrow H_0^{r+1}(\mathbb{T}^d), L 0 2 ( T d ) ⟶ H 0 r + 1 ( T d ) ,
and, by the compact Sobolev embedding H 0 r + 1 ( T d ) ⋐ H 0 r ( T d ) H_0^{r+1}(\mathbb{T}^d)\Subset H_0^r(\mathbb{T}^d) H 0 r + 1 ( T d ) ⋐ H 0 r ( T d ) , it is compact as a map L 0 2 ( T d ) → H 0 r ( T d ) L_0^2(\mathbb{T}^d)\to H_0^r(\mathbb{T}^d) L 0 2 ( T d ) → H 0 r ( T d ) .
The estimates above show that L ρ S L_\rho^S L ρ S is continuous as a map from C 0 ∞ ( T d ) C_0^\infty(\mathbb{T}^d) C 0 ∞ ( T d ) to C 0 ∞ ( T d ) C_0^\infty(\mathbb{T}^d) C 0 ∞ ( T d ) . Suppose, towards a contradiction, that it were surjective. By injectivity, it would be a continuous linear bijection of the Fréchet space C 0 ∞ ( T d ) C_0^\infty(\mathbb{T}^d) C 0 ∞ ( T d ) onto itself. The Fréchet open mapping theorem applied to this map gives a continuous inverse. Applying the continuity of that inverse to the L 2 L^2 L 2 -seminorm, and using the Sobolev seminorms that generate the topology, yields r ≥ 0 r\geq0 r ≥ 0 and C > 0 C>0 C > 0 such that
∥ φ ∥ L 2 = ∥ ( L ρ S ) − 1 L ρ S φ ∥ L 2 ≤ C ∥ L ρ S φ ∥ H r for every φ ∈ C 0 ∞ ( T d ) . (115) \|\varphi\|_{L^2}=\left\|(L_\rho^S)^{-1}L_\rho^S\varphi\right\|_{L^2}\leq C\left\|L_\rho^S\varphi\right\|_{H^r}\qquad\text{for every }\varphi\in C_0^\infty(\mathbb{T}^d). \tag*{(115)} ∥ φ ∥ L 2 = ( L ρ S ) − 1 L ρ S φ L 2 ≤ C L ρ S φ H r for every φ ∈ C 0 ∞ ( T d ) . (115)
Choose an L 2 L^2 L 2 -orthonormal sequence ( φ n ) n ≥ 1 ⊂ C 0 ∞ ( T d ) (\varphi_n)_{n\geq1}\subset C_0^\infty(\mathbb{T}^d) ( φ n ) n ≥ 1 ⊂ C 0 ∞ ( T d ) , for instance φ n ( x ) = 2 cos ( 2 π n x 1 ) \varphi_n(x)=\sqrt{2}\cos(2\pi n x_1) φ n ( x ) = 2 cos ( 2 π n x 1 ) . It converges weakly to zero in L 0 2 ( T d ) L_0^2(\mathbb{T}^d) L 0 2 ( T d ) . Compactness gives ∥ L ρ S φ n ∥ H r → 0 \|L_\rho^S\varphi_n\|_{H^r}\to0 ∥ L ρ S φ n ∥ H r → 0 , contradicting (115). Thus L ρ S L_\rho^S L ρ S is not surjective.
Differentiability in the Local Mobility Calculus
Due to the lack of a differentiable structure on the space of densities in the smooth local mobility calculus developed earlier, we provide a workaround in that framework that naturally extends the definition of Fréchet differentiability in Otto–Wasserstein geometry on the space of smooth probability densities on closed manifolds.
Local first variations and the regular gradient. For ρ ∈ P m ρ , B \rho\in\mathcal{P}_{m_\rho,B} ρ ∈ P m ρ , B , we define the actual admissible variations at ρ \rho ρ by
V ρ : = { η ∈ C c ∞ ( R d ) ∩ T ρ P m ρ , B : ∫ R d η d x = 0 , ∃ ε > 0 such that ρ + t η ∈ P m ρ , B for ∣ t ∣ < ε } . (116) \mathcal{V}_\rho:= \left\{ \eta\in C^\infty_c(\mathbb{R}^d) \cap T_\rho\mathcal{P}_{m_\rho,B} : \int_{\mathbb{R}^d} \eta\,\mathrm{d}x = 0,\ \exists\varepsilon> 0\ \text{such that}\ \rho+ t\eta\in\mathcal{P}_{m_\rho,B}\ \text{for } |t| < \varepsilon\right\}. \tag*{(116)} V ρ := { η ∈ C c ∞ ( R d ) ∩ T ρ P m ρ , B : ∫ R d η d x = 0 , ∃ ε > 0 such that ρ + t η ∈ P m ρ , B for ∣ t ∣ < ε } . (116)
Then we say that F : P m ρ , B → R \mathcal{F} : \mathcal{P}_{m_\rho,B} \to\mathbb{R} F : P m ρ , B → R has a local regular first variation at ρ ∈ P m ρ , B \rho\in\mathcal{P}_{m_\rho,B} ρ ∈ P m ρ , B if there is a class [ f ρ ] ∈ T ρ ∗ P m ρ , B [f_\rho] \in T_\rho^*\mathcal{P}_{m_\rho,B} [ f ρ ] ∈ T ρ ∗ P m ρ , B such that, for every η ∈ V ρ \eta\in\mathcal{V}_\rho η ∈ V ρ ,
lim t → 0 F ( ρ + t η ) − F ( ρ ) t = ∫ R d f ρ η d x . (117) \lim_{t \to0} \frac{\mathcal{F}(\rho+t\eta)-\mathcal{F}(\rho)}{t} = \int_{\mathbb{R}^d} f_\rho\,\eta\,\mathrm{d}x. \tag*{(117)} t → 0 lim t F ( ρ + t η ) − F ( ρ ) = ∫ R d f ρ η d x . (117)
We then write [ f ρ ] = [ δ F δ ρ ( ρ ) ] [f_\rho] = \left[\frac{\delta\mathcal{F}}{\delta\rho}(\rho)\right] [ f ρ ] = [ δ ρ δ F ( ρ ) ] . Its associated regular extension is the linear map
ℓ F , ρ r e g : T ρ P m ρ , B → R , ℓ F , ρ r e g ( η ) : = ∫ R d f ρ η d x . (118) \ell^{\mathrm{reg}}_{\mathcal{F},\rho}: T_\rho\mathcal{P}_{m_\rho,B} \to\mathbb{R}, \qquad\ell^{\mathrm{reg}}_{\mathcal{F},\rho}(\eta) := \int_{\mathbb{R}^d} f_\rho\,\eta\,\mathrm{d}x. \tag*{(118)} ℓ F , ρ reg : T ρ P m ρ , B → R , ℓ F , ρ reg ( η ) := ∫ R d f ρ η d x . (118)
We first give a sufficient condition for this regular extension to be unique.
Lemma B.1 (Uniqueness of the local regular first variation). Let ρ ∈ P m ρ , B \rho\in\mathcal{P}_{m_\rho,B} ρ ∈ P m ρ , B , and suppose that F \mathcal{F} F admits a local regular first variation at ρ \rho ρ in the sense of (117). If
{ η ∈ C c ∞ ( R d ) : ∫ R d η d x = 0 } ⊆ V ρ \left\{ \eta\in C^\infty_c(\mathbb{R}^d) : \int_{\mathbb{R}^d} \eta\,\mathrm{d}x = 0 \right\} \subseteq\mathcal{V}_\rho { η ∈ C c ∞ ( R d ) : ∫ R d η d x = 0 } ⊆ V ρ where V ρ \mathcal{V}_\rho V ρ is as in (116), then the following holds:
the local regular first variation [ δ F δ ρ ( ρ ) ] \left[\frac{\delta\mathcal{F}}{\delta\rho}(\rho)\right] [ δ ρ δ F ( ρ ) ] is unique,
its associated regular extension ℓ F , ρ r e g \ell^{\mathrm{reg}}_{\mathcal{F},\rho} ℓ F , ρ reg in (118) is uniquely determined.
Proof. Clearly 2 follows directly from 1, so it suffices to prove 1. Let [ Φ ] , [ Ψ ] ∈ T ρ ∗ P m ρ , B [\Phi], [\Psi] \in T_\rho^*\mathcal{P}_{m_\rho,B} [ Φ ] , [ Ψ ] ∈ T ρ ∗ P m ρ , B both represent a local regular first variation of F \mathcal{F} F at ρ \rho ρ . By the assumption on admissible variations,
∫ R d ( Φ − Ψ ) η d x = 0 \int_{\mathbb{R}^d}(\Phi-\Psi)\eta\,\mathrm{d}x = 0 ∫ R d ( Φ − Ψ ) η d x = 0
for every η ∈ C c ∞ ( R d ) \eta\in C^\infty_c(\mathbb{R}^d) η ∈ C c ∞ ( R d ) with zero integral. For any η ∈ C c ∞ ( R d ) \eta\in C^\infty_c(\mathbb{R}^d) η ∈ C c ∞ ( R d ) and i = 1 , … , d i = 1,\ldots,d i = 1 , … , d , the function ∂ i η \partial_i\eta ∂ i η has zero integral. Hence
0 = ∫ R d ( Φ − Ψ ) ∂ i η d x = − ∫ R d ∂ i ( Φ − Ψ ) η d x . 0 = \int_{\mathbb{R}^d}(\Phi-\Psi)\partial_i\eta\,\mathrm{d}x = -\int_{\mathbb{R}^d}\partial_i(\Phi-\Psi)\eta\,\mathrm{d}x. 0 = ∫ R d ( Φ − Ψ ) ∂ i η d x = − ∫ R d ∂ i ( Φ − Ψ ) η d x .
It follows that ∇ ( Φ − Ψ ) = 0 \nabla(\Phi-\Psi)=0 ∇ ( Φ − Ψ ) = 0 , so Φ − Ψ \Phi-\Psi Φ − Ψ is constant on R d \mathbb{R}^d R d . Thus [ Φ ] = [ Ψ ] [\Phi]=[\Psi] [ Φ ] = [ Ψ ] .
For the rest of the section we assume always that the assumptions of B.1 hold. Under these assumptions we set
D F ( ρ ) [ η ] : = ℓ F , ρ r e g [ η ] ∀ η ∈ T ρ P m ρ , B . D\mathcal{F}(\rho)[\eta] := \ell^{\mathrm{reg}}_{\mathcal{F},\rho}[\eta] \qquad\forall\eta\in T_\rho\mathcal{P}_{m_\rho,B}. D F ( ρ ) [ η ] := ℓ F , ρ reg [ η ] ∀ η ∈ T ρ P m ρ , B .
For a functional F \mathcal{F} F having a local regular first variation at ρ \rho ρ in the sense of (117) we define its gradient at the point ρ \rho ρ with respect to the local mobility metric G m ρ , B G_{m_{\rho,B}} G m ρ , B as in 4.4 as the unique tangent vector grad G m ρ , B F ( ρ ) \operatorname{grad}_{G_{m_{\rho,B}}}\mathcal{F}(\rho) grad G m ρ , B F ( ρ ) so that
G m ρ , B ( grad G m ρ , B F ( ρ ) , η ˙ ) = D F ( ρ ) [ η ˙ ] ∀ η ˙ ∈ T ρ P m ρ , B (119) G_{m_{\rho,B}}\left(\operatorname{grad}_{G_{m_{\rho,B}}}\mathcal{F}(\rho),\dot{\eta}\right)=D\mathcal{F}(\rho)[\dot{\eta}]\qquad\forall\dot{\eta}\in T_{\rho}\mathcal{P}_{m_{\rho,B}} \tag*{(119)} G m ρ , B ( grad G m ρ , B F ( ρ ) , η ˙ ) = D F ( ρ ) [ η ˙ ] ∀ η ˙ ∈ T ρ P m ρ , B (119)
holds. With this notation, we can prove the following.
Lemma B.2 (Regular local-mobility gradient). If F \mathcal{F} F has a local regular first variation at ρ \rho ρ in the sense of (117), then its gradient with respect to the local mobility metric G m ρ , B G_{m_{\rho,B}} G m ρ , B as in 4.4 at the point ρ \rho ρ is given by
grad G m ρ , B F ( ρ ) = L m ρ , B [ δ F δ ρ ( ρ ) ] = − ∇ ⋅ ( m ρ , B ∇ δ F δ ρ ( ρ ) ) . \operatorname{grad}_{G_{m_{\rho,B}}}\mathcal{F}(\rho)=L_{m_{\rho,B}}\left[\frac{\delta\mathcal{F}}{\delta\rho}(\rho)\right]=-\nabla\cdot\left(m_{\rho,B}\nabla\frac{\delta\mathcal{F}}{\delta\rho}(\rho)\right). grad G m ρ , B F ( ρ ) = L m ρ , B [ δ ρ δ F ( ρ ) ] = − ∇ ⋅ ( m ρ , B ∇ δ ρ δ F ( ρ ) ) .
Proof. Let η ˙ = L m ρ , B [ Φ η ˙ ] \dot{\eta}=L_{m_{\rho,B}}[\Phi_{\dot{\eta}}] η ˙ = L m ρ , B [ Φ η ˙ ] . Then (35), (32), (118) and (119) imply
G m ρ , B ( L m ρ , B [ δ F δ ρ ( ρ ) ] , η ˙ ) = ∫ R d δ F δ ρ ( ρ ) η ˙ d x = D F ( ρ ) [ η ˙ ] . G_{m_{\rho,B}}\left(L_{m_{\rho,B}}\left[\frac{\delta\mathcal{F}}{\delta\rho}(\rho)\right],\dot{\eta}\right)=\int_{\mathbb{R}^{d}}\frac{\delta\mathcal{F}}{\delta\rho}(\rho)\dot{\eta}\,\mathrm{d}x=D\mathcal{F}(\rho)[\dot{\eta}]. G m ρ , B ( L m ρ , B [ δ ρ δ F ( ρ ) ] , η ˙ ) = ∫ R d δ ρ δ F ( ρ ) η ˙ d x = D F ( ρ ) [ η ˙ ] .
Combined with Theorem 4.3, this completes the proof. □
As shown in (21) and (22), Otto–Wasserstein and softmax–Wasserstein geometry are instances of local-mobility geometry.
Lemma B.3. For both geometries, the Otto–Wasserstein and softmax–Wasserstein geometries, the assumption on admissible variations in Lemma B.1 is satisfied. Consequently, the conclusions of Lemma B.1 hold.
Proof. Let ρ ∈ P m ρ , B \rho\in\mathcal{P}_{m_{\rho,B}} ρ ∈ P m ρ , B and η ˙ ∈ C c ∞ ( R d ) \dot{\eta}\in C^{\infty}_{c}(\mathbb{R}^{d}) η ˙ ∈ C c ∞ ( R d ) satisfy ∫ R d η ˙ d x = 0 \int_{\mathbb{R}^{d}}\dot{\eta}\,\mathrm{d}x=0 ∫ R d η ˙ d x = 0 . Since η ˙ / ρ \dot{\eta}/\rho η ˙ / ρ is bounded, for sufficiently small ∣ t ∣ |t| ∣ t ∣ ,
1 2 ρ ≤ ρ + t η ˙ ≤ 3 2 ρ . \frac{1}{2}\rho\leq\rho+t\dot{\eta}\leq\frac{3}{2}\rho. 2 1 ρ ≤ ρ + t η ˙ ≤ 2 3 ρ .
For Otto mobility, these bounds apply directly to the mobility. For softmax mobility, multiplying by exp ( z T A x ) > 0 \exp(z^{\mathsf{T}}Ax)>0 exp ( z T A x ) > 0 and integrating in z z z gives
1 2 Z ρ ≤ Z ρ + t η ˙ ≤ 3 2 Z ρ , 1 3 m ρ ≤ m ρ + t η ˙ ≤ 3 m ρ . \frac{1}{2}Z_{\rho}\leq Z_{\rho+t\dot{\eta}}\leq\frac{3}{2}Z_{\rho},\qquad\frac{1}{3}m_{\rho}\leq m_{\rho+t\dot{\eta}}\leq3m_{\rho}. 2 1 Z ρ ≤ Z ρ + t η ˙ ≤ 2 3 Z ρ , 3 1 m ρ ≤ m ρ + t η ˙ ≤ 3 m ρ .
Moreover, Z ρ + t η ˙ − Z ρ Z_{\rho+t\dot{\eta}}-Z_{\rho} Z ρ + t η ˙ − Z ρ is smooth by compact support of η ˙ \dot{\eta} η ˙ . Thus the perturbed mobility remains smooth and positive, with finite mass.
Uniform comparison also preserves the weighted Poincaré inequality:
inf c ∈ R ∫ R d ∣ Φ − c ∣ 2 d m ρ + t η ˙ ≤ 3 inf c ∈ R ∫ R d ∣ Φ − c ∣ 2 d m ρ ≤ 9 C ρ ∫ R d ( ∇ Φ ) T B ∇ Φ d m ρ + t η ˙ . \inf_{c\in\mathbb{R}}\int_{\mathbb{R}^{d}}|\Phi-c|^{2}\,\mathrm{d}m_{\rho+t\dot{\eta}}\leq3\inf_{c\in\mathbb{R}}\int_{\mathbb{R}^{d}}|\Phi-c|^{2}\,\mathrm{d}m_{\rho}\leq9C_{\rho}\int_{\mathbb{R}^{d}}(\nabla\Phi)^{\mathsf{T}}B\nabla\Phi\,\mathrm{d}m_{\rho+t\dot{\eta}}. c ∈ R inf ∫ R d ∣Φ − c ∣ 2 d m ρ + t η ˙ ≤ 3 c ∈ R inf ∫ R d ∣Φ − c ∣ 2 d m ρ ≤ 9 C ρ ∫ R d ( ∇Φ ) T B ∇Φ d m ρ + t η ˙ .
Normalization follows from ∫ η ˙ d x = 0 \int\dot{\eta}\,\mathrm{d}x=0 ∫ η ˙ d x = 0 , while compact support ensures ∫ ∣ η ˙ ∣ 2 / m ρ d x < ∞ \int|\dot{\eta}|^{2}/m_{\rho}\,\mathrm{d}x<\infty ∫ ∣ η ˙ ∣ 2 / m ρ d x < ∞ . Hence η ˙ \dot{\eta} η ˙ is an actual admissible variation. The conclusion follows from Lemma B.1. □
We can relate local regular first variations to ordinary differentiation in Otto calculus.
Example B.4 (Otto calculus on P ∞ ( M ) \mathcal{P}^{\infty}(\mathcal{M}) P ∞ ( M ) ). Let M \mathcal{M} M be a closed, connected Riemannian manifold. The space P ∞ ( M ) \mathcal{P}^{\infty}(\mathcal{M}) P ∞ ( M ) is open in the affine Fréchet space of smooth functions with unit integral, and its tangent vectors are the smooth functions with zero integral.
For a C 1 C^{1} C 1 functional F : P ∞ ( M ) → R \mathcal{F}\colon\mathcal{P}^{\infty}(\mathcal{M})\to\mathbb{R} F : P ∞ ( M ) → R in the Bastiani sense, a local regular first variation at ρ \rho ρ exists if and only if D F ( ρ ) D\mathcal{F}(\rho) D F ( ρ ) has a smooth representative:
D F ( ρ ) [ η ˙ ] = ∫ M δ F δ ρ ( ρ ) η ˙ d v o l M , η ˙ ∈ T ρ P ∞ ( M ) . D\mathcal{F}(\rho)[\dot{\eta}]=\int_{\mathcal{M}}\frac{\delta\mathcal{F}}{\delta\rho}(\rho)\dot{\eta}\,\mathrm{dvol}_{\mathcal{M}},\qquad\dot{\eta}\in T_{\rho}\mathcal{P}^{\infty}(\mathcal{M}). D F ( ρ ) [ η ˙ ] = ∫ M δ ρ δ F ( ρ ) η ˙ dvol M , η ˙ ∈ T ρ P ∞ ( M ) . For m ρ = ρ m_{\rho}=\rho m ρ = ρ and the identity mobility tensor, the corresponding regular gradient is the usual Otto gradient,
grad G W F ( ρ ) = − div M ( ρ ∇ M δ F δ ρ ( ρ ) ) . \operatorname{grad}_{\mathsf{GW}} \mathcal{F}(\rho)=-\operatorname{div}_{\mathcal{M}}\left(\rho\nabla_{\mathcal{M}}\frac{\delta\mathcal{F}}{\delta\rho}(\rho)\right). grad GW F ( ρ ) = − div M ( ρ ∇ M δ ρ δ F ( ρ ) ) . Thus, (117) agrees with ordinary differentiation whenever the differential has a smooth representative. The C 1 C^{1} C 1 assumption alone does not guarantee such a representative: the differential may instead be distributional.
Regular Hamiltonian variations in local mobility. Consider pairs ( ρ , [ Φ ] ) ∈ P m ρ , B × T ρ ∗ P m ρ , B (\rho,[\Phi])\in\mathcal{P}_{m_{\rho},B}\times T_{\rho}^{*}\mathcal{P}_{m_{\rho},B} ( ρ , [ Φ ]) ∈ P m ρ , B × T ρ ∗ P m ρ , B . Before moving on, we want to stress that this is only a state set and we don’t assume the existence of a cotangent-bundle structure. Next we want to define a notion of differentiability for the Hamiltonian we are interested in that is:
H ( ρ , [ Φ ] ) : = 1 2 ⟨ [ Φ ] , L m ρ , B [ Φ ] ⟩ + F ( ρ ) = 1 2 ∫ R d ( ∇ Φ ) T B ∇ Φ d m ρ + F ( ρ ) , ( ρ , [ Φ ] ) ∈ P m ρ , B × T ρ ∗ P m ρ , B . (120) \mathcal{H}(\rho,[\Phi]):=\frac{1}{2}\langle[\Phi],L_{m_{\rho},B}[\Phi]\rangle+\mathcal{F}(\rho)=\frac{1}{2}\int_{\mathbb{R}^{d}}(\nabla\Phi)^{\mathsf{T}}B\nabla\Phi\,\mathrm{d}m_{\rho}+\mathcal{F}(\rho),\qquad(\rho,[\Phi])\in\mathcal{P}_{m_{\rho},B}\times T_{\rho}^{*}\mathcal{P}_{m_{\rho},B}. \tag*{(120)} H ( ρ , [ Φ ]) := 2 1 ⟨[ Φ ] , L m ρ , B [ Φ ]⟩ + F ( ρ ) = 2 1 ∫ R d ( ∇Φ ) T B ∇Φ d m ρ + F ( ρ ) , ( ρ , [ Φ ]) ∈ P m ρ , B × T ρ ∗ P m ρ , B . (120)
We always assume in the following that F \mathcal{F} F has a local regular first variation at ρ \rho ρ in the sense of (117). For fixed [ Φ ] ∈ C ∞ ( R d ) / R [\Phi]\in C^{\infty}(\mathbb{R}^{d})/\mathbb{R} [ Φ ] ∈ C ∞ ( R d ) / R , we define the actual admissible variations compatible with [ Φ ] [\Phi] [ Φ ] by
V ρ , Φ : = { η ∈ V ρ : [ Φ ] ∈ T ρ + t η ∗ P m ρ , B for all sufficiently small ∣ t ∣ } . \mathcal{V}_{\rho,\Phi}:=\left\{\eta\in\mathcal{V}_{\rho}:[\Phi]\in T_{\rho+t\eta}^{*}\mathcal{P}_{m_{\rho},B}\ \text{for all sufficiently small }|t|\right\}. V ρ , Φ := { η ∈ V ρ : [ Φ ] ∈ T ρ + t η ∗ P m ρ , B for all sufficiently small ∣ t ∣ } .
Assume that F \mathcal{F} F has a local regular first variation at ρ \rho ρ , represented by [ f ρ ] = [ δ F δ ρ ( ρ ) ] ∈ T ρ ∗ P m ρ , B [f_{\rho}]=\left[\frac{\delta\mathcal{F}}{\delta\rho}(\rho)\right]\in T_{\rho}^{*}\mathcal{P}_{m_{\rho},B} [ f ρ ] = [ δ ρ δ F ( ρ ) ] ∈ T ρ ∗ P m ρ , B . We say that the kinetic part of H \mathcal{H} H has a fixed-ambient regular sectional variation at ρ \rho ρ if there is a class [ h ρ , Φ ] ∈ T ρ ∗ P m ρ , B [h_{\rho,\Phi}]\in T_{\rho}^{*}\mathcal{P}_{m_{\rho},B} [ h ρ , Φ ] ∈ T ρ ∗ P m ρ , B such that, for every η ∈ V ρ , Φ \eta\in\mathcal{V}_{\rho,\Phi} η ∈ V ρ , Φ ,
d d t ∣ t = 0 1 2 ∫ R d ( ∇ Φ ) T B ∇ Φ d m ρ + t η = ∫ R d h ρ , Φ η d x . (121) \left.\frac{\mathrm{d}}{\mathrm{d}t}\right|_{t=0}\frac{1}{2}\int_{\mathbb{R}^{d}}(\nabla\Phi)^{\mathsf{T}}B\nabla\Phi\,\mathrm{d}m_{\rho+t\eta}=\int_{\mathbb{R}^{d}}h_{\rho,\Phi}\eta\,\mathrm{d}x. \tag*{(121)} d t d t = 0 2 1 ∫ R d ( ∇Φ ) T B ∇Φ d m ρ + t η = ∫ R d h ρ , Φ η d x . (121)
Here, fixed ambient means that [ Φ ] [\Phi] [ Φ ] is kept fixed as an element of C ∞ ( R d ) / R C^{\infty}(\mathbb{R}^{d})/\mathbb{R} C ∞ ( R d ) / R while ρ \rho ρ is varied. In this case, the associated regular extensions are the linear maps
∂ Φ r e g H ( ρ , [ Φ ] ) : T ρ ∗ P m ρ , B ⟶ R , ∂ Φ r e g H ( ρ , [ Φ ] ) [ Ψ ] : = ⟨ [ Ψ ] , L m ρ , B [ Φ ] ⟩ , (122) \partial_{\Phi}^{\mathrm{reg}}\mathcal{H}(\rho,[\Phi]):T_{\rho}^{*}\mathcal{P}_{m_{\rho},B}\longrightarrow\mathbb{R},\qquad
\partial_{\Phi}^{\mathrm{reg}}\mathcal{H}(\rho,[\Phi])[\Psi]:=\langle[\Psi],L_{m_{\rho},B}[\Phi]\rangle, \tag*{(122)} ∂ Φ reg H ( ρ , [ Φ ]) : T ρ ∗ P m ρ , B ⟶ R , ∂ Φ reg H ( ρ , [ Φ ]) [ Ψ ] := ⟨[ Ψ ] , L m ρ , B [ Φ ]⟩ , (122)
∂ ρ r e g H ( ρ , [ Φ ] ) : T ρ P m ρ , B ⟶ R , ∂ ρ r e g H ( ρ , [ Φ ] ) [ η ] : = ∫ R d ( f ρ + h ρ , Φ ) η d x . (123) \partial_{\rho}^{\mathrm{reg}}\mathcal{H}(\rho,[\Phi]):T_{\rho}\mathcal{P}_{m_{\rho},B}\longrightarrow\mathbb{R},\qquad
\partial_{\rho}^{\mathrm{reg}}\mathcal{H}(\rho,[\Phi])[\eta]:=\int_{\mathbb{R}^{d}}(f_{\rho}+h_{\rho,\Phi})\eta\,\mathrm{d}x. \tag*{(123)} ∂ ρ reg H ( ρ , [ Φ ]) : T ρ P m ρ , B ⟶ R , ∂ ρ reg H ( ρ , [ Φ ]) [ η ] := ∫ R d ( f ρ + h ρ , Φ ) η d x . (123)
Indeed, one can prove the following lemma similarly to Lemma B.1.
Lemma B.5 (Uniqueness of the regular Hamiltonian variations). Let ρ ∈ P m ρ , B \rho\in\mathcal{P}_{m_\rho,B} ρ ∈ P m ρ , B and [ Φ ] ∈ T ρ ∗ P m ρ , B [\Phi] \in T_\rho^*\mathcal{P}_{m_\rho,B} [ Φ ] ∈ T ρ ∗ P m ρ , B . Suppose that F \mathcal{F} F admits a local regular first variation at ρ \rho ρ in the sense of (117) and that the kinetic part of H \mathcal{H} H admits a fixed-ambient regular sectional variation at ρ \rho ρ in the sense of (121). If
{ η ∈ C c ∞ ( R d ) : ∫ R d η d x = 0 } ⊆ V ρ , Φ , \left\{ \eta\in C^\infty_c(\mathbb{R}^d) : \int_{\mathbb{R}^d} \eta\,\mathrm{d}x = 0 \right\} \subseteq\mathcal{V}_{\rho,\Phi}, { η ∈ C c ∞ ( R d ) : ∫ R d η d x = 0 } ⊆ V ρ , Φ , then the following holds:
The classes [ f ρ ] [f_\rho] [ f ρ ] and [ h ρ , Φ ] [h_{\rho,\Phi}] [ h ρ , Φ ] , and hence δ r e g H δ ρ ( ρ , [ Φ ] ) \frac{\delta^{\mathrm{reg}}\mathcal{H}}{\delta\rho}(\rho,[\Phi]) δ ρ δ reg H ( ρ , [ Φ ]) are uniquely determined.
The regular extensions in (122) and (123) are uniquely determined.
Now we can prove the following lemma.
Lemma B.6 (Representatives of the regular Hamiltonian variations). Suppose H \mathcal{H} H is as in (120) so that H \mathcal{H} H has a fixed-ambient regular sectional variation at ρ \rho ρ in the sense of (121). Then, we have
d d s ∣ s = 0 H ( ρ , [ Φ + s Ψ ] ) = ∂ Φ r e g H ( ρ , [ Φ ] ) [ Ψ ] ∀ [ Ψ ] ∈ T ρ ∗ P m ρ , B , \left.\frac{\mathrm{d}}{\mathrm{d}s}\right|_{s=0} \mathcal{H}(\rho,[\Phi+s\Psi]) = \partial^{\mathrm{reg}}_{\Phi}\mathcal{H}(\rho,[\Phi])[\Psi] \qquad\forall[\Psi] \in T_\rho^*\mathcal{P}_{m_\rho,B}, d s d s = 0 H ( ρ , [ Φ + s Ψ ]) = ∂ Φ reg H ( ρ , [ Φ ]) [ Ψ ] ∀ [ Ψ ] ∈ T ρ ∗ P m ρ , B , d d t ∣ t = 0 H ( ρ + t η , [ Φ ] ) = ∂ ρ r e g H ( ρ , [ Φ ] ) [ η ] ∀ η ∈ V ρ , Φ . \left.\frac{\mathrm{d}}{\mathrm{d}t}\right|_{t=0} \mathcal{H}(\rho+t\eta,[\Phi]) = \partial^{\mathrm{reg}}_{\rho}\mathcal{H}(\rho,[\Phi])[\eta] \qquad\forall\eta\in\mathcal{V}_{\rho,\Phi}. d t d t = 0 H ( ρ + t η , [ Φ ]) = ∂ ρ reg H ( ρ , [ Φ ]) [ η ] ∀ η ∈ V ρ , Φ .
Lemma B.6 justifies the notation
δ r e g H δ Φ ( ρ , [ Φ ] ) = L m ρ , B [ Φ ] , δ r e g H δ ρ ( ρ , [ Φ ] ) = [ f ρ + h ρ , Φ ] . \frac{\delta^{\mathrm{reg}}\mathcal{H}}{\delta\Phi}(\rho,[\Phi]) = L_{m_\rho,B}[\Phi], \qquad\frac{\delta^{\mathrm{reg}}\mathcal{H}}{\delta\rho}(\rho,[\Phi]) = [f_\rho+h_{\rho,\Phi}]. δ Φ δ reg H ( ρ , [ Φ ]) = L m ρ , B [ Φ ] , δ ρ δ reg H ( ρ , [ Φ ]) = [ f ρ + h ρ , Φ ] .
Proof. The first identity follows by expanding the kinetic part of H ( ρ , [ Φ + s Ψ ] ) \mathcal{H}(\rho,[\Phi+s\Psi]) H ( ρ , [ Φ + s Ψ ]) in s s s . The second identity follows by adding the prescribed local regular first variation of F \mathcal{F} F to the fixed-ambient regular sectional variation of the kinetic part of H \mathcal{H} H . □ \square □
With these notions of regular variation, we can state the accelerated Newton equations. A fixed-ambient regular damped Hamiltonian trajectory is a pair of functions ( ρ t , Φ t ) (\rho_t,\Phi_t) ( ρ t , Φ t ) , smooth in ( t , x ) (t,x) ( t , x ) , for which the regular variation classes have been specified at every time and such that
ρ t ∈ P m ρ , B , [ Φ t ] ∈ T ρ t ∗ P m ρ , B , \rho_t \in\mathcal{P}_{m_{\rho},B}, \qquad[\Phi_t] \in T_{\rho_t}^*\mathcal{P}_{m_{\rho},B}, ρ t ∈ P m ρ , B , [ Φ t ] ∈ T ρ t ∗ P m ρ , B ,
and
∂ t ρ t = L m ρ t , B [ Φ t ] , [ ∂ t Φ t ] + α t [ Φ t ] = − [ f ρ t + h ρ t , Φ t ] in C ∞ ( R d ) / R . (124) \begin{aligned}
\partial_t\rho_t &= L_{m_{\rho_t},B}[\Phi_t],\\
[\partial_t\Phi_t]+\alpha_t[\Phi_t] &= -[f_{\rho_t}+h_{\rho_t,\Phi_t}] \qquad\text{in } C^\infty(\mathbb{R}^d)/\mathbb{R}. \tag*{(124)}
\end{aligned} ∂ t ρ t [ ∂ t Φ t ] + α t [ Φ t ] = L m ρ t , B [ Φ t ] , = − [ f ρ t + h ρ t , Φ t ] in C ∞ ( R d ) / R . (124)
Before recovering the ordinary damped Hamiltonian equations in Otto–Wasserstein geometry, we examine the analog of Lemma B.3 for variations compatible with a fixed potential. For the geometries in (21) and (22), the following lemma shows that the admissibility assumption of Lemma B.5 holds automatically in Otto–Wasserstein geometry and characterizes the additional compatibility condition in softmax–Wasserstein geometry.
Lemma B.8 (Compatible variations in Otto and softmax geometry). For Otto–Wasserstein geometry, the admissibility assumption in Lemma B.5 holds for every ρ ∈ P m ρ , B \rho\in\mathcal{P}_{m_\rho,B} ρ ∈ P m ρ , B and [ Φ ] ∈ T ρ ∗ P m ρ , B [\Phi] \in T_\rho^*\mathcal{P}_{m_\rho,B} [ Φ ] ∈ T ρ ∗ P m ρ , B . For softmax–Wasserstein geometry, a compactly supported smooth zero-integral variation η \eta η belongs to V ρ , Φ \mathcal{V}_{\rho,\Phi} V ρ , Φ if and only if
( ∇ Z η Z ρ ) T B ∇ Φ ∈ L 2 ( m ρ ) , Z η ( x ) : = ∫ R d exp ( z T A x ) η ( z ) d z . (125) \left(\nabla\frac{Z_\eta}{Z_\rho}\right)^{\mathsf{T}} B\nabla\Phi\in L^2(m_\rho), \qquad Z_\eta(x) := \int_{\mathbb{R}^d} \exp(z^{\mathsf{T}}Ax)\eta(z)\,\mathrm{d}z. \tag*{(125)} ( ∇ Z ρ Z η ) T B ∇Φ ∈ L 2 ( m ρ ) , Z η ( x ) := ∫ R d exp ( z T A x ) η ( z ) d z . (125) Note that in the softmax–Wasserstein geometry a sufficient condition is ∇ ( Z η Z ρ ) ∈ L ∞ ( R d ) \nabla\left(\frac{Z_\eta}{Z_\rho}\right) \in L^\infty(\mathbb{R}^d) ∇ ( Z ρ Z η ) ∈ L ∞ ( R d ) .
Proof. The Otto–Wasserstein case follows along the lines of Lemma B.3, using the compact support of η / ρ \eta/\rho η / ρ . We therefore consider softmax mobility.
By the density-admissibility argument, η ∈ V ρ \eta\in\mathcal{V}_\rho η ∈ V ρ , and m ρ + t η m_{\rho+t\eta} m ρ + t η and m ρ m_\rho m ρ are uniformly comparable for sufficiently small ∣ t ∣ |t| ∣ t ∣ . Thus the weighted Sobolev spaces coincide, and it remains to verify the operator-domain condition.
The product rule gives
L m ρ + t η , B [ Φ ] m ρ + t η = L m ρ , B [ Φ ] m ρ − ( ∇ log m ρ + t η m ρ ) T B ∇ Φ . \frac{L_{m_{\rho+t\eta},B}[\Phi]}{m_{\rho+t\eta}} = \frac{L_{m_\rho,B}[\Phi]}{m_\rho} - \left(\nabla\log\frac{m_{\rho+t\eta}}{m_\rho}\right)^{\mathsf{T}} B\nabla\Phi. m ρ + t η L m ρ + t η , B [ Φ ] = m ρ L m ρ , B [ Φ ] − ( ∇ log m ρ m ρ + t η ) T B ∇Φ.
where
∇ log m ρ + t η m ρ = t ∇ ( η / ρ ) 1 + t η / ρ − t ∇ ( Z η / Z ρ ) 1 + t Z η / Z ρ . \nabla\log\frac{m_{\rho+t\eta}}{m_\rho} = \frac{t\nabla(\eta/\rho)}{1+t\eta/\rho} - \frac{t\nabla(Z_\eta/Z_\rho)}{1+tZ_\eta/Z_\rho}. ∇ log m ρ m ρ + t η = 1 + t η / ρ t ∇ ( η / ρ ) − 1 + t Z η / Z ρ t ∇ ( Z η / Z ρ ) .
Since
∣ Z η Z ρ ∣ ≤ ∥ η ρ ∥ L ∞ , \left|\frac{Z_\eta}{Z_\rho}\right| \leq\left\|\frac{\eta}{\rho}\right\|_{L^\infty}, Z ρ Z η ≤ ρ η L ∞ ,
both denominators are uniformly bounded and bounded away from zero for small ∣ t ∣ |t| ∣ t ∣ . The first term is smooth and compactly supported, so its pairing with B ∇ Φ B\nabla\Phi B ∇Φ belongs to L 2 ( m ρ ) L^2(m_\rho) L 2 ( m ρ ) . For nonzero such t t t , uniform weight comparison therefore makes the operator-domain condition equivalent to (125).
Finally, boundedness of ∇ ( Z η / Z ρ ) \nabla(Z_\eta/Z_\rho) ∇ ( Z η / Z ρ ) , together with the finite kinetic energy of Φ \Phi Φ , implies (125). If ∇ Φ \nabla\Phi ∇Φ has compact support, the product in that condition is itself smooth and compactly supported. □ \square □
Example B.9 (Hamiltonian calculus on P ∞ ( M ) \mathcal{P}^\infty(\mathcal{M}) P ∞ ( M ) ). Let M \mathcal{M} M be a closed, connected Riemannian manifold. Then,
T ρ ∗ P ∞ ( M ) = C ∞ ( M ) / R , V ρ , Φ = T ρ P ∞ ( M ) . T_\rho^*\mathcal{P}^\infty(\mathcal{M}) = C^\infty(\mathcal{M})/\mathbb{R}, \qquad\mathcal{V}_{\rho,\Phi} = T_\rho\mathcal{P}^\infty(\mathcal{M}). T ρ ∗ P ∞ ( M ) = C ∞ ( M ) / R , V ρ , Φ = T ρ P ∞ ( M ) . Thus, the regular cotangent space is independent of ρ \rho ρ , and every tangent vector is compatible with every smooth potential.
Suppose that F : P ∞ ( M ) → R \mathcal{F} : \mathcal{P}^\infty(\mathcal{M}) \to\mathbb{R} F : P ∞ ( M ) → R is C 1 C^1 C 1 in the Bastiani sense and that its differential has a smooth representative:
D F ( ρ ) [ η ˙ ] = ∫ M δ F δ ρ ( ρ ) η ˙ d v o l M , η ˙ ∈ T ρ P ∞ ( M ) . D\mathcal{F}(\rho)[\dot{\eta}] = \int_{\mathcal{M}} \frac{\delta\mathcal{F}}{\delta\rho}(\rho)\,\dot{\eta}\,\mathrm{dvol}_{\mathcal{M}}, \qquad\dot{\eta} \in T_\rho\mathcal{P}^\infty(\mathcal{M}). D F ( ρ ) [ η ˙ ] = ∫ M δ ρ δ F ( ρ ) η ˙ dvol M , η ˙ ∈ T ρ P ∞ ( M ) .
Then the Hamiltonian
H ( ρ , [ Φ ] ) : = 1 2 ∫ M ρ ∣ ∇ M Φ ∣ 2 d v o l M + F ( ρ ) \mathcal{H}(\rho,[\Phi]) := \frac{1}{2}\int_{\mathcal{M}} \rho\lvert\nabla_{\mathcal{M}}\Phi\rvert^2 \, \mathrm{dvol}_{\mathcal{M}} + \mathcal{F}(\rho) H ( ρ , [ Φ ]) := 2 1 ∫ M ρ ∣ ∇ M Φ ∣ 2 dvol M + F ( ρ )
is C 1 C^1 C 1 on P ∞ ( M ) × C ∞ ( M ) / R \mathcal{P}^{\infty}(\mathcal{M}) \times C^{\infty}(\mathcal{M})/\mathbb{R} P ∞ ( M ) × C ∞ ( M ) / R with
D Φ H ( ρ , [ Φ ] ) [ Ψ ] = ∫ M L ρ W [ Φ ] Ψ d v o l M = ∫ M ρ ∇ M Φ ⋅ ∇ M Ψ d v o l M , D_{\Phi}\mathcal{H}(\rho,[\Phi])[\Psi] = \int_{\mathcal{M}} L_{\rho}^{W}[\Phi]\Psi\, \mathrm{dvol}_{\mathcal{M}} = \int_{\mathcal{M}} \rho\nabla_{\mathcal{M}}\Phi\cdot\nabla_{\mathcal{M}}\Psi\, \mathrm{dvol}_{\mathcal{M}}, D Φ H ( ρ , [ Φ ]) [ Ψ ] = ∫ M L ρ W [ Φ ] Ψ dvol M = ∫ M ρ ∇ M Φ ⋅ ∇ M Ψ dvol M ,
D ρ H ( ρ , [ Φ ] ) [ η ˙ ] = ∫ M ( 1 2 ∣ ∇ M Φ ∣ 2 + δ F δ ρ ( ρ ) ) η ˙ d v o l M . D_{\rho}\mathcal{H}(\rho,[\Phi])[\dot{\eta}] = \int_{\mathcal{M}} \left(\frac{1}{2}\lvert\nabla_{\mathcal{M}}\Phi\rvert^2 + \frac{\delta\mathcal{F}}{\delta\rho}(\rho)\right)\dot{\eta} \, \mathrm{dvol}_{\mathcal{M}}. D ρ H ( ρ , [ Φ ]) [ η ˙ ] = ∫ M ( 2 1 ∣ ∇ M Φ ∣ 2 + δ ρ δ F ( ρ ) ) η ˙ dvol M .
Hence the regular Hamiltonian variations coincide with the ordinary partial differentials. In particular, we have
δ r e g H δ Φ ( ρ , [ Φ ] ) = L ρ W [ Φ ] , δ r e g H δ ρ ( ρ , [ Φ ] ) = [ 1 2 ∣ ∇ M Φ ∣ 2 + δ F δ ρ ( ρ ) ] . \frac{\delta^{\mathrm{reg}}\mathcal{H}}{\delta\Phi}(\rho,[\Phi]) = L_{\rho}^{W}[\Phi], \qquad
\frac{\delta^{\mathrm{reg}}\mathcal{H}}{\delta\rho}(\rho,[\Phi]) = \left[\frac{1}{2}\lvert\nabla_{\mathcal{M}}\Phi\rvert^2 + \frac{\delta\mathcal{F}}{\delta\rho}(\rho)\right]. δ Φ δ reg H ( ρ , [ Φ ]) = L ρ W [ Φ ] , δ ρ δ reg H ( ρ , [ Φ ]) = [ 2 1 ∣ ∇ M Φ ∣ 2 + δ ρ δ F ( ρ ) ] .
Consequently, (124) reduces to the damped Otto–Wasserstein Hamiltonian system
{ ∂ t ρ t + div M ( ρ t ∇ M Φ t ) = 0 , [ ∂ t Φ t ] + α t [ Φ t ] + [ 1 2 ∣ ∇ M Φ t ∣ 2 + δ F δ ρ ( ρ t ) ] = 0 in C ∞ ( M ) / R . \begin{cases}
\partial_t \rho_t + \operatorname{div}_{\mathcal{M}}(\rho_t\nabla_{\mathcal{M}}\Phi_t) = 0,\\
[\partial_t\Phi_t] + \alpha_t[\Phi_t] + \left[\frac{1}{2}\lvert\nabla_{\mathcal{M}}\Phi_t\rvert^2 + \frac{\delta\mathcal{F}}{\delta\rho}(\rho_t)\right] = 0 \quad\text{in } C^{\infty}(\mathcal{M})/\mathbb{R}.
\end{cases} { ∂ t ρ t + div M ( ρ t ∇ M Φ t ) = 0 , [ ∂ t Φ t ] + α t [ Φ t ] + [ 2 1 ∣ ∇ M Φ t ∣ 2 + δ ρ δ F ( ρ t ) ] = 0 in C ∞ ( M ) / R .
The second equation holds modulo spatial constants; a time-dependent additive gauge for Φ t \Phi_t Φ t gives an equivalent pointwise equation.
Differentiability in the Quotient Stein Geometry
Local first variations and the regular Stein gradient. For ρ ∈ P K A \rho\in\mathcal{P}_{K_A} ρ ∈ P K A , we define the actual admissible variations at ρ \rho ρ by
V ρ : = { η ∈ C c ∞ ( R d ) ∩ T ρ P K A : ∃ ε > 0 such that ρ + t η ∈ P K A for ∣ t ∣ < ε } . \mathcal{V}_{\rho} := \left\{\eta\in C_c^{\infty}(\mathbb{R}^d) \cap T_{\rho}\mathcal{P}_{K_A} : \exists\varepsilon> 0 \text{ such that } \rho+ t\eta\in\mathcal{P}_{K_A} \text{ for } \lvert t\rvert< \varepsilon\right\}. V ρ := { η ∈ C c ∞ ( R d ) ∩ T ρ P K A : ∃ ε > 0 such that ρ + t η ∈ P K A for ∣ t ∣ < ε } .
In the present setting this simplifies to:
V ρ = { η ∈ C c ∞ ( R d ) : ∫ R d η d x = 0 } . \mathcal{V}_{\rho} = \left\{\eta\in C_c^{\infty}(\mathbb{R}^d) : \int_{\mathbb{R}^d} \eta\, \mathrm{d}x = 0\right\}. V ρ = { η ∈ C c ∞ ( R d ) : ∫ R d η d x = 0 } .
Indeed, the constraints in the definition of P K A \mathcal{P}_{K_A} P K A as in (46) are preserved under small perturbations. Since ρ \rho ρ is smooth and strictly positive, it has a positive minimum on the compact support of η \eta η . Consequently, η / ρ ∈ L 2 ( ρ ) \eta/\rho\in L^2(\rho) η / ρ ∈ L 2 ( ρ ) , and ρ + t η \rho+ t\eta ρ + t η remains strictly positive for all sufficiently small ∣ t ∣ \lvert t\rvert ∣ t ∣ . Its integral remains equal to one. Moreover, the second-moment condition and the condition
∫ R d ∣ x ∣ 2 ∣ ∇ ( ρ + t η ) ∣ 2 ρ + t η d x < ∞ \int_{\mathbb{R}^d} \lvert x\rvert^2 \frac{\lvert\nabla(\rho+t\eta)\rvert^2}{\rho+t\eta} \, \mathrm{d}x < \infty ∫ R d ∣ x ∣ 2 ρ + t η ∣ ∇ ( ρ + t η ) ∣ 2 d x < ∞
are preserved: outside the support of η \eta η , the density is unchanged, while on its support the denominator is bounded below by a positive constant.
We say that F : P K A → R \mathcal{F}:P_{K_A}\to\mathbb{R} F : P K A → R has a local regular first variation at ρ \rho ρ if there is a class [ f ρ ] ∈ T ρ ∗ P K A [f_\rho]\in T_\rho^*P_{K_A} [ f ρ ] ∈ T ρ ∗ P K A such that, for every η ∈ V ρ \eta\in\mathcal{V}_\rho η ∈ V ρ ,
lim t → 0 F ( ρ + t η ) − F ( ρ ) t = ∫ R d f ρ η d x . (126) \lim_{t\to0}\frac{\mathcal{F}(\rho+t\eta)-\mathcal{F}(\rho)}{t}=\int_{\mathbb{R}^d}f_\rho\eta\,\mathrm{d}x. \tag*{(126)} t → 0 lim t F ( ρ + t η ) − F ( ρ ) = ∫ R d f ρ η d x . (126)
We then write [ f ρ ] = [ δ F δ ρ ( ρ ) ] [f_\rho]=\left[\frac{\delta\mathcal{F}}{\delta\rho}(\rho)\right] [ f ρ ] = [ δ ρ δ F ( ρ ) ] . Its associated regular extension is the linear map
ℓ F , ρ r e g : T ρ P K A ⟶ R , ℓ F , ρ r e g ( η ˙ ) : = ∫ R d f ρ η ˙ d x . \ell^{\mathrm{reg}}_{\mathcal{F},\rho}:T_\rho P_{K_A}\longrightarrow\mathbb{R},\qquad\ell^{\mathrm{reg}}_{\mathcal{F},\rho}(\dot{\eta}):=\int_{\mathbb{R}^d}f_\rho\dot{\eta}\,\mathrm{d}x. ℓ F , ρ reg : T ρ P K A ⟶ R , ℓ F , ρ reg ( η ˙ ) := ∫ R d f ρ η ˙ d x .
This integral is finite, since
∣ ∫ R d f ρ η ˙ d x ∣ ≤ ∥ f ρ ∥ L 2 ( ρ ) ∥ η ˙ ρ ∥ L 2 ( ρ ) . \left|\int_{\mathbb{R}^d}f_\rho\dot{\eta}\,\mathrm{d}x\right|\leq\lVert f_\rho\rVert_{L^2(\rho)}\left\lVert\frac{\dot{\eta}}{\rho}\right\rVert_{L^2(\rho)}. ∫ R d f ρ η ˙ d x ≤ ∥ f ρ ∥ L 2 ( ρ ) ρ η ˙ L 2 ( ρ ) .
Moreover, it is independent of the representative of [ f ρ ] [f_\rho] [ f ρ ] , because every η ˙ ∈ T ρ P K A \dot{\eta}\in T_\rho P_{K_A} η ˙ ∈ T ρ P K A has zero integral.
Remark C.1 (Uniqueness of the local regular first variation). The local regular first variation is unique modulo constants. Indeed, if [ Φ ] , [ Ψ ] ∈ T ρ ∗ P K A [\Phi],[\Psi]\in T_\rho^*P_{K_A} [ Φ ] , [ Ψ ] ∈ T ρ ∗ P K A both represent the first variation, then
∫ R d ( Φ − Ψ ) η d x = 0 \int_{\mathbb{R}^d}(\Phi-\Psi)\eta\,\mathrm{d}x=0 ∫ R d ( Φ − Ψ ) η d x = 0 for every η ∈ C c ∞ ( R d ) \eta\in C^\infty_c(\mathbb{R}^d) η ∈ C c ∞ ( R d ) with ∫ η = 0 \int\eta=0 ∫ η = 0 . Hence Φ − Ψ \Phi-\Psi Φ − Ψ is constant. Consequently, [ f ρ ] [f_\rho] [ f ρ ] and ℓ F , ρ r e g \ell^{\mathrm{reg}}_{\mathcal{F},\rho} ℓ F , ρ reg are uniquely determined.
To define the Stein gradient, we restrict this regular extension to T ρ e f f P K A T_\rho^{\mathrm{eff}}P_{K_A} T ρ eff P K A . By using (52) we obtain the corresponding effective cotangent class [ f ρ ] ∈ T ρ ∗ , e f f P K A [f_\rho]\in T_\rho^{*,\mathrm{eff}}P_{K_A} [ f ρ ] ∈ T ρ ∗ , eff P K A . This class determines the restriction of ℓ F , ρ r e g \ell^{\mathrm{reg}}_{\mathcal{F},\rho} ℓ F , ρ reg to the effective tangent space. Indeed, if [ Ψ ] ∈ ker L ρ K A [\Psi]\in\ker L_\rho^{K_A} [ Ψ ] ∈ ker L ρ K A and η ˙ = L ρ K A [ Φ ] \dot{\eta}=L_\rho^{K_A}[\Phi] η ˙ = L ρ K A [ Φ ] , the Green identity gives
∫ R d Ψ η ˙ d x = tr ( M ρ ( Ψ ) ⊤ A M ρ ( Φ ) ) = 0. \int_{\mathbb{R}^d}\Psi\dot{\eta}\,\mathrm{d}x=\operatorname{tr}\left(M_\rho(\Psi)^\top A M_\rho(\Phi)\right)=0. ∫ R d Ψ η ˙ d x = tr ( M ρ ( Ψ ) ⊤ A M ρ ( Φ ) ) = 0.
With this notation, we can prove the following.
Lemma C.2 (Regular Stein gradient). If F \mathcal{F} F has a local regular first variation at ρ \rho ρ in the sense of (126), then there is a unique vector
grad G ρ K A F ( ρ ) ∈ T ρ e f f P K A \operatorname{grad}_{G_\rho^{K_A}}\mathcal{F}(\rho)\in T_\rho^{\mathrm{eff}}P_{K_A} grad G ρ K A F ( ρ ) ∈ T ρ eff P K A such that
G ρ K A ( grad G ρ K A F ( ρ ) , η ˙ ) = ℓ F , ρ r e g ( η ˙ ) for every η ˙ ∈ T ρ e f f P K A . G_\rho^{K_A}\left(\operatorname{grad}_{G_\rho^{K_A}}\mathcal{F}(\rho),\dot{\eta}\right)=\ell^{\mathrm{reg}}_{\mathcal{F},\rho}(\dot{\eta})\qquad\text{for every }\dot{\eta}\in T_\rho^{\mathrm{eff}}P_{K_A}. G ρ K A ( grad G ρ K A F ( ρ ) , η ˙ ) = ℓ F , ρ reg ( η ˙ ) for every η ˙ ∈ T ρ eff P K A . This regular Stein gradient is given by
grad G ρ K A F ( ρ ) = − ∇ ⋅ ( ρ M ρ ( δ F δ ρ ( ρ ) ) ⊤ A x ) . \operatorname{grad}_{G_\rho^{K_A}}\mathcal{F}(\rho)=-\nabla\cdot\left(\rho M_\rho\left(\frac{\delta\mathcal{F}}{\delta\rho}(\rho)\right)^\top A x\right). grad G ρ K A F ( ρ ) = − ∇ ⋅ ( ρ M ρ ( δ ρ δ F ( ρ ) ) ⊤ A x ) .
Proof. Since [ f ρ ] ∈ T ρ ∗ P K A [f_{\rho}] \in T_{\rho}^{*}\mathcal{P}_{K_A} [ f ρ ] ∈ T ρ ∗ P K A , we have L ρ K A [ f ρ ] ∈ T ρ e f f P K A L_{\rho}^{K_A}[f_{\rho}] \in T_{\rho}^{\mathrm{eff}}\mathcal{P}_{K_A} L ρ K A [ f ρ ] ∈ T ρ eff P K A . Let η ˙ ∈ T ρ e f f P K A \dot{\eta} \in T_{\rho}^{\mathrm{eff}}\mathcal{P}_{K_A} η ˙ ∈ T ρ eff P K A , and choose [ Φ ] ∈ T ρ ∗ P K A [\Phi] \in T_{\rho}^{*}\mathcal{P}_{K_A} [ Φ ] ∈ T ρ ∗ P K A such that η ˙ = L ρ K A [ Φ ] \dot{\eta}=L_{\rho}^{K_A}[\Phi] η ˙ = L ρ K A [ Φ ] . By using the polarization of the Stein metric and the Green identity, we obtain
G ρ K A ( L ρ K A [ f ρ ] , η ˙ ) = tr ( M ρ ( f ρ ) T A M ρ ( Φ ) ) = ∫ R d f ρ η ˙ d x = ℓ F , ρ r e g ( η ˙ ) . G_{\rho}^{K_A}\left(L_{\rho}^{K_A}[f_{\rho}],\dot{\eta}\right)=\operatorname{tr}\left(M_{\rho}(f_{\rho})^{\mathsf{T}} A M_{\rho}(\Phi)\right)=\int_{\mathbb{R}^{d}}f_{\rho}\dot{\eta}\,\mathrm{d}x=\ell_{F,\rho}^{\mathrm{reg}}(\dot{\eta}). G ρ K A ( L ρ K A [ f ρ ] , η ˙ ) = tr ( M ρ ( f ρ ) T A M ρ ( Φ ) ) = ∫ R d f ρ η ˙ d x = ℓ F , ρ reg ( η ˙ ) .
Thus L ρ K A [ f ρ ] L_{\rho}^{K_A}[f_{\rho}] L ρ K A [ f ρ ] satisfies the defining gradient identity. Uniqueness follows from the positive definiteness of G ρ K A G_{\rho}^{K_A} G ρ K A on T ρ e f f P K A T_{\rho}^{\mathrm{eff}}\mathcal{P}_{K_A} T ρ eff P K A .
Remark C.3. The regular Stein gradient is defined through the regular extension ℓ F , ρ r e g \ell_{F,\rho}^{\mathrm{reg}} ℓ F , ρ reg . The directional limits in (126) are required only for η ∈ V ρ \eta\in\mathcal{V}_{\rho} η ∈ V ρ . Their existence does not by itself imply a chain rule along arbitrary curves in P K A \mathcal{P}_{K_A} P K A .
Regular Hamiltonian variations in the bilinear Stein setting. Consider pairs ( ρ , [ Φ ] ) (\rho,[\Phi]) ( ρ , [ Φ ]) satisfying
ρ ∈ P K A , [ Φ ] ∈ T ρ ∗ P K A . \rho\in\mathcal{P}_{K_A}, \qquad[\Phi] \in T_{\rho}^{*}\mathcal{P}_{K_A}. ρ ∈ P K A , [ Φ ] ∈ T ρ ∗ P K A .
We regard this collection only as a state set and do not assume a cotangent-bundle structure. For such a pair, define
H ( ρ , [ Φ ] ) : = 1 2 ⟨ [ Φ ] , L ρ K A [ Φ ] ⟩ ρ + F ( ρ ) = 1 2 tr ( M ρ ( Φ ) T A M ρ ( Φ ) ) + F ( ρ ) . (127) \mathcal{H}(\rho,[\Phi]) := \frac{1}{2}\langle[\Phi],L_{\rho}^{K_A}[\Phi]\rangle_{\rho}+\mathcal{F}(\rho)=\frac{1}{2}\operatorname{tr}\left(M_{\rho}(\Phi)^{\mathsf{T}}AM_{\rho}(\Phi)\right)+\mathcal{F}(\rho). \tag*{(127)} H ( ρ , [ Φ ]) := 2 1 ⟨[ Φ ] , L ρ K A [ Φ ] ⟩ ρ + F ( ρ ) = 2 1 tr ( M ρ ( Φ ) T A M ρ ( Φ ) ) + F ( ρ ) . (127)
Throughout this paragraph, assume that F \mathcal{F} F has a local regular first variation at ρ \rho ρ in the sense of (126), represented by
[ f ρ ] = [ δ F δ ρ ( ρ ) ] ∈ T ρ ∗ P K A . [f_{\rho}]=\left[\frac{\delta\mathcal{F}}{\delta\rho}(\rho)\right]\in T_{\rho}^{*}\mathcal{P}_{K_A}. [ f ρ ] = [ δ ρ δ F ( ρ ) ] ∈ T ρ ∗ P K A .
For fixed [ Φ ] ∈ T ρ ∗ P K A [\Phi]\in T_{\rho}^{*}\mathcal{P}_{K_A} [ Φ ] ∈ T ρ ∗ P K A , define the actual admissible variations compatible with [ Φ ] [\Phi] [ Φ ] by
V ρ , Φ : = { η ∈ V ρ : [ Φ ] ∈ T ρ + t η ∗ P K A for all sufficiently small ∣ t ∣ } . \mathcal{V}_{\rho,\Phi}:=\{\eta\in\mathcal{V}_{\rho}:[\Phi]\in T_{\rho+t\eta}^{*}\mathcal{P}_{K_A}\text{ for all sufficiently small }|t|\}. V ρ , Φ := { η ∈ V ρ : [ Φ ] ∈ T ρ + t η ∗ P K A for all sufficiently small ∣ t ∣ } .
In the present setting,
V ρ , Φ = V ρ . \mathcal{V}_{\rho,\Phi}=\mathcal{V}_{\rho}. V ρ , Φ = V ρ .
Indeed, every η ∈ V ρ \eta\in\mathcal{V}_{\rho} η ∈ V ρ is compactly supported. For sufficiently small ∣ t ∣ |t| ∣ t ∣ , the ratio ( ρ + t η ) / ρ (\rho+t\eta)/\rho ( ρ + t η ) / ρ is bounded above and bounded away from zero. Consequently, the weighted Sobolev spaces H 1 ( ρ + t η ) H^{1}(\rho+t\eta) H 1 ( ρ + t η ) and H 1 ( ρ ) H^{1}(\rho) H 1 ( ρ ) coincide as sets, so Φ \Phi Φ remains admissible.
We say that the kinetic part of H \mathcal{H} H has a fixed-ambient regular sectional variation at ρ \rho ρ if there is a class
[ h ρ , Φ ] ∈ T ρ ∗ P K A [h_{\rho,\Phi}]\in T_{\rho}^{*}\mathcal{P}_{K_A} [ h ρ , Φ ] ∈ T ρ ∗ P K A
such that, for every η ∈ V ρ , Φ \eta\in\mathcal{V}_{\rho,\Phi} η ∈ V ρ , Φ ,
d d t ∣ t = 0 1 2 tr ( M ρ + t η ( Φ ) T A M ρ + t η ( Φ ) ) = ∫ R d h ρ , Φ η d x . \left.\frac{\mathrm{d}}{\mathrm{d}t}\right|_{t=0}\frac{1}{2}\operatorname{tr}\left(M_{\rho+t\eta}(\Phi)^{\mathsf{T}}AM_{\rho+t\eta}(\Phi)\right)=\int_{\mathbb{R}^{d}}h_{\rho,\Phi}\eta\,\mathrm{d}x. d t d t = 0 2 1 tr ( M ρ + t η ( Φ ) T A M ρ + t η ( Φ ) ) = ∫ R d h ρ , Φ η d x .
Here, fixed ambient means that [ Φ ] [\Phi] [ Φ ] is kept fixed as an element of C ∞ ( R d ) / R C^{\infty}(\mathbb{R}^{d})/\mathbb{R} C ∞ ( R d ) / R while ρ \rho ρ is varied. In this case, the associated regular extensions are the linear maps
∂ Φ r e g H ( ρ , [ Φ ] ) [ Ψ ] : = ⟨ [ Ψ ] , L ρ K A [ Φ ] ⟩ ρ , [ Ψ ] ∈ T ρ ∗ P K A , \partial_{\Phi}^{\mathrm{reg}}\mathcal{H}(\rho,[\Phi])[\Psi]:=\langle[\Psi],L_{\rho}^{K_A}[\Phi]\rangle_{\rho}, \qquad[\Psi]\in T_{\rho}^{*}\mathcal{P}_{K_A}, ∂ Φ reg H ( ρ , [ Φ ]) [ Ψ ] := ⟨[ Ψ ] , L ρ K A [ Φ ] ⟩ ρ , [ Ψ ] ∈ T ρ ∗ P K A ,
∂ ρ r e g H ( ρ , [ Φ ] ) [ η ˙ ] : = ∫ R d ( f ρ + h ρ , Φ ) η ˙ d x , η ˙ ∈ T ρ P K A . \partial^{\mathrm{reg}}_{\rho}\mathcal{H}(\rho,[\Phi])[\dot{\eta}] := \int_{\mathbb{R}^{d}}(f_{\rho}+h_{\rho,\Phi})\dot{\eta}\,\mathrm{d}x,\qquad\dot{\eta}\in T_{\rho}\mathcal{P}_{K_A}. ∂ ρ reg H ( ρ , [ Φ ]) [ η ˙ ] := ∫ R d ( f ρ + h ρ , Φ ) η ˙ d x , η ˙ ∈ T ρ P K A .
With this notation, we can prove the following.
Lemma C.4 (Representatives of the regular Stein Hamiltonian variations). The kinetic part of H \mathcal{H} H has a fixed-ambient regular sectional variation at ρ \rho ρ if and only if
[ ( ∇ Φ ) T M ρ ( Φ ) T A x ] ∈ T ρ ∗ P K A . \left[(\nabla\Phi)^{\mathsf{T}}M_{\rho}(\Phi)^{\mathsf{T}}Ax\right]\in T_{\rho}^{*}\mathcal{P}_{K_A}. [ ( ∇Φ ) T M ρ ( Φ ) T A x ] ∈ T ρ ∗ P K A . In this case, its representative may be chosen as
h ρ , Φ ( x ) = ∇ Φ ( x ) T M ρ ( Φ ) T A x , h_{\rho,\Phi}(x)=\nabla\Phi(x)^{\mathsf{T}}M_{\rho}(\Phi)^{\mathsf{T}}Ax, h ρ , Φ ( x ) = ∇Φ ( x ) T M ρ ( Φ ) T A x , and
d d s ∣ s = 0 H ( ρ , [ Φ + s Ψ ] ) = ∂ Φ r e g H ( ρ , [ Φ ] ) [ Ψ ] , [ Ψ ] ∈ T ρ ∗ P K A , \left.\frac{\mathrm{d}}{\mathrm{d}s}\right|_{s=0}\mathcal{H}(\rho,[\Phi+s\Psi])=\partial_{\Phi}^{\mathrm{reg}}\mathcal{H}(\rho,[\Phi])[\Psi],\qquad[\Psi]\in T_{\rho}^{*}\mathcal{P}_{K_A}, d s d s = 0 H ( ρ , [ Φ + s Ψ ]) = ∂ Φ reg H ( ρ , [ Φ ]) [ Ψ ] , [ Ψ ] ∈ T ρ ∗ P K A , d d t ∣ t = 0 H ( ρ + t η , [ Φ ] ) = ∂ ρ r e g H ( ρ , [ Φ ] ) [ η ] , η ∈ V ρ , Φ . \left.\frac{\mathrm{d}}{\mathrm{d}t}\right|_{t=0}\mathcal{H}(\rho+t\eta,[\Phi])=\partial_{\rho}^{\mathrm{reg}}\mathcal{H}(\rho,[\Phi])[\eta],\qquad\eta\in\mathcal{V}_{\rho,\Phi}. d t d t = 0 H ( ρ + t η , [ Φ ]) = ∂ ρ reg H ( ρ , [ Φ ]) [ η ] , η ∈ V ρ , Φ .
Proof. By linearity of the matrix-moment operator in the potential,
M ρ ( Φ + s Ψ ) = M ρ ( Φ ) + s M ρ ( Ψ ) . M_{\rho}(\Phi+s\Psi)=M_{\rho}(\Phi)+sM_{\rho}(\Psi). M ρ ( Φ + s Ψ ) = M ρ ( Φ ) + s M ρ ( Ψ ) .
Expanding the quadratic kinetic term and using the symmetry of A A A therefore gives
d d s ∣ s = 0 H ( ρ , [ Φ + s Ψ ] ) = tr ( M ρ ( Ψ ) T A M ρ ( Φ ) ) = ⟨ [ Ψ ] , L ρ K A [ Φ ] ⟩ ρ , \left.\frac{\mathrm{d}}{\mathrm{d}s}\right|_{s=0}\mathcal{H}(\rho,[\Phi+s\Psi])=\operatorname{tr}\left(M_{\rho}(\Psi)^{\mathsf{T}}AM_{\rho}(\Phi)\right)=\langle[\Psi],L_{\rho}^{K_A}[\Phi]\rangle_{\rho}, d s d s = 0 H ( ρ , [ Φ + s Ψ ]) = tr ( M ρ ( Ψ ) T A M ρ ( Φ ) ) = ⟨[ Ψ ] , L ρ K A [ Φ ] ⟩ ρ ,
where the last equality is the Green identity. Similarly, for every η ∈ V ρ \eta\in\mathcal{V}_{\rho} η ∈ V ρ ,
M ρ + t η ( Φ ) = M ρ ( Φ ) + t ∫ R d x ∇ Φ ( x ) T η ( x ) d x . M_{\rho+t\eta}(\Phi)=M_{\rho}(\Phi)+t\int_{\mathbb{R}^{d}}x\nabla\Phi(x)^{\mathsf{T}}\eta(x)\,\mathrm{d}x. M ρ + t η ( Φ ) = M ρ ( Φ ) + t ∫ R d x ∇Φ ( x ) T η ( x ) d x .
The additional integral is finite because η \eta η is compactly supported. Differentiating the quadratic expression yields
d d t ∣ t = 0 1 2 tr ( M ρ + t η ( Φ ) T A M ρ + t η ( Φ ) ) = ∫ R d ∇ Φ ( x ) T M ρ ( Φ ) T A x η ( x ) d x . \left.\frac{\mathrm{d}}{\mathrm{d}t}\right|_{t=0}\frac{1}{2}\operatorname{tr}\left(M_{\rho+t\eta}(\Phi)^{\mathsf{T}}AM_{\rho+t\eta}(\Phi)\right)=\int_{\mathbb{R}^{d}}\nabla\Phi(x)^{\mathsf{T}}M_{\rho}(\Phi)^{\mathsf{T}}Ax\eta(x)\,\mathrm{d}x. d t d t = 0 2 1 tr ( M ρ + t η ( Φ ) T A M ρ + t η ( Φ ) ) = ∫ R d ∇Φ ( x ) T M ρ ( Φ ) T A x η ( x ) d x .
Since V ρ \mathcal{V}_{\rho} V ρ contains every compactly supported smooth function with zero integral, this smooth representative is uniquely determined modulo constants. Thus a regular sectional variation exists precisely when its class belongs to T ρ ∗ P K A T_{\rho}^{*}\mathcal{P}_{K_A} T ρ ∗ P K A . Adding the prescribed local regular first variation of F \mathcal{F} F proves the second identity.
With these notions of regular variation in place, we can state the damped Stein Hamiltonian equations. A fixed-ambient regular damped Hamiltonian trajectory is a pair of functions ( ρ t , Φ t ) (\rho_t,\Phi_t) ( ρ t , Φ t ) , smooth in ( t , x ) (t,x) ( t , x ) , such that
ρ t ∈ P K A , [ Φ t ] ∈ T ρ t ∗ P K A , \rho_t \in\mathcal{P}_{K_A}, \qquad[\Phi_t] \in T_{\rho_t}^{*}\mathcal{P}_{K_A}, ρ t ∈ P K A , [ Φ t ] ∈ T ρ t ∗ P K A ,
the preceding regular variations exist at every time, and
{ ∂ t ρ t = L ρ t K A [ Φ t ] , [ ∂ t Φ t ] + α t [ Φ t ] = − [ f ρ t + h ρ t , Φ t ] in C ∞ ( R d ) / R . (128) \begin{cases}
\partial_t\rho_t = L_{\rho_t}^{K_A}[\Phi_t],\\
[\partial_t\Phi_t] + \alpha_t[\Phi_t] = -[f_{\rho_t}+h_{\rho_t,\Phi_t}] \quad\text{in } C^\infty(\mathbb{R}^d)/\mathbb{R}. \tag*{(128)}
\end{cases} { ∂ t ρ t = L ρ t K A [ Φ t ] , [ ∂ t Φ t ] + α t [ Φ t ] = − [ f ρ t + h ρ t , Φ t ] in C ∞ ( R d ) / R . (128)
Equivalently, we get
{ ∂ t ρ t + ∇ ⋅ ( ρ t M ρ t ( Φ t ) ⊤ A x ) = 0 , [ ∂ t Φ t ] + α t [ Φ t ] + [ ∇ Φ t ( x ) ⊤ M ρ t ( Φ t ) ⊤ A x + δ F δ ρ ( ρ t ) ( x ) ] = 0 in C ∞ ( R d ) / R . \begin{cases}
\partial_t\rho_t+\nabla\cdot\left(\rho_t M_{\rho_t}(\Phi_t)^\top A x\right)=0,\\
[\partial_t\Phi_t]+\alpha_t[\Phi_t]+\left[\nabla\Phi_t(x)^\top M_{\rho_t}(\Phi_t)^\top A x+\frac{\delta\mathcal{F}}{\delta\rho}(\rho_t)(x)\right]=0
\quad\text{in } C^\infty(\mathbb{R}^d)/\mathbb{R}.
\end{cases} { ∂ t ρ t + ∇ ⋅ ( ρ t M ρ t ( Φ t ) ⊤ A x ) = 0 , [ ∂ t Φ t ] + α t [ Φ t ] + [ ∇ Φ t ( x ) ⊤ M ρ t ( Φ t ) ⊤ A x + δ ρ δ F ( ρ t ) ( x ) ] = 0 in C ∞ ( R d ) / R .
A time-dependent additive gauge for Φ t \Phi_t Φ t may be chosen to write the second equation as a pointwise equality of functions.
Remark C.6 (Dependence on the ambient potential class). At a fixed density ρ \rho ρ , the Hamiltonian depends only on the effective cotangent class
[ Φ ] ∈ T ρ ∗ , e f f P K A . [\Phi]\in T_{\rho}^{*,\mathrm{eff}}\mathcal{P}_{K_A}. [ Φ ] ∈ T ρ ∗ , eff P K A .
Its fixed-ambient density variation, however, generally depends on the full class [ Φ ] [\Phi] [ Φ ] . Indeed, if [ Ψ ] ∈ ker L ρ K A [\Psi]\in\ker L_{\rho}^{K_A} [ Ψ ] ∈ ker L ρ K A , then M ρ ( Ψ ) = 0 M_\rho(\Psi)=0 M ρ ( Ψ ) = 0 , so
H ( ρ , [ Φ + Ψ ] ) = H ( ρ , [ Φ ] ) . \mathcal{H}(\rho,[\Phi+\Psi])=\mathcal{H}(\rho,[\Phi]). H ( ρ , [ Φ + Ψ ]) = H ( ρ , [ Φ ]) .
Nevertheless, the corresponding smooth kinetic representatives satisfy
h ρ , Φ + Ψ ( x ) − h ρ , Φ ( x ) = ∇ Ψ ( x ) ⊤ M ρ ( Φ ) ⊤ A x , h_{\rho,\Phi+\Psi}(x)-h_{\rho,\Phi}(x)=\nabla\Psi(x)^\top M_\rho(\Phi)^\top A x, h ρ , Φ + Ψ ( x ) − h ρ , Φ ( x ) = ∇Ψ ( x ) ⊤ M ρ ( Φ ) ⊤ A x ,
which need not be constant. Thus (128) is formulated using the full potential classes [ Φ t ] [\Phi_t] [ Φ t ] and the fixed-ambient convention. The construction does not define a density differential depending only on the effective cotangent class.
Proofs of Section 4
For convenience, we give the proof of the following functional analytic Lemma 4.2 in the spirit of [19 Lemma A.8] .
Proof of Lemma 4.2. 1 The mean functional Φ ⟼ ∫ R d Φ d m ρ \Phi\longmapsto\int_{\mathbb{R}^d}\Phi\,\mathrm{d}m_\rho Φ ⟼ ∫ R d Φ d m ρ is continuous on H 1 ( m ρ , B ) H^1(m_\rho,B) H 1 ( m ρ , B ) . Hence its kernel H 0 1 ( m ρ , B ) H_0^1(m_\rho,B) H 0 1 ( m ρ , B ) is closed and therefore a Hilbert space. Next, by using the local-mobility Poincaré inequality (26) we have
∥ Φ ∥ m ρ , B 2 ≤ ∥ Φ ∥ H 1 ( m ρ , B ) 2 ≤ ( 1 + C ρ ) ∥ Φ ∥ m ρ , B 2 ∀ Φ ∈ H 0 1 ( m ρ , B ) . \|\Phi\|_{m_\rho,B}^2\leq\|\Phi\|_{H^1(m_\rho,B)}^2\leq(1+C_\rho)\|\Phi\|_{m_\rho,B}^2
\qquad\forall\Phi\in H_0^1(m_\rho,B). ∥Φ ∥ m ρ , B 2 ≤ ∥Φ ∥ H 1 ( m ρ , B ) 2 ≤ ( 1 + C ρ ) ∥Φ ∥ m ρ , B 2 ∀Φ ∈ H 0 1 ( m ρ , B ) .
Thus the two norms ∥ ⋅ ∥ m ρ , B \|\cdot\|_{m_\rho,B} ∥ ⋅ ∥ m ρ , B and ∥ ⋅ ∥ H 1 ( m ρ , B ) \|\cdot\|_{H^1(m_\rho,B)} ∥ ⋅ ∥ H 1 ( m ρ , B ) are equivalent on H 0 1 ( m ρ , B ) H_0^1(m_\rho,B) H 0 1 ( m ρ , B ) . 2: Let χ ∈ C c ∞ ( R d ) \chi\in C^\infty_c(\mathbb{R}^d) χ ∈ C c ∞ ( R d ) satisfy 0 ≤ χ ≤ 1 0 \le\chi\le1 0 ≤ χ ≤ 1 , χ = 1 \chi= 1 χ = 1 on B 1 B_1 B 1 , and supp ( χ ) ⊂ B 2 \operatorname{supp}(\chi) \subset B_2 supp ( χ ) ⊂ B 2 . Define χ R ( x ) : = χ ( x / R ) \chi_R(x) := \chi(x/R) χ R ( x ) := χ ( x / R ) . Then
χ R = 1 on B R , supp ( χ R ) ⊂ B 2 R , ∣ ∇ χ R ∣ ≤ C R . (129) \chi_R = 1 \text{ on } B_R,\qquad\operatorname{supp}(\chi_R) \subset B_{2R},\qquad|\nabla\chi_R| \le\frac{C}{R}. \tag*{(129)} χ R = 1 on B R , supp ( χ R ) ⊂ B 2 R , ∣∇ χ R ∣ ≤ R C . (129)
For Φ ∈ H 1 ( m ρ , B ) \Phi\in H^1(m_\rho,B) Φ ∈ H 1 ( m ρ , B ) , set Φ R : = χ R Φ \Phi_R := \chi_R\Phi Φ R := χ R Φ . We have
∥ Φ − Φ R ∥ L 2 ( m ρ ) 2 = ∫ R d ∣ ( 1 − χ R ) Φ ∣ 2 d m ρ ≤ ∫ R d ∖ B R ∣ Φ ∣ 2 d m ρ → 0 as R → ∞ . \|\Phi-\Phi_R\|_{L^2(m_\rho)}^2 = \int_{\mathbb{R}^d}|(1-\chi_R)\Phi|^2\,\mathrm{d}m_\rho\le\int_{\mathbb{R}^d\setminus B_R}|\Phi|^2\,\mathrm{d}m_\rho\to0 \quad\text{as } R\to\infty. ∥Φ − Φ R ∥ L 2 ( m ρ ) 2 = ∫ R d ∣ ( 1 − χ R ) Φ ∣ 2 d m ρ ≤ ∫ R d ∖ B R ∣Φ ∣ 2 d m ρ → 0 as R → ∞.
Moreover,
∇ ( Φ R − Φ ) = ( χ R − 1 ) ∇ Φ + Φ ∇ χ R . (130) \nabla(\Phi_R-\Phi)=(\chi_R-1)\nabla\Phi+\Phi\nabla\chi_R. \tag*{(130)} ∇ ( Φ R − Φ ) = ( χ R − 1 ) ∇Φ + Φ∇ χ R . (130)
Thus, denoting the largest eigenvalue of B B B by λ max ( B ) \lambda_{\max}(B) λ m a x ( B ) , we obtain by using (129) and (130)
∥ ∇ Φ − ∇ Φ R ∥ L 2 ( m ρ , B , R d ) 2 = ∫ R d ∇ ( Φ R − Φ ) ⊤ B ∇ ( Φ R − Φ ) d m ρ \|\nabla\Phi-\nabla\Phi_R\|_{L^2(m_\rho,B,\mathbb{R}^d)}^2
= \int_{\mathbb{R}^d}\nabla(\Phi_R-\Phi)^\top B\nabla(\Phi_R-\Phi)\,\mathrm{d}m_\rho ∥∇Φ − ∇ Φ R ∥ L 2 ( m ρ , B , R d ) 2 = ∫ R d ∇ ( Φ R − Φ ) ⊤ B ∇ ( Φ R − Φ ) d m ρ
≤ 2 ∫ R d ∖ B R ∇ Φ ⊤ B ∇ Φ d m ρ + 2 C 2 λ max ( B ) R 2 ∫ R d ∣ Φ ∣ 2 d m ρ , \le2\int_{\mathbb{R}^d\setminus B_R}\nabla\Phi^\top B\nabla\Phi\,\mathrm{d}m_\rho+\frac{2C^2\lambda_{\max}(B)}{R^2}\int_{\mathbb{R}^d}|\Phi|^2\,\mathrm{d}m_\rho, ≤ 2 ∫ R d ∖ B R ∇ Φ ⊤ B ∇Φ d m ρ + R 2 2 C 2 λ m a x ( B ) ∫ R d ∣Φ ∣ 2 d m ρ ,
as both integrals are uniformly bounded from above for all R > 0 R>0 R > 0 and the right-hand side converges to zero as R → ∞ R\to\infty R → ∞ . Consequently by definition of the norm on H 1 ( m ρ , B ) H^1(m_\rho,B) H 1 ( m ρ , B ) as in (23) ,
Φ R → Φ in H 1 ( m ρ , B ) . (131) \Phi_R\to\Phi\quad\text{in }H^1(m_\rho,B). \tag*{(131)} Φ R → Φ in H 1 ( m ρ , B ) . (131)
For each fixed R R R , the function Φ R \Phi_R Φ R has compact support. Since m ρ m_\rho m ρ is smooth and strictly positive, for every compact set K ⊂ R d K\subset\mathbb{R}^d K ⊂ R d there exist constants 0 < c K ≤ C K < ∞ 0<c_K\le C_K<\infty 0 < c K ≤ C K < ∞ such that
c K ≤ m ρ ( x ) ≤ C K ∀ x ∈ K . c_K\le m_\rho(x)\le C_K\qquad\forall x\in K. c K ≤ m ρ ( x ) ≤ C K ∀ x ∈ K .
Moreover, since B = B ⊤ ≻ 0 B=B^\top\succ0 B = B ⊤ ≻ 0 and denoting the smallest eigenvalue of B B B by λ min ( B ) \lambda_{\min}(B) λ m i n ( B ) ,
λ min ( B ) ∣ v ∣ 2 ≤ v ⊤ B v ≤ λ max ( B ) ∣ v ∣ 2 ∀ v ∈ R d . \lambda_{\min}(B)|v|^2\le v^\top Bv\le\lambda_{\max}(B)|v|^2\qquad\forall v\in\mathbb{R}^d. λ m i n ( B ) ∣ v ∣ 2 ≤ v ⊤ B v ≤ λ m a x ( B ) ∣ v ∣ 2 ∀ v ∈ R d .
Consequently, for every u u u supported in K K K ,
c K min { 1 , λ min ( B ) } ∥ u ∥ H 1 ( R d ) 2 ≤ ∥ u ∥ H 1 ( m ρ , B ) 2 ≤ C K max { 1 , λ max ( B ) } ∥ u ∥ H 1 ( R d ) 2 . c_K\min\{1,\lambda_{\min}(B)\}\|u\|_{H^1(\mathbb{R}^d)}^2\le\|u\|_{H^1(m_\rho,B)}^2\le C_K\max\{1,\lambda_{\max}(B)\}\|u\|_{H^1(\mathbb{R}^d)}^2. c K min { 1 , λ m i n ( B )} ∥ u ∥ H 1 ( R d ) 2 ≤ ∥ u ∥ H 1 ( m ρ , B ) 2 ≤ C K max { 1 , λ m a x ( B )} ∥ u ∥ H 1 ( R d ) 2 .
Thus, on every compact set K ⊂ R d K\subset\mathbb{R}^d K ⊂ R d , the weighted Sobolev norm
∥ u ∥ H 1 ( m ρ , B ; K ) 2 : = ∫ K ∣ u ∣ 2 d m ρ + ∫ K ∇ u ⊤ B ∇ u d m ρ \|u\|_{H^1(m_\rho,B;K)}^2:=\int_K|u|^2\,\mathrm{d}m_\rho+\int_K\nabla u^\top B\nabla u\,\mathrm{d}m_\rho ∥ u ∥ H 1 ( m ρ , B ; K ) 2 := ∫ K ∣ u ∣ 2 d m ρ + ∫ K ∇ u ⊤ B ∇ u d m ρ
is equivalent to the usual unweighted Sobolev norm
∥ u ∥ H 1 ( K ) 2 : = ∫ K ∣ u ∣ 2 d x + ∫ K ∣ ∇ u ∣ 2 d x . \|u\|_{H^1(K)}^2:=\int_K|u|^2\,\mathrm{d}x+\int_K|\nabla u|^2\,\mathrm{d}x. ∥ u ∥ H 1 ( K ) 2 := ∫ K ∣ u ∣ 2 d x + ∫ K ∣∇ u ∣ 2 d x .
Standard mollification therefore provides ψ R ∈ C c ∞ ( R d ) \psi_R\in C^\infty_c(\mathbb{R}^d) ψ R ∈ C c ∞ ( R d ) such that
∥ ψ R − Φ R ∥ H 1 ( m ρ , B ) ≤ 1 R . (132) \|\psi_R-\Phi_R\|_{H^1(m_\rho,B)}\le\frac{1}{R}. \tag*{(132)} ∥ ψ R − Φ R ∥ H 1 ( m ρ , B ) ≤ R 1 . (132)
It follows from (131) and (132) in combination with the triangle inequality that
ψ R → Φ in H 1 ( m ρ , B ) . \psi_R \to\Phi\qquad\text{in } H^1(m_\rho,B). ψ R → Φ in H 1 ( m ρ , B ) .
Hence C c ∞ ( R d ) C^\infty_c(\mathbb{R}^d) C c ∞ ( R d ) is dense in H 1 ( m ρ , B ) H^1(m_\rho,B) H 1 ( m ρ , B ) .
It remains to check that ψ − 1 m ρ ( R d ) ∫ R d ψ d m ρ \psi-\frac{1}{m_\rho(\mathbb{R}^d)}\int_{\mathbb{R}^d}\psi\,\mathrm{d}m_\rho ψ − m ρ ( R d ) 1 ∫ R d ψ d m ρ for ψ ∈ C c ∞ ( R d ) \psi\in C^\infty_c(\mathbb{R}^d) ψ ∈ C c ∞ ( R d ) are dense in H 0 1 ( m ρ , B ) H^1_0(m_\rho,B) H 0 1 ( m ρ , B ) . By density of C c ∞ ( R d ) C^\infty_c(\mathbb{R}^d) C c ∞ ( R d ) in H 1 ( m ρ , B ) H^1(m_\rho,B) H 1 ( m ρ , B ) for each Φ ∈ H 0 1 ( m ρ , B ) \Phi\in H^1_0(m_\rho,B) Φ ∈ H 0 1 ( m ρ , B ) we can choose ψ k ∈ C c ∞ ( R d ) \psi_k\in C^\infty_c(\mathbb{R}^d) ψ k ∈ C c ∞ ( R d ) such that ψ k → Φ \psi_k\to\Phi ψ k → Φ in H 1 ( m ρ , B ) H^1(m_\rho,B) H 1 ( m ρ , B ) . Set
c k : = 1 m ρ ( R d ) ∫ R d ψ k d m ρ , ψ ~ k : = ψ k − c k . c_k:=\frac{1}{m_\rho(\mathbb{R}^d)}\int_{\mathbb{R}^d}\psi_k\,\mathrm{d}m_\rho,\qquad\widetilde{\psi}_k:=\psi_k-c_k. c k := m ρ ( R d ) 1 ∫ R d ψ k d m ρ , ψ k := ψ k − c k .
Then ψ ~ k ∈ H 0 1 ( m ρ , B ) \widetilde{\psi}_k\in H^1_0(m_\rho,B) ψ k ∈ H 0 1 ( m ρ , B ) . Since ψ k → Φ \psi_k\to\Phi ψ k → Φ in H 1 ( m ρ , B ) H^1(m_\rho,B) H 1 ( m ρ , B ) , we have ψ k → Φ \psi_k\to\Phi ψ k → Φ in L 2 ( m ρ ) L^2(m_\rho) L 2 ( m ρ ) . Therefore, by the Cauchy–Schwarz inequality,
∣ ∫ R d ( ψ k − Φ ) d m ρ ∣ ≤ m ρ ( R d ) 1 / 2 ∥ ψ k − Φ ∥ L 2 ( m ρ ) → 0 as k → ∞ . (133) \left|\int_{\mathbb{R}^d}(\psi_k-\Phi)\,\mathrm{d}m_\rho\right|\leq m_\rho(\mathbb{R}^d)^{1/2}\|\psi_k-\Phi\|_{L^2(m_\rho)}\to0\qquad\text{as }k\to\infty. \tag*{(133)} ∫ R d ( ψ k − Φ ) d m ρ ≤ m ρ ( R d ) 1/2 ∥ ψ k − Φ ∥ L 2 ( m ρ ) → 0 as k → ∞. (133)
Since Φ ∈ H 0 1 ( m ρ , B ) \Phi\in H^1_0(m_\rho,B) Φ ∈ H 0 1 ( m ρ , B ) , it has zero m ρ m_\rho m ρ -mean. Thus we obtain from (133):
c k = 1 m ρ ( R d ) ∫ R d ψ k d m ρ → 0. c_k=\frac{1}{m_\rho(\mathbb{R}^d)}\int_{\mathbb{R}^d}\psi_k\,\mathrm{d}m_\rho\to0. c k = m ρ ( R d ) 1 ∫ R d ψ k d m ρ → 0.
We therefore conclude by using ψ ~ k = ψ k − c k \widetilde{\psi}_k=\psi_k-c_k ψ k = ψ k − c k that
∥ ψ ~ k − Φ ∥ H 1 ( m ρ , B ) ≤ ∥ ψ k − Φ ∥ H 1 ( m ρ , B ) + ∣ c k ∣ m ρ ( R d ) 1 / 2 → 0 as k → ∞ . \|\widetilde{\psi}_k-\Phi\|_{H^1(m_\rho,B)}\leq\|\psi_k-\Phi\|_{H^1(m_\rho,B)}+|c_k|m_\rho(\mathbb{R}^d)^{1/2}\to0\qquad\text{as }k\to\infty. ∥ ψ k − Φ ∥ H 1 ( m ρ , B ) ≤ ∥ ψ k − Φ ∥ H 1 ( m ρ , B ) + ∣ c k ∣ m ρ ( R d ) 1/2 → 0 as k → ∞.
This finishes the proof of 2. □
Proof of Theorem 4.3. 1. For Ψ ∈ C c ∞ ( R d ) \Psi\in C^\infty_c(\mathbb{R}^d) Ψ ∈ C c ∞ ( R d ) , integration by parts gives
∫ R d ( L m ρ , B Φ ) Ψ d x = ∫ R d ∇ Φ T B ∇ Ψ d m ρ . (134) \int_{\mathbb{R}^d}(L_{m_\rho,B}\Phi)\Psi\,\mathrm{d}x=\int_{\mathbb{R}^d}\nabla\Phi^{\mathsf{T}}B\nabla\Psi\,\mathrm{d}m_\rho. \tag*{(134)} ∫ R d ( L m ρ , B Φ ) Ψ d x = ∫ R d ∇ Φ T B ∇Ψ d m ρ . (134)
This identity extends to every Ψ ∈ H 1 ( m ρ , B ) \Psi\in H^1(m_\rho,B) Ψ ∈ H 1 ( m ρ , B ) by the following arguments: the left-hand side is continuous with respect to the H 1 ( m ρ , B ) H^1(m_\rho,B) H 1 ( m ρ , B ) -norm because L m ρ , B Φ m ρ ∈ L 2 ( m ρ ) \frac{L_{m_\rho,B}\Phi}{m_\rho}\in L^2(m_\rho) m ρ L m ρ , B Φ ∈ L 2 ( m ρ ) , indeed,
∣ ∫ R d ( L m ρ , B Φ ) Ψ d x ∣ = ∣ ∫ R d L m ρ , B Φ m ρ Ψ d m ρ ∣ ≤ ∥ L m ρ , B Φ m ρ ∥ L 2 ( m ρ ) ∥ Ψ ∥ L 2 ( m ρ ) . \left|\int_{\mathbb{R}^d}(L_{m_\rho,B}\Phi)\Psi\,\mathrm{d}x\right|=\left|\int_{\mathbb{R}^d}\frac{L_{m_\rho,B}\Phi}{m_\rho}\Psi\,\mathrm{d}m_\rho\right|\leq\left\|\frac{L_{m_\rho,B}\Phi}{m_\rho}\right\|_{L^2(m_\rho)}\|\Psi\|_{L^2(m_\rho)}. ∫ R d ( L m ρ , B Φ ) Ψ d x = ∫ R d m ρ L m ρ , B Φ Ψ d m ρ ≤ m ρ L m ρ , B Φ L 2 ( m ρ ) ∥Ψ ∥ L 2 ( m ρ ) .
The right-hand side is also continuous due to the Cauchy–Schwarz inequality
∣ ∫ R d ∇ Φ T B ∇ Ψ d m ρ ∣ ≤ ∥ ∇ Φ ∥ L 2 ( m ρ , B ; R d ) ∥ ∇ Ψ ∥ L 2 ( m ρ , B ; R d ) . \left|\int_{\mathbb{R}^d}\nabla\Phi^{\mathsf{T}}B\nabla\Psi\,\mathrm{d}m_\rho\right|\leq\|\nabla\Phi\|_{L^2(m_\rho,B;\mathbb{R}^d)}\|\nabla\Psi\|_{L^2(m_\rho,B;\mathbb{R}^d)}. ∫ R d ∇ Φ T B ∇Ψ d m ρ ≤ ∥∇Φ ∥ L 2 ( m ρ , B ; R d ) ∥∇Ψ ∥ L 2 ( m ρ , B ; R d ) .
By Lemma 4.2, the space C c ∞ ( R d ) C^\infty_c(\mathbb{R}^d) C c ∞ ( R d ) is dense in H 1 ( m ρ , B ) H^1(m_\rho,B) H 1 ( m ρ , B ) . Thus, for every Ψ ∈ H 1 ( m ρ , B ) \Psi\in H^1(m_\rho,B) Ψ ∈ H 1 ( m ρ , B ) , there exists a sequence Ψ k ∈ C c ∞ ( R d ) \Psi_k\in C^\infty_c(\mathbb{R}^d) Ψ k ∈ C c ∞ ( R d ) such that Ψ k → Ψ \Psi_k\to\Psi Ψ k → Ψ in H 1 ( m ρ , B ) H^1(m_\rho,B) H 1 ( m ρ , B ) . Applying the integration-by-parts identity (134) to each Ψ k \Psi_k Ψ k and passing to the limit, proves (32) for every Ψ ∈ H 1 ( m ρ , B ) \Psi\in H^1(m_\rho,B) Ψ ∈ H 1 ( m ρ , B ) .
The proof is divided into three parts. We first prove that L m ρ , B L_{m_\rho,B} L m ρ , B is well-defined, then that it is
surjective and injective.
Well-definedness: Since m ρ ( R d ) < ∞ m_\rho(\mathbb{R}^d)<\infty m ρ ( R d ) < ∞ , the constant function 1 1 1 belongs to H 1 ( m ρ , B ) H^1(m_\rho,B) H 1 ( m ρ , B ) . Taking Ψ = 1 \Psi=1 Ψ = 1 in (32) gives
∫ R d L m ρ , B Φ d x = 0. \int_{\mathbb{R}^d} L_{m_\rho,B}\Phi\,\mathrm{d}x=0. ∫ R d L m ρ , B Φ d x = 0.
By definition of T ρ ∗ P m ρ , B T_\rho^*\mathcal{P}_{m_\rho,B} T ρ ∗ P m ρ , B in (31) we have L m ρ , B Φ m ρ ∈ L 2 ( m ρ ) \frac{L_{m_\rho,B}\Phi}{m_\rho}\in L^2(m_\rho) m ρ L m ρ , B Φ ∈ L 2 ( m ρ ) . Since m ρ m_\rho m ρ and Φ \Phi Φ are smooth,
L m ρ , B Φ = − ∇ ⋅ ( m ρ B ∇ Φ ) L_{m_\rho,B}\Phi=-\nabla\cdot(m_\rho B\nabla\Phi) L m ρ , B Φ = − ∇ ⋅ ( m ρ B ∇Φ )
is also smooth. Therefore, L m ρ , B Φ ∈ T ρ P m ρ , B L_{m_\rho,B}\Phi\in T_\rho\mathcal{P}_{m_\rho,B} L m ρ , B Φ ∈ T ρ P m ρ , B . Furthermore, for every c ∈ R c\in\mathbb{R} c ∈ R ,
L m ρ , B ( Φ + c ) = L m ρ , B Φ . (135) L_{m_\rho,B}(\Phi+c)=L_{m_\rho,B}\Phi. \tag*{(135)} L m ρ , B ( Φ + c ) = L m ρ , B Φ. (135)
Thus, the value of L m ρ , B [ Φ ] L_{m_\rho,B}[\Phi] L m ρ , B [ Φ ] is independent of the representative of [ Φ ] [\Phi] [ Φ ] , and the operator in (33) is well defined. Its linearity follows directly from its definition.
Surjectivity: By Lemma 4.2 , we know that the bilinear form
a ρ ( Φ , Ψ ) : = ∫ R d ∇ Φ T B ∇ Ψ d m ρ a_\rho(\Phi,\Psi):=\int_{\mathbb{R}^d}\nabla\Phi^{\mathsf T}B\nabla\Psi\,\mathrm{d}m_\rho a ρ ( Φ , Ψ ) := ∫ R d ∇ Φ T B ∇Ψ d m ρ
is bounded and coercive on ( H 0 1 ( m ρ , B ) , ∥ ⋅ ∥ H 1 ( m ρ , B ) ) \left(H^1_0(m_\rho,B),\|\cdot\|_{H^1(m_\rho,B)}\right) ( H 0 1 ( m ρ , B ) , ∥ ⋅ ∥ H 1 ( m ρ , B ) ) . For every ρ ˙ ∈ T ρ P m ρ , B \dot{\rho}\in T_\rho\mathcal{P}_{m_\rho,B} ρ ˙ ∈ T ρ P m ρ , B , the linear functional
ℓ ρ ˙ : H 0 1 ( m ρ , B ) → R , ℓ ρ ˙ ( Ψ ) : = ∫ R d ρ ˙ Ψ d x . \ell_{\dot{\rho}}:H^1_0(m_\rho,B)\to\mathbb{R},\qquad\ell_{\dot{\rho}}(\Psi):=\int_{\mathbb{R}^d}\dot{\rho}\Psi\,\mathrm{d}x. ℓ ρ ˙ : H 0 1 ( m ρ , B ) → R , ℓ ρ ˙ ( Ψ ) := ∫ R d ρ ˙ Ψ d x .
is continuous on H 0 1 ( m ρ , B ) H^1_0(m_\rho,B) H 0 1 ( m ρ , B ) , since we have by (26) that
∣ ℓ ρ ˙ ( Ψ ) ∣ = ∣ ∫ R d ρ ˙ m ρ Ψ d m ρ ∣ ≤ ∥ ρ ˙ m ρ ∥ L 2 ( m ρ ) ∥ Ψ ∥ L 2 ( m ρ ) ≤ C ρ ∥ ρ ˙ m ρ ∥ L 2 ( m ρ ) ∥ Ψ ∥ m ρ , B . \left|\ell_{\dot{\rho}}(\Psi)\right|=\left|\int_{\mathbb{R}^d}\frac{\dot{\rho}}{m_\rho}\Psi\,\mathrm{d}m_\rho\right|\leq\left\|\frac{\dot{\rho}}{m_\rho}\right\|_{L^2(m_\rho)}\|\Psi\|_{L^2(m_\rho)}\leq\sqrt{C_\rho}\left\|\frac{\dot{\rho}}{m_\rho}\right\|_{L^2(m_\rho)}\|\Psi\|_{m_\rho,B}. ∣ ℓ ρ ˙ ( Ψ ) ∣ = ∫ R d m ρ ρ ˙ Ψ d m ρ ≤ m ρ ρ ˙ L 2 ( m ρ ) ∥Ψ ∥ L 2 ( m ρ ) ≤ C ρ m ρ ρ ˙ L 2 ( m ρ ) ∥Ψ ∥ m ρ , B .
Then, by the Lax–Milgram theorem, there exists a unique Φ ρ ˙ ∈ H 0 1 ( m ρ , B ) \Phi_{\dot{\rho}}\in H^1_0(m_\rho,B) Φ ρ ˙ ∈ H 0 1 ( m ρ , B ) such that
∫ R d ∇ Φ ρ ˙ T B ∇ Ψ d m ρ = ∫ R d ρ ˙ Ψ d x for all Ψ ∈ H 0 1 ( m ρ , B ) . (136) \int_{\mathbb{R}^d}\nabla\Phi_{\dot{\rho}}^{\mathsf T}B\nabla\Psi\,\mathrm{d}m_\rho=\int_{\mathbb{R}^d}\dot{\rho}\Psi\,\mathrm{d}x\quad\text{for all }\Psi\in H^1_0(m_\rho,B). \tag*{(136)} ∫ R d ∇ Φ ρ ˙ T B ∇Ψ d m ρ = ∫ R d ρ ˙ Ψ d x for all Ψ ∈ H 0 1 ( m ρ , B ) . (136)
Next, let ψ ∈ C c ∞ ( R d ) \psi\in C^\infty_c(\mathbb{R}^d) ψ ∈ C c ∞ ( R d ) and define its centered version Ψ \Psi Ψ by (28) , in particular, ∇ Ψ = ∇ ψ \nabla\Psi=\nabla\psi ∇Ψ = ∇ ψ . Furthermore, since ρ ˙ ∈ T ρ P m ρ , B \dot{\rho}\in T_\rho\mathcal{P}_{m_\rho,B} ρ ˙ ∈ T ρ P m ρ , B , it has zero mean and therefore
∫ R d ρ ˙ Ψ d x = ∫ R d ρ ˙ ψ d x − 1 m ρ ( R d ) ∫ R d ψ d m ρ ∫ R d ρ ˙ d x = ∫ R d ρ ˙ ψ d x . \int_{\mathbb{R}^d}\dot{\rho}\Psi\,\mathrm{d}x=\int_{\mathbb{R}^d}\dot{\rho}\psi\,\mathrm{d}x-\frac{1}{m_\rho(\mathbb{R}^d)}\int_{\mathbb{R}^d}\psi\,\mathrm{d}m_\rho\int_{\mathbb{R}^d}\dot{\rho} \,\mathrm{d}x=\int_{\mathbb{R}^d}\dot{\rho}\psi\,\mathrm{d}x. ∫ R d ρ ˙ Ψ d x = ∫ R d ρ ˙ ψ d x − m ρ ( R d ) 1 ∫ R d ψ d m ρ ∫ R d ρ ˙ d x = ∫ R d ρ ˙ ψ d x .
In combination with (136) , this implies
∫ R d ∇ Φ ρ ˙ T B ∇ ψ d m ρ = ∫ R d ρ ˙ ψ d x for all ψ ∈ C c ∞ ( R d ) . \int_{\mathbb{R}^d}\nabla\Phi_{\dot{\rho}}^{\mathsf T}B\nabla\psi\,\mathrm{d}m_\rho=\int_{\mathbb{R}^d}\dot{\rho}\psi\,\mathrm{d}x\quad\text{for all }\psi\in C^\infty_c(\mathbb{R}^d). ∫ R d ∇ Φ ρ ˙ T B ∇ ψ d m ρ = ∫ R d ρ ˙ ψ d x for all ψ ∈ C c ∞ ( R d ) .
By the definition of distributional derivatives, this is equivalent to
− ∇ ⋅ ( m ρ B ∇ Φ ρ ˙ ) = ρ ˙ in D ′ ( R d ) . (137) -\nabla\cdot(m_{\rho}B\nabla\Phi_{\dot{\rho}})=\dot{\rho}\quad\text{in }\mathcal{D}'(\mathbb{R}^{d}). \tag*{(137)} − ∇ ⋅ ( m ρ B ∇ Φ ρ ˙ ) = ρ ˙ in D ′ ( R d ) . (137)
Next, we verify that the weak solution Φ ρ ˙ \Phi_{\dot{\rho}} Φ ρ ˙ is smooth. Let K ⊂ R d K\subset\mathbb{R}^{d} K ⊂ R d be compact. Since m ρ m_{\rho} m ρ is smooth and strictly positive, there exists c K > 0 c_{K}>0 c K > 0 such that m ρ ( x ) ≥ c K m_{\rho}(x)\ge c_{K} m ρ ( x ) ≥ c K for all x ∈ K x\in K x ∈ K . Therefore, we obtain for every x ∈ K x\in K x ∈ K and ξ ∈ R d \xi\in\mathbb{R}^{d} ξ ∈ R d that
ξ T m ρ ( x ) B ξ ≥ c K λ min ( B ) ∣ ξ ∣ 2 . \xi^{\mathsf{T}}m_{\rho}(x)B\xi\ge c_{K}\lambda_{\min}(B)|\xi|^{2}. ξ T m ρ ( x ) B ξ ≥ c K λ m i n ( B ) ∣ ξ ∣ 2 .
Hence the coefficient matrix m ρ B m_{\rho}B m ρ B is smooth and uniformly elliptic on every compact subset of R d \mathbb{R}^{d} R d . Since m ρ m_{\rho} m ρ is smooth and strictly positive and B B B is symmetric positive definite, the coefficient matrix m ρ B m_{\rho}B m ρ B is smooth and uniformly elliptic on every relatively compact open subset of R d \mathbb{R}^{d} R d . These local lower bounds also imply that Φ ρ ˙ ∈ H 1 ( m ρ , B ) \Phi_{\dot{\rho}}\in H^{1}(m_{\rho},B) Φ ρ ˙ ∈ H 1 ( m ρ , B ) belongs to H l o c 1 ( R d ) H^{1}_{\mathrm{loc}}(\mathbb{R}^{d}) H loc 1 ( R d ) . Thus, since ρ ˙ ∈ C ∞ ( R d ) \dot{\rho}\in C^{\infty}(\mathbb{R}^{d}) ρ ˙ ∈ C ∞ ( R d ) , standard interior elliptic regularity applied to (137) yields Φ ρ ˙ ∈ C ∞ ( R d ) \Phi_{\dot{\rho}}\in C^{\infty}(\mathbb{R}^{d}) Φ ρ ˙ ∈ C ∞ ( R d ) . Moreover, by (137) it holds L m ρ , B Φ ρ ˙ = ρ ˙ L_{m_{\rho},B}\Phi_{\dot{\rho}}=\dot{\rho} L m ρ , B Φ ρ ˙ = ρ ˙ . Consequently, using ρ ˙ ∈ T ρ P m ρ , B \dot{\rho}\in T_{\rho}\mathcal{P}_{m_{\rho},B} ρ ˙ ∈ T ρ P m ρ , B , we have
L m ρ , B Φ ρ ˙ m ρ = ρ ˙ m ρ ∈ L 2 ( m ρ ) . \frac{L_{m_{\rho},B}\Phi_{\dot{\rho}}}{m_{\rho}}=\frac{\dot{\rho}}{m_{\rho}}\in L^{2}(m_{\rho}). m ρ L m ρ , B Φ ρ ˙ = m ρ ρ ˙ ∈ L 2 ( m ρ ) .
Together with Φ ρ ˙ ∈ C ∞ ( R d ) ∩ H 1 ( m ρ , B ) \Phi_{\dot{\rho}}\in C^{\infty}(\mathbb{R}^{d})\cap H^{1}(m_{\rho},B) Φ ρ ˙ ∈ C ∞ ( R d ) ∩ H 1 ( m ρ , B ) , this proves that [ Φ ρ ˙ ] ∈ T ρ ∗ P m ρ , B [\Phi_{\dot{\rho}}]\in T_{\rho}^{*}\mathcal{P}_{m_{\rho},B} [ Φ ρ ˙ ] ∈ T ρ ∗ P m ρ , B . Finally, by (135), we conclude L m ρ , B [ Φ ρ ˙ ] = ρ ˙ L_{m_{\rho},B}[\Phi_{\dot{\rho}}]=\dot{\rho} L m ρ , B [ Φ ρ ˙ ] = ρ ˙ , which proves the surjectivity of L m ρ , B L_{m_{\rho},B} L m ρ , B .
Injectivity: Suppose that [ Φ ] ∈ T ρ ∗ P m ρ , B [\Phi]\in T_{\rho}^{*}\mathcal{P}_{m_{\rho},B} [ Φ ] ∈ T ρ ∗ P m ρ , B and L m ρ , B [ Φ ] = 0 L_{m_{\rho},B}[\Phi]=0 L m ρ , B [ Φ ] = 0 . Applying (32) with Ψ = Φ \Psi=\Phi Ψ = Φ , we obtain
0 = ∫ R d ∇ Φ T B ∇ Φ d m ρ . 0=\int_{\mathbb{R}^{d}}\nabla\Phi^{\mathsf{T}}B\nabla\Phi\,\mathrm{d}m_{\rho}. 0 = ∫ R d ∇ Φ T B ∇Φ d m ρ .
Since m ρ > 0 m_{\rho}>0 m ρ > 0 and B ≻ 0 B\succ0 B ≻ 0 , the integrand is nonnegative and can vanish only if ∇ Φ = 0 \nabla\Phi=0 ∇Φ = 0 . Thus, Φ \Phi Φ is constant on R d \mathbb{R}^{d} R d , and hence [ Φ ] = 0 [\Phi]=0 [ Φ ] = 0 . This proves the injectivity of L m ρ , B L_{m_{\rho},B} L m ρ , B and finishes the proof. □
Proof of Corollary 4.6. First using [ δ F δ ρ ] ∈ T ρ ∗ P m ρ , B \left[\frac{\delta F}{\delta\rho}\right]\in T_{\rho}^{*}\mathcal{P}_{m_{\rho},B} [ δ ρ δ F ] ∈ T ρ ∗ P m ρ , B in combination with the first part of Theorem 4.3 shows that
− ∇ ⋅ ( m ρ B ∇ δ F δ ρ ) ∈ T ρ P m ρ , B . -\nabla\cdot\left(m_{\rho}B\nabla\frac{\delta F}{\delta\rho}\right)\in T_{\rho}\mathcal{P}_{m_{\rho},B}. − ∇ ⋅ ( m ρ B ∇ δ ρ δ F ) ∈ T ρ P m ρ , B .
For any η ˙ ∈ T ρ P m ρ , B \dot{\eta}\in T_{\rho}\mathcal{P}_{m_{\rho},B} η ˙ ∈ T ρ P m ρ , B with associated local-mobility potential Φ η ˙ \Phi_{\dot{\eta}} Φ η ˙ we get by (35) and integration by parts
G m ρ , B ( L m ρ , B [ δ F δ ρ ] , η ˙ ) = ∫ R d ( ∇ δ F δ ρ ) T B ∇ Φ η ˙ d m ρ = ∫ R d δ F δ ρ η ˙ d x = D F ( ρ ) [ η ˙ ] . G_{m_{\rho},B}\left(L_{m_{\rho},B}\left[\frac{\delta F}{\delta\rho}\right],\dot{\eta}\right)=\int_{\mathbb{R}^{d}}\left(\nabla\frac{\delta F}{\delta\rho}\right)^{\mathsf{T}}B\nabla\Phi_{\dot{\eta}}\,\mathrm{d}m_{\rho}=\int_{\mathbb{R}^{d}}\frac{\delta F}{\delta\rho}\dot{\eta}\,\mathrm{d}x=DF(\rho)[\dot{\eta}]. G m ρ , B ( L m ρ , B [ δ ρ δ F ] , η ˙ ) = ∫ R d ( ∇ δ ρ δ F ) T B ∇ Φ η ˙ d m ρ = ∫ R d δ ρ δ F η ˙ d x = D F ( ρ ) [ η ˙ ] .
Thus, (36) satisfies the defining gradient identity. Uniqueness follows from the positive definiteness of G m ρ , B G_{m_{\rho},B} G m ρ , B . □
Proofs for Section 5
Proof of Lemma 5.1. 1. Linearity is immediate, and constants do not change M ρ M_{\rho} M ρ , so the formula does not depend on the representative. Next, a quick computation confirms for every C ∈ R d × d C\in\mathbb{R}^{d\times d} C ∈ R d × d that
− ∇ ⋅ ( ρ C x ) ρ = − tr ( C ) − ∇ log ρ ( x ) T C x . (138) \frac{-\nabla\cdot(\rho Cx)}{\rho}=-\operatorname{tr}(C)-\nabla\log\rho(x)^{\mathsf{T}}Cx. \tag*{(138)} ρ − ∇ ⋅ ( ρC x ) = − tr ( C ) − ∇ log ρ ( x ) T C x . (138)
For C = M ρ ( Ψ ) T A C=M_{\rho}(\Psi)^{\mathsf{T}}A C = M ρ ( Ψ ) T A , this becomes
L ρ K A [ Ψ ] ρ = − tr ( M ρ ( Ψ ) T A ) − ∇ log ρ ( x ) T M ρ ( Ψ ) T A x . (139) \frac{L_{\rho}^{K_A}[\Psi]}{\rho}=-\operatorname{tr}\left(M_{\rho}(\Psi)^{\mathsf{T}}A\right)-\nabla\log\rho(x)^{\mathsf{T}}M_{\rho}(\Psi)^{\mathsf{T}}Ax. \tag*{(139)} ρ L ρ K A [ Ψ ] = − tr ( M ρ ( Ψ ) T A ) − ∇ log ρ ( x ) T M ρ ( Ψ ) T A x . (139)
Using (139) in combination with (46) , we obtain
∥ L ρ K A [ Ψ ] ρ ∥ L 2 ( ρ ) ≤ [ d + ( ∫ R d ∣ x ∣ 2 ∣ ∇ log ρ ∣ 2 d ρ ) 1 / 2 ] ∥ A ∥ o p ∥ M ρ ( Ψ ) ∥ F < ∞ . (140) \left\|\frac{L_{\rho}^{K_A}[\Psi]}{\rho}\right\|_{L^2(\rho)}
\leq\left[\sqrt{d}+\left(\int_{\mathbb{R}^d}|x|^2|\nabla\log\rho|^2\,d\rho\right)^{1/2}\right]\|A\|_{\mathrm{op}}\|M_{\rho}(\Psi)\|_{\mathrm{F}}<\infty. \tag*{(140)} ρ L ρ K A [ Ψ ] L 2 ( ρ ) ≤ [ d + ( ∫ R d ∣ x ∣ 2 ∣∇ log ρ ∣ 2 d ρ ) 1/2 ] ∥ A ∥ op ∥ M ρ ( Ψ ) ∥ F < ∞. (140)
The output is smooth and satisfies the required score condition.
Further, L ρ K A [ Ψ ] L_{\rho}^{K_A}[\Psi] L ρ K A [ Ψ ] belongs to T ρ P K A T_{\rho}\mathcal{P}_{K_A} T ρ P K A for the following reason: let χ R \chi_R χ R be a smooth cutoff with 0 ≤ χ R ≤ 1 0\leq\chi_R\leq1 0 ≤ χ R ≤ 1 , equal to 1 1 1 on B R B_R B R , supported in B 2 R B_{2R} B 2 R , and satisfying ∣ ∇ χ R ∣ ≤ C / R |\nabla\chi_R|\leq C/R ∣∇ χ R ∣ ≤ C / R . The score bound makes the output integrable, and distributional integration by parts gives
∣ ∫ R d L ρ K A [ Ψ ] d x ∣ = lim R → ∞ ∣ ∫ R d ρ M ρ ( Ψ ) T A x ⋅ ∇ χ R d x ∣ \left|\int_{\mathbb{R}^d}L_{\rho}^{K_A}[\Psi]\,dx\right|
=\lim_{R\to\infty}\left|\int_{\mathbb{R}^d}\rho M_{\rho}(\Psi)^{\mathsf{T}}Ax\cdot\nabla\chi_R\,dx\right| ∫ R d L ρ K A [ Ψ ] d x = R → ∞ lim ∫ R d ρ M ρ ( Ψ ) T A x ⋅ ∇ χ R d x
≤ 2 C ∥ M ρ ( Ψ ) T A ∥ o p lim R → ∞ ∫ { ∣ x ∣ > R } ρ ( x ) d x = 0. \leq2C\|M_{\rho}(\Psi)^{\mathsf{T}}A\|_{\mathrm{op}}\lim_{R\to\infty}\int_{\{|x|>R\}}\rho(x)\,dx=0. ≤ 2 C ∥ M ρ ( Ψ ) T A ∥ op R → ∞ lim ∫ { ∣ x ∣ > R } ρ ( x ) d x = 0.
If Φ \Phi Φ is compactly supported, integration by parts yields
⟨ [ Φ ] , L ρ K A [ Ψ ] ⟩ ρ = ∫ R d ∇ Φ T M ρ ( Ψ ) T A x d ρ = tr ( M ρ ( Φ ) T A M ρ ( Ψ ) ) . (141) \langle[\Phi],L_{\rho}^{K_A}[\Psi]\rangle_{\rho}
=\int_{\mathbb{R}^d}\nabla\Phi^{\mathsf{T}}M_{\rho}(\Psi)^{\mathsf{T}}Ax\,d\rho
=\operatorname{tr}\left(M_{\rho}(\Phi)^{\mathsf{T}}AM_{\rho}(\Psi)\right). \tag*{(141)} ⟨[ Φ ] , L ρ K A [ Ψ ] ⟩ ρ = ∫ R d ∇ Φ T M ρ ( Ψ ) T A x d ρ = tr ( M ρ ( Φ ) T A M ρ ( Ψ ) ) . (141)
For general Φ ∈ C ∞ ∩ H 1 ( ρ ) \Phi\in C^\infty\cap H^1(\rho) Φ ∈ C ∞ ∩ H 1 ( ρ ) , the cutoffs Φ R : = χ R Φ \Phi_R:=\chi_R\Phi Φ R := χ R Φ satisfy
∥ Φ R − Φ ∥ L 2 ( ρ ) → 0 , and ∥ ∇ Φ R − ∇ Φ ∥ L 2 ( ρ ) ≤ ∥ ( 1 − χ R ) ∇ Φ ∥ L 2 ( ρ ) + C R ∥ Φ 1 { R < ∣ x ∣ < 2 R } ∥ L 2 ( ρ ) → 0. \|\Phi_R-\Phi\|_{L^2(\rho)}\to0,\qquad\text{and}\qquad
\|\nabla\Phi_R-\nabla\Phi\|_{L^2(\rho)}
\leq\|(1-\chi_R)\nabla\Phi\|_{L^2(\rho)}+\frac{C}{R}\|\Phi\mathbf{1}_{\{R<|x|<2R\}}\|_{L^2(\rho)}\to0. ∥ Φ R − Φ ∥ L 2 ( ρ ) → 0 , and ∥∇ Φ R − ∇Φ ∥ L 2 ( ρ ) ≤ ∥ ( 1 − χ R ) ∇Φ ∥ L 2 ( ρ ) + R C ∥Φ 1 { R < ∣ x ∣ < 2 R } ∥ L 2 ( ρ ) → 0.
Thus Φ R → Φ \Phi_R\to\Phi Φ R → Φ in H 1 ( ρ ) H^1(\rho) H 1 ( ρ ) . By Cauchy–Schwarz and (140) ,
∣ ⟨ [ Φ R − Φ ] , L ρ K A [ Ψ ] ⟩ ρ ∣ ≤ ∥ Φ R − Φ ∥ L 2 ( ρ ) ∥ L ρ K A [ Ψ ] ρ ∥ L 2 ( ρ ) → 0. (142) \left|\left\langle[\Phi_R-\Phi],L_{\rho}^{K_A}[\Psi]\right\rangle_{\rho}\right|
\leq\|\Phi_R-\Phi\|_{L^2(\rho)}
\left\|\frac{L_{\rho}^{K_A}[\Psi]}{\rho}\right\|_{L^2(\rho)}
\to0. \tag*{(142)} ⟨ [ Φ R − Φ ] , L ρ K A [ Ψ ] ⟩ ρ ≤ ∥ Φ R − Φ ∥ L 2 ( ρ ) ρ L ρ K A [ Ψ ] L 2 ( ρ ) → 0. (142)
Next, using
∥ M ρ ( Ψ ) T A x ∥ L 2 ( ρ ) ≤ ∥ M ρ ( Ψ ) T A ∥ o p ( ∫ R d ∣ x ∣ 2 d ρ ) 1 / 2 < ∞ \|M_{\rho}(\Psi)^{\mathsf{T}}Ax\|_{L^2(\rho)}
\leq\|M_{\rho}(\Psi)^{\mathsf{T}}A\|_{\mathrm{op}}
\left(\int_{\mathbb{R}^d}|x|^2\,d\rho\right)^{1/2}<\infty ∥ M ρ ( Ψ ) T A x ∥ L 2 ( ρ ) ≤ ∥ M ρ ( Ψ ) T A ∥ op ( ∫ R d ∣ x ∣ 2 d ρ ) 1/2 < ∞
in combination with (142) , we conclude that the gradient integral in (140) satisfies
∣ ∫ R d ∇ ( Φ R − Φ ) T M ρ ( Ψ ) T A x d ρ ∣ ≤ ∥ ∇ Φ R − ∇ Φ ∥ L 2 ( ρ ) ∥ M ρ ( Ψ ) T A ∥ o p ( ∫ R d ∣ x ∣ 2 d ρ ) 1 / 2 . \left|\int_{\mathbb{R}^d}\nabla(\Phi_R-\Phi)^{\mathsf{T}}M_{\rho}(\Psi)^{\mathsf{T}}Ax\,d\rho\right|
\leq\|\nabla\Phi_R-\nabla\Phi\|_{L^2(\rho)}
\|M_{\rho}(\Psi)^{\mathsf{T}}A\|_{\mathrm{op}}
\left(\int_{\mathbb{R}^d}|x|^2\,d\rho\right)^{1/2}. ∫ R d ∇ ( Φ R − Φ ) T M ρ ( Ψ ) T A x d ρ ≤ ∥∇ Φ R − ∇Φ ∥ L 2 ( ρ ) ∥ M ρ ( Ψ ) T A ∥ op ( ∫ R d ∣ x ∣ 2 d ρ ) 1/2 .
Thus, passing to the limit in (141) proves (51).
Proof of Lemma 5.2. The inclusion from right to left follows from the definition of L ρ K A L_{\rho}^{K_A} L ρ K A . Conversely, if L ρ K A [ Φ ] = 0 L_{\rho}^{K_A}[\Phi] = 0 L ρ K A [ Φ ] = 0 , the Green identity in item 2 of Lemma 5.1, with Ψ = Φ \Psi= \Phi Ψ = Φ , gives
0 = tr ( M ρ ( Φ ) T A M ρ ( Φ ) ) = ∥ A 1 / 2 M ρ ( Φ ) ∥ F 2 0 = \operatorname{tr}\left(M_{\rho}(\Phi)^{\mathsf{T}} A M_{\rho}(\Phi)\right) = \left\|A^{1/2}M_{\rho}(\Phi)\right\|_{\mathrm{F}}^2 0 = tr ( M ρ ( Φ ) T A M ρ ( Φ ) ) = A 1/2 M ρ ( Φ ) F 2
with the Frobenius norm ∥ ⋅ ∥ F \|\cdot\|_{\mathrm{F}} ∥ ⋅ ∥ F . Since A A A is positive definite, M ρ ( Φ ) = 0 M_{\rho}(\Phi) = 0 M ρ ( Φ ) = 0 . For infinite-dimensionality, choose pairwise disjoint open balls B n ⊂ R d B_n \subset\mathbb{R}^d B n ⊂ R d and nonzero φ n ∈ C c ∞ ( B n ) \varphi_n \in C^\infty_{\mathrm{c}}(B_n) φ n ∈ C c ∞ ( B n ) . Their classes [ φ n ] [\varphi_n] [ φ n ] are linearly independent in T ρ ∗ P K A T_{\rho}^{*}\mathcal{P}_{K_A} T ρ ∗ P K A , so this cotangent space is infinite-dimensional. On the other hand, L ρ K A L_{\rho}^{K_A} L ρ K A factors through M ‾ ρ \overline{M}_{\rho} M ρ , whose range lies in R d × d \mathbb{R}^{d \times d} R d × d . Hence ran L ρ K A \operatorname{ran} L_{\rho}^{K_A} ran L ρ K A is finite-dimensional. By the first isomorphism theorem,
T ρ ∗ P K A / ker L ρ K A ≅ ran L ρ K A . T_{\rho}^{*}\mathcal{P}_{K_A} / \ker L_{\rho}^{K_A} \cong\operatorname{ran} L_{\rho}^{K_A}. T ρ ∗ P K A / ker L ρ K A ≅ ran L ρ K A .
Thus, ker L ρ K A \ker L_{\rho}^{K_A} ker L ρ K A has finite codimension in an infinite-dimensional space and is itself infinite-dimensional.
Proof of Theorem 5.3. If two classes in T ρ ∗ P K A T_{\rho}^{*}\mathcal{P}_{K_A} T ρ ∗ P K A determine the same quotient class, their difference belongs to ker L ρ K A \ker L_{\rho}^{K_A} ker L ρ K A , so (53) is well-defined. Its kernel is zero by the definition of the quotient, and it is onto by the definition T ρ e f f P K A = ran L ρ K A T_{\rho}^{\mathrm{eff}}\mathcal{P}_{K_A} = \operatorname{ran} L_{\rho}^{K_A} T ρ eff P K A = ran L ρ K A . Hence it is an isomorphism. The final identification follows from Lemma 5.2 and the first isomorphism theorem applied to M ρ M_{\rho} M ρ .
Proofs for Section 6
Proof of Proposition 6.2. We first compute the variation along compactly supported density perturbations. Let η ∈ C c ∞ ( R d ) \eta\in C^\infty_{\mathrm{c}}(\mathbb{R}^d) η ∈ C c ∞ ( R d ) satisfy ∫ R d η d x = 0 \int_{\mathbb{R}^d} \eta\,\mathrm{d}x = 0 ∫ R d η d x = 0 . By Lemma B.3, we have that ρ + ε η \rho+ \varepsilon\eta ρ + ε η belongs to P S M \mathcal{P}_{\mathrm{SM}} P SM for sufficiently small ∣ ε ∣ |\varepsilon| ∣ ε ∣ . Since η / ρ \eta/\rho η / ρ is bounded, these perturbed densities and their normalizers are uniformly comparable with ρ \rho ρ and Z ρ Z_{\rho} Z ρ , respectively. Consequently, their interaction energies remain finite.
Symmetry of A A A in (58) gives
F ( ρ + ε η ) − F ( ρ ) = − ε ∫ R d Z ρ η d x − ε 2 2 ∬ R d × R d exp ( y T A x ) η ( x ) η ( y ) d x d y . \mathcal{F}(\rho+ \varepsilon\eta) - \mathcal{F}(\rho) = -\varepsilon\int_{\mathbb{R}^d} Z_{\rho}\eta\,\mathrm{d}x - \frac{\varepsilon^2}{2} \iint_{\mathbb{R}^d \times\mathbb{R}^d} \exp\left(y^{\mathsf{T}}Ax\right)\eta(x)\eta(y)\,\mathrm{d}x\,\mathrm{d}y. F ( ρ + ε η ) − F ( ρ ) = − ε ∫ R d Z ρ η d x − 2 ε 2 ∬ R d × R d exp ( y T A x ) η ( x ) η ( y ) d x d y .
The integrals are absolutely convergent because η \eta η has compact support and Z ρ Z_{\rho} Z ρ is smooth. Thus, the first variation is represented by − Z ρ -Z_{\rho} − Z ρ . Since this identity holds for every compactly supported zero-integral test, any regular first variation must equal [ − Z ρ ] [-Z_{\rho}] [ − Z ρ ] . By (44), this proves the necessity of (59).
For the converse, suppose that (59) holds. Finite energy gives
∫ R d Z ρ 2 m ρ d x = ∫ R d ρ Z ρ d x < ∞ . \int_{\mathbb{R}^d} Z_{\rho}^{2}m_{\rho}\,\mathrm{d}x = \int_{\mathbb{R}^d} \rho Z_{\rho}\,\mathrm{d}x < \infty. ∫ R d Z ρ 2 m ρ d x = ∫ R d ρ Z ρ d x < ∞.
It remains to verify the weighted gradient condition in (44).
Choose χ R ∈ C ∞ ( R d ) \chi_R \in C^\infty(\mathbb{R}^d) χ R ∈ C ∞ ( R d ) with 0 ≤ χ R ≤ 1 0 \le\chi_R \le1 0 ≤ χ R ≤ 1 , χ R = 1 \chi_R = 1 χ R = 1 on B R B_R B R , supp χ R ⊂ B 2 R \operatorname{supp}\chi_R \subset B_{2R} supp χ R ⊂ B 2 R , and ∣ ∇ χ R ∣ ≤ C / R |\nabla\chi_R| \le C/R ∣∇ χ R ∣ ≤ C / R . Multiplying L ρ S M Z ρ L^{\mathrm{SM}}_\rho Z_\rho L ρ SM Z ρ by χ R 2 Z ρ \chi_R^2 Z_\rho χ R 2 Z ρ , integrating by parts, and applying Young’s inequality gives
∫ R d χ R 2 ∇ Z ρ T B ∇ Z ρ m ρ d x ≤ 2 ( ∫ R d Z ρ 2 m ρ d x ) 1 / 2 ( ∫ R d ∣ L ρ S M Z ρ ∣ 2 m ρ d x ) 1 / 2 + C R 2 ∫ R d Z ρ 2 m ρ d x . \int_{\mathbb{R}^d} \chi_R^2 \nabla Z_\rho^\mathsf{T} B \nabla Z_\rho\,m_\rho\,dx
\le2\left(\int_{\mathbb{R}^d} Z_\rho^2m_\rho\,dx\right)^{1/2}
\left(\int_{\mathbb{R}^d}\frac{|L^{\mathrm{SM}}_\rho Z_\rho|^2}{m_\rho}\,dx\right)^{1/2}
+\frac{C}{R^2}\int_{\mathbb{R}^d}Z_\rho^2m_\rho\,dx. ∫ R d χ R 2 ∇ Z ρ T B ∇ Z ρ m ρ d x ≤ 2 ( ∫ R d Z ρ 2 m ρ d x ) 1/2 ( ∫ R d m ρ ∣ L ρ SM Z ρ ∣ 2 d x ) 1/2 + R 2 C ∫ R d Z ρ 2 m ρ d x .
where C C C is independent of R R R . The right-hand side is bounded uniformly for R ≥ 1 R \ge1 R ≥ 1 . Letting R → ∞ R \to\infty R → ∞ therefore yields
∫ R d ∇ Z ρ T B ∇ Z ρ m ρ d x < ∞ . \int_{\mathbb{R}^d}\nabla Z_\rho^\mathsf{T}B\nabla Z_\rho\,m_\rho\,dx < \infty. ∫ R d ∇ Z ρ T B ∇ Z ρ m ρ d x < ∞.
Together with (59) , this proves [ Z ρ ] ∈ T ρ ∗ P S M [Z_\rho] \in T_\rho^*P_{\mathrm{SM}} [ Z ρ ] ∈ T ρ ∗ P SM . The variation computed above is therefore a local regular first variation in the sense of (117) . Its extension to the regular tangent space is
ℓ F , ρ r e g ( η ) = − ∫ R d Z ρ η d x , η ∈ T ρ P S M . \ell^{\mathrm{reg}}_{\mathcal{F},\rho}(\eta)=-\int_{\mathbb{R}^d}Z_\rho\eta\,dx,\qquad\eta\in T_\rho P_{\mathrm{SM}}. ℓ F , ρ reg ( η ) = − ∫ R d Z ρ η d x , η ∈ T ρ P SM .
This integral is finite by weighted Cauchy–Schwarz, using Z ρ ∈ L 2 ( m ρ ) Z_\rho\in L^2(m_\rho) Z ρ ∈ L 2 ( m ρ ) and the tangent-space condition in (42) . □
Proof of Corollary 6.4. Proposition 6.2 identifies the regular first variation of F \mathcal{F} F with [ − Z ρ ] [-Z_\rho] [ − Z ρ ] . By the metric definition (45) ,
G ρ S M ( L ρ S M [ − Z ρ ] , η ) = − ∫ R d Z ρ η d x = ℓ F , ρ r e g ( η ) , η ∈ T ρ P S M . G_\rho^{\mathrm{SM}}(L_\rho^{\mathrm{SM}}[-Z_\rho],\eta)
=-\int_{\mathbb{R}^d}Z_\rho\eta\,dx
=\ell^{\mathrm{reg}}_{\mathcal{F},\rho}(\eta),\qquad\eta\in T_\rho P_{\mathrm{SM}}. G ρ SM ( L ρ SM [ − Z ρ ] , η ) = − ∫ R d Z ρ η d x = ℓ F , ρ reg ( η ) , η ∈ T ρ P SM .
Hence grad G ρ S M F ( ρ ) = L ρ S M [ − Z ρ ] \operatorname{grad}_{G_\rho^{\mathrm{SM}}}\mathcal{F}(\rho)=L_\rho^{\mathrm{SM}}[-Z_\rho] grad G ρ SM F ( ρ ) = L ρ SM [ − Z ρ ] . Differentiating the normalizer in (57) and using B A = V BA=V B A = V gives
B ∇ Z ρ ( x ) = V ∫ R d y exp ( y T A x ) ρ ( y ) d y = Z ρ ( x ) Γ ρ ( x ) . B\nabla Z_\rho(x)=V\int_{\mathbb{R}^d}y\exp(y^\mathsf{T}Ax)\rho(y)\,dy=Z_\rho(x)\Gamma_\rho(x). B ∇ Z ρ ( x ) = V ∫ R d y exp ( y T A x ) ρ ( y ) d y = Z ρ ( x ) Γ ρ ( x ) .
Substituting this identity into (43) proves (60) . The gradient-flow formulation follows by inserting this identity into (56) at densities satisfying the assumptions. □
Proof of Lemma 6.5. By (41) , Z ρ Z_\rho Z ρ is smooth and strictly positive. Moreover, [ Φ ] ∈ T ρ ∗ P S M [\Phi]\in T_\rho^*P_{\mathrm{SM}} [ Φ ] ∈ T ρ ∗ P SM gives
∫ R d ∇ Φ T B ∇ Φ m ρ d x < ∞ . \int_{\mathbb{R}^d}\nabla\Phi^\mathsf{T}B\nabla\Phi\,m_\rho\,dx<\infty. ∫ R d ∇ Φ T B ∇Φ m ρ d x < ∞.
We use this integrability to control the nonlocal term in the second equation of (61) . Strict positivity of ρ \rho ρ and invertibility of A A A imply that, for every r > 0 r>0 r > 0 , there exists c r > 0 c_r>0 c r > 0 such that
Z ρ ( x ) ≥ c r e r ∣ x ∣ ( x ∈ R d ) . (143) Z_\rho(x)\ge c_r e^{r|x|}\qquad(x\in\mathbb{R}^d). \tag*{(143)} Z ρ ( x ) ≥ c r e r ∣ x ∣ ( x ∈ R d ) . (143)
Indeed, choose s > 1 s>1 s > 1 such that ( s − 1 ) σ min ( A ) > r (s-1)\sigma_{\min}(A)>r ( s − 1 ) σ m i n ( A ) > r . For x ≠ 0 x\ne0 x = 0 , integrate over the unit ball centered at s A x / ∣ A x ∣ sAx/|Ax| s A x /∣ A x ∣ . These balls lie in a fixed compact set on which ρ \rho ρ has a positive minimum, and every point y y y in the corresponding ball satisfies
y T A x ≥ ( s − 1 ) ∣ A x ∣ > r ∣ x ∣ . y^{\mathsf T}Ax\ge(s-1)|Ax|>r|x|. y T A x ≥ ( s − 1 ) ∣ A x ∣ > r ∣ x ∣.
This proves (142) for x ≠ 0 x\ne0 x = 0 , decreasing c r c_r c r also includes x = 0 x=0 x = 0 . The nonlocal integral in (61) can be written as
∫ R d exp ( y T A x ) Z ρ ( y ) ∇ Φ ( y ) T B ∇ Φ ( y ) m ρ ( y ) d y . \int_{\mathbb{R}^d}\frac{\exp(y^{\mathsf T}Ax)}{Z_\rho(y)}\nabla\Phi(y)^{\mathsf T}B\nabla\Phi(y)m_\rho(y)\,\mathrm{d}y. ∫ R d Z ρ ( y ) exp ( y T A x ) ∇Φ ( y ) T B ∇Φ ( y ) m ρ ( y ) d y .
For x x x in a fixed compact set, the factor exp ( y T A x ) / Z ρ ( y ) \exp(y^{\mathsf T}Ax)/Z_\rho(y) exp ( y T A x ) / Z ρ ( y ) and each of its spatial derivatives are bounded uniformly in y y y by (142). The weighted gradient integrability of Φ \Phi Φ therefore justifies differentiation under this integral. Thus, the nonlocal term is smooth and finite. The remaining spatial terms in (61) are smooth because ρ \rho ρ , Φ \Phi Φ , and Z ρ Z_\rho Z ρ are smooth and Z ρ > 0 Z_\rho>0 Z ρ > 0 . This proves the pointwise formulation whenever α ( t ) \alpha(t) α ( t ) is finite. □
Proof of Theorem 6.7. We first compute the variation of (63) with respect to the potential. For [ Ψ ] ∈ T ρ ∗ P S M [\Psi]\in T_\rho^*\mathsf{PSM} [ Ψ ] ∈ T ρ ∗ PSM , differentiation and the Green identity for (43) give
d d ε ∣ ε = 0 H ( ρ , [ Φ + ε Ψ ] ) = ∫ R d ∇ Ψ T B ∇ Φ m ρ d x = ⟨ [ Ψ ] , L ρ S M [ Φ ] ⟩ . \left.\frac{\mathrm{d}}{\mathrm{d}\varepsilon}\right|_{\varepsilon=0}\mathcal{H}(\rho,[\Phi+\varepsilon\Psi])=\int_{\mathbb{R}^d}\nabla\Psi^{\mathsf T}B\nabla\Phi\,m_\rho\,\mathrm{d}x=\langle[\Psi],L_\rho^{\mathsf{SM}}[\Phi]\rangle. d ε d ε = 0 H ( ρ , [ Φ + ε Ψ ]) = ∫ R d ∇ Ψ T B ∇Φ m ρ d x = ⟨[ Ψ ] , L ρ SM [ Φ ]⟩ .
Consequently, the regular potential derivative in the sense of (122) is
δ r e g H δ Φ ( ρ , [ Φ ] ) = L ρ S M [ Φ ] ∈ T ρ P S M . (144) \frac{\delta^{\mathrm{reg}}\mathcal{H}}{\delta\Phi}(\rho,[\Phi])=L_\rho^{\mathsf{SM}}[\Phi]\in T_\rho\mathsf{PSM}. \tag*{(144)} δ Φ δ reg H ( ρ , [ Φ ]) = L ρ SM [ Φ ] ∈ T ρ PSM . (144)
Next, we vary the density while keeping [ Φ ] [\Phi] [ Φ ] fixed. Let η ∈ C c ∞ ( R d ) \eta\in C_c^\infty(\mathbb{R}^d) η ∈ C c ∞ ( R d ) satisfy ∫ R d η d x = 0 \int_{\mathbb{R}^d}\eta\,\mathrm{d}x=0 ∫ R d η d x = 0 , and set
Z η ( x ) : = ∫ R d exp ( y T A x ) η ( y ) d y . Z_\eta(x):=\int_{\mathbb{R}^d}\exp(y^{\mathsf T}Ax)\eta(y)\,\mathrm{d}y. Z η ( x ) := ∫ R d exp ( y T A x ) η ( y ) d y .
By Lemma B.8, condition (65) ensures that [ Φ ] ∈ T ρ + ε η ∗ P S M [\Phi]\in T_{\rho+\varepsilon\eta}^*\mathsf{PSM} [ Φ ] ∈ T ρ + ε η ∗ PSM for sufficiently small ∣ ε ∣ |\varepsilon| ∣ ε ∣ . The kinetic energy can therefore be evaluated along these density variations. The identity
m ρ + ε η = m ρ 1 + ε η / ρ 1 + ε Z η / Z ρ , ∣ Z η Z ρ ∣ ≤ ∥ η ρ ∥ ∞ , m_{\rho+\varepsilon\eta}=m_\rho\frac{1+\varepsilon\eta/\rho}{1+\varepsilon Z_\eta/Z_\rho},\qquad\left|\frac{Z_\eta}{Z_\rho}\right|\le\left\|\frac{\eta}{\rho}\right\|_\infty, m ρ + ε η = m ρ 1 + ε Z η / Z ρ 1 + ε η / ρ , Z ρ Z η ≤ ρ η ∞ ,
justifies differentiation under the kinetic-energy integral, since η / ρ \eta/\rho η / ρ is bounded and the kinetic energy is finite. It follows that
d d ε ∣ ε = 0 1 2 ∫ R d ∇ Φ T B ∇ Φ m ρ + ε η d x = 1 2 ∫ R d ∇ Φ T B ∇ Φ m ρ ( η ρ − Z η Z ρ ) d x . \left.\frac{\mathrm{d}}{\mathrm{d}\varepsilon}\right|_{\varepsilon=0}\frac{1}{2}\int_{\mathbb{R}^d}\nabla\Phi^{\mathsf T}B\nabla\Phi\,m_{\rho+\varepsilon\eta}\,\mathrm{d}x
=\frac{1}{2}\int_{\mathbb{R}^d}\nabla\Phi^{\mathsf T}B\nabla\Phi\,m_\rho\left(\frac{\eta}{\rho}-\frac{Z_\eta}{Z_\rho}\right)\,\mathrm{d}x. d ε d ε = 0 2 1 ∫ R d ∇ Φ T B ∇Φ m ρ + ε η d x = 2 1 ∫ R d ∇ Φ T B ∇Φ m ρ ( ρ η − Z ρ Z η ) d x .
In the term containing Z η Z_\eta Z η , Fubini’s theorem rewrites the right-hand side as the pairing of η \eta η with the expression inside the brackets in (64). The interchange is justified by
Z ∣ η ∣ Z ρ ≤ ∥ η ρ ∥ ∞ . \frac{Z_{|\eta|}}{Z_\rho}\le\left\|\frac{\eta}{\rho}\right\|_\infty. Z ρ Z ∣ η ∣ ≤ ρ η ∞ .
and the finite kinetic energy. Condition (64) therefore makes this density variation a regular cotangent class. Compactly supported zero-integral tests determine the class uniquely.
Adding the interaction-energy variation from Proposition 6.2 gives
δ r e g H δ ρ ( ρ , [ Φ ] ) = [ ∇ Φ ( x ) ⊤ B ∇ Φ ( x ) 2 Z ρ ( x ) − 1 2 ∫ R d exp ( y ⊤ A x ) ∇ Φ ( y ) ⊤ B ∇ Φ ( y ) Z ρ ( y ) 2 ρ ( y ) d y − Z ρ ( x ) ] . (145) \frac{\delta^{\mathrm{reg}}\mathcal{H}}{\delta\rho}(\rho,[\Phi]) =
\left[
\frac{\nabla\Phi(x)^{\top}B\nabla\Phi(x)}{2Z_{\rho}(x)}
-\frac{1}{2}\int_{\mathbb{R}^{d}}\exp(y^{\top}Ax)
\frac{\nabla\Phi(y)^{\top}B\nabla\Phi(y)}{Z_{\rho}(y)^{2}}\rho(y)\,\mathrm{d}y
-Z_{\rho}(x)
\right]. \tag*{(145)} δ ρ δ reg H ( ρ , [ Φ ]) = [ 2 Z ρ ( x ) ∇Φ ( x ) ⊤ B ∇Φ ( x ) − 2 1 ∫ R d exp ( y ⊤ A x ) Z ρ ( y ) 2 ∇Φ ( y ) ⊤ B ∇Φ ( y ) ρ ( y ) d y − Z ρ ( x ) ] . (145)
Both contributions belong to T ρ ∗ P S M T_{\rho}^{*}\mathcal{P}_{\mathrm{SM}} T ρ ∗ P SM , so their pairing with a regular tangent vector defines the regular density derivative in the sense of (123) .
Finally, substituting (144) and (145) into (66) yields (61) , with the potential equation understood modulo spatial constants.
Proof of Lemma 6.11. The proof consists of computing the connection of (68) . Fix U , Q ∈ R N × d \mathrm{U},\mathrm{Q}\in\mathbb{R}^{N\times d} U , Q ∈ R N × d and regard them as constant vector fields. The Koszul formula then reduces to
2 g X ( ∇ X g ( U , U ) , Q ) = 2 D g X [ U ] ( U , Q ) − D g X [ Q ] ( U , U ) . (146) 2g_{X}\left(\nabla^{g}_{X}(\mathrm{U},\mathrm{U}),\mathrm{Q}\right)
=2Dg_{X}[\mathrm{U}](\mathrm{U},\mathrm{Q})-Dg_{X}[\mathrm{Q}](\mathrm{U},\mathrm{U}). \tag*{(146)} 2 g X ( ∇ X g ( U , U ) , Q ) = 2 D g X [ U ] ( U , Q ) − D g X [ Q ] ( U , U ) . (146)
We compute the two derivatives by varying the base point X \mathrm{X} X in (68) , while keeping the matrix arguments fixed.
For the first derivative, (72) gives
D g X [ U ] ( U , Q ) = 1 N ⟨ D δ z X [ U ] U B − 1 , Q ⟩ F . Dg_{X}[\mathrm{U}](\mathrm{U},\mathrm{Q})
=\frac{1}{N}\left\langle D_{\delta z_{X}[\mathrm{U}]}\mathrm{U}B^{-1},\mathrm{Q}\right\rangle_{F}. D g X [ U ] ( U , Q ) = N 1 ⟨ D δ z X [ U ] U B − 1 , Q ⟩ F .
For the second derivative, use s i = U i ⊤ B − 1 U i s_i=\mathrm{U}_i^{\top}B^{-1}\mathrm{U}_i s i = U i ⊤ B − 1 U i from the lemma. Then
D g X [ Q ] ( U , U ) = 1 N ∑ i = 1 N s i δ z X [ Q ] i . Dg_{X}[\mathrm{Q}](\mathrm{U},\mathrm{U})
=\frac{1}{N}\sum_{i=1}^{N}s_i\delta z_{X}[\mathrm{Q}]_i. D g X [ Q ] ( U , U ) = N 1 i = 1 ∑ N s i δ z X [ Q ] i .
Expanding the directional derivative in (72) and using the symmetry of A A A and M M M yields
∑ i = 1 N s i δ z X [ Q ] i = ⟨ ( D s M + M D s ) X A , Q ⟩ F . \sum_{i=1}^{N}s_i\delta z_{X}[\mathrm{Q}]_i
=\left\langle(D_sM+MD_s)XA,\mathrm{Q}\right\rangle_F. i = 1 ∑ N s i δ z X [ Q ] i = ⟨ ( D s M + M D s ) X A , Q ⟩ F .
Substituting these expressions into (146) and using (68) gives
D z ∇ X g ( U , U ) B − 1 = D δ z X [ U ] U B − 1 − 1 2 ( D s M + M D s ) X A . D_z\nabla^{g}_{X}(\mathrm{U},\mathrm{U})B^{-1}
=D_{\delta z_{X}[\mathrm{U}]}\mathrm{U}B^{-1}
-\frac{1}{2}(D_sM+MD_s)XA. D z ∇ X g ( U , U ) B − 1 = D δ z X [ U ] U B − 1 − 2 1 ( D s M + M D s ) X A .
Multiplication by D z − 1 D_z^{-1} D z − 1 on the left and B B B on the right, together with A B = V ⊤ AB=V^{\top} A B = V ⊤ , proves (73) .
First-Order Linear Attention: Geometry and Analysis
We first verify that the covariance family (78) belongs to P K A \mathcal{P}_{K_A} P K A . We then prove the three lemmas concerning its covariance coordinates, induced metric, and gradient-flow formulation. Throughout, only second moments are required.
Radiality of η \eta η gives centeredness, and the change of variables x = S 1 / 2 z x=S^{1/2}z x = S 1/2 z gives covariance S S S and the weighted-score bound
∫ R d ∣ x ∣ 2 ∣ ∇ log ρ S ( x ) ∣ 2 d ρ S ( x ) ≤ ∥ S ∥ o p ∥ S − 1 ∥ o p ∫ R d ∣ z ∣ 2 ∣ ∇ log η ( z ) ∣ 2 d η ( z ) . \int_{\mathbb{R}^{d}} |x|^{2}|\nabla\log\rho_{S}(x)|^{2}\,\mathrm{d}\rho_{S}(x)
\le\|S\|_{\mathrm{op}}\|S^{-1}\|_{\mathrm{op}}
\int_{\mathbb{R}^{d}} |z|^{2}|\nabla\log\eta(z)|^{2}\,\mathrm{d}\eta(z). ∫ R d ∣ x ∣ 2 ∣∇ log ρ S ( x ) ∣ 2 d ρ S ( x ) ≤ ∥ S ∥ op ∥ S − 1 ∥ op ∫ R d ∣ z ∣ 2 ∣∇ log η ( z ) ∣ 2 d η ( z ) .
Thus ρ S ∈ P K A \rho_{S}\in\mathcal{P}_{K_A} ρ S ∈ P K A .
Proof of Lemma 7.1 . We first prove that Λ S \Lambda_{S} Λ S is an isomorphism. Diagonalize S A S SAS S A S and denote its eigenvalues by λ i > 0 \lambda_i>0 λ i > 0 . In this eigenbasis,
( Λ S ( P ) ) i j = ( λ i + λ j ) P i j , (\Lambda_{S}(P))_{ij}=(\lambda_i+\lambda_j)P_{ij}, ( Λ S ( P ) ) ij = ( λ i + λ j ) P ij ,
which proves invertibility.
To establish injectivity of D ι ( S ) D\iota(S) D ι ( S ) , we first identify suitable representatives of the effective cotangent classes. For a regular potential Ψ \Psi Ψ , set ψ ( z ) = Ψ ( S 1 / 2 z ) \psi(z)=\Psi(S^{1/2}z) ψ ( z ) = Ψ ( S 1/2 z ) . Integration along the infinitesimal rotations z i ∂ j − z j ∂ i z_i\partial_j-z_j\partial_i z i ∂ j − z j ∂ i gives
∫ R d z i ∂ j ψ d η = ∫ R d z j ∂ i ψ d η . \int_{\mathbb{R}^{d}} z_i\partial_j\psi\,\mathrm{d}\eta
=
\int_{\mathbb{R}^{d}} z_j\partial_i\psi\,\mathrm{d}\eta. ∫ R d z i ∂ j ψ d η = ∫ R d z j ∂ i ψ d η .
Radial cutoffs justify this identity, with the limit controlled by ∣ z ∣ , ∣ ∇ ψ ∣ ∈ L 2 ( η ) |z|,|\nabla\psi|\in L^{2}(\eta) ∣ z ∣ , ∣∇ ψ ∣ ∈ L 2 ( η ) . It follows that
S − 1 M ρ S ( Ψ ) = S − 1 / 2 M η ( ψ ) S − 1 / 2 ∈ Sym ( d ) . S^{-1}M_{\rho_S}(\Psi)=S^{-1/2}M_{\eta}(\psi)S^{-1/2}\in\operatorname{Sym}(d). S − 1 M ρ S ( Ψ ) = S − 1/2 M η ( ψ ) S − 1/2 ∈ Sym ( d ) .
For the reverse inclusion, fix P ∈ Sym ( d ) P\in\operatorname{Sym}(d) P ∈ Sym ( d ) and take
Ψ R ( x ) = 1 2 χ R ( x ) x T P x , \Psi_R(x)=\frac{1}{2}\chi_R(x)x^{\mathsf T}Px, Ψ R ( x ) = 2 1 χ R ( x ) x T P x ,
where χ R \chi_R χ R is smooth, equals one on B R B_R B R , vanishes outside B 2 R B_{2R} B 2 R , and satisfies ∣ ∇ χ R ∣ ≤ C / R |\nabla\chi_R|\le C/R ∣∇ χ R ∣ ≤ C / R . Then
∥ M ρ S ( Ψ R ) − S P ∥ F ≤ C P ∫ ∣ x ∣ > R ∣ x ∣ 2 d ρ S ( x ) ⟶ 0. \|M_{\rho_S}(\Psi_R)-SP\|_{\mathrm F}
\le C_P\int_{|x|>R}|x|^2\,\mathrm{d}\rho_S(x)\longrightarrow0. ∥ M ρ S ( Ψ R ) − S P ∥ F ≤ C P ∫ ∣ x ∣ > R ∣ x ∣ 2 d ρ S ( x ) ⟶ 0.
The range of M ρ S M_{\rho_S} M ρ S on C c ∞ ( R d ) C^\infty_c(\mathbb{R}^{d}) C c ∞ ( R d ) is a finite-dimensional linear subspace and hence closed. There is therefore a compactly supported smooth representative Ψ S , P \Psi_{S,P} Ψ S , P satisfying
M ρ S ( Ψ S , P ) = S P . M_{\rho_S}(\Psi_{S,P})=SP. M ρ S ( Ψ S , P ) = S P .
The cutoff estimate and the closed-range argument use only the second moment, not radiality. Thus, for every ρ ∈ P K A \rho\in\mathcal{P}_{K_A} ρ ∈ P K A and P ∈ Sym ( d ) P\in\operatorname{Sym}(d) P ∈ Sym ( d ) , they also give
Ψ ρ , P ∈ C c ∞ ( R d ) , M ρ ( Ψ ρ , P ) = S ( ρ ) P . \Psi_{\rho,P}\in C^\infty_c(\mathbb{R}^{d}),\qquad M_{\rho}(\Psi_{\rho,P})=S(\rho)P. Ψ ρ , P ∈ C c ∞ ( R d ) , M ρ ( Ψ ρ , P ) = S ( ρ ) P .
Next, radiality and polar decomposition imply
G # ρ S = ρ G S G T ( G ∈ GL ( d ) ) . (147) G_{\#}\rho_S=\rho_{GSG^{\mathsf T}}\qquad(G\in\operatorname{GL}(d)). \tag*{(147)} G # ρ S = ρ GS G T ( G ∈ GL ( d )) . (147)
Differentiating this identity along a matrix curve with G 0 = I d G_0=I_d G 0 = I d and G ˙ 0 = P S A \dot{G}_0=P S A G ˙ 0 = P S A yields
D ι ( S ) [ Λ S ( P ) ] = − ∇ ⋅ ( ρ S P S A x ) = L ρ S K A [ Ψ S , P ] . D\iota(S)[\Lambda_S(P)] = -\nabla\cdot(\rho_S P S A x) = L_{\rho_S}^{K_A}[\Psi_{S,P}]. D ι ( S ) [ Λ S ( P )] = − ∇ ⋅ ( ρ S P S A x ) = L ρ S K A [ Ψ S , P ] .
If this tangent vector vanishes, the Green identity (51) gives
0 = tr ( ( S P ) ⊤ A ( S P ) ) = ∥ A 1 / 2 S P ∥ F 2 . 0=\operatorname{tr}\left((SP)^\top A(SP)\right)=\left\lVert A^{1/2}SP\right\rVert_F^2. 0 = tr ( ( S P ) ⊤ A ( S P ) ) = A 1/2 S P F 2 .
Hence P = 0 P=0 P = 0 . Since Λ S \Lambda_S Λ S is an isomorphism, D ι ( S ) D\iota(S) D ι ( S ) is injective.
Proof of Lemma 7.2. Fix S ∈ Sym + + ( d ) S\in\operatorname{Sym}_{++}(d) S ∈ Sym ++ ( d ) and H , K ∈ Sym ( d ) H,K\in\operatorname{Sym}(d) H , K ∈ Sym ( d ) , and set
P = Λ S − 1 ( H ) , Q = Λ S − 1 ( K ) . P=\Lambda_S^{-1}(H),\qquad Q=\Lambda_S^{-1}(K). P = Λ S − 1 ( H ) , Q = Λ S − 1 ( K ) .
The tangent representation established in the proof of Lemma 7.1 , together with the Green identity (51) , gives
G ρ S K A ( D ι ( S ) [ H ] , D ι ( S ) [ K ] ) = tr ( M ρ S ( Ψ S , P ) ⊤ A M ρ S ( Ψ S , Q ) ) = tr ( P S A S Q ) = 1 2 tr ( H Λ S − 1 ( K ) ) . \begin{aligned}
G_{\rho_S}^{K_A}\left(D\iota(S)[H],D\iota(S)[K]\right)
=\operatorname{tr}\left(M_{\rho_S}(\Psi_{S,P})^\top A M_{\rho_S}(\Psi_{S,Q})\right) \\
&=\operatorname{tr}(P S A S Q) \\
&=\frac{1}{2}\operatorname{tr}\left(H\Lambda_S^{-1}(K)\right).
\end{aligned} G ρ S K A ( D ι ( S ) [ H ] , D ι ( S ) [ K ] ) = tr ( M ρ S ( Ψ S , P ) ⊤ A M ρ S ( Ψ S , Q ) ) = tr ( P S A S Q ) = 2 1 tr ( H Λ S − 1 ( K ) ) .
This proves (81) . For completeness, symmetry follows by transposition and cyclicity of the trace, while
g S lin ( H , H ) = ∥ A 1 / 2 S P ∥ F 2 > 0 ( H ≠ 0 ) . g_S^{\operatorname{lin}}(H,H)=\left\lVert A^{1/2}SP\right\rVert_F^2>0\qquad(H\ne0). g S lin ( H , H ) = A 1/2 S P F 2 > 0 ( H = 0 ) .
Finally, smooth dependence of Λ S − 1 \Lambda_S^{-1} Λ S − 1 on S S S shows that (80) defines a smooth Riemannian metric on Sym + + ( d ) \operatorname{Sym}_{++}(d) Sym ++ ( d ) .
Proof of Proposition 7.4. We first compute the gradient of F lin F^{\operatorname{lin}} F lin . For every H ∈ Sym ( d ) H\in\operatorname{Sym}(d) H ∈ Sym ( d ) ,
g S lin ( − Λ S ( V ) , H ) = − 1 2 tr ( V H ) = D F lin ( S ) [ H ] . g_S^{\operatorname{lin}}(-\Lambda_S(V),H)=-\frac{1}{2}\operatorname{tr}(VH)=DF^{\operatorname{lin}}(S)[H]. g S lin ( − Λ S ( V ) , H ) = − 2 1 tr ( V H ) = D F lin ( S ) [ H ] .
Thus
grad g lin F lin ( S ) = − Λ S ( V ) . \operatorname{grad}_{g_{\operatorname{lin}}}F^{\operatorname{lin}}(S)=-\Lambda_S(V). grad g lin F lin ( S ) = − Λ S ( V ) .
Let S t ∈ Sym + + ( d ) S_t\in\operatorname{Sym}_{++}(d) S t ∈ Sym ++ ( d ) be a C 1 C^1 C 1 curve and set ρ t = ρ S t \rho_t=\rho_{S_t} ρ t = ρ S t . The tangent identity established in the proof of Lemma 7.1 gives
∂ t ρ t + ∇ ⋅ ( ρ t V S t A x ) = D ι ( S t ) [ S ˙ t − Λ S t ( V ) ] . \partial_t\rho_t+\nabla\cdot(\rho_t V S_t A x)
=D\iota(S_t)\left[\dot{S}_t-\Lambda_{S_t}(V)\right]. ∂ t ρ t + ∇ ⋅ ( ρ t V S t A x ) = D ι ( S t ) [ S ˙ t − Λ S t ( V ) ] .
Since D ι ( S t ) D\iota(S_t) D ι ( S t ) is injective, the density curve solves (77) if and only if
S ˙ t = Λ S t ( V ) = − grad g lin F lin ( S t ) , \dot{S}_t=\Lambda_{S_t}(V)=-\operatorname{grad}_{g_{\operatorname{lin}}}F^{\operatorname{lin}}(S_t), S ˙ t = Λ S t ( V ) = − grad g lin F lin ( S t ) ,
which is (82) . The initial data correspond under the parametrization ι \iota ι .
Proof of Corollary 7.12. If V ⪯ 0 V\preceq0 V ⪯ 0 , the scalar second moment is nonincreasing, since
d d t tr ( S t ) = 2 tr ( V S t A S t ) ≤ 0. \frac{\mathrm{d}}{\mathrm{d}t}\operatorname{tr}(S_t)=2\operatorname{tr}(V S_t A S_t)\le0. d t d tr ( S t ) = 2 tr ( V S t A S t ) ≤ 0.
Thus the second moment remains bounded by its initial value. Proposition 7.10 therefore implies T max = ∞ T_{\max}=\infty T m a x = ∞ .
Accelerated Linear Attention: Geometry and Hamiltonian Dynamics
Proof of Lemma 7.15. Since ∇ Φ P ( x ) = P x \nabla\Phi_P(x)=Px ∇ Φ P ( x ) = P x and ρ S \rho_S ρ S has covariance S S S , the moment map in (49) satisfies
M ρ S ( Φ P ) = S P . (148) M_{\rho_S}(\Phi_P)=SP. \tag*{(148)} M ρ S ( Φ P ) = S P . (148)
Differentiating (146) at G = I d G=I_d G = I d in the direction P S A PSA P S A gives
D ι ( S ) [ Λ S ( P ) ] = − ∇ ⋅ ( ρ S P S A x ) = L ρ S K [ Φ P ] . D\iota(S)[\Lambda_S(P)]=-\nabla\cdot(\rho_SPSAx)=L^K_{\rho_S}[\Phi_P]. D ι ( S ) [ Λ S ( P )] = − ∇ ⋅ ( ρ S P S A x ) = L ρ S K [ Φ P ] .
Thus, the first equation in (87) is equivalent to
D ι ( S t ) [ S ˙ t − Λ S t ( P t ) ] = 0. D\iota(S_t)[\dot{S}_t-\Lambda_{S_t}(P_t)]=0. D ι ( S t ) [ S ˙ t − Λ S t ( P t )] = 0.
The injectivity of D ι ( S t ) D\iota(S_t) D ι ( S t ) , established in Appendix G.1, then gives the first equation in (88).
For the second equation, (147) and symmetry of A , S , P A,S,P A , S , P give
∇ Φ P ( x ) ⊤ M ρ S ( Φ P ) ⊤ A x = x ⊤ P 2 S A x = 1 2 x ⊤ ( P 2 S A + A S P 2 ) x . (149) \nabla\Phi_P(x)^\top M_{\rho_S}(\Phi_P)^\top Ax=x^\top P^2SAx=\frac{1}{2}x^\top(P^2SA+ASP^2)x. \tag*{(149)} ∇ Φ P ( x ) ⊤ M ρ S ( Φ P ) ⊤ A x = x ⊤ P 2 S A x = 2 1 x ⊤ ( P 2 S A + A S P 2 ) x . (149)
Substituting (148) into the second equation in (87) therefore yields
1 2 x ⊤ ( P ˙ t + α t P t + P t 2 S t A + A S t P t 2 − V ) x = 0 for all x ∈ R d . \frac{1}{2}x^\top\left(\dot{P}_t+\alpha_tP_t+P_t^2S_tA+AS_tP_t^2-V\right)x=0\qquad\text{for all }x\in\mathbb{R}^d. 2 1 x ⊤ ( P ˙ t + α t P t + P t 2 S t A + A S t P t 2 − V ) x = 0 for all x ∈ R d .
The matrix in parentheses is symmetric, so this identity is equivalent to the second equation in (88). The initial conditions follow from ρ t = ρ S t \rho_t=\rho_{S_t} ρ t = ρ S t and Φ t = Φ P t \Phi_t=\Phi_{P_t} Φ t = Φ P t .
Proof of Lemma 7.16. Differentiating the Hamiltonian in (89) gives
∂ Π H l i n ( S , Π ) = 2 ( S A S Π + Π S A S ) , ∂ S H l i n ( S , Π ) = 2 ( A S Π 2 + Π 2 S A ) − 1 2 V . \partial_{\Pi}H^{\mathrm{lin}}(S,\Pi)=2(SAS\Pi+\Pi SAS),\qquad\partial_SH^{\mathrm{lin}}(S,\Pi)=2(AS\Pi^2+\Pi^2SA)-\frac{1}{2}V. ∂ Π H lin ( S , Π ) = 2 ( S A S Π + Π S A S ) , ∂ S H lin ( S , Π ) = 2 ( A S Π 2 + Π 2 S A ) − 2 1 V .
Substituting Π = P / 2 \Pi=P/2 Π = P /2 into the damped Hamiltonian equations gives precisely (88).
Proof of Proposition 7.17. We first identify the Lagrangian associated with (89). For S ∈ S y m + + ( d ) S\in\mathrm{Sym}_{++}(d) S ∈ Sym ++ ( d ) and H ∈ S y m ( d ) H\in\mathrm{Sym}(d) H ∈ Sym ( d ) , set
L ( S , H ) : = 1 2 g S l i n ( H , H ) − F l i n ( S ) . \mathcal{L}(S,H):=\frac{1}{2}g_S^{\mathrm{lin}}(H,H)-F^{\mathrm{lin}}(S). L ( S , H ) := 2 1 g S lin ( H , H ) − F lin ( S ) .
By (80), its derivative with respect to H H H satisfies
D H L ( S , H ) [ K ] = g S l i n ( H , K ) = 1 2 tr ( Λ S − 1 ( H ) K ) ( K ∈ S y m ( d ) ) . D_H\mathcal{L}(S,H)[K]=g_S^{\mathrm{lin}}(H,K)=\frac{1}{2}\operatorname{tr}\left(\Lambda_S^{-1}(H)K\right)\qquad(K\in\mathrm{Sym}(d)). D H L ( S , H ) [ K ] = g S lin ( H , K ) = 2 1 tr ( Λ S − 1 ( H ) K ) ( K ∈ Sym ( d )) .
Consequently, the momentum matrix associated with H = Λ S ( P ) H=\Lambda_S(P) H = Λ S ( P ) is Π = P / 2 \Pi=P/2 Π = P /2 . The Legendre transform of L \mathcal{L} L is therefore
tr ( Π H ) − L ( S , H ) = 2 tr ( Π S A S Π ) − 1 2 tr ( V S ) = H l i n ( S , Π ) . \operatorname{tr}(\Pi H)-\mathcal{L}(S,H)=2\operatorname{tr}(\Pi SAS\Pi)-\frac{1}{2}\operatorname{tr}(VS)=H^{\mathrm{lin}}(S,\Pi). tr ( Π H ) − L ( S , H ) = 2 tr ( Π S A S Π ) − 2 1 tr ( V S ) = H lin ( S , Π ) .
It follows that the damped Hamiltonian equations in Lemma 7.16 are equivalent to the damped Euler–Lagrange equation
d d t ∂ H L ( S t , S ˙ t ) − ∂ S L ( S t , S ˙ t ) = − α t ∂ H L ( S t , S ˙ t ) . \frac{\mathrm{d}}{\mathrm{d}t}\partial_H\mathcal{L}(S_t,\dot{S}_t)-\partial_S\mathcal{L}(S_t,\dot{S}_t)=-\alpha_t\partial_H\mathcal{L}(S_t,\dot{S}_t). d t d ∂ H L ( S t , S ˙ t ) − ∂ S L ( S t , S ˙ t ) = − α t ∂ H L ( S t , S ˙ t ) .
For the kinetic energy 1 2 g S l i n ( H , H ) \frac{1}{2}g^{\mathrm{lin}}_{S}(H,H) 2 1 g S lin ( H , H ) , the Euler–Lagrange expression is the metric dual of the covariant acceleration. Moreover, ∂ H ˙ L ( S t , S ˙ t ) \partial_{\dot{H}}\mathcal{L}(S_t,\dot{S}_t) ∂ H ˙ L ( S t , S ˙ t ) is the metric dual of S ˙ t \dot{S}_t S ˙ t . Thus, the displayed equation is equivalent to (90).
If ( S , P ) (S,P) ( S , P ) solves (88), then its first equation gives S ∈ C 2 S \in C^2 S ∈ C 2 . Lemma 7.16 and the preceding equivalence show that S S S satisfies (90).
Conversely, let S ∈ C 2 ( I ; Sym + + ( d ) ) S \in C^2(I;\operatorname{Sym}_{++}(d)) S ∈ C 2 ( I ; Sym ++ ( d )) satisfy (90). Since Λ S − 1 \Lambda_S^{-1} Λ S − 1 depends smoothly on S S S , the curve
P t = Λ S t − 1 ( S ˙ t ) P_t=\Lambda_{S_t}^{-1}(\dot{S}_t) P t = Λ S t − 1 ( S ˙ t )
belongs to C 1 ( I ; Sym ( d ) ) C^1(I;\operatorname{Sym}(d)) C 1 ( I ; Sym ( d )) . The same equivalence shows that ( S , P / 2 ) (S,P/2) ( S , P /2 ) satisfies the damped Hamiltonian equations, and hence ( S , P ) (S,P) ( S , P ) solves (88).
Proof of Theorem 7.18. 1. The right-hand side of (88) is continuous in time and locally Lipschitz in ( S , P ) (S,P) ( S , P ) . It also preserves symmetry. The Picard–Lindelöf theorem therefore gives a unique maximal C 1 C^1 C 1 solution with initial values ( S 0 , P 0 ) (S_0,P_0) ( S 0 , P 0 ) . Let G G G solve (91). Direct differentiation and uniqueness for the resulting linear matrix equation give
S t = G t S 0 G t ⊤ ≻ 0 , S_t=G_tS_0G_t^{\top}\succ0, S t = G t S 0 G t ⊤ ≻ 0 ,
since G t G_t G t is invertible and S 0 ≻ 0 S_0\succ0 S 0 ≻ 0 .
Since det G t > 0 \det G_t>0 det G t > 0 , the pushforward in (92) has density
ρ t ( x ) = ρ 0 ( G t − 1 x ) det G t . (150) \rho_t(x)=\frac{\rho_0(G_t^{-1}x)}{\det G_t}. \tag*{(150)} ρ t ( x ) = det G t ρ 0 ( G t − 1 x ) . (150)
A change of variables gives S ( ρ t ) = G t S 0 G t ⊤ = S t S(\rho_t)=G_tS_0G_t^{\top}=S_t S ( ρ t ) = G t S 0 G t ⊤ = S t . Since ∇ Φ t ( x ) = P t x \nabla\Phi_t(x)=P_tx ∇ Φ t ( x ) = P t x , it follows
M ρ t ( Φ t ) = ∫ R d x x ⊤ P t ρ t ( x ) d x = S t P t . M_{\rho_t}(\Phi_t)=\int_{\mathbb{R}^d}xx^{\top}P_t\rho_t(x)\,dx=S_tP_t. M ρ t ( Φ t ) = ∫ R d x x ⊤ P t ρ t ( x ) d x = S t P t .
The integral defining M ρ t ( Φ t ) M_{\rho_t}(\Phi_t) M ρ t ( Φ t ) is absolutely convergent by the finite second moment of ρ t \rho_t ρ t , and t ↦ S t P t t\mapsto S_tP_t t ↦ S t P t is continuous. This verifies condition 3 of the strong solution definition. Next differentiating (149) in time yields
∂ t ρ t = − tr ( P t S t A ) ρ t − ( P t S t A x ) ⋅ ∇ ρ t = − ∇ ⋅ ( ρ t P t S t A x ) , \partial_t\rho_t=-\operatorname{tr}(P_tS_tA)\rho_t-(P_tS_tAx)\cdot\nabla\rho_t=-\nabla\cdot(\rho_tP_tS_tAx), ∂ t ρ t = − tr ( P t S t A ) ρ t − ( P t S t A x ) ⋅ ∇ ρ t = − ∇ ⋅ ( ρ t P t S t A x ) ,
which is the first equation in (87). For the second equation, a direct computation gives
∂ t Φ t + α t Φ t + ∇ Φ t ⊤ M ρ t ( Φ t ) ⊤ A x − 1 2 x ⊤ V x = 1 2 x ⊤ ( P ˙ t + α t P t + P t 2 S t A + A S t P t 2 − V ) x = 0. \partial_t\Phi_t+\alpha_t\Phi_t+\nabla\Phi_t^{\top}M_{\rho_t}(\Phi_t)^{\top}Ax-\frac{1}{2}x^{\top}Vx
=\frac{1}{2}x^{\top}\left(\dot{P}_t+\alpha_tP_t+P_t^2S_tA+AS_tP_t^2-V\right)x=0. ∂ t Φ t + α t Φ t + ∇ Φ t ⊤ M ρ t ( Φ t ) ⊤ A x − 2 1 x ⊤ V x = 2 1 x ⊤ ( P ˙ t + α t P t + P t 2 S t A + A S t P t 2 − V ) x = 0.
Both identities hold in C ( K ) C(K) C ( K ) for every compact K ⊂ R d K\subset\mathbb{R}^d K ⊂ R d , proving condition 4. The initial conditions follow from G 0 = I d G_0=I_d G 0 = I d and P ∣ t = 0 = P 0 P|_{t=0}=P_0 P ∣ t = 0 = P 0 , verifying condition 2. Finally, since G , P G,P G , P are C 1 C^1 C 1 and ρ 0 \rho_0 ρ 0 is smooth, one can check that condition 1 holds. Thus, ( ρ , Φ ) (\rho,\Phi) ( ρ , Φ ) is a strong solution.
Let ( ρ , Φ ) (\rho,\Phi) ( ρ , Φ ) be any strong solution with initial values ( ρ 0 , Φ 0 ) (\rho_0,\Phi_0) ( ρ 0 , Φ 0 ) , where Φ 0 = Φ P 0 \Phi_0=\Phi_{P_0} Φ 0 = Φ P 0 . By condition 3, the coefficient t ↦ M ρ t ( Φ t ) ⊤ A t\mapsto M_{\rho_t}(\Phi_t)^{\top}A t ↦ M ρ t ( Φ t ) ⊤ A is locally integrable. Linear ODE theory therefore gives a unique, locally absolutely continuous and invertible fundamental matrix satisfying
G ˙ t = M ρ t ( Φ t ) ⊤ A G t , G 0 = I d . (151) \dot{G}_t=M_{\rho_t}(\Phi_t)^{\top}AG_t,\qquad G_0=I_d. \tag*{(151)} G ˙ t = M ρ t ( Φ t ) ⊤ A G t , G 0 = I d . (151)
Applying the chain rule to t ↦ Φ t ( G t y ) t \mapsto\Phi_t(G_ty) t ↦ Φ t ( G t y ) and using the second equation in (87) , we obtain
d d t Φ t ( G t y ) + α t Φ t ( G t y ) = 1 2 y T G t T V G t y , Φ 0 ( G 0 y ) = 1 2 y T P 0 y . \frac{\mathrm{d}}{\mathrm{d}t}\Phi_t(G_ty)+\alpha_t\Phi_t(G_ty)=\frac{1}{2}y^{\mathsf T}G_t^{\mathsf T}VG_ty,\qquad\Phi_0(G_0y)=\frac{1}{2}y^{\mathsf T}P_0y. d t d Φ t ( G t y ) + α t Φ t ( G t y ) = 2 1 y T G t T V G t y , Φ 0 ( G 0 y ) = 2 1 y T P 0 y .
The integrating-factor formula gives the unique solution of this scalar linear ODE. Returning to x = G t y x=G_ty x = G t y yields
Φ t ( x ) = 1 2 x T G t − T [ exp ( − ∫ 0 t α r d r ) P 0 + ∫ 0 t exp ( − ∫ s t α r d r ) G s T V G s d s ] G t − 1 x . (152) \Phi_t(x)=\frac{1}{2}x^{\mathsf T}G_t^{-\mathsf T}\left[\exp\left(-\int_0^t\alpha_r\,\mathrm{d}r\right)P_0+\int_0^t\exp\left(-\int_s^t\alpha_r\,\mathrm{d}r\right)G_s^{\mathsf T}VG_s\,\mathrm{d}s\right]G_t^{-1}x. \tag*{(152)} Φ t ( x ) = 2 1 x T G t − T [ exp ( − ∫ 0 t α r d r ) P 0 + ∫ 0 t exp ( − ∫ s t α r d r ) G s T V G s d s ] G t − 1 x . (152)
Thus Φ t = Φ P t \Phi_t=\Phi_{P_t} Φ t = Φ P t for a symmetric, locally absolutely continuous matrix P t P_t P t . Similarly, applying the chain rule and using the density equation in (87) gives
d d t [ det G t ρ t ( G t y ) ] = 0. \frac{\mathrm{d}}{\mathrm{d}t}\left[\det G_t\,\rho_t(G_ty)\right]=0. d t d [ det G t ρ t ( G t y ) ] = 0.
The initial condition therefore yields (150) . In particular, ρ t \rho_t ρ t is a probability density with finite second moment, and a change of variables gives
S t : = S ( ρ t ) = G t S 0 G t T . S_t:=S(\rho_t)=G_tS_0G_t^{\mathsf T}. S t := S ( ρ t ) = G t S 0 G t T .
Since ∇ Φ t ( x ) = P t x \nabla\Phi_t(x)=P_tx ∇ Φ t ( x ) = P t x , we also have M ρ t ( Φ t ) = S t P t M_{\rho_t}(\Phi_t)=S_tP_t M ρ t ( Φ t ) = S t P t .
Next, differentiating the second-moment identity S t = S ( ρ t ) = G t S 0 G t T S_t=S(\rho_t)=G_tS_0G_t^{\mathsf T} S t = S ( ρ t ) = G t S 0 G t T and the expression for the quadratic coefficient P t P_t P t , here P t P_t P t denotes the matrix between x T x^{\mathsf T} x T and x x x on the right-hand side of (151), gives that ( S t , P t ) (S_t,P_t) ( S t , P t ) solves (88) almost everywhere. Since S S S , P P P , and α \alpha α are continuous, classical bootstrapping implies that the matrix equations in (88) hold at every time. Substituting M ρ t ( Φ t ) = S t P t M_{\rho_t}(\Phi_t)=S_tP_t M ρ t ( Φ t ) = S t P t into (151) and using the symmetry of S t S_t S t and P t P_t P t gives
G ˙ t = P t S t A G t , G 0 = I d , \dot{G}_t=P_tS_tAG_t,\qquad G_0=I_d, G ˙ t = P t S t A G t , G 0 = I d ,
which is precisely (91) . Uniqueness of the solution to the matrix ODE system (88) proves that ( ρ , Φ ) (\rho,\Phi) ( ρ , Φ ) has the representation (92) .
4. 4. 4. The representation (150) shows that ρ t \rho_t ρ t remains smooth and strictly positive. Preservation of unit mass also follows from (150) by a change of variables. Similarly,
∫ R d ∣ x ∣ 4 d ρ t ( x ) ≤ ∥ G t ∥ o p 4 ∫ R d ∣ y ∣ 4 d ρ 0 ( y ) . (153) \int_{\mathbb{R}^d}|x|^4\,\mathrm{d}\rho_t(x)\leq\lVert G_t\rVert_{\mathrm{op}}^4\int_{\mathbb{R}^d}|y|^4\,\mathrm{d}\rho_0(y). \tag*{(153)} ∫ R d ∣ x ∣ 4 d ρ t ( x ) ≤ ∥ G t ∥ op 4 ∫ R d ∣ y ∣ 4 d ρ 0 ( y ) . (153)
Applying the chain rule to (150) also yields
∫ R d ∣ x ∣ 2 ∣ ∇ log ρ t ( x ) ∣ 2 d ρ t ( x ) ≤ ∥ G t ∥ o p 2 ∥ G t − 1 ∥ o p 2 ∫ R d ∣ y ∣ 2 ∣ ∇ log ρ 0 ( y ) ∣ 2 d ρ 0 ( y ) . (154) \int_{\mathbb{R}^d}|x|^2|\nabla\log\rho_t(x)|^2\,\mathrm{d}\rho_t(x)\leq\lVert G_t\rVert_{\mathrm{op}}^2\lVert G_t^{-1}\rVert_{\mathrm{op}}^2\int_{\mathbb{R}^d}|y|^2|\nabla\log\rho_0(y)|^2\,\mathrm{d}\rho_0(y). \tag*{(154)} ∫ R d ∣ x ∣ 2 ∣∇ log ρ t ( x ) ∣ 2 d ρ t ( x ) ≤ ∥ G t ∥ op 2 ∥ G t − 1 ∥ op 2 ∫ R d ∣ y ∣ 2 ∣∇ log ρ 0 ( y ) ∣ 2 d ρ 0 ( y ) . (154)
Since ρ 0 ∈ P A 4 \rho_0\in\mathcal{P}_A^4 ρ 0 ∈ P A 4 , the right-hand sides of (153) and (154) are finite for every t ∈ [ 0 , T max ) t\in[0,T_{\max}) t ∈ [ 0 , T m a x ) . Together with smoothness, strict positivity, and unit mass, these estimates prove that ρ t ∈ P A 4 \rho_t\in\mathcal{P}_A^4 ρ t ∈ P A 4 throughout this interval.
Next, we verify that ∂ t ρ t ∈ T ρ t e f f P K A \partial_t\rho_t\in T^{\mathrm{eff}}_{\rho_t}\mathcal{P}_{K_A} ∂ t ρ t ∈ T ρ t eff P K A , with T ρ t e f f P K A T^{\mathrm{eff}}_{\rho_t}\mathcal{P}_{K_A} T ρ t eff P K A as in (52) . Using the first equation in (87) together with the weighted-score estimate (154) , we obtain
∂ t ρ t ρ t = − tr ( P t S t A ) − ( P t S t A x ) ⋅ ∇ log ρ t ∈ L 2 ( ρ t ) . \frac{\partial_t\rho_t}{\rho_t}=-\operatorname{tr}(P_tS_tA)-(P_tS_tAx)\cdot\nabla\log\rho_t\in L^2(\rho_t). ρ t ∂ t ρ t = − tr ( P t S t A ) − ( P t S t A x ) ⋅ ∇ log ρ t ∈ L 2 ( ρ t ) .
By a cutoff argument similar to that in the proof of Lemma 5.1, we obtain ∫ R d ∂ t ρ t d x = 0 \int_{\mathbb{R}^d} \partial_t\rho_t\,\mathrm{d}x=0 ∫ R d ∂ t ρ t d x = 0 . Thus ∂ t ρ t ∈ T ρ t P K A \partial_t\rho_t\in T_{\rho_t}\mathcal{P}_{K_A} ∂ t ρ t ∈ T ρ t P K A , where the latter space is defined as in (47). The finite fourth moment of ρ t \rho_t ρ t gives Φ P t ∈ H 1 ( ρ t ) \Phi_{P_t}\in H^1(\rho_t) Φ P t ∈ H 1 ( ρ t ) , so [ Φ P t ] ∈ T ρ t ∗ P K A [\Phi_{P_t}]\in T_{\rho_t}^*\mathcal{P}_{K_A} [ Φ P t ] ∈ T ρ t ∗ P K A , as defined in (48). The first equation in (87), together with the definition of L ρ t K A L^{K_A}_{\rho_t} L ρ t K A in (50), gives
∂ t ρ t = L ρ t K A [ Φ P t ] . \partial_t\rho_t=L^{K_A}_{\rho_t}[\Phi_{P_t}]. ∂ t ρ t = L ρ t K A [ Φ P t ] .
Thus ∂ t ρ t ∈ T ρ t e f f P K A \partial_t\rho_t\in T^{\mathrm{eff}}_{\rho_t}\mathcal{P}_{K_A} ∂ t ρ t ∈ T ρ t eff P K A , which completes the proof. □
Proof of Corollary 7.21. Fix 0 < T < T max 0<T<T_{\max} 0 < T < T m a x . Let ( S ( n ) , P ( n ) ) (S^{(n)},P^{(n)}) ( S ( n ) , P ( n ) ) and ( S , P ) (S,P) ( S , P ) solve (88) with initial values ( S ( ρ 0 ( n ) ) , P 0 ( n ) ) (S(\rho_0^{(n)}),P_0^{(n)}) ( S ( ρ 0 ( n ) ) , P 0 ( n ) ) and ( S ( ρ 0 ) , P 0 ) (S(\rho_0),P_0) ( S ( ρ 0 ) , P 0 ) , respectively. Let G ( n ) G^{(n)} G ( n ) and G G G solve (91) for these matrix pairs with initialization I d I_d I d .
Convergence of ρ 0 ( n ) \rho_0^{(n)} ρ 0 ( n ) to ρ 0 \rho_0 ρ 0 in W 2 W_2 W 2 implies S ( ρ 0 ( n ) ) → S ( ρ 0 ) S(\rho_0^{(n)})\to S(\rho_0) S ( ρ 0 ( n ) ) → S ( ρ 0 ) in the Frobenius norm. Continuous dependence for (88) and (91) [37 Chapter V] therefore gives existence of S ( n ) , P ( n ) , G ( n ) S^{(n)},P^{(n)},G^{(n)} S ( n ) , P ( n ) , G ( n ) on [ 0 , T ] [0,T] [ 0 , T ] for all sufficiently large n n n and uniform convergence to S , P , G S,P,G S , P , G in the Frobenius norm. The right-hand sides of these ODEs then also give uniform convergence of their time derivatives. By Theorem 7.18, the strong solutions ( ρ ( n ) , Φ ( n ) ) (\rho^{(n)},\Phi^{(n)}) ( ρ ( n ) , Φ ( n ) ) consequently exist on [ 0 , T ] [0,T] [ 0 , T ] .
Using the pushforward representation (92) and the triangle inequality for W 2 W_2 W 2 , we obtain
W 2 ( ρ t ( n ) , ρ t ) ≤ ∥ G t ( n ) ∥ o p W 2 ( ρ 0 ( n ) , ρ 0 ) + ∥ G t ( n ) − G t ∥ o p ( ∫ R d ∣ x ∣ 2 d ρ 0 ( x ) ) 1 / 2 . W_2(\rho_t^{(n)},\rho_t)\leq\lVert G_t^{(n)}\rVert_{\mathrm{op}}W_2(\rho_0^{(n)},\rho_0)+\lVert G_t^{(n)}-G_t\rVert_{\mathrm{op}}\left(\int_{\mathbb{R}^d}|x|^2\,\mathrm{d}\rho_0(x)\right)^{1/2}. W 2 ( ρ t ( n ) , ρ t ) ≤ ∥ G t ( n ) ∥ op W 2 ( ρ 0 ( n ) , ρ 0 ) + ∥ G t ( n ) − G t ∥ op ( ∫ R d ∣ x ∣ 2 d ρ 0 ( x ) ) 1/2 .
Uniform convergence of G ( n ) G^{(n)} G ( n ) makes the matrices G t ( n ) G_t^{(n)} G t ( n ) uniformly bounded on [ 0 , T ] [0,T] [ 0 , T ] . Together with the initial W 2 W_2 W 2 convergence, this estimate therefore proves convergence in C ( [ 0 , T ] ; W 2 ) C([0,T];W_2) C ([ 0 , T ] ; W 2 ) .
Since G t G_t G t is invertible throughout the compact interval [ 0 , T ] [0,T] [ 0 , T ] , the inverse matrices ( G t ( n ) ) − 1 (G_t^{(n)})^{-1} ( G t ( n ) ) − 1 also converge uniformly to G t − 1 G_t^{-1} G t − 1 . Applying (149) and using ρ 0 ( n ) → ρ 0 \rho_0^{(n)}\to\rho_0 ρ 0 ( n ) → ρ 0 in C l o c 1 ( R d ) C^1_{\mathrm{loc}}(\mathbb{R}^d) C loc 1 ( R d ) now gives
ρ ( n ) ⟶ ρ in C ( [ 0 , T ] ; C l o c 1 ) . \rho^{(n)}\longrightarrow\rho\quad\text{in }C([0,T];C^1_{\mathrm{loc}}). ρ ( n ) ⟶ ρ in C ([ 0 , T ] ; C loc 1 ) .
The first equation in (87), with M ρ t ( Φ t ) = S t P t M_{\rho_t}(\Phi_t)=S_tP_t M ρ t ( Φ t ) = S t P t , expresses ∂ t ρ t \partial_t\rho_t ∂ t ρ t in terms of S t , P t , ρ t S_t,P_t,\rho_t S t , P t , ρ t and ∇ ρ t \nabla\rho_t ∇ ρ t . The same identity holds for ( ρ ( n ) , Φ ( n ) ) (\rho^{(n)},\Phi^{(n)}) ( ρ ( n ) , Φ ( n ) ) . The convergence already established for these matrices and functions therefore gives
∂ t ρ ( n ) ⟶ ∂ t ρ in C ( [ 0 , T ] ; C l o c 0 ) . \partial_t\rho^{(n)}\longrightarrow\partial_t\rho\quad\text{in }C([0,T];C^0_{\mathrm{loc}}). ∂ t ρ ( n ) ⟶ ∂ t ρ in C ([ 0 , T ] ; C loc 0 ) .
Finally, the potential formulas in (92) read Φ t ( n ) ( x ) = 1 2 x ⊤ P t ( n ) x \Phi_t^{(n)}(x)=\frac{1}{2}x^\top P_t^{(n)}x Φ t ( n ) ( x ) = 2 1 x ⊤ P t ( n ) x and Φ t ( x ) = 1 2 x ⊤ P t x \Phi_t(x)=\frac{1}{2}x^\top P_tx Φ t ( x ) = 2 1 x ⊤ P t x . Uniform convergence of P ( n ) P^{(n)} P ( n ) and P ˙ ( n ) \dot P^{(n)} P ˙ ( n ) therefore gives
Φ ( n ) ⟶ Φ in C ( [ 0 , T ] ; C l o c 1 ) ∩ C 1 ( [ 0 , T ] ; C l o c 0 ) , \Phi^{(n)}\longrightarrow\Phi\quad\text{in }C([0,T];C^1_{\mathrm{loc}})\cap C^1([0,T];C^0_{\mathrm{loc}}), Φ ( n ) ⟶ Φ in C ([ 0 , T ] ; C loc 1 ) ∩ C 1 ([ 0 , T ] ; C loc 0 ) ,
completing the proof of (93). □
Proof of Proposition 7.22. We first record the energy estimate used in both parts. Define
E ( S , P ) : = H l i n ( S , P / 2 ) = 1 2 tr ( P S A S P ) − 1 2 tr ( V S ) . (155) E(S,P):=\mathcal{H}^{\mathrm{lin}}(S,P/2)=\frac{1}{2}\operatorname{tr}(PSASP)-\frac{1}{2}\operatorname{tr}(VS). \tag*{(155)} E ( S , P ) := H lin ( S , P /2 ) = 2 1 tr ( P S A S P ) − 2 1 tr ( V S ) . (155)
Differentiating along (88) gives
d d t E ( S t , P t ) = − α t ∥ A 1 / 2 S t P t ∥ F 2 ≤ 0. (156) \frac{\mathrm{d}}{\mathrm{d}t}E(S_t,P_t)=-\alpha_t\lVert A^{1/2}S_tP_t\rVert_F^2\leq0. \tag*{(156)} d t d E ( S t , P t ) = − α t ∥ A 1/2 S t P t ∥ F 2 ≤ 0. (156)
Suppose that T max < ∞ T_{\max}<\infty T m a x < ∞ and that S t S_t S t remains bounded on [ 0 , T max ) [0,T_{\max}) [ 0 , T m a x ) . By (154) and (155),
∥ A 1 / 2 S t P t ∥ F 2 ≤ 2 E ( S 0 , P 0 ) + tr ( V S t ) . \left\|A^{1/2}S_tP_t\right\|_{\mathrm{F}}^2\le2E(S_0,P_0)+\operatorname{tr}(VS_t). A 1/2 S t P t F 2 ≤ 2 E ( S 0 , P 0 ) + tr ( V S t ) .
Since A ≻ 0 A\succ0 A ≻ 0 and S t S_t S t is bounded, this bounds S t P t S_tP_t S t P t and P t S t = ( S t P t ) T P_tS_t=(S_tP_t)^{\mathsf{T}} P t S t = ( S t P t ) T .
Differentiating S t − 1 S_t^{-1} S t − 1 using (88) yields
d d t S t − 1 = − S t − 1 ( P t S t ) A − A ( S t P t ) S t − 1 . \frac{\mathrm{d}}{\mathrm{d}t}S_t^{-1}=-S_t^{-1}(P_tS_t)A-A(S_tP_t)S_t^{-1}. d t d S t − 1 = − S t − 1 ( P t S t ) A − A ( S t P t ) S t − 1 .
Gronwall’s inequality therefore bounds S t − 1 S_t^{-1} S t − 1 on [ 0 , T max ) [0,T_{\max}) [ 0 , T m a x ) . Consequently, P t = S t − 1 ( S t P t ) P_t=S_t^{-1}(S_tP_t) P t = S t − 1 ( S t P t ) is also bounded. The pair ( S t , P t ) (S_t,P_t) ( S t , P t ) thus remains in a compact subset of Sym + + ( d ) × Sym ( d ) \operatorname{Sym}_{++}(d)\times\operatorname{Sym}(d) Sym ++ ( d ) × Sym ( d ) . The standard continuation theorem for (88) extends the solution beyond T max T_{\max} T m a x , contradicting maximality; see [37 ] .
Finally, Theorem 7.18 identifies S t S_t S t with the second-moment matrix of ρ t \rho_t ρ t . Since S t ≻ 0 S_t\succ0 S t ≻ 0 ,
∥ S t ∥ F ≤ tr ( S t ) = ∫ R d ∣ x ∣ 2 d ρ t ( x ) ≤ d ∥ S t ∥ F . \left\|S_t\right\|_{\mathrm{F}}\le\operatorname{tr}(S_t)=\int_{\mathbb{R}^d}|x|^2\,\mathrm{d}\rho_t(x)\le\sqrt{d}\left\|S_t\right\|_{\mathrm{F}}. ∥ S t ∥ F ≤ tr ( S t ) = ∫ R d ∣ x ∣ 2 d ρ t ( x ) ≤ d ∥ S t ∥ F .
This proves both assertions in (94).
If V ⪯ 0 V\preceq0 V ⪯ 0 , both terms in (154) are nonnegative. Thus (155) bounds S t P t S_tP_t S t P t throughout the existence interval. Writing the first equation in (88) as
S ˙ t = ( P t S t ) A S t + S t A ( S t P t ) , \dot{S}_t=(P_tS_t)AS_t+S_tA(S_tP_t), S ˙ t = ( P t S t ) A S t + S t A ( S t P t ) ,
Gronwall’s inequality gives
∥ S t ∥ F ≤ ∥ S 0 ∥ F e C t \left\|S_t\right\|_{\mathrm{F}}\le\left\|S_0\right\|_{\mathrm{F}}e^{Ct} ∥ S t ∥ F ≤ ∥ S 0 ∥ F e C t
for a finite constant C C C . Hence S t S_t S t cannot become unbounded in finite time. Part 1 therefore implies T max = ∞ T_{\max}=\infty T m a x = ∞ .
Proof of Theorem 7.23. Set y i ( t ) : = log s i ( t ) y_i(t):=\log s_i(t) y i ( t ) := log s i ( t ) . Eliminating p i p_i p i from (95) gives
y ¨ i + α t y ˙ i = 2 a i v i e y i , y i ( 0 ) = log s i , 0 , y ˙ i ( 0 ) = 0. (157) \ddot{y}_i+\alpha_t\dot{y}_i=2a_iv_ie^{y_i},\qquad y_i(0)=\log s_{i,0},\quad\dot{y}_i(0)=0. \tag*{(157)} y ¨ i + α t y ˙ i = 2 a i v i e y i , y i ( 0 ) = log s i , 0 , y ˙ i ( 0 ) = 0. (157)
Fix an index i i i with v i > 0 v_i>0 v i > 0 and set c i = 2 a i v i > 0 c_i=2a_iv_i>0 c i = 2 a i v i > 0 . The integrating-factor formula for (156) gives
y ˙ i ( t ) = c i ∫ 0 t exp ( − ∫ s t α r d r ) e y i ( s ) d s > 0 ( t > 0 ) . \dot{y}_i(t)=c_i\int_0^t\exp\left(-\int_s^t\alpha_r\,\mathrm{d}r\right)e^{y_i(s)}\,\mathrm{d}s>0\qquad(t>0). y ˙ i ( t ) = c i ∫ 0 t exp ( − ∫ s t α r d r ) e y i ( s ) d s > 0 ( t > 0 ) .
Therefore s i = e y i s_i=e^{y_i} s i = e y i is strictly increasing.
Consider the maximal solution of the i i i th scalar system in (95), and denote its maximal existence time by T i T_i T i . Suppose first that T i = ∞ T_i=\infty T i = ∞ . Monotonicity of y i y_i y i and boundedness of α \alpha α give, for t ≥ 1 t\ge1 t ≥ 1 ,
y ˙ i ( t ) ≥ c i e y i ( 0 ) ∫ t − 1 t exp ( − α ‾ ( t − s ) ) d s . \dot{y}_i(t)\ge c_i e^{y_i(0)}\int_{t-1}^t\exp\bigl(-\overline{\alpha}(t-s)\bigr)\,\mathrm{d}s. y ˙ i ( t ) ≥ c i e y i ( 0 ) ∫ t − 1 t exp ( − α ( t − s ) ) d s .
The right-hand side is a positive constant independent of t t t , so y i ( t ) → ∞ y_i(t)\to\infty y i ( t ) → ∞ as t → ∞ t\to\infty t → ∞ .
To compare this growth with the damping, set r i = y ˙ i e − y i / 2 r_i=\dot{y}_i e^{-y_i/2} r i = y ˙ i e − y i /2 . (157) gives
r ˙ i = e y i / 2 ( c i − r i 2 2 ) − α t r i . \dot{r}_i=e^{y_i/2}\left(c_i-\frac{r_i^2}{2}\right)-\alpha_t r_i. r ˙ i = e y i /2 ( c i − 2 r i 2 ) − α t r i .
Choose 0 < k < 2 c i 0<k<\sqrt{2c_i} 0 < k < 2 c i . Once y i y_i y i is sufficiently large, whenever 0 ≤ r i ≤ k 0\le r_i\le k 0 ≤ r i ≤ k we have
r ˙ i ≥ 1 2 ( c i − k 2 2 ) e y i / 2 ≥ c i − k 2 / 2 2 k y ˙ i . \dot{r}_i\ge\frac{1}{2}\left(c_i-\frac{k^2}{2}\right)e^{y_i/2}\ge\frac{c_i-k^2/2}{2k}\dot{y}_i. r ˙ i ≥ 2 1 ( c i − 2 k 2 ) e y i /2 ≥ 2 k c i − k 2 /2 y ˙ i .
Thus r i r_i r i cannot remain below k k k while y i y_i y i tends to infinity. Moreover, at r i = k r_i=k r i = k its derivative is positive, so it cannot subsequently cross below k k k . Consequently, for all sufficiently large times,
d d t e − y i / 2 = − 1 2 r i ≤ − k 2 . \frac{\mathrm{d}}{\mathrm{d}t}e^{-y_i/2}=-\frac{1}{2}r_i\le-\frac{k}{2}. d t d e − y i /2 = − 2 1 r i ≤ − 2 k .
This contradicts positivity of e − y i / 2 e^{-y_i/2} e − y i /2 on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) . Hence T i < ∞ T_i<\infty T i < ∞ .
Applying Proposition 7.22 in dimension one gives lim sup t ↑ T i s i ( t ) = ∞ \limsup_{t\uparrow T_i}s_i(t)=\infty lim sup t ↑ T i s i ( t ) = ∞ . Since s i s_i s i is increasing, it follows that s i ( t ) → ∞ s_i(t)\to\infty s i ( t ) → ∞ as t ↑ T i t\uparrow T_i t ↑ T i . The preceding argument also shows that, once s i s_i s i is sufficiently large, s ˙ i = r i s i 3 / 2 ≥ k s i 3 / 2 \dot{s}_i=r_i s_i^{3/2}\ge k s_i^{3/2} s ˙ i = r i s i 3/2 ≥ k s i 3/2 .
Every component with v i ≤ 0 v_i\le0 v i ≤ 0 is global by Proposition 7.22 in dimension one. Since the scalar systems are decoupled,
T max = min 1 ≤ i ≤ d T i . T_{\max}=\min_{1\le i\le d}T_i. T m a x = 1 ≤ i ≤ d min T i .
If some v i > 0 v_i>0 v i > 0 , this minimum is finite and attained by a positive-forcing component. For such an index,
s i ( t ) ⟶ ∞ , ∫ R d ∣ x ∣ 2 d ρ t ( x ) = ∑ j = 1 d s j ( t ) ⟶ ∞ s_i(t)\longrightarrow\infty,\qquad\int_{\mathbb{R}^d}|x|^2\,\mathrm{d}\rho_t(x)=\sum_{j=1}^{d}s_j(t)\longrightarrow\infty s i ( t ) ⟶ ∞ , ∫ R d ∣ x ∣ 2 d ρ t ( x ) = j = 1 ∑ d s j ( t ) ⟶ ∞
as t ↑ T max t\uparrow T_{\max} t ↑ T m a x . The equality of the matrix and PDE lifespans follows from Theorem 7.18 .
If every v i ≤ 0 v_i\le0 v i ≤ 0 , then V ⪯ 0 V\preceq0 V ⪯ 0 . Global existence follows directly from Proposition 7.22 .
Finite-Token Layers and Effective Depth
Proof of Proposition 7.24. 1. Define X ( t ) = X 0 G t ⊤ X(t)=X_0G_t^{\top} X ( t ) = X 0 G t ⊤ and Y ( t ) = X ( t ) P t Y(t)=X(t)P_t Y ( t ) = X ( t ) P t . The matrix G t G_t G t is invertible, since it solves the linear system (91) with G 0 = I d G_0=I_d G 0 = I d . Differentiating G t − 1 S t G t − ⊤ G_t^{-1}S_tG_t^{-\top} G t − 1 S t G t − ⊤ using (88) and (91) shows that it is constant. Its initial value is S 0 = X 0 ⊤ X 0 / N S_0=X_0^{\top}X_0/N S 0 = X 0 ⊤ X 0 / N , so
1 N X ( t ) ⊤ X ( t ) = G t S 0 G t ⊤ = S t . \frac{1}{N}X(t)^{\top}X(t)=G_tS_0G_t^{\top}=S_t. N 1 X ( t ) ⊤ X ( t ) = G t S 0 G t ⊤ = S t .
Using this identity and (91) , we obtain
X ˙ = X A S P = 1 N X A X ⊤ Y . \dot{X}=XASP=\frac{1}{N}XAX^{\top}Y. X ˙ = X A S P = N 1 X A X ⊤ Y .
Differentiating Y = X P Y=XP Y = X P and substituting (88) similarly gives
Y ˙ = − α Y − X P 2 S A + X V = − α Y − 1 N Y Y ⊤ X A + X V . \dot{Y}=-\alpha Y-XP^2SA+XV=-\alpha Y-\frac{1}{N}YY^{\top}XA+XV. Y ˙ = − α Y − X P 2 S A + X V = − α Y − N 1 Y Y ⊤ X A + X V .
Here, the last equality uses P = P ⊤ P=P^{\top} P = P ⊤ and X ⊤ X / N = S X^{\top}X/N=S X ⊤ X / N = S . The initial values follow from G 0 = I d G_0=I_d G 0 = I d and P 0 = 0 P_0=0 P 0 = 0 . Thus ( X , Y ) (X,Y) ( X , Y ) solves (96) . Its right-hand side is continuous in time and locally Lipschitz in ( X , Y ) (X,Y) ( X , Y ) , which proves uniqueness.
Since S t S_t S t is positive definite, the identity S t = X ( t ) ⊤ X ( t ) / N S_t = X(t)^\top X(t)/N S t = X ( t ) ⊤ X ( t ) / N in (97) gives
∥ S t ∥ F ≤ tr ( S t ) = 1 N ∥ X ( t ) ∥ F 2 . (158) \lVert S_t\rVert_F \le\operatorname{tr}(S_t) = \frac{1}{N}\lVert X(t)\rVert^2_F. \tag*{(158)} ∥ S t ∥ F ≤ tr ( S t ) = N 1 ∥ X ( t ) ∥ F 2 . (158)
If S t S_t S t is unbounded as t ↑ T max t \uparrow T_{\max} t ↑ T m a x , then (158) implies that ∥ X ( t ) ∥ F \lVert X(t)\rVert_F ∥ X ( t ) ∥ F is also unbounded. A solution extending beyond T max T_{\max} T m a x would be continuous and hence bounded on [ 0 , T max ] [0,T_{\max}] [ 0 , T m a x ] , contradicting this unboundedness of ∥ X ( t ) ∥ F \lVert X(t)\rVert_F ∥ X ( t ) ∥ F . Since part 1 gives a solution on [ 0 , T max ) [0,T_{\max}) [ 0 , T m a x ) , its maximal existence time is exactly T max T_{\max} T m a x .
Proof of Lemma 7.25. Fix 0 < T < T max 0<T<T_{\max} 0 < T < T m a x . Since α ∈ C 1 \alpha\in C^1 α ∈ C 1 , the solution Z = ( X , Y ) Z=(X,Y) Z = ( X , Y ) of (96) belongs to C 2 ( [ 0 , T ] ) C^2([0,T]) C 2 ([ 0 , T ]) . On the closed unit neighbourhood of Z ( [ 0 , T ] ) Z([0,T]) Z ([ 0 , T ]) , the right-hand side of (96) is Lipschitz in ( X , Y ) (X,Y) ( X , Y ) , uniformly for t ∈ [ 0 , T ] t \in[0,T] t ∈ [ 0 , T ] . The classical Euler estimate [76 §2.7] yields a constant C T ≥ 0 C_T \ge0 C T ≥ 0 such that
max ℓ h ≤ T ∥ Z ℓ h − Z ( ℓ h ) ∥ ≤ C T h (159) \max_{\ell h\le T}\lVert Z^h_\ell-Z(\ell h)\rVert\le C_T h \tag*{(159)} ℓ h ≤ T max ∥ Z ℓ h − Z ( ℓ h )∥ ≤ C T h (159)
for all sufficiently small h h h .
It remains to estimate the second moments. Using S ( ℓ h ) = X ( ℓ h ) ⊤ X ( ℓ h ) / N S(\ell h)=X(\ell h)^\top X(\ell h)/N S ( ℓ h ) = X ( ℓ h ) ⊤ X ( ℓ h ) / N from (97) and the definition of S ℓ h S^h_\ell S ℓ h in (99) , we obtain
∥ S ℓ h − S ( ℓ h ) ∥ F ≤ ∥ X ℓ h ∥ F + ∥ X ( ℓ h ) ∥ F N ∥ X ℓ h − X ( ℓ h ) ∥ F . \lVert S^h_\ell-S(\ell h)\rVert_F \le\frac{\lVert X^h_\ell\rVert_F+\lVert X(\ell h)\rVert_F}{N}\lVert X^h_\ell-X(\ell h)\rVert_F. ∥ S ℓ h − S ( ℓ h ) ∥ F ≤ N ∥ X ℓ h ∥ F + ∥ X ( ℓ h ) ∥ F ∥ X ℓ h − X ( ℓ h ) ∥ F .
The continuity of X X X on [ 0 , T ] [0,T] [ 0 , T ] and (159) bound both token matrices uniformly for ℓ h ≤ T \ell h\le T ℓ h ≤ T and 0 < h < h T 0<h<h_T 0 < h < h T . Since ∥ X ℓ h − X ( ℓ h ) ∥ F ≤ ∥ Z ℓ h − Z ( ℓ h ) ∥ \lVert X^h_\ell-X(\ell h)\rVert_F\le\lVert Z^h_\ell-Z(\ell h)\rVert ∥ X ℓ h − X ( ℓ h ) ∥ F ≤ ∥ Z ℓ h − Z ( ℓ h )∥ , the second-moment error is also bounded by a constant times h h h . Combining these estimates proves (100) .
Proof of Theorem 7.26. Fix R > s ∗ R>s_* R > s ∗ . For the chosen positive-forcing component, (95) gives p i ∗ ( t ) > 0 p_{i_*}(t)>0 p i ∗ ( t ) > 0 for 0 < t < T max 0<t<T_{\max} 0 < t < T m a x : at every zero of p i ∗ p_{i_*} p i ∗ , its derivative equals v i ∗ > 0 v_{i_*}>0 v i ∗ > 0 . Consequently, s ′ ( t R ) > 0 s'(t_R)>0 s ′ ( t R ) > 0 . By continuity of s ′ s' s ′ , we can choose δ R > 0 \delta_R>0 δ R > 0 and m R > 0 m_R>0 m R > 0 with [ t R − δ R , t R + δ R ] ⊂ ( 0 , T max ) [t_R-\delta_R,t_R+\delta_R]\subset(0,T_{\max}) [ t R − δ R , t R + δ R ] ⊂ ( 0 , T m a x ) such that
s ′ ( t ) ≥ m R for all t ∈ [ t R − δ R , t R + δ R ] . (160) s'(t)\ge m_R \qquad\text{for all }t\in[t_R-\delta_R,t_R+\delta_R]. \tag*{(160)} s ′ ( t ) ≥ m R for all t ∈ [ t R − δ R , t R + δ R ] . (160)
Next, we compare the discrete second-moment matrices S ℓ h S^h_\ell S ℓ h with the continuous matrices S ( ℓ h ) S(\ell h) S ( ℓ h ) . Applying (100) with T = t R + δ R T=t_R+\delta_R T = t R + δ R gives a constant C R > 0 C_R>0 C R > 0 such that
max 0 ≤ ℓ h ≤ t R + δ R ∥ S ℓ h − S ( ℓ h ) ∥ F ≤ C R h \max_{0\le\ell h\le t_R+\delta_R}\lVert S^h_\ell-S(\ell h)\rVert_F\le C_Rh 0 ≤ ℓ h ≤ t R + δ R max ∥ S ℓ h − S ( ℓ h ) ∥ F ≤ C R h
for all sufficiently small h > 0 h>0 h > 0 . Using S ℓ h = ( X ℓ h ) ⊤ X ℓ h / N S^h_\ell=(X^h_\ell)^\top X^h_\ell/N S ℓ h = ( X ℓ h ) ⊤ X ℓ h / N and (101) , we can express the error in the chosen direction as
∣ 1 N ∥ X ℓ h e i ∗ ∥ 2 2 − s ( ℓ h ) ∣ = ∣ e i ∗ ⊤ ( S ℓ h − S ( ℓ h ) ) e i ∗ ∣ ≤ C R h . (161) \left|\frac{1}{N}\lVert X^h_\ell e_{i_*}\rVert^2_2-s(\ell h)\right|=\left|e_{i_*}^{\top}(S^h_\ell-S(\ell h))e_{i_*}\right|\le C_Rh. \tag*{(161)} N 1 ∥ X ℓ h e i ∗ ∥ 2 2 − s ( ℓ h ) = e i ∗ ⊤ ( S ℓ h − S ( ℓ h )) e i ∗ ≤ C R h . (161)
The last inequality follows from ∥ e i ∗ ∥ 2 = 1 \lVert e_{i_*}\rVert_2=1 ∥ e i ∗ ∥ 2 = 1 and the Frobenius norm bound above.
We now locate the first discrete crossing within a time window of width proportional to h h h around t R t_R t R . Choose c R > C R / m R c_R>C_R/m_R c R > C R / m R . By (160) , a time displacement of c R h c_Rh c R h near t R t_R t R changes the continuous moment by at least m R c R h > C R h m_Rc_Rh>C_Rh m R c R h > C R h , exceeding the discretization error in (161) . First, we observe that, for sufficiently small h > 0 h > 0 h > 0 , both t R − c R h t_R-c_Rh t R − c R h and t R + ( c R + 1 ) h t_R+(c_R+1)h t R + ( c R + 1 ) h belong to [ t R − δ R , t R + δ R ] [t_R-\delta_R,t_R+\delta_R] [ t R − δ R , t R + δ R ] . Next, for every layer with ℓ h ≤ t R − c R h \ell h \le t_R-c_Rh ℓ h ≤ t R − c R h , monotonicity of s s s , the derivative bound (160) , and the error estimate (161) give
1 N ∥ X ℓ h e i ∗ ∥ 2 2 ≤ s ( t R − c R h ) + C R h ≤ R − ( m R c R − C R ) h < R . \frac{1}{N}\lVert X^\mathit{h}_\ell e_{i^*}\rVert_2^2 \le s(t_R-c_Rh)+C_Rh \le R-(m_Rc_R-C_R)h<R. N 1 ∥ X ℓ h e i ∗ ∥ 2 2 ≤ s ( t R − c R h ) + C R h ≤ R − ( m R c R − C R ) h < R .
Thus no layer of (98) with ℓ h ≤ t R − c R h \ell h \le t_R-c_Rh ℓ h ≤ t R − c R h reaches the threshold defining L R ( h ) L_R(h) L R ( h ) in (103) .
For the upper bound, take ℓ \ell ℓ to be the smallest integer satisfying ℓ h ≥ t R + c R h \ell h \ge t_R+c_Rh ℓ h ≥ t R + c R h . Since consecutive grid times differ by h h h ,
t R + c R h ≤ ℓ h < t R + ( c R + 1 ) h . t_R+c_Rh \le\ell h < t_R+(c_R+1)h. t R + c R h ≤ ℓ h < t R + ( c R + 1 ) h .
In particular, this grid time lies in the interval where (160) holds. Applying (160) and (161) at this layer gives
1 N ∥ X ℓ h e i ∗ ∥ 2 2 ≥ s ( t R + c R h ) − C R h ≥ R + ( m R c R − C R ) h > R . \frac{1}{N}\lVert X^\mathit{h}_\ell e_{i^*}\rVert_2^2 \ge s(t_R+c_Rh)-C_Rh \ge R+(m_Rc_R-C_R)h>R. N 1 ∥ X ℓ h e i ∗ ∥ 2 2 ≥ s ( t R + c R h ) − C R h ≥ R + ( m R c R − C R ) h > R .
Therefore L R ( h ) L_R(h) L R ( h ) is finite, and the two bounds yield
t R − c R h < h L R ( h ) ≤ t R + ( c R + 1 ) h . (162) t_R-c_Rh<hL_R(h)\le t_R+(c_R+1)h. \tag*{(162)} t R − c R h < h L R ( h ) ≤ t R + ( c R + 1 ) h . (162)
This proves h L R ( h ) = t R + O R ( h ) hL_R(h)=t_R+O_R(h) h L R ( h ) = t R + O R ( h ) .
Finally, the crossing times t R t_R t R increase with R R R . They cannot remain bounded away from T max T_{\max} T m a x , because s s s is bounded on every compact interval contained in [ 0 , T max ) [0,T_{\max}) [ 0 , T m a x ) . Hence t R ↑ T max t_R\uparrow T_{\max} t R ↑ T m a x . Taking the limits in (162) , we obtain
lim R → ∞ lim h ↓ 0 h L R ( h ) = lim R → ∞ t R = T max . \lim_{R\to\infty}\lim_{h\downarrow0}hL_R(h)=\lim_{R\to\infty}t_R=T_{\max}. R → ∞ lim h ↓ 0 lim h L R ( h ) = R → ∞ lim t R = T m a x .
Proof of Corollary 7.28. Choose R > max { s ∗ , M 2 } R>\max\{s_*,M^2\} R > max { s ∗ , M 2 } . By Theorem 7.26 , h L R ( h ) → t R < T max hL_R(h)\to t_R<T_{\max} h L R ( h ) → t R < T m a x . Thus, for sufficiently small h h h , the layer ℓ = L R ( h ) \ell=L_R(h) ℓ = L R ( h ) satisfies ℓ h < T max \ell h<T_{\max} ℓ h < T m a x . By (103) and ∥ e i ∗ ∥ 2 = 1 \lVert e_{i^*}\rVert_2=1 ∥ e i ∗ ∥ 2 = 1 ,
M 2 < R ≤ 1 N ∥ X ℓ h e i ∗ ∥ 2 2 ≤ max 1 ≤ j ≤ N ∥ x j , ℓ h ∥ 2 2 . M^2<R\le\frac{1}{N}\lVert X^\mathit{h}_\ell e_{i^*}\rVert_2^2\le\max_{1\le j\le N}\lVert x^\mathit{h}_{j,\ell}\rVert_2^2. M 2 < R ≤ N 1 ∥ X ℓ h e i ∗ ∥ 2 2 ≤ 1 ≤ j ≤ N max ∥ x j , ℓ h ∥ 2 2 .
This proves the claim.
Proof of Corollary 7.30. Fix M > 0 M>0 M > 0 and set h = T / L h=T/L h = T / L . By Corollary 7.28 , for every sufficiently large L L L , there is a layer ℓ \ell ℓ with ℓ h < T max ≤ T \ell h<T_{\max}\le T ℓ h < T m a x ≤ T at which a token norm exceeds M M M . Since T = L h T=Lh T = L h , this layer satisfies ℓ ≤ L \ell\le L ℓ ≤ L . Hence B L ( T ) > M B_L(T)>M B L ( T ) > M for all sufficiently large L L L . As M M M was arbitrary, the conclusion follows.
Proof of Corollary 7.31. Applying Lemma 7.25 with h = T / L h=T/L h = T / L bounds the approximation error by C T T / L C_TT/L C T T / L for all sufficiently large L L L . Moreover,
B L ( T ) ≤ max 0 ≤ t ≤ T ∥ X ( t ) ∥ F + C T T L . B_L(T)\le\max_{0\le t\le T}\lVert X(t)\rVert_F+\frac{C_TT}{L}. B L ( T ) ≤ 0 ≤ t ≤ T max ∥ X ( t ) ∥ F + L C T T .
The maximum on the right is finite because X X X is continuous on [ 0 , T ] [0,T] [ 0 , T ] . This proves both conclusions.
Implementation Details
Implementation Details. We collect the implementation details for the experiments in Section 9. The model configurations and training protocols are specified separately for the softmax and linear-attention experiments.
Datasets and tokenization. We use fixed GPT-2 BPE tokenizations of TinyStories [28 ] and OpenWebText, using tiktoken’s gpt2 encoding with 50,257 token IDs and a model vocabulary padded to 50,304. We use the supplied TinyStories training and validation splits. For OpenWebText, we split the documents of Skylion007/openwebtext into 99.5% training and 0.5% validation using a shuffled split with preprocessing seed 1337. Each document is encoded separately with encode_ordinary, followed by one end-of-text token (ID 50,256). The resulting documents are concatenated within each split and stored as a fixed stream of unsigned 16-bit token IDs. Input windows and their one-token-shifted targets may cross document boundaries; the end-of-text token is retained in the prediction task, and attention uses the causal mask without an additional document mask. The primary softmax, linear-attention and looping studies draw training windows from shuffled non-overlapping blocks, with a seeded boundary offset and reshuffling at each epoch. The normalized-sigmoid and native-L2 studies instead sample valid window starts uniformly with replacement, using a deterministic generator keyed by the run seed and optimizer update. Each comparison pairs the sampling rule and batch sequence across methods. The numbers of tokens in the preprocessed training and validation sets are listed in Table 5 . These dataset sizes are distinct from the training-token budgets used in the individual experiments.
Table 5. Numbers of tokens after GPT-2 BPE preprocessing.
MLP and optimizer. The MLP consists of a linear map from width d d d to 4 d 4d 4 d , a GELU activation, and a linear map back to d d d . All models use zero dropout and bias-free linear maps and LayerNorms. The optimizer is AdamW with peak learning rate 6 × 10 − 4 6 \times10^{-4} 6 × 1 0 − 4 , scalar-group learning-rate multiplier 10, and momentum coefficients ( 0.9 , 0.95 ) (0.9, 0.95) ( 0.9 , 0.95 ) . Weight decay is 0.1 only for the token and position embeddings, including the velocity embeddings when present; the tied output weight is the same parameter as the token-state embedding and is counted once. Attention and MLP matrices, LayerNorm scales, and learned scalar coefficients have zero weight decay. Training uses global gradient clipping at 1, 10% warm-up, and cosine decay to 0.1 of the peak learning rate.
Initialization. For an input token a i a_i a i at position i i i , the initial token state is x i ( 0 ) = E X ( a i ) + P X ( i ) x_i^{(0)} = E_X(a_i) + P_X(i) x i ( 0 ) = E X ( a i ) + P X ( i ) , where E X E_X E X and P X P_X P X are learned token and position embedding tables. All momentum variants additionally use v i ( 0 ) = E V ( a i ) + P V ( i ) v_i^{(0)} = E_V(a_i) + P_V(i) v i ( 0 ) = E V ( a i ) + P V ( i ) , with separate learned tables E V E_V E V , P V P_V P V ; these are not tied to E X E_X E X , P X P_X P X and the initial velocity is not zero. The ordinary baseline B has no velocity tables or stream. Linear-map and embedding weights are initialized from N ( 0 , 0.02 2 ) \mathcal{N}(0, 0.02^2) N ( 0 , 0.0 2 2 ) ; LayerNorm scales start at one, with normalization constant 10 − 5 10^{-5} 1 0 − 5 and no bias. The output-head weight is tied to the token-state embedding E X E_X E X , not to the velocity embedding. For a run with seed s s s , the velocity tables are initialized in the order E V , P V E_V, P_V E V , P V using a separate CPU random generator with seed s + 10,000,019 s+10{,}000{,}019 s + 10 , 000 , 019 , so paired momentum variants receive identical initial velocity tables without altering the other model parameters.
Learned update coefficients. For each trainable block k k k and branch b ∈ { A , M } b \in\{A, M\} b ∈ { A , M } (attention or MLP), the coefficients are scalar parameters broadcast over tokens and channels. Writing θ β , k b \theta^b_{\beta,k} θ β , k b , θ γ , k b \theta^b_{\gamma,k} θ γ , k b , and θ μ , k b \theta^b_{\mu,k} θ μ , k b for their unconstrained trainable parameters, we use
β k b = σ ( θ β , k b ) , γ k b = softplus ( θ γ , k b ) + 10 − 6 , μ k b = σ ( θ μ , k b ) , \beta^b_k = \sigma(\theta^b_{\beta,k}), \qquad\gamma^b_k = \operatorname{softplus}(\theta^b_{\gamma,k}) + 10^{-6}, \qquad\mu^b_k = \sigma(\theta^b_{\mu,k}), β k b = σ ( θ β , k b ) , γ k b = softplus ( θ γ , k b ) + 1 0 − 6 , μ k b = σ ( θ μ , k b ) ,
where σ ( u ) = ( 1 + exp ( − u ) ) − 1 \sigma(u) = (1+\exp(-u))^{-1} σ ( u ) = ( 1 + exp ( − u ) ) − 1 and softplus ( u ) = log ( 1 + exp ( u ) ) \operatorname{softplus}(u) = \log(1+\exp(u)) softplus ( u ) = log ( 1 + exp ( u )) . The initial coefficient values are β k b = 0.9 \beta^b_k = 0.9 β k b = 0.9 , γ k b = 1 \gamma^b_k = 1 γ k b = 1 , and μ k b = 0.9 \mu^b_k = 0.9 μ k b = 0.9 wherever look-ahead is enabled. For H0 _0 0 , H1 _1 1 , and R2 _2 2 , attention look-ahead is instead fixed at μ k b = 0 \mu^b_k = 0 μ k b = 0 , with no trainable attention look-ahead parameter; MLP look-ahead is retained. Attention and MLP coefficients are learned separately, as are the coefficients of distinct trainable blocks. Their unconstrained parameters use ten times the base learning rate and no weight decay. These are learned update coefficients, not coefficients constrained to a prescribed continuous-time damping law.
Looping and parameter sharing. In Table 3 , both layouts execute eight logical block applications. The tied layout applies two distinct trainable blocks in the order (0, 1, 0, 1, 0, 1, 0, 1). Sharing includes all attention and MLP weights, position and velocity LayerNorms, and learned update coefficients within each block. For each method and seed, the untied layout starts from eight separate copies of the same two initialized blocks in this alternating order; the copies subsequently train independently. Thus the tied and untied models implement the same function at initialization, rather than starting from eight independently sampled blocks. Token and velocity embeddings, the final LayerNorm and the output head are not duplicated across repetitions. The token and velocity streams are initialized once per forward pass and continue through all eight applications without resets or repeated input injection. Training backpropagates through all eight applications without detaching the streams; gradients from repeated uses accumulate on the shared parameters.
Conventions Paired methods always share the training batches, tokenizer, data splits, and validation targets. The reported endpoint is the final evaluation, rather than the best checkpoint. Peak memory uses only complete dataset/seed blocks.
Hardware and model output. All runs use the same RTX A6000 cohort2 and IEEE FP32 arithmetic without TF32, with no method-specific tuning. All models apply a final LayerNorm to the token states, followed by a linear output head that produces the logits for next-token prediction.
Cost conventions. GPU-hours are the mean allocated GPU time per completed run, computed from the scheduler’s elapsed allocation time. Each run uses one GPU. Means are taken over the paired seeds separately for TinyStories and OpenWebText. Peak MiB denotes peak PyTorch-allocated GPU tensor memory. Parameter counts include all trainable embeddings, blocks, and final normalization, counting shared weights only once.