betaFeedback

WorksMathematics

SympFormer: Inertial Attention Blocks from Damped Hamiltonian Dynamics on Spaces of Probability Measures

Abstract

The attention mechanism underpins modern large language models, yet their dynamics remain only partially understood. Recent perspectives interpret attention blocks as interacting particle systems, whose mean-field limits correspond to gradient flows of energy functionals on spaces of probability densities on Rd\mathbb R^d equipped with Wasserstein-type metrics. We extend this viewpoint by deriving accelerated attention blocks from inertial, Nesterov-type damped Hamiltonian dynamics on these density spaces, developing the geometry and differential calculus needed to make the construction rigorous. In particular, we introduce a local-mobility geometry for the Otto-Wasserstein and softmax-Wasserstein settings, and a quotient Stein geometry that resolves the degeneracy of the linear-attention kernel to yield a well-defined metric and differential calculus. We provide rigorous foundations when softmax attention admits a gradient-flow formulation and when its inertial counterpart admits a damped Hamiltonian on explicitly characterized domains. For linear attention, we obtain a finite-dimensional solution theory with sharp conditions for global existence versus finite-time second-moment blow-up. We further prove a quantitative link between continuous moment blow-up and the discrete-layer depth at which token norms cross a prescribed threshold. Finally, motivated by the finite-token geometry, we propose geometric corrections to token and momentum updates. In decoder-only, nanoGPT-style experiments on TinyStories and OpenWebText, these corrections lower validation negative log-likelihood on next-token prediction relative to momentum-only controls.

Introduction

This paper develops foundations of infinite-dimensional Nesterov acceleration and connects it to the mathematical theory of deep transformers—two fields that at first glance appear largely unrelated. To motivate this connection, we first consider the role of attention in transformers.

The transformer architecture introduced in Attention Is All You Need [77] forms the backbone of modern large language models. Its attention layers update token representations by allowing tokens to exchange information across a sequence. The widespread use and societal importance of these systems make a mathematical understanding of their behavior important. Although attention has become an active subject of mathematical research, with contributions to approximation theory, stability, learning, and the dynamics of token interactions [17, 24, 30, 41, 45, 81, 89], even the behavior of the original transformer architecture remains only partially understood.

Attention layers can be viewed as discrete dynamical systems for token representations [24, 81] and, in a continuous-depth limit, as interacting particle systems [35, 51, 68]. Under suitable assumptions, a subsequent mean-field limit describes the evolution of the token distribution through a non-linear transport equation, allowing PDE methods to be used to study collective behavior [13, 16]. We provide further details in Section 2 and illustrate this passage in Figure 2. In their recent work, Castin–Ablin–Carrillo–Peyré [16] establish well-posedness of the mean-field PDE for softmax attention for compactly supported initial data and study Gaussian initial data separately. General non-Gaussian initial densities with unbounded support remain outside their analysis. As in their work, we consider probability densities on Rd\mathbb{R}^{d}, allowing unbounded support, which lets us study growth in token magnitudes, which sphere-constrained models exclude [13, 33, 34, 35, 56].

Residual attention $X_i^{(k)} \mapsto X_i^{(k+1)}$, particle dynamics $\dot{X}_i = \Gamma_X(X_i)$, and density evolution $\partial_t \rho = -\nabla \cdot (\rho\Gamma_\rho)$, with arrows labeled $h \to 0$ and $N \to \infty$

Figure 2. From attention layers to transport dynamics: with shared weights, the formal continuous-depth limit turns residual updates into an interacting particle system. A suitably normalized mean-field limit then describes the token distribution through a continuity equation.

The transformer PDE arising in the mean-field limit can also be interpreted formally as a gradient flow of an interaction energy on a space of probability densities equipped with a suitable Riemannian metric. For softmax attention, Castin et al. [16, Sec. 5] develop this interpretation, but explicitly leave its rigorous geometric justification open. Before moving on, we highlight several obstacles to making this gradient-flow interpretation rigorous. The first concerns differentiation itself: the admissible density classes on Rd\mathbb{R}^{d} are not assumed to carry a smooth manifold structure. Therefore the relevant tangent and cotangent spaces and notions of differentiation must be specified explicitly. A further difficulty comes from weak Riemannian metrics, for which even smooth energies need not admit a gradient vector field; see § 3.1.1. To define the Riemannian metrics considered here, we follow a guiding principle: associate a differential operator, called the mobility operator, with the attention kernel and use its inverse to define the metric. Making this construction rigorous requires identifying suitable function spaces on which this operator is invertible. Figure 3 illustrates this passage from attention kernels to Riemannian metrics.

Diagram showing the correspondence from an attention kernel to a mobility operator and Riemannian metric

Figure 3. From attention to geometry: an attention kernel gives rise to a mobility operator. The key step is to identify suitable function spaces on which this operator is invertible. Its inverse then defines the metric underlying a rigorous analog of Otto calculus.

A natural extension of this viewpoint is inspired by Nesterov acceleration [60] in finite-dimensional optimization, which, roughly speaking, introduces momentum to accelerate gradient descent; see § 2.2. On spaces of probability densities, this leads us to study infinite-dimensional damped Newton systems, a particular class of damped Hamiltonian systems. Newton dynamics on such spaces also arise in geometric hydrodynamics; see Khesin–Misiołek–Modin [40]. Wang and Li [84] study such damped Newton dynamics as accelerated gradient flows on spaces of probability densities. However, their geometric construction has serious analytic gaps even in the Otto–Wasserstein and Stein settings; see § 3.1 and Appendix § 7.4. The authors of [71] use this framework to derive inertial PDEs for linear and softmax attention, but leave the underlying geometric construction at a formal level.

These connections motivate the following question: *Can PDE methods, functional analysis, and geometry applied to the mean-field limit yield new insights into the dynamics of transformers with finitely many tokens and layers?

Main results. Our paper makes the following main contributions, whose connections we summarize in Figure 1.

Softmax and linear attention; Continuous dynamics and discrete layers; Geometric foundations on $\mathbb{R}^{d}$; Local-mobility geometry; Otto–Wasserstein and softmax-Wasserstein; {xref}`Theorem 4.3`; Quotient Stein geometry; Linear attention; {xref}`Theorem 5.3`; Function spaces, metrics, and differential calculus; Rigorous softmax formulations; Conditions for gradient and damped Hamiltonian equations; {xref}`Corollary 6.4`; {xref}`Theorem 6.7`; Linear-attention solution theory; Existence, uniqueness, and moment blow-up; {xref}`Theorem 7.18`; {xref}`Theorem 7.23`; Direct finite-token geometry; Explicit geometric corrections to token and momentum updates; {xref}`Lemma 6.11`; PDE growth predicts layer depth; Thresholds for large token norms at a fixed number of tokens; {xref}`Theorem 7.26`; {xref}`Corollary 7.30`; Geometric corrections improve prediction; Softmax and linear architectures compared with momentum controls; Tables 2 and 4

Figure 1. From attention geometry to analysis and architecture. The local-mobility and quotient Stein constructions provide function spaces and a differential calculus for attention dynamics on Rd\mathbb{R}^{d}. For softmax attention, they identify when the equations admit regular gradient and Hamiltonian formulations. For inertial linear attention under the stated assumptions, continuous blow-up predicts large token norms in discrete layers, even at a fixed number of tokens. A direct finite-token geometric construction motivates corrections to token and momentum updates. The reported experiments evaluate their predictive gains and additional computational cost.

  1. Foundations for infinite-dimensional Nesterov acceleration. For linear and softmax attention, we develop the geometric foundations needed to make the formal gradient-flow and damped Hamiltonian formulations rigorous on specified domains (Corollary 6.4, Theorem 6.7, Proposition 7.4, and Lemma 7.16). This provides the rigorous justification left open in [16 Sec. 5.4]. For softmax attention, we introduce a local-mobility geometry, which includes both Otto–Wasserstein and softmax–Wasserstein geometry on smooth probability densities over Rd\mathbb{R}^{d} (Theorem 4.3). For linear attention, we introduce a quotient Stein geometry adapted to the bilinear attention kernel (Theorem 5.3). Since a differentiable structure on the spaces of probability densities involved is not given a priori, we develop the associated differential calculus directly on these spaces (Appendices § 7.5 and § 7.6).

  1. Well-posedness and blow-up for the linear-attention transformer PDE and its accelerated counterpart. By choosing coordinates adapted to the evolving density, we reduce the infinite-dimensional dynamics of both equations to finite-dimensional matrix ODEs. The resulting two-way PDE–ODE correspondences yield existence and uniqueness of strong solutions for admissible non-Gaussian initial data with potentially unbounded support (Theorems 7.6 and 7.18). In the accelerated case, we require a quadratic initial momentum potential. These reductions also give conditions for global-in-time existence and finite-time second-moment blow-up (Proposition 7.13 and Theorem 7.23).

  1. From PDE blow-up to growth of token norms across layers. For accelerated linear attention at fixed token count, we use continuous moment growth to predict the first explicit-Euler layer crossing a prescribed directional second-moment threshold. The prediction has a bounded error in layer count as the step size tends to zero (Theorem 7.26). As a corollary, continuum moment blow-up implies that every prescribed token-norm bound is exceeded in the discrete layers for sufficiently small step sizes (Corollary 7.28). To the best of our knowledge, these are the first results linking continuum moment blow-up to quantitative threshold-depth predictions and arbitrarily large token norms in discrete attention networks at fixed token count.

  1. Numerical evaluation of geometric attention updates. We derive corrections to token and momentum updates from the finite-token softmax metric (Lemma 6.11). On TinyStories and OpenWebText, we compare practical versions of these updates with matched momentum-only controls. The corrections give small reductions in mean validation loss in some settings, at increased computational cost (§ 7.2).

We discuss the relation to previous work in § 7.3.

Organization. Section 2 recalls the interpretation of attention layers as interacting particle systems and introduces their continuum descriptions. § 2.2 reviews Nesterov acceleration and motivates its extension to attention through geometric formulations of density dynamics. The obstruction to using the full formal tangent space in Stein geometry is established in Appendix § 7.4. § 3.1 and § 3.3 develop the local-mobility and quotient Stein frameworks, respectively. They identify suitable function spaces, establish invertibility of the corresponding mobility operators,

and define the induced metrics and gradients. The notions of differentiability needed for the gradient and Hamiltonian formulations are developed in Appendices § 7.5 and § 7.6. § 4.1 treats softmax attention. It distinguishes pointwise well-definedness from regular gradient and Hamiltonian formulations, identifies their admissibility conditions, and derives the finite-token geometric correction used in the numerical section. § 4.2 applies the quotient Stein framework to linear attention. It introduces covariance coordinates, establishes the ODE–PDE correspondence and strong well-posedness, and studies global existence and finite-time moment blow-up. It then relates continuous growth to activation thresholds and effective depth in discrete attention layers. § 7.1 introduces the discretizations and resulting architectures, and § 7.2 compares their linear and softmax variants in language-modeling experiments. We discuss the relation to previous work in § 7.3.

I Attention Models and Acceleration

Transformers as Dynamical Systems

In this section, we recall the mean-field description of transformer dynamics and discuss possible geometric interpretations of the resulting PDE; see, e.g., [16, 35]. We first specify the attention weights and residual layers, then describe the formal passage to continuous-depth particle dynamics and their mean-field continuity equation, as illustrated in Figure 2. Finally, we discuss the geometric requirements for a gradient-flow interpretation and the obstacles to making it rigorous.

Token configurations and attention weights. Transformers act on token configurations

X:=(X1,…,XN)∈(Rd)N.\mathbf{X} := (X_1,\ldots,X_N) \in(\mathbb{R}^{d})^N.

The literature also considers embeddings in (Sd)N(\mathbb{S}^{d})^N; see, e.g., [13, 18, 21, 34], which this paper does not handle. Self-attention combines information across tokens, whereas MLP layers apply tokenwise non-linear transformations. Let Q,K∈Rm×dQ, K \in\mathbb{R}^{m \times d} and V∈Rd×dV \in\mathbb{R}^{d \times d}, m≤dm \le d, denote the query, key, and value matrices, and set A:=K⊤Q∈Rd×dA := K^{\top}Q \in\mathbb{R}^{d \times d}. For the geometric constructions in § 4.2 and § 4.1, we assume that m=dm = d and that AA is a symmetric matrix. In the following, we ignore the causal masking, layer normalization, the standard attention scaling, the output projection, and the multi-head structure usually used in transformer architectures.

Recall that the softmax function is defined for w∈Rdw \in\mathbb{R}^d by

softmax⁡(w):=(exp⁡(wi)∑j=1dexp⁡(wj))i=1d.\operatorname{softmax}(w) := \left(\frac{\exp(w_i)}{\sum_{j=1}^{d}\exp(w_j)}\right)_{i=1}^{d}.

We consider the following two choices of attention weights κ=κX:Rd×Rd→R\kappa= \kappa_X : \mathbb{R}^d \times\mathbb{R}^d \to\mathbb{R}:

κX(Xi,Xj)={exp⁡((Xj)⊤AXi)∑ℓ=1Nexp⁡((Xℓ)⊤AXi)=softmax⁡j((Xℓ)⊤AXi)ℓ=1Nsoftmax self-attention,(Xj)⊤AXilinear self-attention.(1)\kappa_X(X_i,X_j) = \begin{cases} \frac{\exp\left((X_j)^\top A X_i\right)}{\sum_{\ell=1}^{N}\exp\left((X_\ell)^\top A X_i\right)} = \operatorname{softmax}_j\left((X_\ell)^\top A X_i\right)_{\ell=1}^{N} & \text{softmax self-attention},\\ (X_j)^\top A X_i & \text{linear self-attention}. \end{cases} \tag*{(1)}

We use the term “linear self-attention” for unnormalized dot-product attention, to distinguish it from other linear self-attention mechanisms [39]. Note that the evaluation cost of transformers with softmax attention scales quadratically with sequence length. Kernel attention addresses this by replacing softmax with a more general kernel function. For example, [66] aims to identify kernels that retain the expressivity of attention while enabling quasi-linear computation.

Residual attention and MLP layers. Let (X10,…,XN0)(X_1^0,\ldots,X_N^0) be the input tokens, let h>0h > 0 be the step size, and let k∈Nk \in\mathbb{N} denote the layer index. The attention matrices usually differ between layers. For the continuous-depth description below, we instead keep QQ, KK, and VV fixed across layers (“weight sharing”), as in architectures such as ALBERT [43]. The residual attention sublayer is

Xi(k+1)=Xi(k)+h∑j=1NκX(Xi(k),Xj(k))VXj(k),(2)X_i^{(k+1)} = X_i^{(k)} + h \sum_{j=1}^{N}\kappa_X\left(X_i^{(k)},X_j^{(k)}\right)VX_j^{(k)}, \tag*{(2)}

where we have to replace hh by h/Nh/N for linear attention. For weight matrices W,W~∈Rd×dW,\widetilde{W} \in\mathbb{R}^{d \times d}, a bias b∈Rdb \in\mathbb{R}^d, and a non-linear function σ:R→R\sigma: \mathbb{R} \to\mathbb{R} applied componentwise, the residual MLP sublayer is

Xi(k+1)=Xi(k)+hWσ(W~Xi(k)+b),(3)X_i^{(k+1)} = X_i^{(k)} + hW\sigma\left(\widetilde{W}X_i^{(k)} + b\right), \tag*{(3)}

where the MLP parameters may differ between layers. The updates (2) and (3) describe the two sublayers separately. In the following passage to particle and density dynamics, we consider only the attention update (2).

Particle dynamics. Given input tokens X(0)=(X1(0),…,XN(0))∈(Rd)NX(0) = (X_1(0),\ldots,X_N(0)) \in(\mathbb{R}^d)^N, we interpret the attention iteration as a discretization in depth. Formally, as the step size h→0h \to0, this gives

X˙i(t)=ΓX(t)(Xi(t)),1≤i≤N,(4)\dot{X}_i(t) = \Gamma_{X(t)}(X_i(t)), \qquad1 \leq i \leq N, \tag*{(4)}

where ΓX:Rd→Rd\Gamma_X : \mathbb{R}^d \to\mathbb{R}^d is the attention vector field determined by QQ, KK, VV, and the current token configuration X(t)X(t). For softmax attention, it is given by

ΓX(x):=∑j=1Nexp⁡(Qx⋅KXj)∑ℓ=1Nexp⁡(Qx⋅KXℓ)VXj,for x∈Rd.\Gamma_X(x) := \sum_{j=1}^{N}\frac{\exp\left(Qx \cdot KX_j\right)}{\sum_{\ell=1}^{N}\exp\left(Qx \cdot KX_\ell\right)}VX_j, \qquad\text{for } x \in\mathbb{R}^d.

The velocity of each token in (4) is a similarity-weighted average of the value vectors. The resulting dynamics form a non-linear interacting particle system.

Density continuity equation. For a probability density ρ\rho, the corresponding attention field is

Γρ:Rd→Rd,x↦∫Vyκρ(x,y)ρ(y) dy,\Gamma_\rho:\mathbb{R}^d\to\mathbb{R}^d,\qquad x\mapsto\int_V y\kappa_\rho(x,y)\rho(y)\,\mathrm{d}y,

for all ρ\rho such that this integral is finite. The two choices in (1) lead to the density-dependent weights

κρ(x,y):={exp⁡(y⊤Ax)∫exp⁡(z⊤Ax)ρ(z) dz,softmax self-attention,y⊤Ax,linear self-attention.\kappa_\rho(x,y):= \begin{cases} \dfrac{\exp(y^\top A x)}{\int\exp(z^\top A x)\rho(z)\,\mathrm{d}z}, & \text{softmax self-attention},\\ y^\top A x, & \text{linear self-attention}. \end{cases}

In the formal mean-field limit, the self-attention dynamics (4) are described on Rd\mathbb{R}^d by the continuity equation

∂tρt=−∇⋅(ρtΓρt),t>0,(5)\partial_t\rho_t=-\nabla\cdot(\rho_t\Gamma_{\rho_t}),\qquad t>0, \tag*{(5)}

see [16, 35] and also [13, 35, 68, 90].

Geometry of the mean-field limit. Under suitable assumptions on the attention matrices, the mean-field equation (5) admits a formal gradient-flow interpretation. Recall that a Riemannian gradient flow on a manifold M\mathcal{M} is specified by a pair (F,G)(\mathcal{F},G): a smooth functional F:M→R\mathcal{F}:\mathcal{M}\to\mathbb{R} and a Riemannian metric GG. Thus, identifying an energy is only part of the construction; the geometry determines the corresponding direction of steepest descent.

Making this interpretation rigorous requires specifying the underlying space of probability densities. A central distinction is whether these densities have compact support or extend over all of Rd\mathbb{R}^d. In the latter case, additional integrability and coercivity conditions are needed to justify the formal geometric operators. For softmax attention, a general rigorous justification of this geometry for non-compactly supported densities was left open in [16 Sec. 5].

This geometric viewpoint provides a way to introduce momentum into attention dynamics, following the principle of Nesterov acceleration, as discussed in the next section. Discretizing the resulting inertial equations yields attention layers that propagate both token positions and a momentum variable across network depth.

We close this subsection by recalling the guiding question of this paper:

Can PDE methods, functional analysis, and geometry applied to the mean-field limit yield new insights into the dynamics of transformers with finitely many tokens and layers?

Nesterov's Acceleration Method and Attention Models

The aim of this section is to explain how Nesterov acceleration provides a way to introduce momentum into attention dynamics and to identify the geometric constructions needed to justify this approach.

We first recall Nesterov acceleration in Euclidean space and its continuous-time formulation as the damped Hamiltonian system (10). We then describe the corresponding construction for smooth probability densities on compact manifolds, leading to the accelerated Otto–Wasserstein equations (13). This construction relies on the mobility operator being invertible between suitable cotangent and tangent spaces. We explain why this requirement presents an obstruction in Stein geometry.

Finally, we connect this viewpoint to the linear and softmax attention models introduced in Section 2. Their kernels give rise to different mobility operators and hence different formal geometric descriptions. We state the resulting accelerated systems in (17) and (19), and identify the function-space and differentiability questions that must be resolved to make them rigorous. These questions motivate the local-mobility calculus developed in § 3.1 and the quotient Stein geometry developed in § 3.3.

Nesterov Acceleration on Rd\mathbb{R}^{d}

Nesterov acceleration [60] is a well-known technique for improving the convergence rate of gradient descent methods. For an objective function F∈C1(Rd;R)F \in C^{1}(\mathbb{R}^{d};\mathbb{R}), we consider the optimization problem

min⁡x∈RdF(x).(6)\min_{x \in\mathbb{R}^{d}} F(x). \tag*{(6)}

The gradient descent sequence (x(k))k∈N(x^{(k)})_{k \in\mathbb{N}} starting at x0∈Rdx_{0} \in\mathbb{R}^{d} with step size τ\tau is given by

{x(k+1)=x(k)−τ∇F(x(k)),x(0)=x0.\begin{cases} x^{(k+1)} = x^{(k)} - \tau\nabla F(x^{(k)}),\\ x^{(0)} = x_{0}. \end{cases}

Nesterov’s accelerated gradient descent is obtained by choosing an extrapolation coefficient as

References

  1. [1]S. Adams, N. Dirr, M. Peletier, and J. Zimmer. Large deviations and gradient flows. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371(2005):20120341, 12 2013.DOI
  2. [2]F. Alimisis, A. Orvieto, G. Becigneul, and A. Lucchi. A continuous-time perspective for modeling acceleration in Riemannian optimization. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1297–1307. PMLR, 26–28 Aug 2020.arxiv.org/abs/1910.10782
  3. [3]L. Ambrosio, N. Gigli, and G. Savaré. Gradient flows: in Metric Spaces and in the Space of Probability Measures. Springer Science & Business Media, 2 edition, 2008.
  4. [4]V. Arnold. Sur la géométrie différentielle des groupes de Lie de dimension infinie et ses applications à l’hydrodynamique des fluides parfaits. Annales de l’Institut Fourier, 16(1):319–361, 1966.DOI
  5. [5]C. J. Atkin. The Hopf–Rinow theorem is false in infinite dimensions. Bulletin of the London Mathematical Society, 7(3):261–266, Nov. 1975.DOI
  6. [6]H. Attouch, X. Goudou, and P. Redont. The heavy ball with friction method, I. the continuous dynamical system: global exploration of the local minima of a real-valued function by asymptotic analysis of a dissipative dynamical system. Communications in Contemporary Mathematics, 02(01):1–34, 2000.DOI
  7. [7]H. Attouch, J. Peypouquet, and P. Redont. Fast convex optimization via inertial dynamics with Hessian driven damping. Journal of Differential Equations, 261(10):5734–5783, 2016.arxiv.org/abs/1601.07113
  8. [8]D. Bakry, I. Gentil, and M. Ledoux. Analysis and geometry of Markov diffusion operators, volume 103. Springer, 2014.DOI
  9. [9]M. Bauer, M. Bruveris, and P. W. Michor. The homogeneous Sobolev metric of order one on diffeomorphism groups on the real line. Journal of Nonlinear Science, 24(5):769–808, 2014.DOI
  10. [10]M. Bauer, P. Harms, and P. W. Michor. Regularity and completeness of half-Lie groups. Journal of the European Mathematical Society, 2025. Published online first.
  11. [11]R. Bhatia, T. Jain, and Y. Lim. On the Bures–Wasserstein distance between positive definite matrices. Expositiones Mathematicae, 37(2):165–191, 2019.arxiv.org/abs/1712.01504
  12. [12]S. Brody, U. Alon, and E. Yahav. On the expressivity role of layernorm in transformers’ attention. In Findings of the Association for Computational Linguistics: ACL 2023, pages 14211–14221, 2023.arxiv.org/abs/2305.02582
  13. [13]M. Burger, S. Kabri, Y. Korolev, T. Roith, and L. Weigand. Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer normalization. Philosophical Transactions A, 383(2298):20240233, 2025.arxiv.org/abs/2501.03096
  14. [14]J. A. Carrillo, Y.-P. Choi, and O. Tse. Convergence to equilibrium in Wasserstein distance for damped Euler equations with interaction forces. Communications in Mathematical Physics, 365(1):329–361, 2019.
  15. [15]J. A. Carrillo, S. Lisini, G. Savaré, and D. Slepčev. Nonlinear mobility continuity equations and generalized displacement convexity. Journal of Functional Analysis, 258(4):1273–1309, 2010.arxiv.org/abs/0901.3978
  16. [16]V. Castin, P. Ablin, J. A. Carrillo, and G. Peyré. A unified perspective on the dynamics of deep transformers. Foundations of Computational Mathematics, 2026. In press.arxiv.org/abs/2501.18322
  17. [17]V. Castin, P. Ablin, and G. Peyré. How smooth is attention? In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 5817–5840. PMLR, 2024.arxiv.org/abs/2312.14820
  18. [18]J. Chemseddine, G. Kornhardt, and G. Steidl. Spherical flows for sampling categorical data. Advances in Neural Information Processing Systems (Spotlight), 2026.
  19. [19]J. Chemseddine, C. Wald, R. Duong, and G. Steidl. Neural sampling from Boltzmann densities: Fisher-Rao curves in the Wasserstein geometry. In International Conference on Learning Representations, volume 2025, pages 10768–10792, 2025.arxiv.org/abs/2410.03282
  20. [20]S. Chen, Q. Li, O. Tse, and S. J. Wright. Accelerating optimization over the space of probability measures. Journal of Machine Learning Research, 26(31):1–40, 2025.
  21. [21]S. Chen, Z. Lin, Y. Polyanskiy, and P. Rigollet. Quantitative clustering in mean-field transformer models. arXiv preprint arXiv:2504.14697, 2025.arxiv.org/abs/2504.14697
  22. [22]M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Ł. Kaiser. Universal transformers. In International Conference on Learning Representations, 2019.arxiv.org/abs/1807.03819
  23. [23]J. Dolbeault, B. Nazaret, and G. Savaré. A new class of transport distances between measures. Calculus of Variations and Partial Differential Equations, 34(2):193–231, 2009.
  24. [24]Y. Dong, J.-B. Cordonnier, and A. Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2793–2803. PMLR, 2021.arxiv.org/abs/2103.03404
  25. [25]A. Duncan, N. Nüsken, and L. Szpruch. On the geometry of Stein variational gradient descent. Journal of Machine Learning Research, 24(56):1–39, 2023.
  26. [26]E. Dupont, A. Doucet, and Y. W. Teh. Augmented neural ODEs. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1904.01681
  27. [27]I. Ekeland. The Hopf-Rinow theorem in infinite dimension. J. Differential Geometry, 13(2):287–301, 1978.DOI
  28. [28]R. Eldan and Y. Li. TinyStories: How small can language models be and still speak coherent English? arXiv preprint arXiv:2305.07759, 2023.arxiv.org/abs/2305.07759
  29. [29]Y. Eliashberg and L. Polterovich. Bi-invariant metrics on the group of Hamiltonian diffeomorphisms. International Journal of Mathematics, 4:727–738, 1993.
  30. [30]T. Furuya, M. V. de Hoop, and G. Peyré. Transformers are universal in-context learners. In International Conference on Learning Representations, 2025.arxiv.org/abs/2408.01367
  31. [31]W. Gangbo, H. K. Kim, and T. Pacini. Differential forms on Wasserstein space and infinite-dimensional Hamiltonian systems. Memoirs of the American Mathematical Society, 211(993):vi+77, 2011.arxiv.org/abs/0807.1065
  32. [32]J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. Bartoldson, B. Kailkhura, A. Bhatele, and T. Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. In Advances in Neural Information Processing Systems, volume 38, pages 41340–41391, 2026.arxiv.org/abs/2502.05171
  33. [33]B. Geshkovski, H. Koubbi, Y. Polyanskiy, and P. Rigollet. Dynamic metastability in the self-attention model, 2024.arxiv.org/abs/2410.06833
  34. [34]B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet. The emergence of clusters in self-attention dynamics. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 36, pages 57026–57037, 2023.DOI
  35. [35]B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet. A mathematical perspective on transformers. Bulletin of the American Mathematical Society, 62(3):427–479, 2025.arxiv.org/abs/2312.10794
  36. [36]R. S. Hamilton. The inverse function theorem of Nash and Moser. Bulletin of the American Mathematical Society, 7(1), July 1982.DOI
  37. [37]P. Hartman. Ordinary Differential Equations, volume 38 of Classics in Applied Mathematics. Society for Industrial and Applied Mathematics, 2002.
  38. [38]R. Jordan, D. Kinderlehrer, and F. Otto. The variational formulation of the Fokker–Planck equation. SIAM Journal on Mathematical Analysis, 29(1):1–17, 1998.
  39. [39]A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156–5165. PMLR, 2020.arxiv.org/abs/2006.16236
  40. [40]B. Khesin, G. Misiołek, and K. Modin. Geometric hydrodynamics and infinite-dimensional newton’s equations. Bulletin of the American Mathematical Society, 58(3):377–442, 2021.DOI
  41. [41]H. Kim, G. Papamakarios, and A. Mnih. The Lipschitz constant of self-attention. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5562–5571. PMLR, 2021.arxiv.org/abs/2006.04710
  42. [42]J. D. Lafferty. The density manifold and configuration space quantization. Transactions of the American Mathematical Society, 305(2):699–741, 1988.
  43. [43]Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. ALBERT: A lite BERT for self-supervised learning of language representations, 2020.arxiv.org/abs/1909.11942
  44. [44]S. Lang. Fundamentals of differential geometry, volume 191 of Graduate Texts in Mathematics. Springer-Verlag, New York, 1999.DOI
  45. [45]S. Li, T. J. Maranzatto, J. Peszek, T. Teolis, S. Akkoc, K. Riedl, S. Ulukus, and N. García Trillos. On the diverse dynamical behaviors arising in deep linear transformers, 2026.
  46. [46]S. Li, Y. Zhang, J. Guo, Q. Gu, and M. Wang. Deeploop: Depth scaling for looped transformers. arXiv preprint arXiv:2607.13491, 2026.arxiv.org/abs/2607.13491
  47. [47]S. Lisini. Characterization of absolutely continuous curves in wasserstein spaces. Calculus of variations and partial differential equations, 28(1):85–120, 2007.DOI
  48. [48]C. Liu and J. Zhu. Riemannian Stein variational gradient descent for Bayesian inference. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 3627–3634. AAAI Press, 2018.arxiv.org/abs/1711.11216
  49. [49]Q. Liu and D. Wang. Stein variational gradient descent: A general purpose Bayesian inference algorithm. Advances in Neural Information Processing Systems, 29, 2016.arxiv.org/abs/1608.04471
  50. [50]T. Liu, P. Ghosal, K. Balasubramanian, and N. Pillai. Towards understanding the dynamics of Gaussian-Stein variational gradient descent. Advances in Neural Information Processing Systems, 36:61234–61291, 2023.arxiv.org/abs/2305.14076
  51. [51]Y. Lu, Z. Li, D. He, Z. Sun, B. Dong, T. Qin, L. Wang, and T.-Y. Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. In Second Workshop on Machine Learning and the Physical Sciences (NeurIPS 2019), Vancouver, Canada., 2019.
  52. [52]L. Maier. Information geometry on the ℓ²-simplex via the q-root transform. Geometric Science of Information, 16033, 2026.
  53. [53]L. Maier. Information geometry via the Q-root transform. arXiv preprint arXiv:2603.20081, 2026.arxiv.org/abs/2603.20081
  54. [54]L. Malagò, L. Montrucchio, and G. Pistone. Wasserstein Riemannian geometry of Gaussian densities. Information Geometry, 1(2):137–179, 2018.DOI
  55. [55]C. Maor. Riemannian geometry of diffeomorphism groups, 2022. Lecture notes from a course held at the Hebrew University.
  56. [56]A. Massucco, L. Del Grande, M. Carioni, C. Brune, and C.-B. Schönlieb. Multi-headed transformer architectures as time-dependent Wasserstein gradient flows. arXiv preprint arXiv:2605.18870, 2026.arxiv.org/abs/2605.18870
  57. [57]P. W. Michor and D. Mumford. Vanishing geodesic distance on spaces of submanifolds and diffeomorphisms. Documenta Mathematica, 10:217–245, 2005.arxiv.org/abs/math/0409303
  58. [58]A. Mielke, M. A. Peletier, and D. M. Renger. On the relation between gradient flows and the large-deviation principle, with applications to Markov chains and diffusion. Potential Analysis, 41(4):1293–1327, 2014.
  59. [59]P. Monmarché. Nesterov acceleration for the Wasserstein minimization of displacement-convex free energies. arXiv preprint arXiv:2605.13186, 2026.arxiv.org/abs/2605.13186
  60. [60]Y. Nesterov. A method for solving the convex programming problem with convergence rate \(O(k^{-2})\). Doklady Akademii Nauk SSSR, 269:543, 1983.
  61. [61]Y. Nesterov. Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, 2004.
  62. [62]H. H. N. Nguyen, T. Nguyen, H. Vo, S. Osher, and T. Vo. Improving neural ordinary differential equations with Nesterov’s accelerated gradient method. Advances in Neural Information Processing Systems, 35:7712–7726, 2022.DOI
  63. [63]T. M. Nguyen, R. G. Baraniuk, R. M. Kirby, S. J. Osher, and B. Wang. Momentum transformer: Closing the performance gap between self-attention and its linearization. In Proceedings of Mathematical and Scientific Machine Learning, volume 190 of PMLR, pages 189–204, 2022.arxiv.org/abs/2208.00579
  64. [64]N. Nüsken and D. R. M. Renger. Stein variational gradient descent: Many-particle and long-time asymptotics. Foundations of Data Science, 5(3):286–320, 2023.arxiv.org/abs/2102.12956
  65. [65]F. Otto. The geometry of dissipative evolution equations: the porous medium equation. Communications in Partial Differential Equations, 26(1-2):101–174, 2001.DOI
  66. [66]N. Rux, J. Hertrich, and S. Neumayer. Quasi linear kernel attention with infinite capacity. arXiv preprint arXiv:2609.35349, 2026.arxiv.org/abs/2609.35349
  67. [67]M. E. Sander, P. Ablin, M. Blondel, and G. Peyré. Momentum residual neural networks. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9276–9287. PMLR, 18–24 Jul 2021.arxiv.org/abs/2102.07870
  68. [68]M. E. Sander, P. Ablin, M. Blondel, and G. Peyré. Sinkformers: Transformers with doubly stochastic attention. In International Conference on Artificial Intelligence and Statistics, pages 3515–3530. PMLR, 2022.arxiv.org/abs/2110.11773
  69. [69]A. Schmeding. An introduction to infinite-dimensional differential geometry, volume 202. Cambridge University Press, 2022.
  70. [70]V. Stein and W. Li. Towards understanding accelerated Stein variational gradient flow—analysis of generalized bilinear kernels for Gaussian target distributions. arXiv preprint arXiv:2509.04008, 2025.arxiv.org/abs/2509.04008
  71. [71]V. Stein, W. Li, and G. Steidl. Sympformer: Accelerated attention blocks via inertial dynamics on density manifolds. arXiv preprint arXiv:2603.16535, 2026.arxiv.org/abs/2603.16535
  72. [72]W. Su, S. Boyd, and E. J. Candes. A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights. Journal of Machine Learning Research, 17(153):1–43, 2016.
  73. [73]A. Taghvaei and P. Mehta. Accelerated flow for probability distributions. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6076–6085. PMLR, 09–15 Jun 2019.arxiv.org/abs/1901.03317
  74. [74]A. Takatsu. Wasserstein geometry of Gaussian measures. Osaka Journal of Mathematics, 48(4):1005–1026, 2011.DOI
  75. [75]J. Tang, Q. Li, and W. Gangbo. Nesterov acceleration in optimizing over probability measures. arXiv preprint arXiv:2607.23008, 2026.arxiv.org/abs/2607.23008
  76. [76]G. Teschl. Ordinary Differential Equations and Dynamical Systems, volume 140 of Graduate Studies in Mathematics. American Mathematical Society, Providence, RI, 2012.DOI
  77. [77]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Walach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.arxiv.org/abs/1706.03762
  78. [78]C. Villani. Topics in optimal transportation, volume 58 of Graduate Studies in Mathematics. American Mathematical Society, 2003.DOI
  79. [79]C. Villani. Optimal transport: old and new, volume 338 of Grundlehren der mathematischen Wissenschaften. Springer Berlin, Heidelberg, 1 edition, 2008.
  80. [80]M.-K. von Renesse. An optimal transport view of Schrödinger’s equation. Canadian Mathematical Bulletin, 55(4):858–869, 2012.
  81. [81]J. Vuckovic, A. Baratin, and R. T. d. Combes. A mathematical theory of attention. arXiv preprint arXiv:2007.02876, 2020.arxiv.org/abs/2007.02876
  82. [82]B. Wang, H. Xia, T. Nguyen, and S. Osher. How does momentum benefit deep neural networks architecture design? A few case studies. Research in the Mathematical Sciences, 9(3):57, 2022.arxiv.org/abs/2110.07034
  83. [83]H. Wang, S. Ma, L. Dong, S. Huang, D. Zhang, and F. Wei. Deepnet: Scaling transformers to 1,000 layers. IEEE Trans. Pattern Anal. Mach. Intell., 46(10):6761–6774, Oct. 2024.arxiv.org/abs/2203.00555
  84. [84]Y. Wang and W. Li. Accelerated information gradient flow. Journal of Scientific Computing, 90:1–47, 2022.
  85. [85]A. Wibisono, A. C. Wilson, and M. I. Jordan. A variational perspective on accelerated methods in optimization. Proceedings of the National Academy of Sciences, 113(47):E7351–E7358, 2016.DOI
  86. [86]H. Xia, V. Suliafu, H. Ji, T. Nguyen, A. Bertozzi, S. Osher, and B. Wang. Heavy ball neural ordinary differential equations. In Advances in Neural Information Processing Systems, volume 34, pages 18646–18659, 2021.arxiv.org/abs/2110.04840
  87. [87]L. Yang, K. Lee, R. Nowak, and D. Papailiopoulos. Looped transformers are better at learning learning algorithms. In International Conference on Learning Representations, volume 2024, pages 42195–42214, 2024.arxiv.org/abs/2311.12424
  88. [88]C. Yildiz, M. Heinonen, and H. Lahdesmaki. Ode2vae: Deep generative second order odes with bayesian neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
  89. [89]C. Yun, S. B. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations, 2020.arxiv.org/abs/1912.10077
  90. [90]Y. D. Zhong, T. Zhang, A. Chakraborty, and B. Dey. A neural ODE interpretation of transformer layers. arXiv preprint arXiv:2212.06011, 2022.arxiv.org/abs/2212.06011
  91. [91]A. Zimin, Y. Polyanskiy, and P. Rigollet. Yuriiformer: A suite of Nesterov-accelerated transformers. arXiv preprint arXiv:2601.23236, 2026.arxiv.org/abs/2601.23236

Paper details

vlnc:W_qcgqxn3h
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents