WorksMathematics
Mean-field Token Steering
Abstract
We consider stochastic token steering through a deep stack of transformer layers as a mean-field control problem for an interacting particle system on the unit sphere, where interactions arise from the attention mechanism. In the mean-field limit, the token population evolves via a controlled integro-partial differential equation where the feedback control acts as a corrective vector field guiding the population toward an aggregate configuration of tokens in the representation space. We deduce the minimum effort control to accomplish this task over a fixed time horizon interpreted as normalized layer depth. We show that the resulting controlled dynamics for the token population can be understood as a mean-field Schrödinger bridge adapted to the sphere geometry. Building on this, we propose a novel generalization of the dynamic Sinkhorn recursion to compute the consistent token evolution. Numerical experiments confirm that the proposed approach achieves exact steering of the token population.
Introduction
The transformer architecture [57] and the self-attention mechanism [6, 44] within it are now widely used in large language models (LLMs). An important reason behind this wide usage and the significant impact made thereby lies in their ability to capture long-range dependencies in data, such as in prompts with a large number of tokens. Notwithstanding the empirical success, the mathematical understanding of how representations evolve through a deep stack of transformer layers via interactions among the tokens is only beginning to emerge now. Still, there remains a significant theoretical gap in understanding what kind of representations can emerge through such evolutions.
One promising approach toward this understanding is to index the layers by time and to consider the evolution of interacting tokens in the large-depth limit as a continuous-time dynamical system. Then, the token ensemble evolves as a system of interacting particles, where a particle represents an individual token embedding [22, 50, 30] (Lu et al. 2019). Mathematically, this evolution is described by a system of coupled ordinary or stochastic differential equations, depending on whether the evolution is deterministic or noisy. Of particular interest is the regime of a large number of tokens (), referred to as the mean-field limit, in which case the dynamics are governed by a nonlinear integro-partial differential equation (PDE) that describes the evolution of the token distribution over the unit sphere , where is the embedding dimension. The spherical geometry arises from the layer normalization. This integro-PDE is parameterized by the query, key, and value matrices , , indexed by layer or time index , and the inverse temperature in the self-attention mechanism. Understanding the mean-field limit is of the essence in practice as the context lengths and depths in LLMs continue to increase. For example, both Claude Opus 5.5 and Gemini frontier models feature 1-million-token context windows, while GPT-4o features a 128k-token context window.
Previous studies on the aforementioned integro-PDE found that the token dynamics can lead to the emergence of a single cluster in the long run [29, 30], and dynamic meta-stability with partial clustering at intermediate times [8, 9] (Geshkovski et al. 2024). In practice, the clustering relates to rank or entropy collapse [21, 58] (Girolandino and Goldt 2026), which has been verified empirically as well as via non-mean-field theoretical studies [45, 1] (Cristiciello et al. 2026).
While existing works on mean-field token dynamics have focused on analysis, a natural question is whether it is possible to design an optimal correction of the nonlocal self-attention dynamics to steer the token population toward a desired aggregate configuration. This work shows why and how this is possible.
The optimal correction we derive can be interpreted as additive control of the noisy mean-field interaction between tokens. By standard Lie-Trotter splitting ([32 p. 47]), the corrective drift associated with additive control can be realized through a feedforward layer concatenated to the self-attention layer ([9 Remark 2.1], [35]). To reduce computational cost, it is preferable to achieve such active steering in a fixed time horizon interpreted as normalized layer depth. In this work, we put forth this new perspective of active guidance of the token distribution by shifting focus from mean-field dynamics to mean-field control.
Contributions We formulate and solve the mean-field token steering problem through the lens of stochastic optimal control. Our specific contributions are the following.
We show that the minimum effort control that steers the interacting token ensemble from a given aggregate configuration to another can be understood as a mean-field Schrödinger bridge adapted to the sphere geometry. This establishes a new connection between two lines of literature: mean-field self-attention dynamics and Schrödinger bridges. While Schrödinger bridges and their Riemannian versions are well-known in diffusion-based generative AI ([20, 52, 4]), the variant of the Schrödinger bridge appearing here is far from standard since it involves both nonlocal self-attention as well as the spherical geometry.
Building on the above connection, we derive the mathematical structure of the minimum-effort control and the optimal evolution of the token distribution, in terms of the solution to a system of coupled nonlinear integro-PDE boundary value problems. We emphasize how these results significantly extend the previously known results for Schrödinger bridges.
To numerically solve the derived system of equations for the minimum-effort active token steering, we propose a novel generalization of the dynamic Sinkhorn recursion (also known as the iterative proportional fitting algorithm). We illustrate the proposed algorithm for steering tokens over and .
The assumptions needed in our development are mild. For instance, we work with normalized self-attention instead of its unnormalized simplification ([49, 35]). Also, we only require the matrices , , to be continuous in . In comparison, the mean-field transformer dynamics literature makes stronger assumptions on these matrices with varying degrees of generality. These include e.g., uniformly continuous and bounded and (identity matrix) in Abella et al. [1], time (i.e., depth) invariance as in , , in Geshkovski et al. [29], time-invariant invertible , , in Bruno et al. [9], symmetric in Sander et al. [50]; Bruno et al. [8]; Geshkovski et al. [30]; Burger et al. [10]; the latter is helpful for a Wasserstein-like gradient flow interpretation but is not needed in our setting.
Related works The idea of identifying layers with time and then deriving a continuous-time flow in the large-depth limit is reminiscent of neural ordinary and stochastic differential equations ([14, 56]). The transformer case differs from that non-interacting setting because long-range interactions among tokens give rise to a mean-field integro-PDE for the token distribution. The transformer then acts as a flow map on , the space of probability distributions on .
We note that a mean-field integro-PDE also arises ([47, 17]) in training a feed-forward network with a single hidden layer of infinite width via stochastic gradient descent. Unlike that case, the transformer mean-field dynamics is not induced by training, but instead by the propagation of the distributional state of the tokens.
In this work, we consider noisy nonlocal interactions among tokens: the presence of noise makes the mean-field integro-PDE second-order ([3]), i.e., a McKean-Vlasov-Fokker-Planck PDE ([46, 54]). Prior literature on stochastic transformers has motivated the noise due to random initialization of the value matrix ([25]), or as computational regularization ([36, 35]).
Control in transformers has been used from different viewpoints: as Low Rank Adaptation (LoRA) for the purpose of fine-tuning of pre-trained models ([59]), as positional encoding and prompt prefixes to counteract mode collapse ([34]), and as training over a parametrized family of transformers ([37, 35]). Different from these works, we formulate mean-field token steering as a minimum-effort distributional control problem. In doing so, we make connections with the broader mean-field Schrödinger bridge literature ([5, 60, 23 b]).
Organization In Section 2, we introduce both the uncontrolled and controlled versions of the transformer mean-field dynamics. In Section 3, we present our main results: the problem formulation, mathematical structure of the Schrödinger system that determines the optimal controller for mean-field token steering, and a computational method to solve the same. Numerical experiments are reported in Section 4. Section 5 closes with concluding remarks.
transformer Mean-field Dynamics
To account for layer normalization, we consider token embeddings that evolve on . Following Rigollet [49]; Shalova and Schlichting [51]; Isobe et al. [35], we model their evolution over a continuum of attention layer depth as the Itô stochastic differential equation:
where denotes the embedding (state) of the -th token for , and is the value matrix.
The kernel denotes the unnormalized attention given by the -th token to the -th token. The corresponding normalization . Specifically, the unnormalized attention kernel
where are the query and key matrices at layer , respectively, and is a scaling parameter that represents the inverse temperature. Both are of size: sequence length, so . In (2) and throughout, denotes the Euclidean inner product.
In (1), the attention aggregated by the -th token amounts to a drift in its own dynamics. The operator projects this drift onto the tangent space , i.e.,
for . Furthermore, denotes a Brownian motion on , and denotes noise intensity.
Assumptions We assume that
A1. the matricial trajectories remain continuous over ,
A2. the Brownian motions are independent.
Mean-field limit To describe the evolution of the token embeddings as a population, we consider the empirical measure
where denotes the Dirac mass concentrated at . In the mean-field limit , interactions among the tokens through the attention mechanism take place via averaged effects (Bolley et al. [7]). When the initial token states are sampled independently from a distribution , then by standard propagation-of-chaos arguments (Sznitman [54]; Chaintron and Diez [13]), the empirical (random) measure converges to a deterministic probability distribution in almost sure sense. Throughout, we use the same symbol to denote a probability law in and its probability density function, i.e., .
As a result of propagation of chaos, (1) converges to the following McKean–Vlasov-type dynamics on (Pascucci et al. [48]):
where
and the attention normalization
We view the nonlocal vector field as the attention-induced mean-field velocity for the token embeddings. In the control context, we will treat this as a prior dynamics resulting from self-attention. We proceed by dropping the index for brevity.
Controlled dynamics and mean-field integro-PDE Recent work by Isobe et al. [35] studies the competition between the token clustering effected by the attention kernel and the feed-forward term, viewing the latter as a trainable controller that can steer tokens away from the clustered distribution. The present work builds on this perspective. Specifically, we consider the question of how to design a suitable controller that steers potentially clustering token dynamics from a prescribed initial distribution toward a desired target distribution. Our analysis below explicitly identifies the optimal controller to accomplish the steering task.
To this end, we consider the controlled dynamics
where represents the steering control input. Equivalently, the token distribution satisfies the mean-field integro-PDE
where denotes the spherical divergence, and is the Laplace-Beltrami operator on . We interpret as a corrective velocity field for the mean-field token dynamics.
Schrödinger Bridges over Attention Dynamics: Formulation and Results
Problem Formulation
We propose to cast the controlled mean-field token steering as a Schrödinger bridge problem, where the control design is informed by two probability laws . We think of as given initial and target token distributions.
Our goal is to identify a controlled evolution over that respects the mean-field token distribution dynamics (7), satisfies , and minimizes the total control effort
where denotes the Euclidean norm. We can interpret the spatial integral
as the instantaneous kinetic energy for the token population, and the space-time integral (8) as the total work done due to the population transfer induced by the control over .
Notice that while appears in the token dynamics (6), any component of normal to the tangent space of the sphere increases control energy cost. Thus, we can restrict ourselves to controllers that belong to the tangent space. We then let the set of admissible control trajectories be defined as
where denotes the round metric norm.
Letting
we now state our main problem as follows.
Determine
subject to
In what follows, we present the structure of the solution for the constrained variational problem (11).
Identification of the Schrödinger Bridge
The minimizer for problem (11) comprises of the optimal control and the optimally controlled for . The optimal control policy is understood as a time-varying feedback, i.e., is, in general, non-homogeneous in . This means that at any given time, the optimal corrective velocity experienced by one token embedding is different from another.
The following Theorem 1 shows that the minimizing pair can be recovered by solving a system of coupled nonlinear integro-PDEs with bilinear boundary conditions. The proof for this result is given in the Supplementary Material Section A.
Theorem 1 (Schrödinger System for Mean-field Token-steering). The minimizing tuple for problem (11) is given by
where , and the positive real-valued functions on satisfy
where denotes the inner product induced by the round metric on , and (14a)–(14b) are subject to the multiplicative endpoint conditions
where
Remark 2. The pair is referred to as Schrödinger factors (Caluya and Halder [12]) since they enable a factorization of ; see (15).
The system described in (14) is a mean-field generalization of the standard Schrödinger systems for the classical bridge problems; see Chetrite et al. [16], Section 5), Léonard [41]. It consists of nonlinear integro-PDEs that are coupled via the nonlocal attention transport and the reaction rate . This structure differs from the standard Schrödinger bridge in that (14a)–(14b) then becomes a system of linear advection-diffusion PDEs in , see e.g., Caluya and Halder [12], Theorem 2.
In the standard (i.e., non-mean-field) Schrödinger bridge, the linearity of (14a)–(14b) can be leveraged in computation via dynamic Sinkhorn recursions (Chen et al. [15], Section 8) over the function pair . In that case, the dynamic Sinkhorn (or iterative proportional fitting (De Bortoli et al. [20])) recursion is known to be contractive with respect to Hilbert’s projective metric (Bushell [11]; Georgiou and Pavon [27]).
In contrast, the nonlocal attention structure in , makes (14) nonlinear, and it is not obvious if and how a Sinkhorn-like algorithm can be designed to compute the minimizing pair (12)–(13). Motivated by this, we propose a novel algorithm to obtain self-consistent solutions to (14).
Computational Approach
Building on Theorem 1, we propose a generalized proportional fitting algorithm to compute the solution to the Schrödinger system (14), enabling mean-field token steering. Our computational methodology is detailed in Algorithm 1, which comprises nested fixed-point iterations indexed by and . Specifically, the inner
| Algorithm 1 | Mean-Field Token Steering Algorithm |
| Require: | . |
| 1: | Initialize and . |
| 2: | for |
| 3: | Compute . |
| 4: | for |
| 5: | Compute |
| . | |
| 6: | Integrate |
| \hfill (15) | |
| backward in time from . | |
| 7: | Set |
| . \hfill (16) | |
| 8: | Integrate |
| \hfill (17) | |
| forward in time from . | |
| 9: | Set |
| . \hfill (18) | |
| 10: | if then |
| 11: | break |
| 12: | end if |
| 13: | end for |
| 14: | Update |
| . \hfill (19) | |
| 15: | if then |
| 16: | break |
| 17: | end if |
| 18: | Set and . |
| 19: | end for |
| 20: | return , and . |
Table 1.
iteration index , and the outer iteration index . The initial guesses , , and may be taken as the solution to the corresponding non-interacting Schrödinger bridge problem, obtained by setting the , in (14) to zero.
In essence, the nested structure in Algorithm 1 handles the two sources of nonlinear coupling from the mean-field velocity and the reaction rate separately. Within the outer iteration that is indexed by , the quantity denotes the -th approximation of the marginal flow. This approximation corresponds to mean-field velocity and normalization . The inner iteration that is indexed by entails solving (16)–(19), but with the fixed , and the reaction rate determined from the approximated marginal flow and delayed evaluation of the preceding Schrödinger factors.
Inner iteration With these quantities fixed, the backward equation for in (15) and the forward equation for in (17) are linear Kolmogorov equations. We use these linear equations for time marching, and update the marched terminal and initial values by imposing the prescribed final and initial marginal densities , as in (18) and (16), respectively.
Outer iteration After the inner iteration completes, the outer loop updates according to (19), using a damped evaluation if found necessary for numerical stability. The nested iterations are repeated until the marginal flow and the Schrödinger pair are jointly self-consistent.
Convergence criteria We take our convergence criteria to be with respect to the Hilbert’s projective metric defined for nonnegative functions in the Banach space as
The evaluation is restricted to the common support of the functions being compared.
Numerical Experiments
We illustrate the proposed mean-field token steering framework on two experiments: one for (circle) and the other for (2-sphere in ). In both experiments, we fix the noise strength parameter , inverse temperature , and Algorithm 1 parameters: tol , . In both, we perform a unimodal-to-bimodal steering over with
for , a convex weight vector , and for . In other words, we steer a von Mises–Fisher distribution toward a two-component mixture of von Mises–Fisher distributions. We detail the other parameters and results for the specific experiments below.
S1 Experiment
Setup We use the following parameters for the initial and final distributions: , , , , , , . We use the self-attention parameters: (here ), and randomly generated
In using Algorithm 1 for this experiment, we time-march the solutions for (15) and (17) using the Lie–Trotter splitting and 600 uniform discretizations of the time horizon . To approximate the spatial derivatives on the circle, we used the finite difference with 150 discretizations of . To approximate the integrals in nonlocal terms, we used the periodic quadrature method.
In applying Algorithm 1 to this experiment, we use inner iterations, and outer iterations.
Results For the numerical experiment setup described above, Figure 1 shows four snapshots of the optimally controlled density (solid curve) computed using Algorithm 1, together with (dashed curves). This plot demonstrates that Algorithm 1 effectively steers the mean-field token distribution over the given horizon . On these snapshots, we also superimposed the optimally controlled evolution of 200 token embeddings (hollow circular markers), showing a higher concentration near the modes of , as expected. These token evolutions were computed via Euler–Maruyama simulation of the controlled dynamics (6) with obtained from Algorithm 1 and with the same time discretization as before.

Figure 1. Evolution of optimally controlled token distribution and tokens for the experiment in § 4.1.
Figure 2 shows convergence of Algorithm 1 with respect to the Hilbert metric (20), for this experiment.

Figure 2. Convergence of Algorithm 1 for the experiment in § 4.1.
An additional experiment is reported in the Supplementary Material Section B.
Experiment
Setup. Here, we use the following parameters for the initial and final distributions: , , ,
, is the unit tangent vector at , rad, , . For the self-attention parameters, we use , fixed (here ), and randomly generated
To apply Algorithm 1 for this experiment, we time-march the solutions for (15) and (17) using the Lie-Trotter splitting as before, with 200 uniform discretizations of the time horizon . To approximate the spatial derivatives on , we used the finite difference with discretizations of . To approximate the integrals in nonlocal terms, we used the weighted quadrature method.
In applying Algorithm 1 to this experiment, we use inner iterations, and outer iterations.
Results. For the numerical experiment setup described above, Figure 3 shows four snapshots of the optimally controlled density (solid contours) computed using Algorithm 1, together with (dashed contours), demonstrating the effectiveness of Algorithm 1. To avoid clutter, we suppressed the contour labels. The hollow circular markers in Figure 3 show 160 optimally controlled token embeddings, computed via Euler-Maruyama simulation of (6) with obtained from Algorithm 1 and with the same time discretization as before.

Figure 3. Evolution of optimally controlled token distribution and tokens for the S experiment in § 4.2.
For this experiment, Figure 4 shows convergence of Algorithm 1 with respect to the Hilbert metric (20).

Figure 4. Convergence of Algorithm 1 for the experiment in § 4.2.
Concluding Remarks
In this work, we have formulated the problem of steering self-attention-driven token dynamics in the mean-field limit with minimum control effort. The formulation, based on the theory of Schrödinger bridges, identifies the minimum-effort control as a feedback law that transports the token distribution between prescribed initial and final configurations over a finite transformer depth horizon. Our analysis characterizes the optimal controller via a forward–backward system of nonlinearly coupled integro-PDEs. To numerically solve such a system, we have proposed a generalized iterative proportional fitting algorithm adapted to the transformer mean-field setting.
Our primary contributions are theoretical: proposing the idea of mean-field token steering via control, connecting the formulation with the mean-field Schrödinger bridge in the presence of attention-induced nonlocal drift, designing a variant of dynamic Sinkhorn recursion to solve the same, and demonstrating the feasibility of computation via simple numerical examples. While we empirically demonstrate local convergence, proving guarantees for the same remains a topic of ongoing research.
This work points towards multiple research avenues and connections within the rapidly evolving theory of transformer dynamics. Connecting the present result to recent analyses of representation collapse and meta-stable multi-clustering [29, 9] is the most direct foreseeable research direction, and may provide a means to preserve the representation structure. Our numerical results support that possibility.
Another connection is the recent line of research on activation steering (see, e.g., Li et al. (2023); Lee et al. [38]; Skifstad et al. [53]; Adila et al. [2]) that aims to guide an LLM during prompt context inference toward a desired semantic direction by modifying the model’s internal representation at selected layers. The work here can be viewed as a distributional analogue of activation steering, where steering is achieved at the level of token-embedding populations. Therefore, an interesting direction for future work is to study how the controller obtained herein can be best realized (or approximated) by sparse interventions at finitely many layers, and how this restriction affects steering performance.
More broadly, extensions of the present framework to finite token populations, discrete layer depths, and multi-head attention indeed constitute an important step toward bridging the current theory with practical transformer architectures.
AI use statement
During the preparation of this work, the authors used generative AI (OpenAI, Anthropic) to assist in developing the code for the numerical solvers and in postprocessing of the figures. The authors reviewed, validated, and edited all the outputs and take full responsibility for the final contents, figures, claims, and artifacts presented here.
PROOF OF THEOREM 1
This section provides the proof of Theorem 1. We start by writing the Lagrangian of Problem (11) as
where is a Lagrange multiplier that enforces the dynamical constraint (11b).
Using integration by parts, we write
Next, define
By the divergence theorem ([39 Theorem 16.48]), we have
since is a closed manifold, i.e., . Hence, substituting (S3) into the previous equation yields
recalling that denotes the inner product induced by the round metric on .
By Green’s second identity ([40 Page 51]), we also obtain that
By (S2), (S4), and (S5), the Lagrangian in (S1) can be rewritten as
Optimization over the controller
The first-order optimality condition for is such that the Gâteaux derivative
for every perturbation , with , . This implies
That is, for almost every ,
Optimization over the marginal flow
Let us substitute (S8) into (S6). This yields the following evaluation at the optimizer .
where
We let be a perturbation of the optimal PDF-valued curve , so that
Further, since the endpoint optimal PDFs are fixed as in (13),
Then, the first-order optimality condition with respect to is such that
Therefore, by (S9)–(S11), we have
Indeed,
Notice that
Therefore, (S14) and (S15) lead to
Recall that
Let us set
Hence,
Note that by using (S18) and (S19), we can rewrite the previous equation as
Thus,
Additionally,
where we have used the fact that and and scalars for any fixed .
Using (S20), we can rewrite (S16) as
By exchanging the variables and in the second integral on the right-hand side of (S21), we obtain
By substituting (S22) into (S13), we obtain
Since , we deduce that the quantity in brackets must be spatially independent. That is,
where
for some .
Since depends only on time, we can absorb it into via the transformation
and write the first-order optimality in as the Hamilton–Jacobi–Bellman equation
Therefore, we can summarize the optimality conditions for Problem (11) as follows.
where is as in (S24).
The forward-backward dynamics
Next, we introduce a change of variable
where is a positive function for all . This transformation is known as the Cole–Hopf ([18, 33, 55]) or Fleming logarithmic transformation ([26]). By (S27), Eq. (S26a) can be written as
recovering (14).
By (S27), (S26c), and the identity
we can obtain the dynamics of as in (16).
Introducing a second nonnegative function through the factorization
as in (15), we deduce that satisfies (17). Lastly, by (15), we can obtain (18) and (19).
Combining the preceding optimality conditions and the defined transformations yields the Schrödinger system stated in the theorem.
Additional Experiment
We report an additional mean-field token-steering experiment over that supplements our results in § 4.1. Here, we focus on comparing the uncontrolled versus optimally controlled evolution of the token distribution in the mean-field (i.e., number of tokens ) limit.
Inspired by the experiment reported in Bruno et al. [8], we set to be uniform on slightly perturbed by white noise. It is known that for this , the uncontrolled mean-field dynamics for certain choice of parameters gives rise to a 3 mode meta-stable clustering. To investigate the possibility of countering this trend via control, we set as an equal-weight mixture of two von Mises-Fisher densities on , given by
with , , for , rad, and . In Figure S1, this is shown as “target”. We use the parameters: , , , (here ), and randomly generated

Figure S1. Evolution of uncontrolled versus optimally controlled token distribution for the experiment in Supplementary Section B.
We follow the space-time discretization as in Bruno et al. [8], and in Figure S1, compare the uncontrolled (i.e., with prior mean-field token dynamics) versus controlled (i.e., with corrected mean-field token dynamics) evolution of the token densities over . The controlled evolution is computed using Algorithm 1 proposed here with the same algorithm parameters as in § 4.1. Figure S1 shows that the optimal control accomplishes the uniform to target bimodal token steering, overcoming the natural tendency of forming 3 modes over the same horizon.
References
- [1]Abella, Á. R., Silvestre, J. P., and Tabuada, P. (2025). Consensus is all you get: The role of attention in transformers. In Forty-second International Conference on Machine Learning.
- [2]Adila, D., Cooper, J., Yun, A., Trost, A., and Sala, F. (2026). Weight updates as activation shifts: A principled framework for steering. In Zhang, T., Dudik, M., Jaggi, M., Agarwal, A., Li, S., Schuurmans, D., Zhu, J., Berkenkamp, F., Dong, H., and Bietti, A., editors, Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 565–590. PMLR.arxiv.org/abs/2603.00425
- [3]Agazzi, A., Bruno, G., García, E. M., Saviōzzi, S., and Romito, M. (2026). Stochastic scaling limits and synchronization by noise in deep transformer models. arXiv preprint arXiv:2604.26898.arxiv.org/abs/2604.26898
- [4]Albergo, M., Boffi, N. M., and Vanden-Eijnden, E. (2025). Stochastic interpolants: A unifying frame- work for flows and diffusions. Journal of Machine Learning Research, 26(209):1–80.
- [5]Backhoff, J., Conforti, G., Gentil, I., and Léonard, C. (2020). The mean field Schrödinger problem: ergodic behavior, entropy estimates and functional inequalities. Probability Theory and Related Fields, 178(1):475–530.
- [6]Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.arxiv.org/abs/1409.0473
- [7]Bolley, F., Cañizo, J. A., and Carrillo, J. A. (2011). Stochastic mean-field limit: Non-Lipschitz forces and swarming. Mathematical Models and Methods in Applied Sciences, 21(11):2179–2210.arxiv.org/abs/1009.5166
- [8]Bruno, G., Pasqualotto, F., and Agazzi, A. (2025). Emergence of meta-stable clustering in mean-field transformer models. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2410.23228
- [9]Bruno, G., Pasqualotto, F., and Agazzi, A. (2026). A multiscale analysis of mean-field transformers in the moderate interaction regime. Advances in Neural Information Processing Systems, 38:133305–133341.arxiv.org/abs/2509.25040
- [10]Burger, M., Kabri, S., Korolev, K., Roith, T., and Weigand, L. (2025). Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer normalization. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 383(2298):20240333.arxiv.org/abs/2501.03096
- [11]Bushell, P. J. (1973). Hilbert’s metric and positive contraction mappings in a Banach space. Archive for Rational Mechanics and Analysis, 52(4):330–338.DOI
- [12]Caluya, K. F. and Halder, A. (2021). Wasserstein proximal algorithms for the Schrödinger bridge problem: Density control with nonlinear drift. IEEE Transactions on Automatic Control, 67(3):1163–1178.arxiv.org/abs/1912.01244
- [13]Chaintron, L.-P. and Diez, A. (2022). Propagation of chaos: A review of models, methods and applications. II. applications. Kinetic and Related Models, 15(6):1017–1173.
- [14]Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. (2018). Neural ordinary differential equations. In Advances in Neural Information Processing Systems, volume 31.arxiv.org/abs/1806.07366
- [15]Chen, Y., Georgiou, T. T., and Pavon, M. (2021). Stochastic control laisons: Richard Sinkhorn meets Gaspard Monge on a Schrödinger bridge. Siam Review, 63(2):249–313.DOI
- [16]Chetrite, R., Muratore-Ginanneschi, P., and Schwieger, K. (2021). E. Schrödinger’s 1931 paper “On the Reversal of the Laws of Nature” [“Über die Umkehrung der Naturgesetze”, Sitzungsberichte der preussischen Akademie der Wissenschaften, physikalisch-mathematische Klasse, 8 N9 144–153]. The European Physical Journal H, 46:28.
- [17]Chizat, L. and Bach, F. (2018). On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in neural information processing systems, 31.arxiv.org/abs/1805.09545
- [18]Cole, J. D. (1951). On a quasi-linear parabolic equation occurring in aerodynamics. Quarterly of Applied Mathematics, 9(3):225–236.DOI
- [20]De Bortoli, V., Thornton, J., Heng, J., and Doucet, A. (2021). Diffusion Schrödinger bridge with applications to score-based generative modeling. Advances in neural information processing systems, 34:17695–17709.
- [21]Dong, Y., Cordonnier, J.-B., and Loukas, A. (2021). Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International conference on machine learning, pages 2793–2803. PMLR.arxiv.org/abs/2103.03404
- [22]Dutta, S., Gautam, T., Chakrabarti, S., and Chakraborty, T. (2021). Redesigning the transformer architecture with insights from multi-particle dynamical systems. Advances in Neural Information Processing Systems, 34:5531–5544.arxiv.org/abs/2109.15142
- [23]Eldesoukey, A., Chen, Y., and Halder, A. (2026a). A generalized Sinkhorn algorithm for mean-field Schrödinger bridge. IEEE Control Systems Letters.
- [25]Fedorov, L., Sander, M. E., Elie, R., Marion, P., and Lauriere, M. (2026). Clustering in deep stochastic transformers. In Forty-third International Conference on Machine Learning.arxiv.org/abs/2601.21942
- [26]Fleming, W. H. (2005). Logarithmic transformations and stochastic control. In Advances in Filtering and Optimal Stochastic Control, pages 131–141. Springer.
- [27]Georgiou, T. T. and Pavon, M. (2015). Positive contraction mappings for classical and quantum Schrödinger systems. Journal of Mathematical Physics, 56,3(3).
- [29]Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. (2023). The emergence of clusters in self-attention dynamics. Advances in Neural Information Processing Systems, 36:57026–57037.DOI
- [30]Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. (2025). A mathematical perspective on transformers. Bulletin of the American Mathematical Society, 62(3):427–479.arxiv.org/abs/2312.10794
- [32]Hairer, E., Lubich, C., and Wanner, G. (2006). Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations, volume 31 of Springer Series in Computational Mathematics. Springer, Berlin, Heidelberg, 2nd edition.
- [33]Hopf, E. (1950). The partial differential equation $u_t + u u_x = \mu u_{xx}$. Communications on Pure and Applied Mathematics, 3(3):201–230.
- [34]Imaizumi, M., Koyama, M., Isobe, N., and Hayashi, K. (2026). Anti mode-collapse in mean-field transformer via auxiliary variables. arXiv preprint arXiv:2605.30229.arxiv.org/abs/2605.30229
- [35]Isobe, N., Inoue, D., and Imaizumi, M. (2026). Training-induced escape from token clustering in a mean-field formulation of transformers. arXiv preprint arXiv:2605.07772.arxiv.org/abs/2605.07772
- [36]Jain, N., Chiang, P.-y., Wen, Y., Kirchenbauer, J., Chu, H.-M., Somepalli, G., Bartoldson, B., Kailkhura, B., Schwarzschild, A., Saha, A., Goldblum, M., Geiping, J., and Goldstein, T. (2024). Neftune: Noisy embeddings improve instruction finetuning. In Kim, B., Yue, Y., Chaudhuri, S., Fragkiadaki, K., Khan, M., and Sun, Y., editors, International Conference on Learning Representations, volume 2024, pages 17566–17591.arxiv.org/abs/2310.05914
- [37]Kan, K., Li, X., Zhang, B., Sahai, T., Osher, S., and Katsoulakis, M. (2026). Optimal control for transformer architectures: Enhancing generalization, robustness and efficiency. Advances in Neural Information Processing Systems, 38:648183–64228.arxiv.org/abs/2505.13499
- [38]Lee, B. W., Padhi, I., Natesan Ramamurthy, K., Miehling, E., Dognin, P., Nagireddy, M., and Dhurandhar, A. (2025). Programming refusal with conditional activation steering. In Yue, Y., Garg, A., Peng, N., Sha, F., and Yu, R., editors, International Conference on Learning Representations, volume 2025, pages 90960–90985.arxiv.org/abs/2409.05907
- [39]Lee, J. M. (2012). Introduction to Smooth Manifolds, volume 218 of Graduate Texts in Mathematics. Springer New York, 2 edition.
- [40]Lee, J. M. (2018). Introduction to Riemannian Manifolds, volume 176 of Graduate Texts in Mathematics. Springer International Publishing. Cham, Switzerland, 2nd edition.DOI
- [41]Léonard, C. (2014). A survey of the schrödinger problem and some of its connections with optimal transport. Discrete and Continuous Dynamical Systems, 34(4):1533–1574.DOI
- [44]Luong, M.-T., Pham, H., and Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1412–1421.arxiv.org/abs/1508.04025
- [45]Markdahl, J., Thunberg, J., and Gonçalves, J. (2017). Almost global consensus on the n-sphere. IEEE Transactions on Automatic Control, 63(6):1664–1675.arxiv.org/abs/1609.04885
- [46]McKean Jr, H. P. (1966). A class of Markov processes associated with nonlinear parabolic equations. Proceedings of the National Academy of Sciences, 56(6):1907–1911.pubmed.ncbi.nlm.nih.gov/16591437
- [47]Mei, S., Montanari, A., and Nguyen, P.-M. (2018). A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665–E7671.DOI
- [48]Pascucci, A., Rondelli, A., and Veretennikov, A. Y. (2026). Existence and uniqueness results for strongly degenerate McKean–Vlasov equations with rough coefficients. Stochastic Processes and their Applications, 199:104978.arxiv.org/abs/2409.14451
- [49]Rigollet, P. (2026). The Mean-Field Dynamics of Transformers, pages 389–404.
- [50]Sander, M. E., Ablin, P., Blondel, M., and Peyré, G. (2022). Sinkformers: Transformers with doubly stochastic attention. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 3515–3530. PMLR.arxiv.org/abs/2110.11773
- [51]Shalova, A. and Schlichting, A. (2026). Solutions of stationary Mckean–Vlasov equation on a high-dimensional sphere and other Riemannian manifolds. Advances in Nonlinear Analysis, 15(1):20250141.arxiv.org/abs/2412.14813
- [52]Shi, Y., De Bortoli, V., Campbell, A., and Doucet, A. (2023). Diffusion Schrödinger bridge matching. Advances in neural information processing systems, 36:62183–62223.arxiv.org/abs/2303.16852
- [53]Skifstad, J., Yang, X. A., and Chou, G. (2026). Local linearity of LLMs enables activation steering via model-based linear optimal control. In Zhang, T., Dudik, M., Jaggi, M., Agarwal, A., Li, S., Schuurmans, D., Zhu, J., Berkenkamp, F., Dong, H., and Bietti, A., editors, Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 113885–113919. PMLR.arxiv.org/abs/2604.19018
- [54]Sznitman, A.-S. (2006). Topics in propagation of chaos. In Ecole d’été de probabilités de Saint-Flour XIX—1989, pages 165–251. Springer.
- [55]Teter, A. and Halder, A. (2025). On the Hopf–Cole transform for control-affine Schrödinger bridges. arXiv preprint arXiv:2503.17640.
- [56]Tzen, B. and Raginsky, M. (2019). Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv:1905.09883.arxiv.org/abs/1905.09883
- [57]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.arxiv.org/abs/1706.03762
- [58]Zhai, S., Likhomanenko, T., Littwin, E., Busbridge, D., Ramapuram, J., Zhang, Y., Gu, J., and Susskind, J. M. (2023). Stabilizing transformer training by preventing attention entropy collapse. In International conference on machine learning, pages 40770–40803. PMLR.arxiv.org/abs/2303.06296
- [59]Zhang, C., Jingpu, C., Xu, Y., and Li, Q. (2024). Parameter-efficient fine-tuning with controls. In Forty-first International Conference on Machine Learning.
- [60]Zhang, Z., Wang, Z., Sun, Y., Li, T., and Zhou, P. (2026). Modeling cell dynamics and interactions with unbalanced mean field Schrödinger bridge. Advances in Neural Information Processing Systems, 38:139518–139558.arxiv.org/abs/2505.11197