betaFeedback

WorksComputer Science

Two-Clustering Regime of Token Dynamics in Causal Attention

Abstract

We partially resolve Conjecture 1 of [Karagodin et al., Clustering in Causal Attention Masking (NeurIPS 2024)] and extend it to time-dependent parameter regimes. The conjecture predicts that causal attention dynamics converge to two clusters aligned with the eigenvectors associated with the largest eigenvalue λmax⁡λ_{\max} of the value transformation. We prove that this behavior holds when λmax⁡>0λ_{\max}>0 and is generally false when λmax⁡≤0λ_{\max}\le0. The resulting two-cluster regime is common in practice and substantially more challenging than the classical single-cluster setting studied previously. A central contribution of our work is the identification of a rigorous dimension reduction phenomenon: despite evolving in a high-dimensional embedding space, token trajectories collapse onto a two-dimensional structure governed by a small number of dominant directions. Building on this insight, we introduce a new analytical framework–instability within the flow–inspired by instability phenomena in physical systems. This framework enables the analysis of steady-regime behavior in non-monotone, non-autonomous attention dynamics. Our theoretical findings are illustrated through simulations and experimental studies using Group Query Attention (GQA) layers.

Introduction

Transformer architectures [78] underpin modern artificial intelligence, achieving strong performance in natural language processing [18, 17, 6, 8, 57, 65], computer vision [21, 66, 62, 47, 1], general machine learning [71, 38, 54, 45], and reinforcement learning [34, 12]. Their success is driven by large-scale pretraining, enabling effective transfer to downstream tasks [64, 19, 63, 74]. By replacing recurrence and convolution with self-attention, Transformers model global token interactions and capture long-range dependencies, yielding rich contextual representations [46, 79, 16, 73, 52, 55, 60]. As a result, Transformers form the core architecture of large language models. Despite their empirical success, a rigorous theoretical understanding of their internal representations remains limited, an issue that becomes increasingly critical as these models continue to scale.

Transformers can be interpreted through a particle-based perspective, where individual tokens are viewed as interacting entities. Adopting this viewpoint, the works of [28, 29] investigate the continuous-time limit of Transformer models, reformulating their behavior as a dynamical system. Within this framework, the authors demonstrate that token representations evolve toward clustered configurations associated with specific attractors determined by their initial conditions, providing a theoretical explanation for the contextual nature of Transformer representations. Further extending this line of analysis, [27] shows that the self-attention dynamics can exhibit metastability, with tokens remaining confined near multi-cluster configurations over exponentially long time horizons. Complementary results are obtained in [7], where a hardmax, attention-only Transformer is examined in a discrete-time setting using similar analytical techniques. Furthermore, the authors in [61] analyze the flow of tokens across attention layers and investigate how different forms of positional encoding - specifically absolute and rotary - affect these dynamical regimes.

Recent work on token dynamics has been extended to causal attention to better reflect its central role in sequential generation. In this autoregressive setting, each token is allowed to attend only to past tokens, enforcing temporal consistency by masking future information during attention computation. This restriction distinguishes causal attention from full attention, which operates on complete sequences and is therefore better suited to tasks such as machine translation. Because of its ability to support step-by-step generation, causal attention is widely used in large-scale generative models, including natural language and image generation systems such as DALL-E [66], VQGAN [25], and Parti [85], as well as multimodal foundation models like Chameleon [72]. More generally, attention mechanisms that limit interactions to subsets of tokens have enabled recent scaling advances, leading to models such as MUSE [11] and AlphaFold 3 [3]. Unlike full attention, the dynamics induced by causal attention cannot be characterized as a mean-field gradient flow as in [27, 29, 28], necessitating different analytical tools.

Background – Causal Attention Dynamics and a Two-Clustering Conjecture

A full Transformer layer typically consists of multi-head self-attention, a position-wise feed-forward network (FFN), normalization layers, and residual connections [78]. Our goal is not to reproduce every component of the Transformer architecture, but rather to clarify and formalize the key mathematical structure that governs its behavior. To this end, we simplify the architecture to only the components relevant for our analysis: a single-head self-attention layer with normalization and a residual connection, omitting the FFN block.

Let the input sequence be represented by token embeddings X=[x1,…,xL]∈RL×dX = [x_1, \ldots, x_L] \in\mathbb{R}^{L \times d}, where LL is the number of tokens and dd is the embedding dimension. With parameter matrices Q,K,V∈RD×DQ, K, V \in\mathbb{R}^{D \times D} used to form queries, keys and values, the attention operator is given by

Attn⁡(X)i=∑j=1L(e⟨Qxi,Kxj⟩∑k=1Le⟨Qxi,Kxk⟩)Vxj,\operatorname{Attn}(X)_i=\sum_{j=1}^{L}\left(\frac{e^{\langle Qx_i,Kx_j\rangle}}{\sum_{k=1}^{L}e^{\langle Qx_i,Kx_k\rangle}}\right)Vx_j,

for i=1,…,Li=1,\ldots,L.

Following [87], we add an RMS normalization layer and a residual connection. Denoting by XtX^t the activations at layer tt, the update to the next layer is

Xt+1=Xt+RMSNorm⁡(Attn⁡(Xt)).(1)X^{t+1}=X^t+\operatorname{RMSNorm}(\operatorname{Attn}(X^t)). \tag*{(1)}

The update rule (1) can be interpreted as a finite-difference approximation of a time derivative, with Xt+1−XtX^{t+1}-X^t playing the role of the increment between consecutive layers, and the RMS normalization playing the role of keeping the token representations lie on the standard unit sphere SD−1S^{D-1}. Hence, the sequence {Xt}t\{X^t\}_t can be regarded as a time-discretization of an underlying continuous-time flow which is given by the dynamical system

∂txi(t)=Pxi(t)(∑j=1Laij(t)V(t)xj(t)),\partial_t x_i(t)=\mathbf{P}_{x_i(t)}\left(\sum_{j=1}^{L}a_{ij}(t)V(t)x_j(t)\right),

with

aij(t)=e⟨Q(t)xi(t),K(t)xj(t)⟩∑k=1Le⟨Q(t)xi(t),K(t)xk(t)⟩,a_{ij}(t)=\frac{e^{\langle Q(t)x_i(t),K(t)x_j(t)\rangle}}{\sum_{k=1}^{L}e^{\langle Q(t)x_i(t),K(t)x_k(t)\rangle}},

and the operator Px(y):=y−⟨x,y⟩x/∥x∥2\mathbf{P}_x(y):=y-\langle x,y\rangle x/\|x\|^2 is the projector that ensures xk(t)x_k(t) remains on the sphere. Finally, by imposing a causal masking constraint, so that each token attends only to its predecessors, we obtain the following causal self-attention equation that we analyze in this work:

∂txi(t)=Pxi(t)(∑j=1iaij(t)V(t)xj(t)),(2)\partial_t x_i(t)=\mathbf{P}_{x_i(t)}\left(\sum_{j=1}^{i}a_{ij}(t)V(t)x_j(t)\right), \tag*{(2)}

with

aij(t)=e⟨Q(t)xi(t),K(t)xj(t)⟩∑k=1ie⟨Q(t)xi(t),K(t)xk(t)⟩,a_{ij}(t)=\frac{e^{\langle Q(t)x_i(t),K(t)x_j(t)\rangle}}{\sum_{k=1}^{i}e^{\langle Q(t)x_i(t),K(t)x_k(t)\rangle}},

for i,j=1,…,Li,j=1,\ldots,L.

The authors in [37] predicted that the eigenvectors of the value matrix plays a central role in determining the qualitative clustering behavior of the causal attention dynamics. In the time-independent setting where QQ, KK, and VV are fixed, the authors in [37] further conjectured the following.

Conjecture 1.1 (Two-clustering configuration of the causal attention dynamics). Assume that QQ, KK, VV are arbitrary time-independent matrices and that VV has dd distinct positive eigenvalues. Let λmax⁡\lambda_{\max} denote the largest eigenvalue, and let ξmax⁡\xi_{\max} be a corresponding unit eigenvector, i.e., ∣ξmax⁡∣=1|\xi_{\max}|=1. Then, for almost every initial condition

X(0)=(x1(0),…,xL(0))∈(Sd−1)L,X(0)=(x_1(0),\ldots,x_L(0))\in(S^{d-1})^L,

the causal attention dynamics (2) converges to a two-cluster configuration:

lim⁡t→+∞xi(t)∈{−ξmax⁡,ξmax⁡},\lim_{t\to+\infty}x_i(t)\in\{-\xi_{\max},\xi_{\max}\},

for all $i=1,\ldots,L.

This two-cluster behavior has been observed empirically for causal attention in [37] and for full attention in [61]. To the best of our knowledge, however, a rigorous proof of this conjecture is not currently available in the literature.

Our main contributions

In this paper, we prove Conjecture 1.1 under a mild condition and further generalize it to settings with time-dependent parameters Q(t)Q(t), K(t)K(t) and V(t)V(t). Our rigorous theoretical analysis reveals several dynamical mechanisms that govern when and how two-cluster configurations emerge, including hierarchical suppression, intrinsic dimensional reduction, and recurrence to the dominant directions ±ξmax⁡\pm\xi_{\max}.

  • Recurrence to the dominant direction ξmax⁡\xi_{\max}. Once the first L−1L-1 tokens stabilize, the last token is forced to repeatedly re-align with ξmax⁡\xi_{\max} along arbitrarily large depths (Theorem 4.2). This recurrent re-alignment creates a “déjà vu” effect: past dominant information is not permanently forgotten, but reappears again and again in the infinite–depth limit.

  • Hierarchical suppression mechanism. We introduce the error-to-transverse ratio κℓ(t)\kappa_{\ell}(t) (see definition in Eq. (6)), a simple but crucial quantity that governs a striking hierarchical suppression: depending on its size, the system either collapses entire blocks of coordinates or locks onto exactly one surviving direction (see Theorem 2.2). The error-to-transverse ratio exactly predicts when a token will suppress all but its most favorable direction.

  • Dimensional reduction. The token dynamics concentrates on a low-dimensional manifold, revealing dimensionality reduction within deep sequence models. Under mild conditions, every nontrivial limiting configuration lies in a two-dimensional coordinate plane and satisfies a single scalar algebraic constraint (Theorem 2.3). This rigidity phenomenon appears to be new and, to our knowledge, has not been rigorously analyzed for attention dynamics in the literature.

Existing analyses of the causal-attention ODE can be categorized by their assumptions on the parameter matrices (V,Q,K)(V,Q,K) and initial data. A summary of the theoretical landscape and the key analytical ideas is provided in Table 1. Our theoretical findings are illustrated through simulations and experimental studies using Group Query Attention (GQA) layers.

WorkParametersInitial DataKey TechniquesTheoretical Results
[37]VV constant Q,KQ, K generalArbitrary on Sd−1\mathbb{S}^{d-1}Explicit x1(t)x_1(t) formula; Schur complement theoremConvergence of x1(t)x_1(t) to ±ξmax⁡\pm\xi_{\max} Full convergence if V=IdV=I_d
[2]V(t)V(t) symmetric Q(t),K(t)Q(t), K(t) generalSingle hemisphereLyapunov function; Input-to-state stabilityGlobal convergence to single cluster ξmax⁡\xi_{\max}
OursV(t)V(t) symmetric Q(t),K(t)Q(t), K(t) generalArbitrary on Sd−1\mathbb{S}^{d-1}Instability analysis; Hierarchy dim. deduction Error-to-traverse ratioGlobal convergence to two clusters ±ξmax⁡\pm\xi_{\max}; Theoretical justification of dimensional deduction

Table 1. Comparison of theoretical analyses for causal-attention dynamics. Our work resolves the two-cluster regime for diagonal V(t)V(t) without hemisphere constraints and provides the first theoretical justification for the dimensional deduction phenomenon conjectured from experiments.

Difficulties from previous approaches. The two-cluster regime with time-varying depth is far more challenging than the classical single-cluster setting, which was the primary focus of the analyses in [37] and [2]. The dynamics no longer admit a monotone Lyapunov functional, and thus every existing technique in the literature – including hemisphere-based Lyapunov methods [2] and Schub-type genericity arguments [37] – completely fails in this regime.

Our new approach. Our key contribution is a new mathematical tool that we call the instability-within-the-flow, inspired by stability theory in physics. The evolution of a token under a time-dependent value matrix V(t)V(t) resembles a small ball rolling on a hill whose summit r(t)r(t) itself moves over time. As the “hilltop” r(t)r(t) drifts, even slowly, the stable and unstable regions experienced by the token shift accordingly, producing a much richer and more delicate dynamical landscape. A rapidly moving hilltop can trap the flow in an interior region (leading to interior clusters), while a stationary hill forces the trajectory toward one of two extremal states {ξmax⁡,−ξmax⁡}\{\xi_{\max},-\xi_{\max}\}. To our knowledge, this structure has not appeared in previous work, and we believe it offers a promising mathematical tool for understanding directional biases and cluster formation in deep sequence models. The full framework is developed in Sections 3 in the main text and in Section E in appendix.

Organization. Section 2 analyzes hierarchical suppression and high-dimensional dimension-reduction mechanisms. Section 3 visualizes causal attention dynamics as the motion of a small ball on a hill. Section 4 proves the two-cluster behavior of tokens. Section 5 validates the theoretical findings in practical Transformers with GQA layers.

Hierarchical Suppression Mechanism and Dimension Reduction

Our first main result is to uncover a high-dimensional reduction mechanism in which the dynamics collapse to exactly two coordinates for arbitrary matrices Q(t)Q(t) and K(t)K(t). Although the dynamics formally reside in a high-dimensional space, the long-time behavior is effectively governed by only a handful of intrinsic degrees of freedom.

The study of the high-dimensional reduction mechanism based on the dynamical system (2) without any additional assumption on the model parameters Q(t)Q(t), K(t)K(t), and V(t)V(t) is challenging. To make the proofs feasible while still maintaining the generality practicality of our theoretical findings, we will use the following assumption during this section (see also in [2]).

Assumption 2.1 (Symmetric and time-invariant eigenspace value matrix). The value matrix V(t)V(t) is a symmetric matrix having dd different eigenvalues and time-invariant eigenspaces. Furthermore, by a suitable linear change of variables, we can write without loss of generality that:

V(t)=diag⁡{λ1(t),λ2(t),…,λd(t)},(3)V(t)=\operatorname{diag}\{\lambda_1(t),\lambda_2(t),\ldots,\lambda_d(t)\}, \tag*{(3)}

so that the eigen-basis coincides with the canonical orthogonal basis {e⃗1,…,e⃗d}\{\vec{e}_1,\ldots,\vec{e}_d\} and

λ1(t)>λ2(t)>⋯>λd(t)for all t≥0.\lambda_1(t)>\lambda_2(t)>\cdots>\lambda_d(t)\quad\text{for all }t\ge0.

Under Assumption 2.1, the largest eigenvalue e⃗1=ξmax⁡\vec{e}_1=\xi_{\max} will be the leading direction selected by the dynamics. Let us fix a solution x(t)=(x1(t),…,xL(t))∈C∞([0,+∞),(Sd−1)L)\mathbf{x}(t)=(x_1(t),\ldots,x_L(t))\in C^\infty([0,+\infty),(\mathbb{S}^{d-1})^L) of the dynamical system (2), subject to the initial condition x(0)∈(Sd−1)L\mathbf{x}(0)\in(\mathbb{S}^{d-1})^L. We will derive our theorems by induction on the number LL of tokens. In particular, we define the error function

εL−1(t)=∑j=1L−1∥xj(t)−sje⃗1∥,\varepsilon_{L-1}(t)=\sum_{j=1}^{L-1}\left\lVert x_j(t)-s_j\vec{e}_1\right\rVert,

where ∥⋅∥\lVert\cdot\rVert denotes the Euclidean norm. Our inductive hypothesis for the first L−1L-1 tokens is

εL−1(t)⟶0as t→∞,(4)\varepsilon_{L-1}(t)\longrightarrow0\qquad\text{as }t\to\infty, \tag*{(4)}

References

  1. [1]Laziz U. Abdullaev, Maksim Tkachenko, and Tan M. Nguyen. Revisiting transformers with insights from image filtering and boosting. SIAM Journal on Imaging Sciences, 2025.arxiv.org/abs/2506.10371
  2. [2]Álvaro Rodríguez Abella, João Pedro Silvestre, and Paulo Tabuada. Consensus is all you get: The role of attention in transformers. In Forty-second International Conference on Machine Learning, 2025.
  3. [3]Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, 630(8016):493–500, 2024.DOI
  4. [4]Juan A Acebrón, Luis L Bonilla, Conrad J Pérez Vicente, Félix Ritort, and Renato Spigler. The kuramoto model: A simple paradigm for synchronization phenomena. Reviews of modern physics, 77(1):137–185, 2005.DOI
  5. [5]Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, Singapore, December 2023. Association for Computational Linguistics.
  6. [6]Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3159–3166, 2019.arxiv.org/abs/1808.04444
  7. [7]Albert Alcalde, Giovanni Fantuzzi, and Enrique Zuazua. Clustering in pure-attention hardmax transformers and its role in sentiment analysis. arXiv preprint arXiv:2407.01602, 2024.DOI
  8. [8]Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference on Learning Representations, 2019.arxiv.org/abs/1809.10853
  9. [9]Han Bao, Ryuichiro Hataya, and Ryo Karakida. Self-attention networks localize when qk-eigenspectrum concentrates. arXiv preprint arXiv:2402.02098, 2024.arxiv.org/abs/2402.02098
  10. [10]Etienne Boursier and Claire Boyer. Softmax as linear attention in the large-prompt regime: a measure-based perspective, 2025.arxiv.org/abs/2512.11784
  11. [11]Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. MUSE: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023.arxiv.org/abs/2301.00704
  12. [12]Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021.arxiv.org/abs/2106.01345
  13. [13]Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1806.07366
  14. [14]Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12020–12030, 2022.arxiv.org/abs/2203.06345
  15. [15]Jingpu Cheng, Qianxiao Li, Ting Lin, and Zuowei Shen. Interpolation, approximation and controllability of deep neural networks. arXiv preprint arXiv:2309.06015, 2023.arxiv.org/abs/2309.06015
  16. [16]Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, Doha, Qatar, October 2014. Association for Computational Linguistics.arxiv.org/abs/1406.1078
  17. [17]Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Anna Korhonen, David Traum, and Luís Márquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy, July 2019. Association for Computational Linguistics.arxiv.org/abs/1901.02860
  18. [18]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019.arxiv.org/abs/1807.03819
  19. [19]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.arxiv.org/abs/1810.04805
  20. [20]Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793–2803. PMLR, 2021.arxiv.org/abs/2103.03404
  21. [21]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
  22. [22]Gbetondji JS Dovonon, Michael M Bronstein, and Matt J Kusner. Setting the record straight on transformer oversmoothing. arXiv preprint arXiv:2401.04301, 2024.arxiv.org/abs/2401.04301
  23. [23]Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407, 2024.arxiv.org/abs/2407.21783
  24. [24]Subhabrata Dutta, Tanya Gautam, Soumen Chakrabarti, and Tanmoy Chakraborty. Redesigning the transformer architecture with insights from multi-particle dynamical systems. Advances in Neural Information Processing Systems, 34:5531–5544, 2021.arxiv.org/abs/2109.15142
  25. [25]Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021.arxiv.org/abs/2012.09841
  26. [26]Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. Rank diminishing in deep neural networks. Advances in Neural Information Processing Systems, 35:33054–33065, 2022.arxiv.org/abs/2206.06072
  27. [27]Borjan Geshkovski, Hugo Koubbi, Yury Polyanskiy, and Philippe Rigollet. Dynamic metastability in the self-attention model. arXiv preprint arXiv:2410.06833, 2024.arxiv.org/abs/2410.06833
  28. [28]Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspective on transformers. arXiv preprint arXiv:2312.10794, 2023.arxiv.org/abs/2312.10794
  29. [29]Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics. Advances in Neural Information Processing Systems, 36, 2024.DOI
  30. [30]Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning perspective on oversmoothing and beyond. arXiv preprint arXiv:2303.06562, 2023.arxiv.org/abs/2303.06562
  31. [31]Seung-Yeal Ha, Jeongho Kim, Jinyeong Park, and Xiongtiao Zhang. Complete cluster predictability of the cucker–smale flocking model on the real line. Archive for Rational Mechanics and Analysis, 231:319–365, 2019.DOI
  32. [32]Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34(1):014004, 2017.DOI
  33. [33]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
  34. [34]Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273–1286, 2021.arxiv.org/abs/2106.02039
  35. [35]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.arxiv.org/abs/2310.06825
  36. [36]Amir Joudaki, Hadi Daneshmand, and Francis Bach. On the impact of activation and normalization in obtaining isometric embeddings at initialization. Advances in Neural Information Processing Systems, 36:39855–39875, 2023.arxiv.org/abs/2305.18399
  37. [37]Nikita Karagodin, Yury Polyanskiy, and Philippe Rigollet. Clustering in causal attention masking. Advances in Neural Information Processing Systems, 37:115652–115681, 2024.arxiv.org/abs/2411.04990
  38. [38]Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1–41, 2022.DOI
  39. [39]Samory Kpotufe and Abdeslam Boulariais. Gradient weights help nonparametric regressors. In Advances in Neural Information Processing Systems, volume 25, 2012.
  40. [40]Ulrich Krause et al. A discrete nonlinear and non-autonomous model of consensus formation. Communications in difference equations, 2000:227–236, 2000.
  41. [41]Yoshiki Kuramoto. Self-entrainment of a population of coupled non-linear oscillators. In International Symposium on Mathematical Problems in Theoretical Physics: January 23–29, 1975, Kyoto University, Kyoto/Japan, pages 420–422. Springer, 1975.
  42. [42]Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436–444, 2015.DOI
  43. [43]Qianxiao Li, Ting Lin, and Zuowei Shen. Deep learning via dynamical systems: An approximation perspective. Journal of the European Mathematical Society, 25(5):1671–1709, 2022.arxiv.org/abs/1912.10382
  44. [44]Hongzhou Lin and Stefanie Jegelka. Resnet with one-neuron hidden layers is a universal approximator. Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1806.10909
  45. [45]Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. AI open, 3:111–132, 2022.arxiv.org/abs/2106.04554
  46. [46]Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In International Conference on Learning Representations, 2017.arxiv.org/abs/1703.03130
  47. [47]Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.arxiv.org/abs/2103.14030
  48. [48]Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv:1906.02762, 2019.arxiv.org/abs/1906.02762
  49. [49]Sebastien Motsch and Eitan Tadmor. Heterophilious dynamics enhances consensus. SIAM review, 56(4):577–621, 2014.
  50. [50]Ho Huu Nghia Nguyen, Tan Nguyen, Huyen Vo, Stanley Osher, and Thieu Vo. Improving neural ordinary differential equations with nesterov’s accelerated gradient method. Advances in Neural Information Processing Systems, 35:7712–7726, 2022.DOI
  51. [51]Tam Nguyen, Tan Nguyen, and Richard Baraniuk. Mitigating over-smoothing in transformers via regularized nonlocal functionals. Advances in Neural Information Processing Systems, 36:80233–80256, 2023.arxiv.org/abs/2312.00751
  52. [52]Tam Minh Nguyen, Tan Minh Nguyen, Dung DD Le, Duy Khuong Nguyen, Viet-Anh Tran, Richard Baraniuk, Nhat Ho, and Stanley Osher. Improving transformers with probabilistic attention keys. In International Conference on Machine Learning, pages 16595–16621. PMLR, 2022.
  53. [53]Tam Minh Nguyen, César A Uribe, Tan Minh Nguyen, and Richard Baraniuk. Pidformer: Transformer meets control theory. In Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2402.15989
  54. [54]Tan Nguyen, Minh Pham, Tam Nguyen, Khai Nguyen, Stanley Osher, and Nhat Ho. Fourier-former: Transformer meets generalized fourier integral theorem. Advances in Neural Information Processing Systems, 35:29319–29335, 2022.DOI
  55. [55]Tam Minh Nguyen, Tam Minh Nguyen, Hai Ngoc Do, Khai Nguyen, Vishwanath Saragadam, Minh Pham, Nguyen Duy Khuong, Nhat Ho, and Stanley Osher. Improving transformer with an admixture of attention heads. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.DOI
  56. [56]Tuan Nguyen, Hirotada Honda, Takashi Sano, Vinh Nguyen, Shugo Nakamura, and Tan Minh Nguyen. From coupled oscillators to graph neural networks: Reducing over-smoothing via a kuramoto model-based approach. In International Conference on Artificial Intelligence and Statistics, pages 2710–2718. PMLR, 2024.arxiv.org/abs/2311.03260
  57. [57]Stefan K Nielsen, Laziz U Abdullaev, Rachel S Teo, and Tan M Nguyen. Elliptical attention. Advances in Neural Information Processing Systems, 37:109748–109789, 2024.arxiv.org/abs/2406.13770
  58. [58]Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27198–27211, 2022.arxiv.org/abs/2206.03126
  59. [59]Lorenzo Noci, Chuning Li, Mufan Li, Bobby He, Thomas Hofmann, Chris J Maddison, and Dan Roy. The shaped transformer: Attention models in the infinite depth-and-width limit. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2306.17759
  60. [60]Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249–2255, Austin, Texas, November 2016. Association for Computational Linguistics.arxiv.org/abs/1606.01933
  61. [61]Duy-Tung Pham, An The Nguyen, Viet-Hoang Tran, Nhan-Phu Chung, Xin T Tong, Tan M Nguyen, and Thieu N Vo. Dynamical properties of tokens in self-attention and effects of positional encoding. Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2512.03058
  62. [62]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
  63. [63]Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
  64. [64]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  65. [65]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.arxiv.org/abs/1910.10683
  66. [66]Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821–8831. Pmlr, 2021.arxiv.org/abs/2102.12092
  67. [67]Domenec Ruiz-Balet and Enrique Zuazua. Neural ode control for classification, approximation, and transport. SIAM Review, 65(3):735–773, 2023.
  68. [68]Michael Scholkemper, Xinyi Wu, Ali Jadbabaie, and Michael T Schaub. Residual connections and normalization can probably prevent oversmoothing in gnns. arXiv preprint arXiv:2406.02997, 2024.arxiv.org/abs/2406.02997
  69. [69]Bingjing Song, Boran Han, Shuai Zhang, Jie Ding, and Mingyi Hong. Unraveling the gradient descent dynamics of transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.arxiv.org/abs/2411.07538
  70. [70]Paulo Tabuada and Bahman Gharesifard. Universal approximation power of deep residual neural networks through the lens of control. IEEE Transactions on Automatic Control, 68(5):2715–2728, 2022.DOI
  71. [71]Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1–28, 2022.
  72. [72]Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024.arxiv.org/abs/2405.09818
  73. [73]Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy, July 2019. Association for Computational Linguistics.arxiv.org/abs/1905.05950
  74. [74]Rachel Teo and Tan Nguyen. Molex: Mixture of layer experts for fine-tuning with sparse upcycling. In International Conference on Learning Representations, volume 2025, pages 70264–70289, 2025.
  75. [75]Rachel S Teo and Tan M Nguyen. Momentumsmoe: Integrating momentum into sparse mixture of experts. Advances in Neural Information Processing Systems, 37:28965–29000, 2024.arxiv.org/abs/2410.14574
  76. [76]Rachel S Teo and Tan M Nguyen. Unveiling the hidden structure of self-attention via kernel principal component analysis. Advances in Neural Information Processing Systems, 37:101393–101427, 2024.arxiv.org/abs/2406.13762
  77. [77]Matthew Thorpe, Tan Minh Nguyen, Hedi Xia, Thomas Strohmer, Andrea Bertozzi, Stanley Osher, and Bao Wang. Grand++: Graph neural diffusion with a source term. In International Conference on Learning Representations, 2022.
  78. [78]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03762
  79. [79]Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. In Tal Linzen, Gregor Chrupała, Yonatan Belinkov, and Dieuwke Hupkes, editors, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63–76, Florence, Italy, August 2019. Association for Computational Linguistics.arxiv.org/abs/1906.04284
  80. [80]Thieu N Vo, Tung D Pham, Xin T Tong, and Tan Minh Nguyen. Demystifying the token dynamics of deep selective state space models. International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.03292
  81. [81]Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. LoRA-pro: Are low-rank adapters properly optimized? In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2407.18242
  82. [82]Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics, June 2018.arxiv.org/abs/1704.05426
  83. [83]Xinyi Wu, Amir Aj orlou, Zihui Wu, and Ali Jadbabaie. Demystifying oversmoothing in attention-based graph neural networks. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2305.16102
  84. [84]Hedi Xia, Vai Suliafu, Hangjie Ji, Tan Nguyen, Andrea Bertozzi, Stanley Osher, and Bao Wang. Heavy ball neural ordinary differential equations. Advances in Neural Information Processing Systems, 34:18646–18659, 2021.arxiv.org/abs/2110.04840
  85. [85]Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunj an Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfey Yang, Burak Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022.arxiv.org/abs/2206.10789
  86. [86]Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pages 40770–40803. PMLR, 2023.arxiv.org/abs/2303.06296
  87. [87]Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1910.07467
  88. [88]Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural odes and invertible residual networks. In International Conference on Machine Learning, pages 11086–11095. PMLR, 2020.arxiv.org/abs/1907.12998
  89. [89]Victor Zhong, Caiming Xiong, and Richard Socher. Seq2Sql: Generating structured queries from natural language using reinforcement learning, 2017.arxiv.org/abs/1709.00103
  90. [90]Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, Fares Obied, and Zhenzhong Lan. Value residual learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28341–28356, Vienna, Austria, July 2025. Association for Computational Linguistics.arxiv.org/abs/2410.17897

Paper details

vlnc:W_ku92xy5e
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents