We partially resolve Conjecture 1 of [Karagodin et al., Clustering in Causal Attention Masking (NeurIPS 2024)] and extend it to time-dependent parameter regimes. The conjecture predicts that causal attention dynamics converge to two clusters aligned with the eigenvectors associated with the largest eigenvalue λmax of the value transformation. We prove that this behavior holds when λmax>0 and is generally false when λmax≤0. The resulting two-cluster regime is common in practice and substantially more challenging than the classical single-cluster setting studied previously. A central contribution of our work is the identification of a rigorous dimension reduction phenomenon: despite evolving in a high-dimensional embedding space, token trajectories collapse onto a two-dimensional structure governed by a small number of dominant directions. Building on this insight, we introduce a new analytical framework–instability within the flow–inspired by instability phenomena in physical systems. This framework enables the analysis of steady-regime behavior in non-monotone, non-autonomous attention dynamics. Our theoretical findings are illustrated through simulations and experimental studies using Group Query Attention (GQA) layers.
Introduction
Transformer architectures [78] underpin modern artificial intelligence, achieving strong performance in natural language processing [18, 17, 6, 8, 57, 65], computer vision [21, 66, 62, 47, 1], general machine learning [71, 38, 54, 45], and reinforcement learning [34, 12]. Their success is driven by large-scale pretraining, enabling effective transfer to downstream tasks [64, 19, 63, 74]. By replacing recurrence and convolution with self-attention, Transformers model global token interactions and capture long-range dependencies, yielding rich contextual representations [46, 79, 16, 73, 52, 55, 60]. As a result, Transformers form the core architecture of large language models. Despite their empirical success, a rigorous theoretical understanding of their internal representations remains limited, an issue that becomes increasingly critical as these models continue to scale.
Transformers can be interpreted through a particle-based perspective, where individual tokens are viewed as interacting entities. Adopting this viewpoint, the works of [28, 29] investigate the continuous-time limit of Transformer models, reformulating their behavior as a dynamical system. Within this framework, the authors demonstrate that token representations evolve toward clustered configurations associated with specific attractors determined by their initial conditions, providing a theoretical explanation for the contextual nature of Transformer representations. Further extending this line of analysis, [27] shows that the self-attention dynamics can exhibit metastability, with tokens remaining confined near multi-cluster configurations over exponentially long time horizons. Complementary results are obtained in [7], where a hardmax, attention-only Transformer is examined in a discrete-time setting using similar analytical techniques. Furthermore, the authors in [61] analyze the flow of tokens across attention layers and investigate how different forms of positional encoding - specifically absolute and rotary - affect these dynamical regimes.
Recent work on token dynamics has been extended to causal attention to better reflect its central role in sequential generation. In this autoregressive setting, each token is allowed to attend only to past tokens, enforcing temporal consistency by masking future information during attention computation. This restriction distinguishes causal attention from full attention, which operates on complete sequences and is therefore better suited to tasks such as machine translation. Because of its ability to support step-by-step generation, causal attention is widely used in large-scale generative models, including natural language and image generation systems such as DALL-E [66], VQGAN [25], and Parti [85], as well as multimodal foundation models like Chameleon [72]. More generally, attention mechanisms that limit interactions to subsets of tokens have enabled recent scaling advances, leading to models such as MUSE [11] and AlphaFold 3 [3]. Unlike full attention, the dynamics induced by causal attention cannot be characterized as a mean-field gradient flow as in [27, 29, 28], necessitating different analytical tools.
Background – Causal Attention Dynamics and a Two-Clustering Conjecture
A full Transformer layer typically consists of multi-head self-attention, a position-wise feed-forward network (FFN), normalization layers, and residual connections [78]. Our goal is not to reproduce every component of the Transformer architecture, but rather to clarify and formalize the key mathematical structure that governs its behavior. To this end, we simplify the architecture to only the components relevant for our analysis: a single-head self-attention layer with normalization and a residual connection, omitting the FFN block.
Let the input sequence be represented by token embeddings X=[x1,…,xL]∈RL×d, where L is the number of tokens and d is the embedding dimension. With parameter matrices Q,K,V∈RD×D used to form queries, keys and values, the attention operator is given by
Following [87], we add an RMS normalization layer and a residual connection. Denoting by Xt the activations at layer t, the update to the next layer is
Xt+1=Xt+RMSNorm(Attn(Xt)).(1)
The update rule (1) can be interpreted as a finite-difference approximation of a time derivative, with Xt+1−Xt playing the role of the increment between consecutive layers, and the RMS normalization playing the role of keeping the token representations lie on the standard unit sphere SD−1. Hence, the sequence {Xt}t can be regarded as a time-discretization of an underlying continuous-time flow which is given by the dynamical system
and the operator Px(y):=y−⟨x,y⟩x/∥x∥2 is the projector that ensures xk(t) remains on the sphere. Finally, by imposing a causal masking constraint, so that each token attends only to its predecessors, we obtain the following causal self-attention equation that we analyze in this work:
The authors in [37] predicted that the eigenvectors of the value matrix plays a central role in determining the qualitative clustering behavior of the causal attention dynamics. In the time-independent setting where Q, K, and V are fixed, the authors in [37] further conjectured the following.
Conjecture 1.1 (Two-clustering configuration of the causal attention dynamics). Assume that Q, K, V are arbitrary time-independent matrices and that V has d distinct positive eigenvalues. Let λmax denote the largest eigenvalue, and let ξmax be a corresponding unit eigenvector, i.e., ∣ξmax∣=1. Then, for almost every initial condition
X(0)=(x1(0),…,xL(0))∈(Sd−1)L,
the causal attention dynamics (2) converges to a two-cluster configuration:
t→+∞limxi(t)∈{−ξmax,ξmax},
for all $i=1,\ldots,L.
This two-cluster behavior has been observed empirically for causal attention in [37] and for full attention in [61]. To the best of our knowledge, however, a rigorous proof of this conjecture is not currently available in the literature.
Our main contributions
In this paper, we prove Conjecture 1.1 under a mild condition and further generalize it to settings with time-dependent parameters Q(t), K(t) and V(t). Our rigorous theoretical analysis reveals several dynamical mechanisms that govern when and how two-cluster configurations emerge, including hierarchical suppression, intrinsic dimensional reduction, and recurrence to the dominant directions ±ξmax.
Recurrence to the dominant directionξmax. Once the first L−1 tokens stabilize, the last token is forced to repeatedly re-align with ξmax along arbitrarily large depths (Theorem 4.2). This recurrent re-alignment creates a “déjà vu” effect: past dominant information is not permanently forgotten, but reappears again and again in the infinite–depth limit.
Hierarchical suppression mechanism. We introduce the error-to-transverse ratioκℓ(t) (see definition in Eq. (6)), a simple but crucial quantity that governs a striking hierarchical suppression: depending on its size, the system either collapses entire blocks of coordinates or locks onto exactly one surviving direction (see Theorem 2.2). The error-to-transverse ratio exactly predicts when a token will suppress all but its most favorable direction.
Dimensional reduction. The token dynamics concentrates on a low-dimensional manifold, revealing dimensionality reduction within deep sequence models. Under mild conditions, every nontrivial limiting configuration lies in a two-dimensional coordinate plane and satisfies a single scalar algebraic constraint (Theorem 2.3). This rigidity phenomenon appears to be new and, to our knowledge, has not been rigorously analyzed for attention dynamics in the literature.
Existing analyses of the causal-attention ODE can be categorized by their assumptions on the parameter matrices (V,Q,K) and initial data. A summary of the theoretical landscape and the key analytical ideas is provided in Table 1. Our theoretical findings are illustrated through simulations and experimental studies using Group Query Attention (GQA) layers.
Instability analysis; Hierarchy dim. deduction Error-to-traverse ratio
Global convergence to two clusters±ξmax; Theoretical justification of dimensional deduction
Table 1. Comparison of theoretical analyses for causal-attention dynamics. Our work resolves the two-cluster regime for diagonal V(t) without hemisphere constraints and provides the first theoretical justification for the dimensional deduction phenomenon conjectured from experiments.
Difficulties from previous approaches. The two-cluster regime with time-varying depth is far more challenging than the classical single-cluster setting, which was the primary focus of the analyses in [37] and [2]. The dynamics no longer admit a monotone Lyapunov functional, and thus every existing technique in the literature – including hemisphere-based Lyapunov methods [2] and Schub-type genericity arguments [37] – completely fails in this regime.
Our new approach. Our key contribution is a new mathematical tool that we call the instability-within-the-flow, inspired by stability theory in physics. The evolution of a token under a time-dependent value matrix V(t) resembles a small ball rolling on a hill whose summit r(t) itself moves over time. As the “hilltop” r(t) drifts, even slowly, the stable and unstable regions experienced by the token shift accordingly, producing a much richer and more delicate dynamical landscape. A rapidly moving hilltop can trap the flow in an interior region (leading to interior clusters), while a stationary hill forces the trajectory toward one of two extremal states {ξmax,−ξmax}. To our knowledge, this structure has not appeared in previous work, and we believe it offers a promising mathematical tool for understanding directional biases and cluster formation in deep sequence models. The full framework is developed in Sections 3 in the main text and in Section E in appendix.
Organization. Section 2 analyzes hierarchical suppression and high-dimensional dimension-reduction mechanisms. Section 3 visualizes causal attention dynamics as the motion of a small ball on a hill. Section 4 proves the two-cluster behavior of tokens. Section 5 validates the theoretical findings in practical Transformers with GQA layers.
Hierarchical Suppression Mechanism and Dimension Reduction
Our first main result is to uncover a high-dimensional reduction mechanism in which the dynamics collapse to exactly two coordinates for arbitrary matrices Q(t) and K(t). Although the dynamics formally reside in a high-dimensional space, the long-time behavior is effectively governed by only a handful of intrinsic degrees of freedom.
The study of the high-dimensional reduction mechanism based on the dynamical system (2) without any additional assumption on the model parameters Q(t), K(t), and V(t) is challenging. To make the proofs feasible while still maintaining the generality practicality of our theoretical findings, we will use the following assumption during this section (see also in [2]).
Assumption 2.1 (Symmetric and time-invariant eigenspace value matrix). The value matrix V(t) is a symmetric matrix having d different eigenvalues and time-invariant eigenspaces. Furthermore, by a suitable linear change of variables, we can write without loss of generality that:
V(t)=diag{λ1(t),λ2(t),…,λd(t)},(3)
so that the eigen-basis coincides with the canonical orthogonal basis {e1,…,ed} and
λ1(t)>λ2(t)>⋯>λd(t)for all t≥0.
Under Assumption 2.1, the largest eigenvalue e1=ξmax will be the leading direction selected by the dynamics. Let us fix a solution x(t)=(x1(t),…,xL(t))∈C∞([0,+∞),(Sd−1)L) of the dynamical system (2), subject to the initial condition x(0)∈(Sd−1)L. We will derive our theorems by induction on the number L of tokens. In particular, we define the error function
εL−1(t)=j=1∑L−1∥xj(t)−sje1∥,
where ∥⋅∥ denotes the Euclidean norm. Our inductive hypothesis for the first L−1 tokens is
εL−1(t)⟶0as t→∞,(4)
References
[1]Laziz U. Abdullaev, Maksim Tkachenko, and Tan M. Nguyen. Revisiting transformers with insights from image filtering and boosting. SIAM Journal on Imaging Sciences, 2025.arxiv.org/abs/2506.10371
[2]Álvaro Rodríguez Abella, João Pedro Silvestre, and Paulo Tabuada. Consensus is all you get: The role of attention in transformers. In Forty-second International Conference on Machine Learning, 2025.
[3]Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, 630(8016):493–500, 2024.DOI
[4]Juan A Acebrón, Luis L Bonilla, Conrad J Pérez Vicente, Félix Ritort, and Renato Spigler. The kuramoto model: A simple paradigm for synchronization phenomena. Reviews of modern physics, 77(1):137–185, 2005.DOI
[5]Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, Singapore, December 2023. Association for Computational Linguistics.
[6]Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3159–3166, 2019.arxiv.org/abs/1808.04444
[7]Albert Alcalde, Giovanni Fantuzzi, and Enrique Zuazua. Clustering in pure-attention hardmax transformers and its role in sentiment analysis. arXiv preprint arXiv:2407.01602, 2024.DOI
[8]Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference on Learning Representations, 2019.arxiv.org/abs/1809.10853
[9]Han Bao, Ryuichiro Hataya, and Ryo Karakida. Self-attention networks localize when qk-eigenspectrum concentrates. arXiv preprint arXiv:2402.02098, 2024.arxiv.org/abs/2402.02098
[10]Etienne Boursier and Claire Boyer. Softmax as linear attention in the large-prompt regime: a measure-based perspective, 2025.arxiv.org/abs/2512.11784
[11]Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. MUSE: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023.arxiv.org/abs/2301.00704
[12]Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021.arxiv.org/abs/2106.01345
[13]Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1806.07366
[14]Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12020–12030, 2022.arxiv.org/abs/2203.06345
[15]Jingpu Cheng, Qianxiao Li, Ting Lin, and Zuowei Shen. Interpolation, approximation and controllability of deep neural networks. arXiv preprint arXiv:2309.06015, 2023.arxiv.org/abs/2309.06015
[16]Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, Doha, Qatar, October 2014. Association for Computational Linguistics.arxiv.org/abs/1406.1078
[17]Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Anna Korhonen, David Traum, and Luís Márquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy, July 2019. Association for Computational Linguistics.arxiv.org/abs/1901.02860
[18]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019.arxiv.org/abs/1807.03819
[19]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.arxiv.org/abs/1810.04805
[20]Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793–2803. PMLR, 2021.arxiv.org/abs/2103.03404
[21]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[22]Gbetondji JS Dovonon, Michael M Bronstein, and Matt J Kusner. Setting the record straight on transformer oversmoothing. arXiv preprint arXiv:2401.04301, 2024.arxiv.org/abs/2401.04301
[23]Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407, 2024.arxiv.org/abs/2407.21783
[24]Subhabrata Dutta, Tanya Gautam, Soumen Chakrabarti, and Tanmoy Chakraborty. Redesigning the transformer architecture with insights from multi-particle dynamical systems. Advances in Neural Information Processing Systems, 34:5531–5544, 2021.arxiv.org/abs/2109.15142
[25]Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021.arxiv.org/abs/2012.09841
[26]Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. Rank diminishing in deep neural networks. Advances in Neural Information Processing Systems, 35:33054–33065, 2022.arxiv.org/abs/2206.06072
[27]Borjan Geshkovski, Hugo Koubbi, Yury Polyanskiy, and Philippe Rigollet. Dynamic metastability in the self-attention model. arXiv preprint arXiv:2410.06833, 2024.arxiv.org/abs/2410.06833
[28]Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspective on transformers. arXiv preprint arXiv:2312.10794, 2023.arxiv.org/abs/2312.10794
[29]Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics. Advances in Neural Information Processing Systems, 36, 2024.DOI
[30]Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning perspective on oversmoothing and beyond. arXiv preprint arXiv:2303.06562, 2023.arxiv.org/abs/2303.06562
[31]Seung-Yeal Ha, Jeongho Kim, Jinyeong Park, and Xiongtiao Zhang. Complete cluster predictability of the cucker–smale flocking model on the real line. Archive for Rational Mechanics and Analysis, 231:319–365, 2019.DOI
[32]Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34(1):014004, 2017.DOI
[33]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
[34]Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273–1286, 2021.arxiv.org/abs/2106.02039
[35]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.arxiv.org/abs/2310.06825
[36]Amir Joudaki, Hadi Daneshmand, and Francis Bach. On the impact of activation and normalization in obtaining isometric embeddings at initialization. Advances in Neural Information Processing Systems, 36:39855–39875, 2023.arxiv.org/abs/2305.18399
[37]Nikita Karagodin, Yury Polyanskiy, and Philippe Rigollet. Clustering in causal attention masking. Advances in Neural Information Processing Systems, 37:115652–115681, 2024.arxiv.org/abs/2411.04990
[38]Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1–41, 2022.DOI
[39]Samory Kpotufe and Abdeslam Boulariais. Gradient weights help nonparametric regressors. In Advances in Neural Information Processing Systems, volume 25, 2012.
[40]Ulrich Krause et al. A discrete nonlinear and non-autonomous model of consensus formation. Communications in difference equations, 2000:227–236, 2000.
[41]Yoshiki Kuramoto. Self-entrainment of a population of coupled non-linear oscillators. In International Symposium on Mathematical Problems in Theoretical Physics: January 23–29, 1975, Kyoto University, Kyoto/Japan, pages 420–422. Springer, 1975.
[42]Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436–444, 2015.DOI
[43]Qianxiao Li, Ting Lin, and Zuowei Shen. Deep learning via dynamical systems: An approximation perspective. Journal of the European Mathematical Society, 25(5):1671–1709, 2022.arxiv.org/abs/1912.10382
[44]Hongzhou Lin and Stefanie Jegelka. Resnet with one-neuron hidden layers is a universal approximator. Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1806.10909
[45]Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. AI open, 3:111–132, 2022.arxiv.org/abs/2106.04554
[46]Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In International Conference on Learning Representations, 2017.arxiv.org/abs/1703.03130
[47]Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.arxiv.org/abs/2103.14030
[48]Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv:1906.02762, 2019.arxiv.org/abs/1906.02762
[50]Ho Huu Nghia Nguyen, Tan Nguyen, Huyen Vo, Stanley Osher, and Thieu Vo. Improving neural ordinary differential equations with nesterov’s accelerated gradient method. Advances in Neural Information Processing Systems, 35:7712–7726, 2022.DOI
[51]Tam Nguyen, Tan Nguyen, and Richard Baraniuk. Mitigating over-smoothing in transformers via regularized nonlocal functionals. Advances in Neural Information Processing Systems, 36:80233–80256, 2023.arxiv.org/abs/2312.00751
[52]Tam Minh Nguyen, Tan Minh Nguyen, Dung DD Le, Duy Khuong Nguyen, Viet-Anh Tran, Richard Baraniuk, Nhat Ho, and Stanley Osher. Improving transformers with probabilistic attention keys. In International Conference on Machine Learning, pages 16595–16621. PMLR, 2022.
[53]Tam Minh Nguyen, César A Uribe, Tan Minh Nguyen, and Richard Baraniuk. Pidformer: Transformer meets control theory. In Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2402.15989
[54]Tan Nguyen, Minh Pham, Tam Nguyen, Khai Nguyen, Stanley Osher, and Nhat Ho. Fourier-former: Transformer meets generalized fourier integral theorem. Advances in Neural Information Processing Systems, 35:29319–29335, 2022.DOI
[55]Tam Minh Nguyen, Tam Minh Nguyen, Hai Ngoc Do, Khai Nguyen, Vishwanath Saragadam, Minh Pham, Nguyen Duy Khuong, Nhat Ho, and Stanley Osher. Improving transformer with an admixture of attention heads. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.DOI
[56]Tuan Nguyen, Hirotada Honda, Takashi Sano, Vinh Nguyen, Shugo Nakamura, and Tan Minh Nguyen. From coupled oscillators to graph neural networks: Reducing over-smoothing via a kuramoto model-based approach. In International Conference on Artificial Intelligence and Statistics, pages 2710–2718. PMLR, 2024.arxiv.org/abs/2311.03260
[57]Stefan K Nielsen, Laziz U Abdullaev, Rachel S Teo, and Tan M Nguyen. Elliptical attention. Advances in Neural Information Processing Systems, 37:109748–109789, 2024.arxiv.org/abs/2406.13770
[58]Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27198–27211, 2022.arxiv.org/abs/2206.03126
[59]Lorenzo Noci, Chuning Li, Mufan Li, Bobby He, Thomas Hofmann, Chris J Maddison, and Dan Roy. The shaped transformer: Attention models in the infinite depth-and-width limit. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2306.17759
[60]Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249–2255, Austin, Texas, November 2016. Association for Computational Linguistics.arxiv.org/abs/1606.01933
[61]Duy-Tung Pham, An The Nguyen, Viet-Hoang Tran, Nhan-Phu Chung, Xin T Tong, Tan M Nguyen, and Thieu N Vo. Dynamical properties of tokens in self-attention and effects of positional encoding. Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2512.03058
[62]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
[63]Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[64]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
[65]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.arxiv.org/abs/1910.10683
[66]Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821–8831. Pmlr, 2021.arxiv.org/abs/2102.12092
[67]Domenec Ruiz-Balet and Enrique Zuazua. Neural ode control for classification, approximation, and transport. SIAM Review, 65(3):735–773, 2023.
[68]Michael Scholkemper, Xinyi Wu, Ali Jadbabaie, and Michael T Schaub. Residual connections and normalization can probably prevent oversmoothing in gnns. arXiv preprint arXiv:2406.02997, 2024.arxiv.org/abs/2406.02997
[69]Bingjing Song, Boran Han, Shuai Zhang, Jie Ding, and Mingyi Hong. Unraveling the gradient descent dynamics of transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.arxiv.org/abs/2411.07538
[70]Paulo Tabuada and Bahman Gharesifard. Universal approximation power of deep residual neural networks through the lens of control. IEEE Transactions on Automatic Control, 68(5):2715–2728, 2022.DOI
[71]Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1–28, 2022.
[72]Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024.arxiv.org/abs/2405.09818
[73]Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy, July 2019. Association for Computational Linguistics.arxiv.org/abs/1905.05950
[74]Rachel Teo and Tan Nguyen. Molex: Mixture of layer experts for fine-tuning with sparse upcycling. In International Conference on Learning Representations, volume 2025, pages 70264–70289, 2025.
[75]Rachel S Teo and Tan M Nguyen. Momentumsmoe: Integrating momentum into sparse mixture of experts. Advances in Neural Information Processing Systems, 37:28965–29000, 2024.arxiv.org/abs/2410.14574
[76]Rachel S Teo and Tan M Nguyen. Unveiling the hidden structure of self-attention via kernel principal component analysis. Advances in Neural Information Processing Systems, 37:101393–101427, 2024.arxiv.org/abs/2406.13762
[77]Matthew Thorpe, Tan Minh Nguyen, Hedi Xia, Thomas Strohmer, Andrea Bertozzi, Stanley Osher, and Bao Wang. Grand++: Graph neural diffusion with a source term. In International Conference on Learning Representations, 2022.
[78]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03762
[79]Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. In Tal Linzen, Gregor Chrupała, Yonatan Belinkov, and Dieuwke Hupkes, editors, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63–76, Florence, Italy, August 2019. Association for Computational Linguistics.arxiv.org/abs/1906.04284
[80]Thieu N Vo, Tung D Pham, Xin T Tong, and Tan Minh Nguyen. Demystifying the token dynamics of deep selective state space models. International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.03292
[81]Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. LoRA-pro: Are low-rank adapters properly optimized? In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2407.18242
[82]Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics, June 2018.arxiv.org/abs/1704.05426
[83]Xinyi Wu, Amir Aj orlou, Zihui Wu, and Ali Jadbabaie. Demystifying oversmoothing in attention-based graph neural networks. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2305.16102
[84]Hedi Xia, Vai Suliafu, Hangjie Ji, Tan Nguyen, Andrea Bertozzi, Stanley Osher, and Bao Wang. Heavy ball neural ordinary differential equations. Advances in Neural Information Processing Systems, 34:18646–18659, 2021.arxiv.org/abs/2110.04840
[85]Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunj an Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfey Yang, Burak Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3):5, 2022.arxiv.org/abs/2206.10789
[86]Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pages 40770–40803. PMLR, 2023.arxiv.org/abs/2303.06296
[87]Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1910.07467
[88]Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural odes and invertible residual networks. In International Conference on Machine Learning, pages 11086–11095. PMLR, 2020.arxiv.org/abs/1907.12998
[89]Victor Zhong, Caiming Xiong, and Richard Socher. Seq2Sql: Generating structured queries from natural language using reinforcement learning, 2017.arxiv.org/abs/1709.00103
[90]Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, Fares Obied, and Zhenzhong Lan. Value residual learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28341–28356, Vienna, Austria, July 2025. Association for Computational Linguistics.arxiv.org/abs/2410.17897