betaFeedback

WorksComputer Science

First-Token Attraction in Mamba Dynamics

Abstract

We study token dynamics in deep Mamba models, with a particular focus on the attracting role of the first token. Unlike attention mechanisms, where token interactions are governed by positive scalar coefficients, Mamba dynamics involve full matrix-valued interaction coefficients with no positivity or definiteness guarantees. This structural difference makes Mamba dynamics substantially harder to analyze and prevents the direct application of existing techniques for attention-based models. To overcome this challenge, we introduce a new analytical framework based on a high-dimensional invariant box and a sharp exit-time argument, which together recover an effective form of sign-definiteness. Using this framework, we characterize the possible limit points of token trajectories and prove explicit exponential convergence rates. Our analysis reveals that the first token has privileged dynamics: its limiting direction attracts subsequent tokens, inducing token clustering and attention-sink phenomena in deep Mamba models. We further show that this attraction mechanism extends to broader sequence dynamics, including state-space models, causal attention, and classical time-series models. Experiments on Falcon-Mamba-7B validate our predictions, showing that concentration on the first token increases with depth and dominates in later layers.

Introduction

Modern sequence modeling has been dominated by transformers, while state space models (SSMs) have emerged as an important alternative. Transformers have set the standard for representation learning, yet their quadratic complexity with respect to sequence length poses challenges for efficiency and scalability [77]. In parallel, SSMs have reemerged as powerful alternatives that combine principles of recurrence and convolution with ideas from classical control theory [34]. Recent innovations in parameterization and implementation have made SSMs substantially more efficient [25, 26, 24, 29, 45, 36, 63, 72], enabling them to capture long-range dependencies effectively [23, 58] and to excel in benchmark tasks such as the Long Range Arena [74]. Beyond efficiency, these models have demonstrated strong empirical performance in domains as varied as audio, speech, and vision [21, 54, 70]. Within this line of development, the Mamba architecture has emerged as a selective SSM with striking empirical results. Its distinctive component, the selective state-space (S6) layer, introduces input-dependent parameters that grant the model content awareness [22]. This design has proven effective across a wide spectrum of applications, including language modeling [67, 41], vision and video analysis [48, 89, 85, 42], medical imaging [50, 80], tabular data [5], reinforcement learning [64], 3D perception [46], graph learning [79], and even NN-dimensional sequence modeling [44]. The breadth of these successes has been linked to Mamba’s ability to generate highly adaptive and expressive representations across modalities [68, 65, 84, 89].

The study of token dynamics—the evolution of token representations within a model—has proven to be a powerful framework for understanding transformers, offering insights into equilibrium structures, stability properties, and clustering phenomena. In contrast, the token dynamics of selective SSMs have received little attention, despite the strong empirical performance of architectures such as Mamba.

Our Contributions

In this paper, we investigate the dynamical properties of tokens in deep pretrained Mamba models. Prior theoretical studies often rely on restrictive conditions such as the absence of normalization and one-dimensional token embeddings (see [78]). We consider the continuous-time limit of pretrained Mamba models without these restrictive assumptions and systematically analyze the asymptotic behavior of token trajectories over time. Our main contributions are summarized as follows:

  • We construct a new theoretical framework based on a high-dimensional invariant box for the Mamba dynamics and establish a sharp exit-time argument to determine when token trajectories remain confined to this domain. This framework overcomes the difficulties arising in previous approaches for attention settings.

  • We characterize the set of limit points of token trajectories in Mamba models and establish explicit exponential convergence rates (Theorems 3.5, 3.3, 3.6 and 3.9). Our theoretical analysis further demonstrates that the first token exerts an attracting influence on the remaining tokens, a phenomenon that we empirically validate.

  • We demonstrate a practical implication of our theoretical results to offer a principled explanation of the attention sink effect in Mamba models via the attraction of the first token.

To validate our theoretical findings and their applications, we conduct extensive experiments by visualizing attention concentration in Falcon-Mamba-7B across different depths. These experimental results confirm the predictive accuracy of our analysis and underscore its practical benefits in real-world scenarios.

Mamba dynamics is far more complicated than causal attention dynamics. Indeed, while our work is inspired by the recent analysis of Transformer dynamics in [35], the mathematical structure of Mamba is much more complex. In the attention setting, the effective interaction coefficients are positive scalars as they are normalized via softmax. The positivity of these interaction coefficients ensures sign-definiteness and enables the use of powerful tools from monotone and cooperative dynamical systems. In contrast, for Mamba, the effective interaction coefficients are full matrices with no positivity or definiteness guarantees. This lack of sign-definiteness is not a minor technical inconvenience; it is an intrinsic structural feature of practical Mamba architectures and invalidates essentially all positivity-based techniques that underpin the current theory of attention dynamics. As a consequence, techniques used for analyzing causal attention dynamics, including hemisphere-based Lyapunov methods [3] and Shub-type genericity arguments [35], completely fail in this architecture.

Our new analytical framework. To overcome the above challenge, we develop a new analytical framework based on a high-dimensional invariant box for the dynamical system governing the continuous-time limit of Mamba, together with a sharp exit-time argument that recovers effective sign-definiteness and determines when token trajectories remain confined to this domain. Within this framework, we characterize all possible token limit points and derive explicit exponential convergence rates toward them. This analysis yields a theoretical explanation for a widely observed empirical phenomenon: the limit point of the first token exerts an attracting influence on subsequent tokens, thereby inducing clustering and attention-sink phenomena. Finally, we extend our analysis of this attracting effect to a broad class of autoregressive sequence models, including state-space models, causal Transformers, and classical time-series models.

Organization. Section 2 derives a dynamical system over the unit sphere arising from the continuous-time limit of Mamba. Section 3 analyzes the limitations of the first token and proves its attraction effect on subsequent tokens. Section 4.1 validates the theory in practical Mamba settings, while Section 5 applies the results to predict attention-sink phenomena in Mamba models.

Mamba Dynamics as a Flow Map on Sphere

Background – S6 layer

The building block of a selective state-space (Mamba) model is a Mamba block (see [13]). At the heart of the Mamba block is the S6 layer, which plays a key role in the success of deep Mamba models.

S6 layer. Formally, an S6 layer is defined as a function that maps a sequence of tokens x=(x1,…,xL)∈RD×L\mathbf{x}=(x_1,\ldots,x_L)\in\mathbb{R}^{D\times L} to another sequence of tokens y=(y1,…,yL)∈RD×L\mathbf{y}=(y_1,\ldots,y_L)\in\mathbb{R}^{D\times L} (with the same number of channels DD). In this context, each vector xl∈RDx_l\in\mathbb{R}^{D} (or yl∈RDy_l\in\mathbb{R}^{D}) represents a token/word, while the whole sequence x=(x1,…,xL)\mathbf{x}=(x_1,\ldots,x_L) (or y=(y1,…,yL)\mathbf{y}=(y_1,\ldots,y_L)) is called a prompt/sentence.

For each d=1,…,Dd=1,\ldots,D, the dd-th channel output y^d=(yd1,…,ydL)∈RL\hat{y}_d=(y_{d1},\ldots,y_{dL})\in\mathbb{R}^{L} is determined recurrently from the corresponding dd-th channel input x^d=(xd1,…,xdL)∈RL\hat{x}_d=(x_{d1},\ldots,x_{dL})\in\mathbb{R}^{L} via a sequence of hidden states hd1,…,hdL∈RNh_{d1},\ldots,h_{dL}\in\mathbb{R}^{N} as follows:

hdl=A‾dl⋅hd,l−1+B‾dl⋅xdl,hd0=0,ydl=Cl⋅hdl,(1)\begin{aligned} h_{dl}&=\overline{A}_{dl}\cdot h_{d,l-1}+\overline{B}_{dl}\cdot x_{dl},\quad h_{d0}=0,\\ y_{dl}&=C_l\cdot h_{dl}, \tag*{(1)} \end{aligned}

for l=1,…,Ll=1,\ldots,L. Here, the matrices A‾dl\overline{A}_{dl}, B‾dl\overline{B}_{dl}, and ClC_l are input-dependent and time-varying, determined by

A‾dl=e−adΔd(xl),B‾dl=Δd(xl)SB⋅xl,andCl=(SC⋅xl)T,(2)\overline{A}_{dl}=e^{-a_d\Delta_d(x_l)},\quad\overline{B}_{dl}=\Delta_d(x_l)S_B\cdot x_l,\quad\text{and}\quad C_l=(S_C\cdot x_l)^{\mathsf{T}}, \tag*{(2)}

where Δd:RD→R\Delta_d:\mathbb{R}^{D}\to\mathbb{R} is the step size function at the dd-th channel and is defined for each u∈RDu\in\mathbb{R}^{D} by

Δd(u)=softplus⁡(SΔ,d⋅u)=ln⁡(1+eSΔ,d⋅u).(3)\Delta_d(u)=\operatorname{softplus}(S_{\Delta,d}\cdot u)=\ln\left(1+e^{S_{\Delta,d}\cdot u}\right). \tag*{(3)}

The hidden positive coefficients ad∈R>0a_d\in\mathbb{R}_{>0}, and the step size vectors SΔ,d∈RDS_{\Delta,d}\in\mathbb{R}^{D} for d=1,…,Dd=1,\ldots,D, as well as the input and output matrices SB,SC∈RN×DS_B,S_C\in\mathbb{R}^{N\times D}, are learnable from input data.

By eliminating the hidden states hdlh_{dl} in system (1) and recall that yl=(y1l,…,yDL)y_l=(y_{1l},\ldots,y_{DL}), we obtain the following form (see also [7]):

yl=∑j=1lAlj(x)xj,(4)y_l=\sum_{j=1}^{l}A_{lj}(\mathbf{x})x_j, \tag*{(4)}

where Alj=diag⁡(A1,lj,…,AD,lj)A_{lj}=\operatorname{diag}(A_{1,lj},\ldots,A_{D,lj}) and SBC=SCTSBS_{BC}=S_C^{\mathsf{T}}S_B as well as

Ad,lj(x)={xlTSBCxl⋅Δd(xl),if l=j,xlTSBCxj⋅Δd(xj)⋅exp⁡(−ad∑k=j+1lΔd(xk)),if l>j.(5)A_{d,lj}(\mathbf{x})= \begin{cases} x_l^{\mathsf{T}}S_{BC}x_l\cdot\Delta_d(x_l), & \text{if }l=j,\\ x_l^{\mathsf{T}}S_{BC}x_j\cdot\Delta_d(x_j)\cdot\exp\left(-a_d\sum_{k=j+1}^{l}\Delta_d(x_k)\right), & \text{if }l>j. \end{cases} \tag*{(5)}

In our context, the matrix SBC=SCTSB∈RD×DS_{BC}=S_C^{\mathsf{T}}S_B\in\mathbb{R}^{D\times D}, which we will refer to as the input-output matrix, plays an essential role in characterizing the dynamical properties of tokens.

Mamba Dynamic as Flow Map on Sphere

All practical implementations of Mamba models incorporate normalization [8], most commonly realized as root mean square (RMS) normalization [87]. In this approach, the sequence of tokens at each layer is normalized by dividing each token by its Euclidean norm. This operation enhances stability by preventing divergence, thereby reducing rounding errors and avoiding numerical overflow. Consequently, the resulting dynamics can be viewed as evolving on the unit sphere SD−1\mathbb{S}^{D-1} throughout.

Follow the framework of [35], a Mamba model including S6 layer and RMS normalization with skip connection can then be viewed as a flow map on (SD−1)L(\mathbb{S}^{D-1})^L of the dynamical system

ddtxl(t)=Pxl(t)(∑j=1lAlj(x(t),t)xj(t)),l=1,…,L,(6)\frac{d}{dt}x_l(t)=\mathbf{P}_{x_l(t)}\left(\sum_{j=1}^{l}A_{lj}(x(t),t)x_j(t)\right),\quad l=1,\ldots,L, \tag*{(6)}
x(0)=(x1(0),…,xL(0))∈(SD−1)L,(7)\mathbf{x}(0)=(x_1(0),\ldots,x_L(0))\in(\mathbb{S}^{D-1})^L, \tag*{(7)}

where

  • for each x∈SD−1x\in\mathbb{S}^{D-1}, the map Px:RD→TxSD−1\mathbf{P}_x:\mathbb{R}^D\to T_x\mathbb{S}^{D-1} is the projection onto the tangent space at xx of the sphere SD−1\mathbb{S}^{D-1} and it is defined by Px(y)=y−⟨y,x⟩x\mathbf{P}_x(y)=y-\langle y,x\rangle x for each y∈RDy\in\mathbb{R}^D;

  • the coefficient AljA_{lj} is an D×DD\times D diagonal matrix Alj=diag⁡(A1,lj,…,AD,lj)A_{lj}=\operatorname{diag}(A_{1,lj},\ldots,A_{D,lj}) with

References

  1. [1]Laziz Abdullaev and Tan Minh Nguyen. Transformer meets twicing: Harnessing unattended residual information. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2503.00687
  2. [2]Laziz U Abdullaev, Maksim Tkachenko, and Tan M Nguyen. Revisiting transformers with insights from image filtering and boosting. SIAM Journal on Imaging Sciences, 2025.arxiv.org/abs/2506.10371
  3. [3]Álvaro Rodríguez Abella, João Pedro Silvestre, and Paulo Tabuada. Consensus is all you get: The role of attention in transformers. In Forty-second International Conference on Machine Learning, 2025.
  4. [4]Juan A Acebrón, Luis L Bonilla, Conrad J Pérez Vicente, Félix Ritort, and Renato Spigler. The kuramoto model: A simple paradigm for synchronization phenomena. Reviews of modern physics, 77(1):137–185, 2005.DOI
  5. [5]Md Atik Ahamed and Qiang Cheng. Mambatab: A plug-and-play model for learning tabular data. In 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR), pages 369–375, 2024.arxiv.org/abs/2401.08867
  6. [6]Albert Alcalde, Giovanni Fantuzzi, and Enrique Zuazua. Clustering in pure-attention hardmax transformers and its role in sentiment analysis. arXiv preprint arXiv:2407.01602, 2024.DOI
  7. [7]Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models. arXiv preprint arXiv:2403.01590, 2024.arxiv.org/abs/2403.01590
  8. [8]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.arxiv.org/abs/1607.06450
  9. [9]Han Bao, Ryuichiro Hataya, and Ryo Karakida. Self-attention networks localize when qk-eigenspectrum concentrates. arXiv preprint arXiv:2402.02098, 2024.arxiv.org/abs/2402.02098
  10. [10]Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.arxiv.org/abs/1806.07366
  11. [11]Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12020–12030, 2022.arxiv.org/abs/2203.06345
  12. [12]Jingpu Cheng, Qianxiao Li, Ting Lin, and Zuowei Shen. Interpolation, approximation and controllability of deep neural networks. arXiv preprint arXiv:2309.06015, 2023.arxiv.org/abs/2309.06015
  13. [13]Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning, pages 10041–10071. PMLR, 2024.arxiv.org/abs/2405.21060
  14. [14]Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793–2803. PMLR, 2021.arxiv.org/abs/2103.03404
  15. [15]Gbetondji JS Dovonon, Michael M Bronstein, and Matt J Kusner. Setting the record straight on transformer oversmoothing. arXiv preprint arXiv:2401.04301, 2024.arxiv.org/abs/2401.04301
  16. [16]Subhabrata Dutta, Tanya Gautam, Soumen Chakrabarti, and Tanmoy Chakraborty. Redesigning the transformer architecture with insights from multi-particle dynamical systems. Advances in Neural Information Processing Systems, 34:5531–5544, 2021.arxiv.org/abs/2109.15142
  17. [17]Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. Rank diminishing in deep neural networks. Advances in Neural Information Processing Systems, 35:33054–33065, 2022.arxiv.org/abs/2206.06072
  18. [18]Borjan Geshkovski, Hugo Koubbi, Yury Polyanskiy, and Philippe Rigollet. Dynamic metastability in the self-attention model. arXiv preprint arXiv:2410.06833, 2024.arxiv.org/abs/2410.06833
  19. [19]Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspective on transformers. arXiv preprint arXiv:2312.10794, 2023.arxiv.org/abs/2312.10794
  20. [20]Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics. Advances in Neural Information Processing Systems, 36, 2024.DOI
  21. [21]Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It’s raw! audio generation with state-space models. In International Conference on Machine Learning, pages 7616–7633. PMLR, 2022.arxiv.org/abs/2202.09729
  22. [22]Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024.arxiv.org/abs/2312.00752
  23. [23]Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474–1487, 2020.arxiv.org/abs/2008.07669
  24. [24]Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971–35983, 2022.arxiv.org/abs/2206.11893
  25. [25]Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022.arxiv.org/abs/2111.00396
  26. [26]Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572–585, 2021.arxiv.org/abs/2110.13985
  27. [27]Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2410.10781
  28. [28]Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning perspective on oversmoothing and beyond. arXiv preprint arXiv:2303.06562, 2023.arxiv.org/abs/2303.06562
  29. [29]Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 35:22982–22994, 2022.arxiv.org/abs/2203.14343
  30. [30]Seung-Yeal Ha, Jeongho Kim, Jinyeong Park, and Xiongtao Zhang. Complete cluster predictability of the cucker–smale flocking model on the real line. Archive for Rational Mechanics and Analysis, 231:319–365, 2019.DOI
  31. [31]Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34(1):014004, 2017.DOI
  32. [32]Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. In International Conference on Machine Learning, pages 21502–21521. PMLR, 2024.arxiv.org/abs/2402.01032
  33. [33]Amir Joudaki, Hadi Deshmand, and Francis Bach. On the impact of activation and normalization in obtaining isometric embeddings at initialization. Advances in Neural Information Processing Systems, 36:39855–39875, 2023.arxiv.org/abs/2305.18399
  34. [34]RE Kalman. A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82(1):35–45, 1960.DOI
  35. [35]Nikita Karagodin, Yury Polyanskiy, and Philippe Rigollet. Clustering in causal attention masking. Advances in Neural Information Processing Systems, 37:115652–115681, 2024.arxiv.org/abs/2411.04990
  36. [36]Chrysoula Kosma, Giannis Nikolentzos, and Michalis Vazirgiannis. Time-parameterized convolutional neural networks for irregularly sampled time series. arXiv preprint arXiv:2308.03210, 2023.arxiv.org/abs/2308.03210
  37. [37]Samory Kpotufe and Abdeslam Boulariais. Gradient weights help nonparametric regressors. In Advances in Neural Information Processing Systems, volume 25, 2012.
  38. [38]Ulrich Krause et al. A discrete nonlinear and non-autonomous model of consensus formation. Communications in difference equations, 2000:227–236, 2000.
  39. [39]Yoshiki Kuramoto. Self-entrainment of a population of coupled non-linear oscillators. In International Symposium on Mathematical Problems in Theoretical Physics: January 23–29, 1975, Kyoto University, Kyoto/Japan, pages 420–422. Springer, 1975.
  40. [40]Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436–444, 2015.DOI
  41. [41]Barak Lenz, Opher Lieber, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M. Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, Gal Cohen, Gal Shachaf, Haim Rozenblum, Hofit Bata, Ido Blass, Inbal Magar, Itay Dalmendigos, Jhonathan Osin, Julie Fadlon, Maria Rozman, Matan Danos, Michael Gokhman, Mor Zuzman, Naama Gidron, Nir Ratner, Noam Gat, Noam Rozen, Oded Fried, Ohad Leshno, Omer Antverg, Omri Abend, Or Dagan, Orit Cohavi, Raz Alon, Ro’i Belson, Roi Cohen, Rom Gilad, Roman Glozman, Shahar Lev, Shai Shalev-Shwartz, Shaked Haim Meirom, Tal Delbari, Tal Ness, Tomer Asida, Tom Ben Gal, Tom Braude, Uriya Pumerantz, Josh Cohen, Yonatan Belinkov, Yuval Globerson, Yuval Peleg Levy, and Yoav Shoham. Jamba: Hybrid transformer-mamba language models. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2403.19887
  42. [42]Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Video-mamba: State space model for efficient video understanding. In European Conference on Computer Vision, pages 237–255. Springer, 2024.arxiv.org/abs/2403.06977
  43. [43]Qianxiao Li, Ting Lin, and Zuowei Shen. Deep learning via dynamical systems: An approximation perspective. Journal of the European Mathematical Society, 25(5):1671–1709, 2022.arxiv.org/abs/1912.10382
  44. [44]Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-nd: Selective state space modeling for multi-dimensional data. In European Conference on Computer Vision, pages 75–92. Springer, 2024.arxiv.org/abs/2402.05892
  45. [45]Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. What makes convolutional models great on long sequence modeling? In The Eleventh International Conference on Learning Representations, 2023.
  46. [46]Dingkang Liang, Xin Zhou, Wei Xu, Xingkui Zhu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. Advances in neural information processing systems, 37:32653–32677, 2025.
  47. [47]Hongzhou Lin and Stefanie Jegelka. Resnet with one-neuron hidden layers is a universal approximator. Advances in neural information processing systems, 31, 2018.
  48. [48]Fusheng Liu and Qianxiao Li. From generalization analysis to optimization designs for state space models. arXiv preprint arXiv:2405.02670, 2024.
  49. [49]Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv:1906.02762, 2019.
  50. [50]Chao Ma and Ziyang Wang. Semi-mamba-unet: Pixel-level contrastive and cross-supervised visual mamba-based unet for semi-supervised medical image segmentation. Knowledge-Based Systems, 300:112203, 2024.
  51. [51]William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. In International Conference on Machine Learning, pages 35492–35506. PMLR, 2024.arxiv.org/abs/2404.08819
  52. [52]Sebastien Motsch and Eitan Tadmor. Heterophilious dynamics enhances consensus. SIAM review, 56(4):577–621, 2014.
  53. [53]Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Christopher Salvi, and Terry Lyons. Theoretical foundations of deep selective state-space models. Advances in Neural Information Processing Systems, 37:127226–127272, 2025.arxiv.org/abs/2402.19047
  54. [54]Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. Advances in neural information processing systems, 35:2846–2861, 2022.DOI
  55. [55]Ho Huu Nghia Nguyen, Tan Nguyen, Huyen Vo, Stanley Osher, and Thieu Vo. Improving neural ordinary differential equations with nesterov’s accelerated gradient method. Advances in Neural Information Processing Systems, 35:7712–7726, 2022.DOI
  56. [56]Tam Nguyen, Tan Nguyen, and Richard Baraniuk. Mitigating over-smoothing in transformers via regularized nonlocal functionals. Advances in Neural Information Processing Systems, 36:80233–80256, 2023.arxiv.org/abs/2312.00751
  57. [57]Tam Minh Nguyen, César A Uribe, Tan Minh Nguyen, and Richard Baraniuk. Pidformer: Transformer meets control theory. In Forty-first International Conference on Machine Learning, 2024.arxiv.org/abs/2402.15989
  58. [58]Thai Khanh Nguyen, Vo Ngoc Bich Uyen, Thieu Vo, Tan Minh Nguyen, and Cuong Pham. MuonSSM: Orthogonalizing state space models for sequence modeling. In Forty-third International Conference on Machine Learning, 2026.
  59. [59]Tuan Nguyen, Hirotada Honda, Takashi Sano, Vinh Nguyen, Shugo Nakamura, and Tan Minh Nguyen. From coupled oscillators to graph neural networks: Reducing over-smoothing via a kuramoto model-based approach. In International Conference on Artificial Intelligence and Statistics, pages 2710–2718. PMLR, 2024.arxiv.org/abs/2311.03260
  60. [60]Stefan K Nielsen, Laziz U Abdullaev, Rachel S Teo, and Tan M Nguyen. Elliptical attention. Advances in Neural Information Processing Systems, 37:109748–109789, 2024.arxiv.org/abs/2406.13770
  61. [61]Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27198–27211, 2022.arxiv.org/abs/2206.03126
  62. [62]Lorenzo Noci, Chuning Li, Mufan Li, Bobby He, Thomas Hofmann, Chris J Maddison, and Dan Roy. The shaped transformer: Attention models in the infinite depth-and-width limit. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2306.17759
  63. [63]Antonio Orvieto, Samuel L Smith, Albert Gu, Anush an Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pages 26670–26698. PMLR, 2023.arxiv.org/abs/2303.06349
  64. [64]Toshihiro Ota. Decision mamba: Reinforcement learning via sequence modeling with selective state spaces. arXiv preprint arXiv:2403.19925, 2024.arxiv.org/abs/2403.19925
  65. [65]Badri Narayana Patro and Vijay Srinivas Agneeswaran. Mamba-360: Survey of state space models as transformer alternative for long sequence modelling: Methods, applications, and challenges. arXiv preprint arXiv:2404.16112, 2024.arxiv.org/abs/2404.16112
  66. [66]Duy-Tung Pham, An The Nguyen, Viet-Hoang Tran, Nhan-Phu Chung, Xin T Tong, Tan M Nguyen, and Thieu N Vo. Dynamical properties of tokens in self-attention and effects of positional encoding. Advances in Neural Information Processing Systems, 2026.arxiv.org/abs/2512.03058
  67. [67]Maciej Pióro, Kamil Cieb iera, Krystian Król, Jan Ludziejewski, and Sebastian Jaszczur. Moe-mamba: Efficient selective state space models with mixture of experts. arXiv preprint arXiv:2401.04081, 2024.arxiv.org/abs/2401.04081
  68. [68]Farnoush Rezaei Jafari, Grégoire Montavon, Klaus-Robert Müller, and Oliver Eberle. Mambalrp: Explaining selective state space sequence models. Advances in Neural Information Processing Systems, 37:118540–118570, 2025.arxiv.org/abs/2406.07592
  69. [69]Domenec Ruiz-Balet and Enrique Zuazua. Neural ode control for classification, approximation, and transport. SIAM Review, 65(3):735–773, 2023.
  70. [70]George Saon, Ankit Gupta, and Xiaodong Cui. Diagonal state space augmented transformers for speech recognition. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.arxiv.org/abs/2302.14120
  71. [71]Michael Scholkemper, Xinyi Wu, Ali Jadbabaie, and Michael T Schaub. Residual connections and normalization can probably prevent oversmoothing in gnns. arXiv preprint arXiv:2406.02997, 2024.arxiv.org/abs/2406.02997
  72. [72]Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2208.04933
  73. [73]Paulo Tabuada and Bahman Gharesifard. Universal approximation power of deep residual neural networks through the lens of control. IEEE Transactions on Automatic Control, 68(5):2715–2728, 2022.DOI
  74. [74]Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In International Conference on Learning Representations, 2021.
  75. [75]Rachel S Teo and Tan M Nguyen. Momentumsmoe: Integrating momentum into sparse mixture of experts. Advances in Neural Information Processing Systems, 37:28965–29000, 2024.arxiv.org/abs/2410.14574
  76. [76]Matthew Thorpe, Tan Minh Nguyen, Hedi Xia, Thomas Strohmer, Andrea Bertozzi, Stanley Osher, and Bao Wang. Grand++: Graph neural diffusion with a source term. In International Conference on Learning Representations, 2022.
  77. [77]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03762
  78. [78]Thieu N Vo, Tung D Pham, Xin T Tong, and Tan Minh Nguyen. Demystifying the token dynamics of deep selective state space models. International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.03292
  79. [79]Chloe Wang, Oleksii Tsepa, Jun Ma, and Bo Wang. Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. arXiv preprint arXiv:2402.00789, 2024.arxiv.org/abs/2402.00789
  80. [80]Ziyang Wang, Jian-Qing Zheng, Yichi Zhang, Ge Cui, and Lei Li. Mamba-unet: Unet-like pure visual mamba for medical image segmentation. arXiv preprint arXiv:2402.05079, 2024.arxiv.org/abs/2402.05079
  81. [81]Xinyi Wu, Amir Ajrlou, Zhihu Wu, and Ali Jadbabaie. Demystifying oversmoothing in attention-based graph neural networks. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2305.16102
  82. [82]Hedi Xia, Vai Suliafu, Hangjie Ji, Tan Nguyen, Andrea Bertozzi, Stanley Osher, and Bao Wang. Heavy ball neural ordinary differential equations. Advances in Neural Information Processing Systems, 34:18646–18659, 2021.arxiv.org/abs/2110.04840
  83. [83]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2309.17453
  84. [84]Rui Xu, Shu Yang, Yihui Wang, Bo Du, and Hao Chen. A survey on vision mamba: Models, applications and challenges. arXiv preprint arXiv:2404.18861, 2024.
  85. [85]Yijun Yang, Zhaohu Xing, and Lei Zhu. Vivim: a video vision mamba for medical video object segmentation. arXiv preprint arXiv:2401.14168, 2024.arxiv.org/abs/2401.14168
  86. [86]Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pages 40770–40803. PMLR, 2023.arxiv.org/abs/2303.06296
  87. [87]Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1910.07467
  88. [88]Han Zhang, Xi Gao, Jacob Unterman, and Tom Arozd. Approximation capabilities of neural odes and invertible residual networks. In International Conference on Machine Learning, pages 11086–11095. PMLR, 2020.arxiv.org/abs/1907.12998
  89. [89]Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In International Conference on Machine Learning, pages 62429–62442. PMLR, 2024.arxiv.org/abs/2401.09417
  90. [90]Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, and Hakim Hacid. Falcon mamba: The first competitive attention-free 7b language model. arXiv preprint arXiv:2410.05355, 2024.arxiv.org/abs/2410.05355

Paper details

vlnc:W_epv4gu66
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 12

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents