betaFeedback

WorksComputer Science

Spectral Degeneration of Self-Attention under Isotropic Score Geometry

Abstract

We study the spectral structure of the self-attention matrix in Transformer models through an idealized spectral model for the pre-softmax score matrix. Motivated by the rotational isotropy of query and key representations, our model replaces the random token-space orientation of the score matrix by its canonical singular-value representative with a logarithmic spike–bulk separation. Empirical diagnostics on pretrained vision and language transformers motivate this model: token-space singular matrices have nearly uniform eigen-angle distributions, and score spectra are dominated by a small number of effective directions. We then prove that the row-wise softmax map transforms this structure into a degenerate attention matrix: dominant modes are preserved, whereas the bulk spectral distributions concentrate at zero. Stable rank remains bounded for bidirectional attention but grows logarithmically for causal attention. Within the canonical model, these results explain how attention can preserve a small number of dominant directions while compressing the remaining bulk modes.

Introduction

Transformers have become the dominant architecture in modern machine learning, powering large language models and state-of-the-art systems in vision, speech, and multimodal learning. Introduced in the seminal work “Attention Is All You Need” [Vaswani et al., 2017], the transformer replaces recurrence and convolution with a simple yet powerful mechanism: self-attention.

Given query and key matrices QQ, KK, self-attention computes a score matrix S=QKT/dqkS = QK^{\mathsf{T}}/\sqrt{d_{qk}}, followed by a row-wise softmax normalization A=softmax⁡(S)A = \operatorname{softmax}(S), which defines how information is aggregated across tokens. This mechanism is at the core of language models such as BERT Devlin et al., [4]; Liu et al. (2019) and GPT-style architectures Radford et al., (2018, 2019); Brown et al. [2] and vision models such as Vision Transformers (ViT) Dosovitskiy et al., [7], enabling scalable modeling of long-range dependencies.

Despite its empirical success, the theoretical understanding of attention remains limited, especially in the trained regime. A central open question is to characterize the structure of the attention matrix AA, which governs information flow and expressivity Bordelon et al., [1]; Sakai et al. (2025). Recent studies Dong et al., [6]; Noci et al. (2022); Nait Saada et al. (2025); Hayase et al. (2026); Sanyal et al. (2026) suggest that attention matrices often exhibit degeneration phenomena, where they become close to low-rank or even rank-one structures in deep layers. Understanding the origin of this degeneration is critical for explaining both the strengths and limitations of transformer models.

In this paper, we propose a new perspective based on the geometry of the query and key matrices. Motivated by isotropy diagnostics in trained representations, we study a canonical spectral model that isolates a mechanism of attention degeneration under softmax normalization. Specifically, we hypothesize that in trained transformers, the query and key matrices exhibit an approximate tokenspace orthogonal invariance, meaning that their token-space singular vectors behave as if they were drawn from a Haar distribution.

This is analogous to the feature-space rotational invariance exploited in RaBitQ [8] (Gao et al. 2025) and TurboQuant [Zandieh et al., 2026], where random rotations in feature space support efficient quantization. A shared orthogonal right rotation, (Q,K)↦(QR,KR)(Q, K) \mapsto(QR, KR), preserves the score matrix SS. In contrast, left rotation Q↦UQQ \mapsto UQ acts on the token dimension and does not preserve SS in the deterministic way, but instead characterizes its statistical structure.

In high dimension this isotropic picture has a concrete implication for the entries Sij=⟨qi,kj⟩/dqdkS_{ij} = \langle q_i, k_j \rangle/ \sqrt{d_q d_k}, where qiq_i and kjk_j are token-level query and key vectors. When distinct-token directions are approximately independent and isotropic, their cosine similarities satisfy ⟨qi,kj⟩/(∥qi∥2∥kj∥2)→0\langle q_i, k_j \rangle/ (\lVert q_i\rVert_2 \lVert k_j\rVert_2) \to0 for i≠ji \ne j, in probability as the feature dimension grows. This concentration describes angular geometry rather than the magnitude of the scaled scores, which also depends on query and key norms. It motivates studying score spectra separately from their orientation. In our spectral surrogate model, we keep the singular values and replace the token-space orientation by the canonical representative Σ\Sigma.

Empirical observation. We support this hypothesis with empirical evidence from both vision and language models: a pretrained Vision Transformer evaluated on Imagenette [Howard, 2019], and GPT-2 evaluated on Wikitext-103. Across these settings, the token-space singular matrices exhibit spectral behavior consistent with Haar-distributed orthogonal matrices. In particular, their eigen-angle distributions are close to uniform on the unit circle, consistent with one spectral consequence of token-space isotropy. Details are discussed in Section 4.

From geometry to degeneration. Motivated by high-dimensional isotropy, we analyze softmax⁡(Σ)\operatorname{softmax}(\Sigma) and its causal counterpart, where the diagonal surrogate Σ\Sigma contains the singular values of SS. We assume that bulk values lie below log⁡ℓ\log\ell and spikes above it, with diverging additive margins.

In the bidirectional model, softmax normalization makes bulk rows asymptotically uniform, while spike weights dominate the normalization and concentrate rows on canonical basis vectors. When the number of spikes satisfies k=o(ℓ)k = o(\ell), the leading k+1k + 1 singular values and eigenvalues converge to 11, while the rest vanish. We extend this spike–bulk analysis to causal attention and generalized row-normalized attention maps.

Main contributions. Our contributions can be summarized as follows:

  • We formulate a canonical diagonal score model motivated by token-space isotropy and provide empirical diagnostics of token-space geometry in pretrained transformers.

  • We characterize the eigenvalue and singular-value spectra of canonical bidirectional attention and show that, under spike–bulk separation, their empirical distributions concentrate on {0,1}\{0,1\}. Our results include a non-asymptotic characterization of the ordered eigenvalues and spectral profiles with explicit error bounds.

  • To the best of our knowledge, we provide the first joint characterization of eigenvalue and singular-value degeneration in causal attention. Under the canonical spike–bulk score model, we derive exact eigenvalues and quantitative singular-value bounds, showing that bounded bulk scores yield harmonic bulk eigenvalue decay and at most logarithmically many bulk singular values above any fixed positive threshold.

Related work

Collapse in depth. Dong et al. [2021], Noci et al. [2022] show that repeated attention in pure attention networks progressively compresses representations, producing doubly exponential rank collapse with depth. These results characterize the cumulative effect of composing attention layers.

Collapse in width. In a different regime, Nait Saada et al. [2025] analyze attention matrices in the width limit by modeling the score matrix SS as an i.i.d. random matrix. They show that the softmax attention matrix behaves like a random Markov matrix and exhibits rank collapse driven by a spectral gap. However, this analysis relies on the strong i.i.d. assumption on SS, which does not reflect the structured nature of trained transformers.

Hayase et al. [2026] study the spectrum of attention matrices at initialization through a Gaussian equivalence framework. Unlike the i.i.d. model of Nait Saada et al. [2025], their analysis accounts for the structured dependence in the score matrix SS, which arises even at initialization from the shared input and weight matrices. They provide a random matrix characterization of attention in the infinite-width limit, capturing these correlations. However, their results remain restricted to randomly initialized networks and do not address the trained regime.

Token-space isotropy observed in both Gaussian-initialized and trained transformers motivates our canonical diagonal score surrogate, which isolates a mechanism of degeneration under row-wise softmax normalization. We derive asymptotic singular-value bounds and a non-asymptotic eigenvalue characterization, and extend the analysis to causal attention and generalized softmax attention with alternative activations.

Main result

Setup

Let X∈Rℓ×dX \in\mathbb{R}^{\ell\times d} be the input sequence matrix, where ℓ\ell is the context length and dd is the hidden dimension. Let WQ,WK∈Rd×dqkW_Q, W_K \in\mathbb{R}^{d \times d_{qk}} be the query and key projection matrices. We define Q=XWQQ = XW_Q, K=XWKK = XW_K, and the (scaled) score matrix S=QKT/dqk∈Rℓ×ℓS = QK^{\mathsf{T}}/\sqrt{d_{qk}} \in\mathbb{R}^{\ell\times\ell}. The attention matrix A∈Rℓ×ℓA \in\mathbb{R}^{\ell\times\ell} is the row stochastic matrix given by the row-wise softmax:

Aij=exp⁡(Sij)∑k=1ℓexp⁡(Sik)A_{ij} = \frac{\exp(S_{ij})}{\sum_{k=1}^{\ell} \exp(S_{ik})}

Token-space isotropy motivates examining the singular value decomposition S=UΣVTS = U\Sigma V^{\mathsf{T}}, with U,V∈O(ℓ)U,V \in O(\ell). We use Σ\Sigma to define the following canonical attention model.

Assumption 1 (Spectral canonical model). Let s1≥s2≥⋯≥sℓ≥0s_1 \ge s_2 \ge\cdots\ge s_\ell\ge0 denote the singular values of the pre-softmax score matrix SS, and let Σ=diag⁡(s1,…,sℓ)\Sigma= \operatorname{diag}(s_1,\ldots,s_\ell). The model attention matrices are defined by

Ab=softmax⁡(Σ),Ac=softmax⁡(Σ+M),A_b = \operatorname{softmax}(\Sigma), \qquad A_c = \operatorname{softmax}(\Sigma+ M),

where MM is the causal mask defined below. We write AA for the model under discussion. In the causal model, the singular values are placed in descending diagonal order.

Assumption 2 (Spectral separation of Σ\Sigma). Let s1≥s2≥⋯≥sℓ≥0s_1 \ge s_2 \ge\cdots\ge s_\ell\ge0 be the diagonal entries of Σ\Sigma. There exist an integer k=k(ℓ)∈{0,…,ℓ}k = k(\ell) \in\{0,\ldots,\ell\} and functions f(ℓ),g(ℓ)→∞f(\ell), g(\ell) \to\infty, with g(ℓ)=o(ℓ)g(\ell) = o(\ell), such that, for all sufficiently large ℓ\ell:

  1. The spikes (i≤ki \le k): The kk spike entries lie above the logarithmic normalization scale by a diverging margin, with esi≥ℓf(ℓ)e^{s_i} \ge\ell f(\ell), and hence

si=Ω(log⁡(ℓf(ℓ))).s_i = \Omega(\log(\ell f(\ell))).
  1. The bulk (i>ki > k): The remaining ℓ−k\ell-k bulk entries lie below the same scale by a diverging margin, with esi≤ℓ/g(ℓ)e^{s_i} \le\ell/g(\ell), and hence

si=O(log⁡(ℓ/g(ℓ))).s_i = O(\log(\ell/g(\ell))).

The singular spectrum exhibits a strict gap between the bulk and the spikes, with no singular value sis_i lying between the two thresholds.

The logarithmic threshold comes from the softmax normalization: Aii=esi/(esi+ℓ−1)A_{ii} = e^{s_i}/(e^{s_i}+\ell-1). The diagonal weight esie^{s_i} competes with ℓ−1\ell-1 unit off-diagonal weights, placing the transition between averaging and diagonal concentration at si≈log⁡ℓs_i \approx\log\ell (Figure 1).

Figure 1. Qualitative spike–bulk separation under Assumption 2. Spike and bulk scores satisfy si≥log⁡(ℓf(ℓ))s_i \ge\log(\ell f(\ell)) and si≤log⁡(ℓ/g(ℓ))s_i \le\log(\ell/g(\ell)), respectively; the open band between them contains no score entries. The dashed curve marks the normalization scale log⁡ℓ\log\ell.

For a fixed head dimension dqkd_{qk}, the score matrix satisfies k≤rank⁡(S)≤dqkk \le\operatorname{rank}(S) \le d_{qk}. Consequently, the number of spikes is sublinear in sequence length, with k=o(ℓ)k = o(\ell).

Motivated by isotropy, the canonical model retains the score singular values and aligns the left and right singular directions with the token axes, isolating the spectral effects of softmax normalization and causal masking. Section 4 compares the model with attention in pretrained transformers.

Attention degeneration

In this subsection, A=Ab=softmax⁡(Σ)A = A_b = \operatorname{softmax}(\Sigma). Throughout the remaining analysis, we use

cℓ=kℓ,εℓ=max⁡{f(ℓ)−1,g(ℓ)−1}c_\ell= \frac{k}{\ell}, \qquad\varepsilon_\ell= \max\{f(\ell)^{-1},g(\ell)^{-1}\}

for the spike fraction and the bidirectional approximation error scale, respectively.

Theorem 1 (Attention Singular Degeneration). Under Assumptions 1 and 2, as ℓ→∞\ell\to\infty, the following hold:

  1. Singular-value profile. Define

α±(x)=1−x2±x−34x2,x∈[0,1].\alpha_{\pm}(x) = \sqrt{1-\frac{x}{2} \pm\sqrt{x-\frac{3}{4}x^2}}, \qquad x \in[0,1].

For 1≤k≤ℓ1 \le k \le\ell, the ordered deterministic profile is

(σ1∗,…,σℓ∗)=(α+(cℓ),1,…,1⏟k−1,α−(cℓ),0,…,0⏟ℓ−k−1).(\sigma_1^*,\ldots,\sigma_\ell^*) = \left(\alpha_+(c_\ell),\underbrace{1,\ldots,1}_{k-1},\alpha_-(c_\ell),\underbrace{0,\ldots,0}_{\ell-k-1}\right).

The endpoint profiles are (1,0,…,0)(1,0,\ldots,0) for k=0k=0 and (1,…,1)(1,\ldots,1) for k=ℓk=\ell. The singular values of AA satisfy the uniform approximation

max⁡1≤i≤ℓ∣σi(A)−σi∗∣=O(εℓ).\max_{1 \le i \le\ell} \left|\sigma_i(A)-\sigma_i^*\right| = O(\varepsilon_\ell).

In particular, σi(A)=O(εℓ)→0\sigma_i(A) = O(\varepsilon_\ell) \to0 for i≥k+2i \ge k+2.

  1. Sublinear spikes. If k=o(ℓ)k = o(\ell), the leading k+1k+1 singular values converge uniformly to 11, with

max⁡1≤i≤k+1∣σi(A)−1∣=O(εℓ+cℓ)→0.\max_{1 \le i \le k+1} \left|\sigma_i(A)-1\right| = O(\varepsilon_\ell+\sqrt{c_\ell}) \to0.
  1. Spectral degeneration. If cℓ→c∈[0,1]c_\ell\to c \in[0,1], the empirical singular-value distribution μASV:=ℓ−1∑i=1ℓδσi(A)\mu_A^{\mathrm{SV}} := \ell^{-1}\sum_{i=1}^{\ell}\delta_{\sigma_i(A)} converges weakly to cδ1+(1−c)δ0c\delta_1+(1-c)\delta_0.

The proof of Theorem 1 gives the exact diagonal-plus-rank-one decomposition

A=D+z1T,(1)A = D + z1^{\mathsf T}, \tag*{(1)}

References

  1. [1]Blake Bordelon, Hamza Chaudhry, and Cengiz Pehlevan. Infinite limits of multi-head transformer dynamics. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 35824–35878. Curran Associates, Inc., 2024. doi: 10.52202/079017-1130. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/3eff068e195daace49955348de9f8398-Paper-Conference.pdf.DOI
  2. [2]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models Are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfc b4967418bfb8ac142f64a-Paper.pdf.arxiv.org/abs/2005.14165
  3. [3]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. doi: 10.1109/CVPR.2009.5206848.DOI
  4. [4]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423/.DOI
  5. [5]Persi Diaconis and Mehrdad Shahshahani. On the eigenvalues of random matrices. Journal of Applied Probability, 31:49–62, 1994. ISSN 00219002. URL http://www.jstor.org/stable/3214948.DOI
  6. [6]Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2793–2803. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/dong21a.html.arxiv.org/abs/2103.03404
  7. [7]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
  8. [8]Jianyang Gao and Cheng Long. RaBiT: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search. In Proceedings of the ACM on Management of Data, volume 2, New York, NY, USA, May 2024. Association for Computing Machinery. doi: 10.1145/3654970. URL https://doi.org/10.1145/3654970.

Paper details

vlnc:W_bkkwwd5w
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents