Introduction
Transformers have become the dominant architecture in modern machine learning, powering large language models and state-of-the-art systems in vision, speech, and multimodal learning. Introduced in the seminal work “Attention Is All You Need ” [Vaswani et al., 2017], the transformer replaces recurrence and convolution with a simple yet powerful mechanism: self-attention .
Given query and key matrices Q Q Q , K K K , self-attention computes a score matrix S = Q K T / d q k S = QK^{\mathsf{T}}/\sqrt{d_{qk}} S = Q K T / d q k , followed by a row-wise softmax normalization A = softmax ( S ) A = \operatorname{softmax}(S) A = softmax ( S ) , which defines how information is aggregated across tokens. This mechanism is at the core of language models such as BERT Devlin et al., [4 ] ; Liu et al. (2019) and GPT-style architectures Radford et al., (2018, 2019); Brown et al. [2 ] and vision models such as Vision Transformers (ViT) Dosovitskiy et al., [7 ] , enabling scalable modeling of long-range dependencies.
Despite its empirical success, the theoretical understanding of attention remains limited, especially in the trained regime . A central open question is to characterize the structure of the attention matrix A A A , which governs information flow and expressivity Bordelon et al., [1 ] ; Sakai et al. (2025). Recent studies Dong et al., [6 ] ; Noci et al. (2022); Nait Saada et al. (2025); Hayase et al. (2026); Sanyal et al. (2026) suggest that attention matrices often exhibit degeneration phenomena , where they become close to low-rank or even rank-one structures in deep layers. Understanding the origin of this degeneration is critical for explaining both the strengths and limitations of transformer models.
In this paper, we propose a new perspective based on the geometry of the query and key matrices. Motivated by isotropy diagnostics in trained representations, we study a canonical spectral model that isolates a mechanism of attention degeneration under softmax normalization. Specifically, we hypothesize that in trained transformers, the query and key matrices exhibit an approximate tokenspace orthogonal invariance , meaning that their token-space singular vectors behave as if they were drawn from a Haar distribution.
This is analogous to the feature-space rotational invariance exploited in RaBitQ [8 ] (Gao et al. 2025) and TurboQuant [Zandieh et al., 2026], where random rotations in feature space support efficient quantization. A shared orthogonal right rotation, ( Q , K ) ↦ ( Q R , K R ) (Q, K) \mapsto(QR, KR) ( Q , K ) ↦ ( QR , K R ) , preserves the score matrix S S S . In contrast, left rotation Q ↦ U Q Q \mapsto UQ Q ↦ U Q acts on the token dimension and does not preserve S S S in the deterministic way, but instead characterizes its statistical structure.
In high dimension this isotropic picture has a concrete implication for the entries S i j = ⟨ q i , k j ⟩ / d q d k S_{ij} = \langle q_i, k_j \rangle/ \sqrt{d_q d_k} S ij = ⟨ q i , k j ⟩ / d q d k , where q i q_i q i and k j k_j k j are token-level query and key vectors. When distinct-token directions are approximately independent and isotropic, their cosine similarities satisfy ⟨ q i , k j ⟩ / ( ∥ q i ∥ 2 ∥ k j ∥ 2 ) → 0 \langle q_i, k_j \rangle/ (\lVert q_i\rVert_2 \lVert k_j\rVert_2) \to0 ⟨ q i , k j ⟩ / (∥ q i ∥ 2 ∥ k j ∥ 2 ) → 0 for i ≠ j i \ne j i = j , in probability as the feature dimension grows. This concentration describes angular geometry rather than the magnitude of the scaled scores, which also depends on query and key norms. It motivates studying score spectra separately from their orientation. In our spectral surrogate model, we keep the singular values and replace the token-space orientation by the canonical representative Σ \Sigma Σ .
Empirical observation. We support this hypothesis with empirical evidence from both vision and language models: a pretrained Vision Transformer evaluated on Imagenette [Howard, 2019], and GPT-2 evaluated on Wikitext-103. Across these settings, the token-space singular matrices exhibit spectral behavior consistent with Haar-distributed orthogonal matrices. In particular, their eigen-angle distributions are close to uniform on the unit circle, consistent with one spectral consequence of token-space isotropy. Details are discussed in Section 4.
From geometry to degeneration. Motivated by high-dimensional isotropy, we analyze softmax ( Σ ) \operatorname{softmax}(\Sigma) softmax ( Σ ) and its causal counterpart, where the diagonal surrogate Σ \Sigma Σ contains the singular values of S S S . We assume that bulk values lie below log ℓ \log\ell log ℓ and spikes above it, with diverging additive margins.
In the bidirectional model, softmax normalization makes bulk rows asymptotically uniform, while spike weights dominate the normalization and concentrate rows on canonical basis vectors. When the number of spikes satisfies k = o ( ℓ ) k = o(\ell) k = o ( ℓ ) , the leading k + 1 k + 1 k + 1 singular values and eigenvalues converge to 1 1 1 , while the rest vanish. We extend this spike–bulk analysis to causal attention and generalized row-normalized attention maps.
Main contributions. Our contributions can be summarized as follows:
We formulate a canonical diagonal score model motivated by token-space isotropy and provide empirical diagnostics of token-space geometry in pretrained transformers.
We characterize the eigenvalue and singular-value spectra of canonical bidirectional attention and show that, under spike–bulk separation, their empirical distributions concentrate on { 0 , 1 } \{0,1\} { 0 , 1 } . Our results include a non-asymptotic characterization of the ordered eigenvalues and spectral profiles with explicit error bounds.
To the best of our knowledge, we provide the first joint characterization of eigenvalue and singular-value degeneration in causal attention. Under the canonical spike–bulk score model, we derive exact eigenvalues and quantitative singular-value bounds, showing that bounded bulk scores yield harmonic bulk eigenvalue decay and at most logarithmically many bulk singular values above any fixed positive threshold.
Related work
Collapse in depth. Dong et al. [2021], Noci et al. [2022] show that repeated attention in pure attention networks progressively compresses representations, producing doubly exponential rank collapse with depth. These results characterize the cumulative effect of composing attention layers.
Collapse in width. In a different regime, Nait Saada et al. [2025] analyze attention matrices in the width limit by modeling the score matrix S S S as an i.i.d. random matrix. They show that the softmax attention matrix behaves like a random Markov matrix and exhibits rank collapse driven by a spectral gap. However, this analysis relies on the strong i.i.d. assumption on S S S , which does not reflect the structured nature of trained transformers.
Hayase et al. [2026] study the spectrum of attention matrices at initialization through a Gaussian equivalence framework. Unlike the i.i.d. model of Nait Saada et al. [2025], their analysis accounts for the structured dependence in the score matrix S S S , which arises even at initialization from the shared input and weight matrices. They provide a random matrix characterization of attention in the infinite-width limit, capturing these correlations. However, their results remain restricted to randomly initialized networks and do not address the trained regime.
Token-space isotropy observed in both Gaussian-initialized and trained transformers motivates our canonical diagonal score surrogate, which isolates a mechanism of degeneration under row-wise softmax normalization. We derive asymptotic singular-value bounds and a non-asymptotic eigenvalue characterization, and extend the analysis to causal attention and generalized softmax attention with alternative activations.
Main result
Setup
Let X ∈ R ℓ × d X \in\mathbb{R}^{\ell\times d} X ∈ R ℓ × d be the input sequence matrix, where ℓ \ell ℓ is the context length and d d d is the hidden dimension. Let W Q , W K ∈ R d × d q k W_Q, W_K \in\mathbb{R}^{d \times d_{qk}} W Q , W K ∈ R d × d q k be the query and key projection matrices. We define Q = X W Q Q = XW_Q Q = X W Q , K = X W K K = XW_K K = X W K , and the (scaled) score matrix S = Q K T / d q k ∈ R ℓ × ℓ S = QK^{\mathsf{T}}/\sqrt{d_{qk}} \in\mathbb{R}^{\ell\times\ell} S = Q K T / d q k ∈ R ℓ × ℓ . The attention matrix A ∈ R ℓ × ℓ A \in\mathbb{R}^{\ell\times\ell} A ∈ R ℓ × ℓ is the row stochastic matrix given by the row-wise softmax:
A i j = exp ( S i j ) ∑ k = 1 ℓ exp ( S i k ) A_{ij} = \frac{\exp(S_{ij})}{\sum_{k=1}^{\ell} \exp(S_{ik})} A ij = ∑ k = 1 ℓ exp ( S ik ) exp ( S ij )
Token-space isotropy motivates examining the singular value decomposition S = U Σ V T S = U\Sigma V^{\mathsf{T}} S = U Σ V T , with U , V ∈ O ( ℓ ) U,V \in O(\ell) U , V ∈ O ( ℓ ) . We use Σ \Sigma Σ to define the following canonical attention model.
Assumption 1 (Spectral canonical model). Let s 1 ≥ s 2 ≥ ⋯ ≥ s ℓ ≥ 0 s_1 \ge s_2 \ge\cdots\ge s_\ell\ge0 s 1 ≥ s 2 ≥ ⋯ ≥ s ℓ ≥ 0 denote the singular values of the pre-softmax score matrix S S S , and let Σ = diag ( s 1 , … , s ℓ ) \Sigma= \operatorname{diag}(s_1,\ldots,s_\ell) Σ = diag ( s 1 , … , s ℓ ) . The model attention matrices are defined by
A b = softmax ( Σ ) , A c = softmax ( Σ + M ) , A_b = \operatorname{softmax}(\Sigma), \qquad A_c = \operatorname{softmax}(\Sigma+ M), A b = softmax ( Σ ) , A c = softmax ( Σ + M ) , where M M M is the causal mask defined below. We write A A A for the model under discussion. In the causal model, the singular values are placed in descending diagonal order.
Assumption 2 (Spectral separation of Σ \Sigma Σ ). Let s 1 ≥ s 2 ≥ ⋯ ≥ s ℓ ≥ 0 s_1 \ge s_2 \ge\cdots\ge s_\ell\ge0 s 1 ≥ s 2 ≥ ⋯ ≥ s ℓ ≥ 0 be the diagonal entries of Σ \Sigma Σ . There exist an integer k = k ( ℓ ) ∈ { 0 , … , ℓ } k = k(\ell) \in\{0,\ldots,\ell\} k = k ( ℓ ) ∈ { 0 , … , ℓ } and functions f ( ℓ ) , g ( ℓ ) → ∞ f(\ell), g(\ell) \to\infty f ( ℓ ) , g ( ℓ ) → ∞ , with g ( ℓ ) = o ( ℓ ) g(\ell) = o(\ell) g ( ℓ ) = o ( ℓ ) , such that, for all sufficiently large ℓ \ell ℓ :
The spikes (i ≤ k i \le k i ≤ k ): The k k k spike entries lie above the logarithmic normalization scale by a diverging margin, with e s i ≥ ℓ f ( ℓ ) e^{s_i} \ge\ell f(\ell) e s i ≥ ℓ f ( ℓ ) , and hence
s i = Ω ( log ( ℓ f ( ℓ ) ) ) . s_i = \Omega(\log(\ell f(\ell))). s i = Ω ( log ( ℓ f ( ℓ ))) . The bulk (i > k i > k i > k ): The remaining ℓ − k \ell-k ℓ − k bulk entries lie below the same scale by a diverging margin, with e s i ≤ ℓ / g ( ℓ ) e^{s_i} \le\ell/g(\ell) e s i ≤ ℓ / g ( ℓ ) , and hence
s i = O ( log ( ℓ / g ( ℓ ) ) ) . s_i = O(\log(\ell/g(\ell))). s i = O ( log ( ℓ / g ( ℓ ))) . The singular spectrum exhibits a strict gap between the bulk and the spikes, with no singular value s i s_i s i lying between the two thresholds.
The logarithmic threshold comes from the softmax normalization: A i i = e s i / ( e s i + ℓ − 1 ) A_{ii} = e^{s_i}/(e^{s_i}+\ell-1) A ii = e s i / ( e s i + ℓ − 1 ) . The diagonal weight e s i e^{s_i} e s i competes with ℓ − 1 \ell-1 ℓ − 1 unit off-diagonal weights, placing the transition between averaging and diagonal concentration at s i ≈ log ℓ s_i \approx\log\ell s i ≈ log ℓ (Figure 1 ).
Figure 1. Qualitative spike–bulk separation under Assumption 2. Spike and bulk scores satisfy s i ≥ log ( ℓ f ( ℓ ) ) s_i \ge\log(\ell f(\ell)) s i ≥ log ( ℓ f ( ℓ )) and s i ≤ log ( ℓ / g ( ℓ ) ) s_i \le\log(\ell/g(\ell)) s i ≤ log ( ℓ / g ( ℓ )) , respectively; the open band between them contains no score entries. The dashed curve marks the normalization scale log ℓ \log\ell log ℓ .
For a fixed head dimension d q k d_{qk} d q k , the score matrix satisfies k ≤ rank ( S ) ≤ d q k k \le\operatorname{rank}(S) \le d_{qk} k ≤ rank ( S ) ≤ d q k . Consequently, the number of spikes is sublinear in sequence length, with k = o ( ℓ ) k = o(\ell) k = o ( ℓ ) .
Motivated by isotropy, the canonical model retains the score singular values and aligns the left and right singular directions with the token axes, isolating the spectral effects of softmax normalization and causal masking. Section 4 compares the model with attention in pretrained transformers.
Attention degeneration
In this subsection, A = A b = softmax ( Σ ) A = A_b = \operatorname{softmax}(\Sigma) A = A b = softmax ( Σ ) . Throughout the remaining analysis, we use
c ℓ = k ℓ , ε ℓ = max { f ( ℓ ) − 1 , g ( ℓ ) − 1 } c_\ell= \frac{k}{\ell}, \qquad\varepsilon_\ell= \max\{f(\ell)^{-1},g(\ell)^{-1}\} c ℓ = ℓ k , ε ℓ = max { f ( ℓ ) − 1 , g ( ℓ ) − 1 }
for the spike fraction and the bidirectional approximation error scale, respectively.
Theorem 1 (Attention Singular Degeneration). Under Assumptions 1 and 2, as ℓ → ∞ \ell\to\infty ℓ → ∞ , the following hold:
Singular-value profile. Define
α ± ( x ) = 1 − x 2 ± x − 3 4 x 2 , x ∈ [ 0 , 1 ] . \alpha_{\pm}(x) = \sqrt{1-\frac{x}{2} \pm\sqrt{x-\frac{3}{4}x^2}}, \qquad x \in[0,1]. α ± ( x ) = 1 − 2 x ± x − 4 3 x 2 , x ∈ [ 0 , 1 ] . For 1 ≤ k ≤ ℓ 1 \le k \le\ell 1 ≤ k ≤ ℓ , the ordered deterministic profile is
( σ 1 ∗ , … , σ ℓ ∗ ) = ( α + ( c ℓ ) , 1 , … , 1 ⏟ k − 1 , α − ( c ℓ ) , 0 , … , 0 ⏟ ℓ − k − 1 ) . (\sigma_1^*,\ldots,\sigma_\ell^*) = \left(\alpha_+(c_\ell),\underbrace{1,\ldots,1}_{k-1},\alpha_-(c_\ell),\underbrace{0,\ldots,0}_{\ell-k-1}\right). ( σ 1 ∗ , … , σ ℓ ∗ ) = α + ( c ℓ ) , k − 1 1 , … , 1 , α − ( c ℓ ) , ℓ − k − 1 0 , … , 0 . The endpoint profiles are ( 1 , 0 , … , 0 ) (1,0,\ldots,0) ( 1 , 0 , … , 0 ) for k = 0 k=0 k = 0 and ( 1 , … , 1 ) (1,\ldots,1) ( 1 , … , 1 ) for k = ℓ k=\ell k = ℓ . The singular values of A A A satisfy the uniform approximation
max 1 ≤ i ≤ ℓ ∣ σ i ( A ) − σ i ∗ ∣ = O ( ε ℓ ) . \max_{1 \le i \le\ell} \left|\sigma_i(A)-\sigma_i^*\right| = O(\varepsilon_\ell). 1 ≤ i ≤ ℓ max ∣ σ i ( A ) − σ i ∗ ∣ = O ( ε ℓ ) . In particular, σ i ( A ) = O ( ε ℓ ) → 0 \sigma_i(A) = O(\varepsilon_\ell) \to0 σ i ( A ) = O ( ε ℓ ) → 0 for i ≥ k + 2 i \ge k+2 i ≥ k + 2 .
Sublinear spikes. If k = o ( ℓ ) k = o(\ell) k = o ( ℓ ) , the leading k + 1 k+1 k + 1 singular values converge uniformly to 1 1 1 , with
max 1 ≤ i ≤ k + 1 ∣ σ i ( A ) − 1 ∣ = O ( ε ℓ + c ℓ ) → 0. \max_{1 \le i \le k+1} \left|\sigma_i(A)-1\right| = O(\varepsilon_\ell+\sqrt{c_\ell}) \to0. 1 ≤ i ≤ k + 1 max ∣ σ i ( A ) − 1 ∣ = O ( ε ℓ + c ℓ ) → 0. Spectral degeneration. If c ℓ → c ∈ [ 0 , 1 ] c_\ell\to c \in[0,1] c ℓ → c ∈ [ 0 , 1 ] , the empirical singular-value distribution μ A S V : = ℓ − 1 ∑ i = 1 ℓ δ σ i ( A ) \mu_A^{\mathrm{SV}} := \ell^{-1}\sum_{i=1}^{\ell}\delta_{\sigma_i(A)} μ A SV := ℓ − 1 ∑ i = 1 ℓ δ σ i ( A ) converges weakly to c δ 1 + ( 1 − c ) δ 0 c\delta_1+(1-c)\delta_0 c δ 1 + ( 1 − c ) δ 0 .
The proof of Theorem 1 gives the exact diagonal-plus-rank-one decomposition
A = D + z 1 T , (1) A = D + z1^{\mathsf T}, \tag*{(1)} A = D + z 1 T , (1)
where 1 ∈ R ℓ 1 \in\mathbb{R}^{\ell} 1 ∈ R ℓ is the all-ones vector, Z i = e s i + ℓ − 1 Z_i=e^{s_i}+\ell-1 Z i = e s i + ℓ − 1 is the row partition function, z = ( z 1 , … , z ℓ ) T z=(z_1,\ldots,z_\ell)^{\mathsf T} z = ( z 1 , … , z ℓ ) T with z i = Z i − 1 z_i=Z_i^{-1} z i = Z i − 1 , and D = diag ( d 1 , … , d ℓ ) D=\operatorname{diag}(d_1,\ldots,d_\ell) D = diag ( d 1 , … , d ℓ ) with d i = 1 − ℓ z i d_i=1-\ell z_i d i = 1 − ℓ z i .
When a bulk is present, all bulk rows perform the same averaging operation in the limit and therefore contribute a single direction. Together with the k k k directions retained by the spike rows, this shared averaging direction forms a ( k + 1 ) (k+1) ( k + 1 ) -dimensional subspace.
From decomposition (1), we derive a closed-form characteristic equation and show that the canonical attention matrix has real eigenvalues despite its asymmetry.
Theorem 2 (Attention Eigen Degeneration). Under Assumptions 1 and 2, write d 1 ≥ ⋯ ≥ d ℓ d_1 \ge\cdots\ge d_\ell d 1 ≥ ⋯ ≥ d ℓ for the diagonal entries of D D D .
Exact spectral structure. For every ℓ \ell ℓ , the eigenvalues λ 1 ( A ) ≥ ⋯ ≥ λ ℓ ( A ) \lambda_1(A) \ge\cdots\ge\lambda_\ell(A) λ 1 ( A ) ≥ ⋯ ≥ λ ℓ ( A ) are real and lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] , with λ 1 ( A ) = 1 \lambda_1(A)=1 λ 1 ( A ) = 1 . For 2 ≤ i ≤ ℓ 2 \le i \le\ell 2 ≤ i ≤ ℓ , they satisfy
{ λ i ( A ) = d i , if d i = d i − 1 , d i < λ i ( A ) < d i − 1 , if d i < d i − 1 . \begin{cases}
\lambda_i(A)=d_i, & \text{if } d_i=d_{i-1},\\
d_i<\lambda_i(A)<d_{i-1}, & \text{if } d_i<d_{i-1}.
\end{cases} { λ i ( A ) = d i , d i < λ i ( A ) < d i − 1 , if d i = d i − 1 , if d i < d i − 1 . For ℓ > 1 \ell>1 ℓ > 1 , this also gives the bounds
ℓ A i i − 1 ℓ − 1 ≤ λ i ( A ) ≤ ℓ A i − 1 , i − 1 − 1 ℓ − 1 , 2 ≤ i ≤ ℓ . \frac{\ell A_{ii}-1}{\ell-1}\le\lambda_i(A)\le\frac{\ell A_{i-1,i-1}-1}{\ell-1},\qquad2\le i\le\ell. ℓ − 1 ℓ A ii − 1 ≤ λ i ( A ) ≤ ℓ − 1 ℓ A i − 1 , i − 1 − 1 , 2 ≤ i ≤ ℓ . Eigenvalue profile. As ℓ → ∞ \ell\to\infty ℓ → ∞ , the spike and bulk eigenvalues satisfy
1 − O ( f ( ℓ ) − 1 ) ≤ d i ≤ λ i ( A ) ≤ d i − 1 < 1 , 2 ≤ i ≤ k , 1-O\left(f(\ell)^{-1}\right)\le d_i\le\lambda_i(A)\le d_{i-1}<1,\qquad2\le i\le k, 1 − O ( f ( ℓ ) − 1 ) ≤ d i ≤ λ i ( A ) ≤ d i − 1 < 1 , 2 ≤ i ≤ k , 0 ≤ d i ≤ λ i ( A ) ≤ d i − 1 = O ( g ( ℓ ) − 1 ) , k + 2 ≤ i ≤ ℓ . 0\le d_i\le\lambda_i(A)\le d_{i-1}=O\left(g(\ell)^{-1}\right),\qquad k+2\le i\le\ell. 0 ≤ d i ≤ λ i ( A ) ≤ d i − 1 = O ( g ( ℓ ) − 1 ) , k + 2 ≤ i ≤ ℓ . If k < ℓ k<\ell k < ℓ , the remaining eigenvalue satisfies
λ k + 1 ( A ) = 1 − c ℓ + O ( ε ℓ ) , \lambda_{k+1}(A)=1-c_\ell+O(\varepsilon_\ell), λ k + 1 ( A ) = 1 − c ℓ + O ( ε ℓ ) , and hence converges to 1 − c 1-c 1 − c whenever c ℓ → c ∈ [ 0 , 1 ] c_\ell\to c\in[0,1] c ℓ → c ∈ [ 0 , 1 ] . In particular, if k = o ( ℓ ) k=o(\ell) k = o ( ℓ ) , the leading k + 1 k+1 k + 1 eigenvalues converge uniformly to 1.
Spectral degeneration. If c ℓ → c ∈ [ 0 , 1 ] c_\ell\to c\in[0,1] c ℓ → c ∈ [ 0 , 1 ] , the empirical eigenvalue distribution μ A e i g : = ℓ − 1 ∑ i = 1 ℓ δ λ i ( A ) \mu_A^{\mathrm{eig}}:=\ell^{-1}\sum_{i=1}^{\ell}\delta_{\lambda_i(A)} μ A eig := ℓ − 1 ∑ i = 1 ℓ δ λ i ( A ) converges weakly to c δ 1 + ( 1 − c ) δ 0 c\delta_1+(1-c)\delta_0 c δ 1 + ( 1 − c ) δ 0 .
Remark 1 (Bounded Bulk). If the bulk scores satisfy s i ≤ B s_i\le B s i ≤ B for i > k i>k i > k , with a constant B ≥ 0 B\ge0 B ≥ 0 independent of ℓ \ell ℓ , then
0 ≤ d i = e s i − 1 e s i + ℓ − 1 ≤ e B − 1 ℓ , i > k . 0\le d_i=\frac{e^{s_i}-1}{e^{s_i}+\ell-1}\le\frac{e^B-1}{\ell},\qquad i>k. 0 ≤ d i = e s i + ℓ − 1 e s i − 1 ≤ ℓ e B − 1 , i > k . Consequently, the O ( g ( ℓ ) − 1 ) O(g(\ell)^{-1}) O ( g ( ℓ ) − 1 ) contributions in Theorems 1–2 improve to O ( ℓ − 1 ) O(\ell^{-1}) O ( ℓ − 1 ) , and the O ( ε ℓ ) O(\varepsilon_\ell) O ( ε ℓ ) profile errors become O ( max { f ( ℓ ) − 1 , ℓ − 1 } ) O(\max\{f(\ell)^{-1},\ell^{-1}\}) O ( max { f ( ℓ ) − 1 , ℓ − 1 }) .
Causal attention
We now turn to causal attention, the masked variant used in decoder architectures [2 ] (Radford et al. 2018; Radford et al. 2019). The lower-triangular mask changes the asymptotic operator structure: even benign bulk scores leave a triangular averaging component rather than the same collapse seen in bidirectional attention. Throughout this subsection, A = A c = softmax ( Σ + M ) A=A_c=\operatorname{softmax}(\Sigma+M) A = A c = softmax ( Σ + M ) , where the causal mask is
M i , j = { − ∞ , for j > i , 0 , else . M_{i,j}=
\begin{cases}
-\infty, & \text{for } j>i,\\
0, & \text{else}.
\end{cases} M i , j = { − ∞ , 0 , for j > i , else .
Under Assumption 2, the first k k k token positions contain the spikes and the remaining ℓ − k \ell-k ℓ − k positions form the bulk. Since the causal matrix is lower triangular, its eigenvalues are exactly its diagonal entries. The spike eigenvalues remain close to one, while harmonic bulk decay follows once the relevant bulk scores are uniformly bounded.
Theorem 3 (Eigenvalues of Causal Attention). Under Assumptions 1 and 2, for any sequence length ℓ \ell ℓ , the eigenvalues of A A A are real and equal to its diagonal entries:
λ i ( A ) = exp ( s i ) exp ( s i ) + i − 1 . \lambda_i(A)=\frac{\exp(s_i)}{\exp(s_i)+i-1}. λ i ( A ) = exp ( s i ) + i − 1 exp ( s i ) . The largest eigenvalue is exactly λ 1 = 1 \lambda_1=1 λ 1 = 1 . As ℓ → ∞ \ell\to\infty ℓ → ∞ ,
λ i ( A ) = Θ ( 1 / i ) for i ∈ { ℓ − r + 1 , … , ℓ } , \lambda_i(A) = \Theta(1/i) \qquad\text{for } i \in\{\ell-r+1,\ldots,\ell\}, λ i ( A ) = Θ ( 1/ i ) for i ∈ { ℓ − r + 1 , … , ℓ } ,
provided that, for some r ≤ ℓ − k r \le\ell-k r ≤ ℓ − k , the last r r r bulk scores are uniformly bounded, i.e., s i = O ( 1 ) s_i = O(1) s i = O ( 1 ) for i ∈ { ℓ − r + 1 , … , ℓ } i \in\{\ell-r+1,\ldots,\ell\} i ∈ { ℓ − r + 1 , … , ℓ } . In particular, if all bulk scores s i s_i s i , i > k i > k i > k , are uniformly bounded, then r = ℓ − k r = \ell-k r = ℓ − k .
Spectral degeneration. If c ℓ → c ∈ [ 0 , 1 ] c_\ell\to c \in[0,1] c ℓ → c ∈ [ 0 , 1 ] , the empirical eigenvalue distribution of A A A converges weakly to c δ 1 + ( 1 − c ) δ 0 c\delta_1 + (1-c)\delta_0 c δ 1 + ( 1 − c ) δ 0 .
The singular values are subtler because triangularity no longer identifies them. The next result describes their spectral degeneration under spectral separation, with sharper bulk bounds when the bulk scores are uniformly bounded.
Theorem 4 (Singular Values of Causal Attention). Under Assumption 1, A A A satisfies, for every sequence length ℓ \ell ℓ ,
1 3 ℓ ≤ σ i ( A ) ≤ ∥ A ∥ o p ≤ 5 , 1 ≤ i ≤ ℓ . \frac{1}{3\ell} \le\sigma_i(A) \le\lVert A\rVert_{\mathrm{op}} \le\sqrt{5}, \qquad1 \le i \le\ell. 3 ℓ 1 ≤ σ i ( A ) ≤ ∥ A ∥ op ≤ 5 , 1 ≤ i ≤ ℓ . Under the additional spectral-separation assumption 2, the following conclusions hold:
Spike subspace. For k ≥ 1 k \ge1 k ≥ 1 and all sufficiently large ℓ \ell ℓ , there exists a ( k − 1 ) (k-1) ( k − 1 ) -dimensional subspace E s p ⊂ R ℓ E_{\mathrm{sp}} \subset\mathbb{R}^{\ell} E sp ⊂ R ℓ such that
( 1 − O ( k ℓ f ( ℓ ) ) ) ∥ v ∥ 2 ≤ ∥ A v ∥ 2 ≤ ( 1 + O ( k ℓ f ( ℓ ) ) ) ∥ v ∥ 2 for all v ∈ E s p . \left(1-O\left(\frac{k}{\ell f(\ell)}\right)\right)\lVert v\rVert_2 \le\lVert Av\rVert_2 \le\left(1+O\left(\frac{k}{\ell f(\ell)}\right)\right)\lVert v\rVert_2 \qquad\text{for all } v \in E_{\mathrm{sp}}. ( 1 − O ( ℓ f ( ℓ ) k ) ) ∥ v ∥ 2 ≤ ∥ A v ∥ 2 ≤ ( 1 + O ( ℓ f ( ℓ ) k ) ) ∥ v ∥ 2 for all v ∈ E sp . Moreover, for $k \ge 1,
σ k ( A ) ≥ 1 − O ( k ℓ f ( ℓ ) ) . \sigma_k(A) \ge1-O\left(\frac{k}{\ell f(\ell)}\right). σ k ( A ) ≥ 1 − O ( ℓ f ( ℓ ) k ) . Spectral degeneration. If c ℓ → c ∈ [ 0 , 1 ] c_\ell\to c \in[0,1] c ℓ → c ∈ [ 0 , 1 ] , the empirical singular-value distribution of A A A converges weakly to c δ 1 + ( 1 − c ) δ 0 c\delta_1 + (1-c)\delta_0 c δ 1 + ( 1 − c ) δ 0 .
If, in addition, there exists a constant B > 0 B > 0 B > 0 , independent of ℓ \ell ℓ , such that s i ≤ B s_i \le B s i ≤ B for all i > k i > k i > k , then there exists a constant C B > 0 C_B > 0 C B > 0 , depending only on B B B , such that the following further estimates hold for all sufficiently large ℓ \ell ℓ :
Bulk singular-value upper bounds. For every ordered bulk index m ∈ { 1 , … , ℓ − k } m \in\{1,\ldots,\ell-k\} m ∈ { 1 , … , ℓ − k } ,
σ k + m ( A ) ≤ C B min { 1 , ℓ − k − m + 1 k + m , log ( ℓ + 1 ) m } . \sigma_{k+m}(A) \le C_B \min\left\{1,\frac{\ell-k-m+1}{k+m},\sqrt{\frac{\log(\ell+1)}{m}}\right\}. σ k + m ( A ) ≤ C B min { 1 , k + m ℓ − k − m + 1 , m log ( ℓ + 1 ) } . Moreover, for every fixed ε > 0 \varepsilon> 0 ε > 0 , # { i > k : σ i ( A ) > ε } ≤ C B log ( ℓ + 1 ) ε 2 \#\{i > k:\sigma_i(A) > \varepsilon\} \le\frac{C_B\log(\ell+1)}{\varepsilon^2} # { i > k : σ i ( A ) > ε } ≤ ε 2 C B l o g ( ℓ + 1 ) . Consequently, for every sequence m ℓ ∈ { 1 , … , ℓ − k } m_\ell\in\{1,\ldots,\ell-k\} m ℓ ∈ { 1 , … , ℓ − k } with m ℓ / log ℓ → ∞ m_\ell/\log\ell\to\infty m ℓ / log ℓ → ∞ ,
σ k + m ℓ ( A ) → 0. \sigma_{k+m_\ell}(A) \to0. σ k + m ℓ ( A ) → 0.
Weak spectral degeneration holds under spectral separation, while the stronger logarithmic bounds use the additional bounded-bulk condition. The causal mask gives the bulk a Hardy-type lower triangular averaging structure, already visible when s i ≡ 0 s_i \equiv0 s i ≡ 0 .
Encoder versus decoder
Within the canonical model, bounded bulk scores give an O ( 1 / ℓ ) O(1/\ell) O ( 1/ ℓ ) bulk term for bidirectional attention, whereas causal attention has at most logarithmically many bulk singular values above any fixed threshold. This contrast suggests a spectral perspective on encoder and decoder representations: concentration in a few directions may favor selective aggregation, while retaining additional directions may support the range of continuations in sequential prediction.
The contrast is already visible when Σ = 0 \Sigma= 0 Σ = 0 : bidirectional attention returns the global average, while causal attention averages each prefix,
A b = 11 T ℓ , ( A c ) i j = 1 { j ≤ i } i . A_b = \frac{\mathbf{1}\mathbf{1}^{\mathsf{T}}}{\ell}, \qquad(A_c)_{ij} = \frac{\mathbf{1}_{\{j\le i\}}}{i}. A b = ℓ 1 1 T , ( A c ) ij = i 1 { j ≤ i } .
Here ∥ A b ∥ F 2 = 1 \lVert A_b\rVert_F^2 = 1 ∥ A b ∥ F 2 = 1 , whereas ∥ A c ∥ F 2 = ∑ i = 1 ℓ 1 / i ∼ log ℓ \lVert A_c\rVert_F^2 = \sum_{i=1}^{\ell} 1/i \sim\log\ell ∥ A c ∥ F 2 = ∑ i = 1 ℓ 1/ i ∼ log ℓ . Their operator norms remain of constant order: ∥ A b ∥ o p = 1 \lVert A_b\rVert_{\mathrm{op}} = 1 ∥ A b ∥ op = 1 and 1 ≤ ∥ A c ∥ o p ≤ 2 1 \le\lVert A_c\rVert_{\mathrm{op}} \le2 1 ≤ ∥ A c ∥ op ≤ 2 . Figure 2 compares their ordered singular values.
Figure 2. Ordered singular values of zero-score attention at ℓ = 512 \ell= 512 ℓ = 512 . Bidirectional attention has rank one, whereas causal prefix averaging retains a nonzero singular-value tail.
To quantify this difference more precisely, we introduce the stable rank srank ( A ) = ∥ A ∥ F 2 / ∥ A ∥ o p 2 \operatorname{srank}(A) = \lVert A\rVert_F^2/\lVert A\rVert_{\mathrm{op}}^2 srank ( A ) = ∥ A ∥ F 2 / ∥ A ∥ op 2 and entropy-based effective ranks . For a nonzero matrix A A A with singular values σ j ( A ) \sigma_j(A) σ j ( A ) , define
erank p ( A ) = exp ( − ∑ j q j ( p ) log q j ( p ) ) , q j ( p ) = σ j ( A ) p ∑ r σ r ( A ) p , p ∈ { 1 , 2 } . \operatorname{erank}_p(A) = \exp\left(-\sum_j q_j^{(p)} \log q_j^{(p)}\right), \qquad q_j^{(p)} = \frac{\sigma_j(A)^p}{\sum_r \sigma_r(A)^p}, \quad p \in\{1,2\}. erank p ( A ) = exp ( − j ∑ q j ( p ) log q j ( p ) ) , q j ( p ) = ∑ r σ r ( A ) p σ j ( A ) p , p ∈ { 1 , 2 } .
Here erank 1 \operatorname{erank}_1 erank 1 is the effective rank of Roy and Vetterli [2007], while erank 2 \operatorname{erank}_2 erank 2 uses squared singular values; we take 0 log 0 = 0 0\log0 = 0 0 log 0 = 0 .
Theorem 5 (Informal: Spectral ranks). Consider canonical bidirectional and causal attention with a fixed number k k k of spectrally separated spikes and bulk scores uniformly bounded by B B B . As ℓ → ∞ \ell\to\infty ℓ → ∞ ,
srank ( A b ) , erank 2 ( A b ) = k + 1 + o ( 1 ) , \operatorname{srank}(A_b),\ \operatorname{erank}_2(A_b) = k + 1 + o(1), srank ( A b ) , erank 2 ( A b ) = k + 1 + o ( 1 ) , srank ( A c ) , erank 2 ( A c ) = Θ B ( log ℓ ) . \operatorname{srank}(A_c),\ \operatorname{erank}_2(A_c) = \Theta_B(\log\ell). srank ( A c ) , erank 2 ( A c ) = Θ B ( log ℓ ) . Moreover, erank 1 ( A c ) = Ω B ( ( log ℓ ) 3 ) \operatorname{erank}_1(A_c) = \Omega_B((\log\ell)^3) erank 1 ( A c ) = Ω B (( log ℓ ) 3 ) , while erank 1 ( A b ) = k + 1 + o ( 1 ) \operatorname{erank}_1(A_b) = k + 1 + o(1) erank 1 ( A b ) = k + 1 + o ( 1 ) when its linear bulk mass and entropy moment vanish.
Precise conditions and bounds are given in Theorems 6 and 7 in Appendix § 10 . These bounds distinguish the two attention patterns despite their common limiting empirical spectral law δ 0 \delta_0 δ 0 . Relative to context length, the stable ranks decay as 1 / ℓ 1/\ell 1/ ℓ and log ℓ / ℓ \log\ell/\ell log ℓ / ℓ , respectively, quantifying how causal masking slows spectral degeneration as context length grows. Further technical remarks on the causal singular-value bounds are collected in Appendix § 9 .5.
Experiment
Our experiments on pretrained GPT-2 and ViT connect the spectral model to trained attention through three questions: how token-space singular factors are oriented, how score strength is distributed across singular directions, and how real and canonical attention spectra compare after softmax.
Eigen-angle diagnostics of token-space singular factors
Imagenette data We examine the pretrained ViT-B/16 model with SWAG weights [Singh et al., 2022], using the validation split of Imagenette. Each 384 × 384 384 \times384 384 × 384 image is mapped to a token matrix X ∈ R 577 × 768 X \in\mathbb{R}^{577 \times768} X ∈ R 577 × 768 at each layer. For every layer input, we compute a full singular value decomposition X = U X Σ X V X T X = U_X\Sigma_XV_X^{\mathsf{T}} X = U X Σ X V X T , and examine the square token-space singular factor U X U_X U X . Its eigenvalues lie on the unit circle, so we write λ j ( U X ) = e i θ j \lambda_j(U_X) = e^{\mathrm{i}\theta_j} λ j ( U X ) = e i θ j with θ j ∈ [ − π , π ] \theta_j \in[-\pi,\pi] θ j ∈ [ − π , π ] . Haar orthogonal matrices have asymptotically uniform empirical eigen-angles [5 Theorem 3] , giving a necessary asymptotic diagnostic for Haar behavior. We measure discrepancies from the uniform distribution using Kullback-Leibler divergence, Kolmogorov distance, and angular Wasserstein distance. Figure 7 shows that the measured discrepancies are consistently close to zero across ViT layers, supporting the isotropy motivation for the spectral canonical model in the vision setting.
Figure 7. Layer-wise discrepancies between the empirical eigen-angle distribution of the ViT token-space singular matrices and the uniform distribution on [ − π , π ] [-\pi,\pi] [ − π , π ] for Imagenette validation inputs. The log-scale box plots show that Kullback-Leibler divergence, Kolmogorov distance, and Wasserstein distance remain close to zero across layers.
Wikitext data We apply the same diagnostic to the token-space singular factor U X U_X U X of GPT-2 hidden-state inputs on the Wikitext-103 validation split [Merity et al., 2017], with sequence lengths from 17 to 471 tokens. Figure 3 shows decreasing discrepancies with sequence length, indicating increasingly uniform empirical eigen-angle distributions over the tested range.
Figure 3. Discrepancies between the GPT-2 token-space singular eigen-angle distribution and the uniform distribution on [ − π , π ] [-\pi,\pi] [ − π , π ] as a function of sequence length on the Wikitext-103 validation split. Linear- and log-scale summaries are shown for Kullback-Leibler divergence, Kolmogorov distance, and Wasserstein distance, with means, interquartile ranges, and 10–90% bands. Kullback-Leibler divergence is calculated by histogram binning with 50 bins.
Effective rank of the score matrix
Our theory allows a nonzero bulk, but the spike–bulk analysis is most informative when a few dominant singular directions capture most of the score matrix energy. We therefore measure the intrinsic rank of GPT-2 score matrices on the Wikitext-103 validation split as ℓ \ell ℓ varies. Figure 4 shows that both rank surrogates remain moderate across the tested sequence lengths rather than scaling linearly with ℓ \ell ℓ , motivating the spike–bulk structure in Assumption 2.
Figure 4. Stable rank and effective rank of GPT-2 score matrices on the Wikitext-103 validation split as a function of sequence length. Curves show the median trend, with interquartile ranges and wider percentile bands.
Real-versus-canonical attention spectra
We examine the spectral canonical model by directly comparing real attention with the attention induced by the score singular values. For each head, we construct A r e a l , h = softmax r o w ( S h + M ) A_{\mathrm{real},h} = \operatorname{softmax}_{\mathrm{row}}(S_h + M) A real , h = softmax row ( S h + M ) and A Σ , h = softmax r o w ( Σ h + M ) A_{\Sigma,h} = \operatorname{softmax}_{\mathrm{row}}(\Sigma_h + M) A Σ , h = softmax row ( Σ h + M ) , using the same mask and placing the score singular values in descending diagonal order. Here M M M is the causal mask for GPT-2 and zero for ViT. These score and attention diagnostics are computed separately for each head of the intact pretrained networks, with aggregation only after the spectral measurements are obtained (see Appendix § 12 .5 for head-resolved summaries). Measure the Wasserstein distance between their empirical singular-value distributions:
W 1 ( μ A r e a l , h S V , μ A Σ , h S V ) = 1 ℓ ∑ j = 1 ℓ ∣ σ j ( A r e a l , h ) − σ j ( A Σ , h ) ∣ . W_1\left(\mu^{\mathrm{SV}}_{A_{\mathrm{real},h}},\mu^{\mathrm{SV}}_{A_{\Sigma,h}}\right) = \frac{1}{\ell}\sum_{j=1}^{\ell}\left|\sigma_j(A_{\mathrm{real},h})-\sigma_j(A_{\Sigma,h})\right|. W 1 ( μ A real , h SV , μ A Σ , h SV ) = ℓ 1 j = 1 ∑ ℓ ∣ σ j ( A real , h ) − σ j ( A Σ , h ) ∣ .
Figure 5 shows a substantial decrease in spectral discrepancy as GPT-2 context length grows from 64 to 1,024. The median distance falls from 0.3002 to 0.0942 across all layers and from 0.3290 to 0.0748 in the last layer. At ℓ = 577 \ell= 577 ℓ = 577 , ViT has layerwise medians between 0.0367 and 0.1037, with greater variation across image–head pairs in early layers (Figure 6). These observations support bulk spectral similarity, with decreasing discrepancy over the tested GPT-2 lengths.
Rank proxies complement this distributional comparison by measuring spectral concentration. At GPT-2 length 1,024, the pooled stable-rank medians are 1.18 for real attention and 27.51 for canonical attention; the corresponding erank1 _1 1 medians are 314.06 and 95.12. Thus proximity of equally weighted spectral distributions coexists with different concentration and entropy profiles. The sampling and numerical details and the relation to leading spectral modes are discussed in Appendix § 12 .3.
Score-level geometry and spectral structure
We apply the eigen-angle diagnostics directly to the actual scores, decomposing S h = U S h Σ h V S h T S_h = U_{S_h}\Sigma_hV_{S_h}^{\mathsf{T}} S h = U S h Σ h V S h T independently for every sample, layer, and head. The analysis covers 1,646 GPT-2 sequences and 3,925 ViT images, with all 12 layers and 12 heads. For both GPT-2 factors, the mean angular W 1 W_1 W 1 decreases from approximately 0.31 in the 17–64 length band to 0.10 in the 385–471 band; the pooled ViT mean is approximately 0.08 for each factor. KL and Kolmogorov discrepancies show similar GPT-2 length trends and remain small across ViT layers (Figure 8 and Figure 9 ). These measurements extend the near-uniform eigen-angle patterns from hidden-state inputs to the computed score singular factors, supporting the isotropy motivation for Assumption 1.
Figure 8. GPT-2 actual-score eigen-angle discrepancies for the left and right singular-vector matrices. KL, Kolmogorov, and angular Wasserstein discrepancies decrease over the displayed sequence lengths.
Figure 9. ViT actual-score eigen-angle discrepancies across layers for both singular-vector factors. Each layer contains 47,100 image–head observations from 3,925 images.
Figure 10 and Figure 11 complement these geometric diagnostics with logarithmic score spectra for one input per model. Each layer pools nonzero singular values after independent per-head decomposition; an empirical-gap split highlights a high-value group above a lower bulk. Together with the modest score-rank proxies in Figure 4 , these descriptive spectra motivate a model with a few dominant score directions. The quantitative thresholds in Assumption 2 specify the asymptotic separation regime analyzed by the theory. The structural bound k ≤ rank ( S h ) ≤ d q k = 64 k \le\operatorname{rank}(S_h) \le d_q k = 64 k ≤ rank ( S h ) ≤ d q k = 64 supplies a length-independent upper bound on the number of spikes which depends on the input and learned projections.
Figure 10. GPT-2 actual score spectra for a representative 1,024-token input. Each layer pools the nonzero singular values from all 12 heads, shown on a logarithmic horizontal axis. The dashed line marks the empirical gap, with larger values colored orange.
Figure 11. ViT actual score spectra for a representative 577-token image. Each layer pools the nonzero singular values from all 12 heads. The logarithmic scale and empirical-gap coloring follow Figure 10 .
Conclusion
Despite the empirical success of self-attention, a spectral analysis of attention matrices under consistent assumptions from initialization to the trained regime has been largely missing. Our central observation is that the token-space singular factors exhibit nearly uniform empirical eigen-angle distributions across a range of Transformer-based models, including both encoder and decoder architectures, in long-sequence regimes. This empirical property motivates our spectral canonical model for the score matrix, while the observed effective ranks motivate the spike–bulk assumption on its singular values. Together these assumptions make the spectrum of the attention matrix amenable to non-asymptotic and asymptotic analysis that characterizes degeneration, or rank collapse, within the canonical model. The theory provides a model mechanism for spectral compression observed in practice, offering one perspective on why this architecture has been so effective in LLMs.
Future work could study how alignment of dominant token subspaces across heads and the value and output projections shape the effective dimension of multi-head attention, including the effects of residual connections and normalization. Isotropy and spike–bulk diagnostics could also test for analogous spectral compression in the input-dependent recurrences of selective state-space models such as Mamba [Gu and Dao, 2024]. This would support comparisons with attention at the level of their learned sequence operators.
Acknowledgments and Disclosure of Funding
We thank the reviewers for their constructive remarks. Jianfeng Yao’s research is supported by NSFC Grant RFIS-A10120230617, the Guangdong Pearl River Talent Program Grant 2023JC10X022, and the Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023B1212010001). The authors declare no competing interests.
Scope of the canonical reduction
The diagonal surrogate is an additional modeling choice. Neither score singular values nor Haar-distributed singular-vector orientations alone justify replacing the attention matrix by its canonical counterpart. The distinction matters both before and after applying a causal mask.
Equality in law and orientation averaging. The statement S = d Σ S \mathrel{\overset{d}{=}} \Sigma S = d Σ , with Σ \Sigma Σ diagonal and nonnegative, would force S S S to be diagonal and nonnegative almost surely. It cannot express a nontrivial bi-orthogonally invariant score law. In particular, if S = U Σ V ⊤ S = U\Sigma V^{\top} S = U Σ V ⊤ , where U U U , V V V are independent Haar orthogonal matrices independent of Σ \Sigma Σ , the average over orientations is not Σ \Sigma Σ . When integrable, E [ S ∣ Σ ] = 0 \mathbb{E}[S \mid\Sigma] = 0 E [ S ∣ Σ ] = 0 by sign symmetry. For both attention patterns, the conditional means are
E [ softmax ( S ) ∣ Σ ] = ℓ − 1 11 ⊤ , E [ softmax ( S + M ) ∣ Σ ] i j = 1 { j ≤ i } i . \mathbb{E}[\operatorname{softmax}(S) \mid\Sigma] = \ell^{-1}\mathbf{1}\mathbf{1}^{\top}, \qquad\mathbb{E}[\operatorname{softmax}(S + M) \mid\Sigma]_{ij} = \frac{\mathbf{1}\{j \le i\}}{i}. E [ softmax ( S ) ∣ Σ ] = ℓ − 1 1 1 ⊤ , E [ softmax ( S + M ) ∣ Σ ] ij = i 1 { j ≤ i } .
Indeed, conditional invariance under column permutations makes the expected weights equal within each row’s unmasked positions, and each row sums to one. For a causal row, use permutations of its first i i i columns. These identities require only conditional column-permutation invariance, and concern mean attention matrices, not expected singular values or effective ranks.
A score-spectrum counterexample. For ℓ ≥ 2 \ell\ge2 ℓ ≥ 2 , let
Σ = t ℓ e 1 e 1 ⊤ , S = t ℓ ℓ 11 ⊤ , t ℓ = 2 log ℓ . \Sigma= t_{\ell}e_{1}e_{1}^{\top}, \qquad S = \frac{t_{\ell}}{\ell}\mathbf{1}\mathbf{1}^{\top}, \qquad t_{\ell} = 2\log\ell. Σ = t ℓ e 1 e 1 ⊤ , S = ℓ t ℓ 1 1 ⊤ , t ℓ = 2 log ℓ .
Both score matrices have singular values ( t ℓ , 0 , … , 0 ) (t_{\ell},0,\ldots,0) ( t ℓ , 0 , … , 0 ) , satisfying Assumption 2 with one spike. Every row of S S S is constant, so softmax ( S ) = ℓ − 1 11 ⊤ \operatorname{softmax}(S) = \ell^{-1}\mathbf{1}\mathbf{1}^{\top} softmax ( S ) = ℓ − 1 1 1 ⊤ has rank one. In contrast, Theorem 1 gives two singular values tending to one for softmax ( Σ ) \operatorname{softmax}(\Sigma) softmax ( Σ ) . Thus score singular values alone do not determine even the limiting number of nonvanishing attention singular values. This example illustrates orientation dependence; it does not assume Haar orientations.
The causal ordering is part of the model. For ℓ = 2 \ell= 2 ℓ = 2 , causal attention with diagonal scores ( t ℓ , 0 ) (t_{\ell},0) ( t ℓ , 0 ) has eigenvalues 1 , 1 / 2 1,1/2 1 , 1/2 , whereas scores ( 0 , t ℓ ) (0,t_{\ell}) ( 0 , t ℓ ) give eigenvalues 1 , e t ℓ / ( e t ℓ + 1 ) 1,e^{t_{\ell}}/(e^{t_{\ell}}+1) 1 , e t ℓ / ( e t ℓ + 1 ) . The score singular values are identical. Consequently, placing the descending singular values along the diagonal and then applying a fixed causal mask also fixes their token positions. The causal results are conditional on that convention.
Interpretation of the diagnostics. Nearly uniform eigen-angles of a chosen SVD factor are consistent with a necessary asymptotic consequence of a Haar law; they do not establish the joint Haar law, independence of left and right factors, or independence from singular values. SVD signs and nullspace completions also affect this diagnostic. Similarly, small score rank proxies do not establish the explicit exponential thresholds of Assumption 2. Appendix F records these empirical qualifications. For example, the small Wasserstein distance between empirical spectral laws does not control their leading singular values or rank proxies. A theorem transferring the canonical results to general score matrices would need additional control of the orientation-dependent softmax error; such a transfer is not proved here.
From individual heads to multi-head representations. For n h n_h n h heads, let A h ∈ R ℓ × ℓ A_h \in\mathbb{R}^{\ell\times\ell} A h ∈ R ℓ × ℓ be the attention matrix and V h ∈ R ℓ × d v V_h \in\mathbb{R}^{\ell\times d_v} V h ∈ R ℓ × d v the value matrix of head h ∈ { 1 , … , n h } h \in\{1,\ldots,n_h\} h ∈ { 1 , … , n h } . Writing Z h = A h V h Z_h = A_hV_h Z h = A h V h and Z = [ Z 1 ∣ ⋯ ∣ Z n h ] Z = [Z_1 \mid\cdots\mid Z_{n_h}] Z = [ Z 1 ∣ ⋯ ∣ Z n h ] , the multi-head output before adding any output bias is
Y = [ A 1 V 1 ∣ ⋯ ∣ A n h V n h ] W O = Z W O , W O ∈ R n h d v × d o u t . Y = [A_1V_1 \mid\cdots\mid A_{n_h}V_{n_h}]W_O = ZW_O,\qquad W_O \in\mathbb{R}^{n_hd_v \times d_{\mathrm{out}}}. Y = [ A 1 V 1 ∣ ⋯ ∣ A n h V n h ] W O = Z W O , W O ∈ R n h d v × d out .
The token-space Gram matrices satisfy
Z Z T = ∑ h = 1 n h A h V h V h T A h T , Y Y T = Z W O W O T Z T . ZZ^{\mathsf{T}} = \sum_{h=1}^{n_h} A_hV_hV_h^{\mathsf{T}}A_h^{\mathsf{T}},\qquad YY^{\mathsf{T}} = ZW_OW_O^{\mathsf{T}}Z^{\mathsf{T}}. Z Z T = h = 1 ∑ n h A h V h V h T A h T , Y Y T = Z W O W O T Z T .
These identities connect the headwise spectral picture to the combined representation. Before the output projection, the column space is the span of the head-output column spaces: aligned subspaces can retain a common low-dimensional representation, whereas distinct subspaces can increase the dimension of the combined span. The singular values also depend on the value matrices and the weighting induced by W O W_O W O . Consequently, headwise attention spectra alone do not determine the singular spectrum of Y Y Y . The head-resolved measurements in Appendix F.5 describe variation in spectral concentration and canonical discrepancy; a theory of the combined output would additionally require a joint description of the head-output subspaces and projections. Extending this description through residual connections and normalization provides a further question at the Transformer-block level.
Generalized softmax attention
Recent work has explored alternative attention activations to reduce computational cost. The canonical results extend to positive activations followed by row normalization. We leave unnormalized attention mechanisms to complementary analyses such as [Liao et al., 2026].
Let γ : R → ( 0 , ∞ ) \gamma: \mathbb{R} \to(0,\infty) γ : R → ( 0 , ∞ ) be nondecreasing on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) . Multiplying γ \gamma γ by a positive constant does not change the normalized attention, so take γ ( 0 ) = 1 \gamma(0)=1 γ ( 0 ) = 1 . Define
A i j γ = γ ( S i j ) ∑ r = 1 ℓ γ ( S i r ) . A_{ij}^{\gamma} = \frac{\gamma(S_{ij})}{\sum_{r=1}^{\ell}\gamma(S_{ir})}. A ij γ = ∑ r = 1 ℓ γ ( S i r ) γ ( S ij ) .
For causal attention, masked entries are set to zero and only unmasked entries enter the row normalization.
Under Assumption 1, evaluate this map on Σ \Sigma Σ . Set
s ~ i = log γ ( s i ) , Σ ~ = diag ( s ~ 1 , … , s ~ ℓ ) . \widetilde{s}_i = \log\gamma(s_i),\qquad\widetilde{\Sigma} = \operatorname{diag}(\widetilde{s}_1,\ldots,\widetilde{s}_{\ell}). s i = log γ ( s i ) , Σ = diag ( s 1 , … , s ℓ ) .
Then s ~ 1 ≥ ⋯ ≥ s ~ ℓ > 0 \widetilde{s}_1 \ge\cdots\ge\widetilde{s}_{\ell} > 0 s 1 ≥ ⋯ ≥ s ℓ > 0 , and the canonical generalized attention matrices are exactly
A b γ = softmax ( Σ ~ ) , A c γ = softmax ( Σ ~ + M ) . A_b^{\gamma} = \operatorname{softmax}(\widetilde{\Sigma}),\qquad A_c^{\gamma} = \operatorname{softmax}(\widetilde{\Sigma}+M). A b γ = softmax ( Σ ) , A c γ = softmax ( Σ + M ) .
Thus every canonical spectral and rank-proxy result applies with s i s_i s i replaced by s ~ i \widetilde{s}_i s i . In particular, spectral separation becomes
γ ( s i ) ≥ ℓ f ( ℓ ) ( i ≤ k ) , γ ( s i ) ≤ ℓ / g ( ℓ ) ( i > k ) , \gamma(s_i) \ge\ell f(\ell)\quad(i \le k),\qquad\gamma(s_i) \le\ell/g(\ell)\quad(i > k), γ ( s i ) ≥ ℓ f ( ℓ ) ( i ≤ k ) , γ ( s i ) ≤ ℓ / g ( ℓ ) ( i > k ) ,
with the same conditions on f f f and g g g . Bounded-bulk conclusions require uniformly bounded γ ( s i ) \gamma(s_i) γ ( s i ) in the bulk; s i ≤ B s_i \le B s i ≤ B implies this through γ ( s i ) ≤ γ ( B ) \gamma(s_i) \le\gamma(B) γ ( s i ) ≤ γ ( B ) . The transition occurs when γ ( s i ) \gamma(s_i) γ ( s i ) is of order ℓ \ell ℓ . Neither strict monotonicity nor continuity nor an inverse activation is needed. Bounded activations cannot meet the spike threshold for any nonempty spike set, but still admit the zero-spike conclusions.
Proofs of main results
Proof of Theorem 1
Proof. Singular-value profile. Set z i = ( e s i + ℓ − 1 ) − 1 z_i=(e^{s_i}+\ell-1)^{-1} z i = ( e s i + ℓ − 1 ) − 1 , d i = 1 − ℓ z i d_i=1-\ell z_i d i = 1 − ℓ z i , D = diag ( d i ) D=\operatorname{diag}(d_i) D = diag ( d i ) , J = ℓ − 1 11 T J=\ell^{-1}11^{\mathsf{T}} J = ℓ − 1 1 1 T , and P = I − J P=I-J P = I − J . Direct row normalization gives the exact identities
A = D + z 1 T = J + D P , d i = e s i − 1 e s i + ℓ − 1 ∈ [ 0 , 1 ) . A=D+z1^{\mathsf{T}}=J+DP,\qquad d_i=\frac{e^{s_i}-1}{e^{s_i}+\ell-1}\in[0,1). A = D + z 1 T = J + D P , d i = e s i + ℓ − 1 e s i − 1 ∈ [ 0 , 1 ) .
Let D ⋆ = diag ( 1 k , 0 ℓ − k ) D^\star=\operatorname{diag}(1_k,0_{\ell-k}) D ⋆ = diag ( 1 k , 0 ℓ − k ) and A ⋆ = J + D ⋆ P A^\star=J+D^\star P A ⋆ = J + D ⋆ P . Spectral separation implies
1 − d i ≤ f ( ℓ ) − 1 ( i ≤ k ) , d i ≤ g ( ℓ ) − 1 ( i > k ) . 1-d_i\le f(\ell)^{-1}\quad(i\le k),\qquad d_i\le g(\ell)^{-1}\quad(i>k). 1 − d i ≤ f ( ℓ ) − 1 ( i ≤ k ) , d i ≤ g ( ℓ ) − 1 ( i > k ) .
Since P P P is an orthogonal projection,
∥ A − A ⋆ ∥ op = ∥ ( D − D ⋆ ) P ∥ op ≤ max { f ( ℓ ) − 1 , g ( ℓ ) − 1 } = ε ℓ . (2) \lVert A-A^\star\rVert_{\operatorname{op}}=\lVert(D-D^\star)P\rVert_{\operatorname{op}}\le\max\{f(\ell)^{-1},g(\ell)^{-1}\}=\varepsilon_\ell. \tag*{(2)} ∥ A − A ⋆ ∥ op = ∥( D − D ⋆ ) P ∥ op ≤ max { f ( ℓ ) − 1 , g ( ℓ ) − 1 } = ε ℓ . (2)
It remains only to compute the singular values of A ⋆ A^\star A ⋆ .
For 1 ≤ k < ℓ 1\le k<\ell 1 ≤ k < ℓ , its first k k k rows are coordinate vectors and its remaining rows equal ℓ − 1 1 T \ell^{-1}1^{\mathsf{T}} ℓ − 1 1 T . Thus
A ⋆ ( A ⋆ ) T = ( I k ℓ − 1 1 k × ( ℓ − k ) ℓ − 1 1 ( ℓ − k ) × k ℓ − 1 1 ( ℓ − k ) × ( ℓ − k ) ) . A^\star(A^\star)^{\mathsf{T}}=
\begin{pmatrix}
I_k & \ell^{-1}1_{k\times(\ell-k)}\\
\ell^{-1}1_{(\ell-k)\times k} & \ell^{-1}1_{(\ell-k)\times(\ell-k)}
\end{pmatrix}. A ⋆ ( A ⋆ ) T = ( I k ℓ − 1 1 ( ℓ − k ) × k ℓ − 1 1 k × ( ℓ − k ) ℓ − 1 1 ( ℓ − k ) × ( ℓ − k ) ) .
The zero-sum spike subspace has eigenvalue one with multiplicity k − 1 k-1 k − 1 , and the zero-sum bulk subspace has eigenvalue zero with multiplicity ℓ − k − 1 \ell-k-1 ℓ − k − 1 . On the remaining two-dimensional subspace, use the orthonormal basis k − 1 / 2 ( 1 k , 0 ) k^{-1/2}(1_k,0) k − 1/2 ( 1 k , 0 ) , ( ℓ − k ) − 1 / 2 ( 0 , 1 ℓ − k ) (\ell-k)^{-1/2}(0,1_{\ell-k}) ( ℓ − k ) − 1/2 ( 0 , 1 ℓ − k ) . The corresponding symmetric matrix is
( 1 c ℓ ( 1 − c ℓ ) c ℓ ( 1 − c ℓ ) 1 − c ℓ ) , \begin{pmatrix}
1 & \sqrt{c_\ell(1-c_\ell)}\\
\sqrt{c_\ell(1-c_\ell)} & 1-c_\ell
\end{pmatrix}, ( 1 c ℓ ( 1 − c ℓ ) c ℓ ( 1 − c ℓ ) 1 − c ℓ ) ,
whose eigenvalues are
a ± ( c ℓ ) = 1 − c ℓ 2 ± c ℓ − 3 4 c ℓ 2 = α ± ( c ℓ ) 2 . a_{\pm}(c_\ell)=1-\frac{c_\ell}{2}\pm\sqrt{c_\ell-\frac{3}{4}c_\ell^2}=\alpha_{\pm}(c_\ell)^2. a ± ( c ℓ ) = 1 − 2 c ℓ ± c ℓ − 4 3 c ℓ 2 = α ± ( c ℓ ) 2 .
For 0 < c ℓ < 1 0<c_\ell<1 0 < c ℓ < 1 , a + ( c ℓ ) > 1 > a − ( c ℓ ) > 0 a_+(c_\ell)>1>a_-(c_\ell)>0 a + ( c ℓ ) > 1 > a − ( c ℓ ) > 0 , so the two exceptional singular values occupy positions 1 1 1 and k + 1 k+1 k + 1 , giving the ordered profile in the theorem. The endpoint profiles follow from A ⋆ = J A^\star=J A ⋆ = J when k = 0 k=0 k = 0 and A ⋆ = I A^\star=I A ⋆ = I when k = ℓ k=\ell k = ℓ . Weyl’s inequality and (2) prove the uniform approximation, including the vanishing tail.
Sublinear spikes. If k = o ( ℓ ) k=o(\ell) k = o ( ℓ ) , then a ± ( c ℓ ) = 1 + O ( c ℓ ) a_{\pm}(c_\ell)=1+O(\sqrt{c_\ell}) a ± ( c ℓ ) = 1 + O ( c ℓ ) , so α ± ( c ℓ ) = 1 + O ( c ℓ ) \alpha_{\pm}(c_\ell)=1+O(\sqrt{c_\ell}) α ± ( c ℓ ) = 1 + O ( c ℓ ) . The other leading profile values are exactly one. Combining this with the uniform approximation gives the stated bound for all 1 ≤ i ≤ k + 1 1\le i\le k+1 1 ≤ i ≤ k + 1 . For k = 0 k=0 k = 0 , the same conclusion follows from σ 1 ⋆ = 1 \sigma_1^\star=1 σ 1 ⋆ = 1 .
Spectral degeneration. The singular profile has k + O ( 1 ) k+O(1) k + O ( 1 ) unit values, ℓ − k + O ( 1 ) \ell-k+O(1) ℓ − k + O ( 1 ) zeros, and at most two exceptional values, all uniformly bounded. Thus, if c ℓ → c ∈ [ 0 , 1 ] c_\ell\to c\in[0,1] c ℓ → c ∈ [ 0 , 1 ] , its empirical distribution converges weakly to c δ 1 + ( 1 − c ) δ 0 c\delta_1+(1-c)\delta_0 c δ 1 + ( 1 − c ) δ 0 , including the endpoint profiles k = 0 , ℓ k=0,\ell k = 0 , ℓ . The uniform singular-value approximation transfers this weak limit to μ A S V \mu_A^{\mathrm{SV}} μ A SV .
Proof of Theorem 2
Proof. Exact spectral structure. Retain D D D , J J J , P P P , z z z from the preceding proof. The case ℓ = 1 \ell=1 ℓ = 1 is immediate, so assume ℓ > 1 \ell>1 ℓ > 1 . Let u = ℓ − 1 / 2 1 u=\ell^{-1/2}1 u = ℓ − 1/2 1 and choose an orthonormal basis matrix Q ∈ R ℓ × ( ℓ − 1 ) Q\in\mathbb{R}^{\ell\times(\ell-1)} Q ∈ R ℓ × ( ℓ − 1 ) for u ⊥ u^\perp u ⊥ . In the orthonormal basis [ u , Q ] [u,Q] [ u , Q ] ,
[ u , Q ] T A [ u , Q ] = ( 1 u T D Q 0 Q T D Q ) . [u,Q]^{\mathsf{T}}A[u,Q]=
\begin{pmatrix}
1 & u^{\mathsf{T}}DQ\\
0 & Q^{\mathsf{T}}DQ
\end{pmatrix}. [ u , Q ] T A [ u , Q ] = ( 1 0 u T D Q Q T D Q ) .
Consequently the eigenvalues of A A A are one together with those of the symmetric compression Q T D Q Q^{\mathsf{T}}DQ Q T D Q . Since 0 ≤ D < I 0\le D<I 0 ≤ D < I , they are real and belong to [ 0 , 1 ] [0,1] [ 0 , 1 ] , with a simple largest eigenvalue 1 1 1 . Cauchy interlacing for a codimension-one compression gives
d i ≤ λ i ( A ) ≤ d i − 1 , 2 ≤ i ≤ ℓ . d_i\le\lambda_i(A)\le d_{i-1},\qquad2\le i\le\ell. d i ≤ λ i ( A ) ≤ d i − 1 , 2 ≤ i ≤ ℓ .
The relation A i i = ℓ − 1 + ( 1 − ℓ − 1 ) d i A_{ii}=\ell^{-1}+(1-\ell^{-1})d_i A ii = ℓ − 1 + ( 1 − ℓ − 1 ) d i yields the displayed bounds in terms of A A A ’s diagonal. For the strict alternative and characteristic equation, applying the determinant lemma to (1) gives, away from the poles d i d_i d i ,
det ( λ I − A ) = ∏ i = 1 ℓ ( λ − d i ) ( 1 − ∑ i = 1 ℓ z i λ − d i ) . (3) \det(\lambda I-A)=\prod_{i=1}^{\ell}(\lambda-d_i)\left(1-\sum_{i=1}^{\ell}\frac{z_i}{\lambda-d_i}\right). \tag*{(3)} det ( λ I − A ) = i = 1 ∏ ℓ ( λ − d i ) ( 1 − i = 1 ∑ ℓ λ − d i z i ) . (3)
Group equal diagonal values into distinct poles δ r \delta_r δ r , with multiplicities m r m_r m r and residues w r = ∑ i : d i = δ r z i > 0 w_r=\sum_{i:d_i=\delta_r}z_i>0 w r = ∑ i : d i = δ r z i > 0 . The secular factor is then 1 − ∑ r w r / ( λ − δ r ) 1-\sum_r w_r/(\lambda-\delta_r) 1 − ∑ r w r / ( λ − δ r ) . Its nonzero residue at each pole cancels exactly one of the m r m_r m r polynomial factors, so δ r \delta_r δ r has algebraic multiplicity exactly m r − 1 m_r-1 m r − 1 . The secular function is strictly increasing between poles and runs from − ∞ -\infty − ∞ to + ∞ +\infty + ∞ between adjacent distinct poles. It has one further root above the largest pole and none below the smallest. This accounts for all ℓ \ell ℓ roots and proves the stated strict interlacing alternative: equality occurs precisely when d i = d i − 1 d_i=d_{i-1} d i = d i − 1 ; otherwise λ i ( A ) \lambda_i(A) λ i ( A ) lies strictly between these two entries.
Eigenvalue profile. Compare with D ⋆ D^\star D ⋆ above. The associated ideal matrix A ⋆ A^\star A ⋆ is block lower triangular, with diagonal blocks I k I_k I k and ℓ − 1 1 ( ℓ − k ) × ( ℓ − k ) \ell^{-1}\mathbf{1}_{(\ell-k)\times(\ell-k)} ℓ − 1 1 ( ℓ − k ) × ( ℓ − k ) . For k < ℓ k<\ell k < ℓ , its ordered eigenvalue list is
1 , … , 1 ⏟ k times , 1 − c ℓ , 0 , … , 0 ⏟ ℓ − k − 1 times . \underbrace{1,\ldots,1}_{k\text{ times}},\quad1-c_\ell,\quad\underbrace{0,\ldots,0}_{\ell-k-1\text{ times}}. k times 1 , … , 1 , 1 − c ℓ , ℓ − k − 1 times 0 , … , 0 .
For k = ℓ k=\ell k = ℓ , it consists of ℓ \ell ℓ ones. Both non-Perron spectra are eigenvalues of symmetric compressions, and
∥ Q ⊤ ( D − D ⋆ ) Q ∥ o p ≤ max { f ( ℓ ) − 1 , g ( ℓ ) − 1 } = ε ℓ . \left\|Q^\top(D-D^\star)Q\right\|_{\mathrm{op}}\leq\max\{f(\ell)^{-1},g(\ell)^{-1}\}=\varepsilon_\ell. Q ⊤ ( D − D ⋆ ) Q op ≤ max { f ( ℓ ) − 1 , g ( ℓ ) − 1 } = ε ℓ .
Hermitian Weyl’s inequality therefore proves the intermediate-eigenvalue estimate uniformly for 0 ≤ k < ℓ 0\leq k<\ell 0 ≤ k < ℓ . For k = 0 k=0 k = 0 this is simply λ 1 = 1 \lambda_1=1 λ 1 = 1 . The sharper spike error O ( f ( ℓ ) − 1 ) O(f(\ell)^{-1}) O ( f ( ℓ ) − 1 ) and bulk error O ( g ( ℓ ) − 1 ) O(g(\ell)^{-1}) O ( g ( ℓ ) − 1 ) follow directly from interlacing and the bounds on d i d_i d i in the preceding proof. If k = o ( ℓ ) k=o(\ell) k = o ( ℓ ) , then c ℓ → 0 c_\ell\to0 c ℓ → 0 , and the spike and intermediate estimates together give uniform convergence of the leading k + 1 k+1 k + 1 eigenvalues to one.
Spectral degeneration. For k < ℓ k<\ell k < ℓ , the ideal eigenvalue profile differs from k k k ones and ℓ − k \ell-k ℓ − k zeros in only one entry, of empirical mass 1 / ℓ 1/\ell 1/ ℓ ; for k = ℓ k=\ell k = ℓ , the profiles coincide. Hence, if c ℓ → c ∈ [ 0 , 1 ] c_\ell\to c\in[0,1] c ℓ → c ∈ [ 0 , 1 ] , its empirical distribution converges weakly to c δ 1 + ( 1 − c ) δ 0 c\delta_1+(1-c)\delta_0 c δ 1 + ( 1 − c ) δ 0 . The same Hermitian perturbation bound gives uniform approximation of the ordered eigenvalues and therefore the asserted weak limit for μ A e i g \mu_A^{\mathrm{eig}} μ A eig .
Proof of Theorem 3
Proof. Write a i = e s i a_i=e^{s_i} a i = e s i . The canonical causal matrix is lower triangular, so its eigenvalues are its diagonal entries:
λ i ( A ) = a i a i + i − 1 = 1 − i − 1 a i + i − 1 . \lambda_i(A)=\frac{a_i}{a_i+i-1}=1-\frac{i-1}{a_i+i-1}. λ i ( A ) = a i + i − 1 a i = 1 − a i + i − 1 i − 1 .
These are already in decreasing order: a i a_i a i is nonincreasing, so ( i − 1 ) / a i (i-1)/a_i ( i − 1 ) / a i is nondecreasing. In particular, λ 1 = 1 \lambda_1=1 λ 1 = 1 . For i ≤ k i\leq k i ≤ k ,
0 ≤ 1 − λ i ( A ) ≤ i − 1 ℓ f ( ℓ ) ≤ k ℓ f ( ℓ ) . (4) 0\leq1-\lambda_i(A)\leq\frac{i-1}{\ell f(\ell)}\leq\frac{k}{\ell f(\ell)}. \tag*{(4)} 0 ≤ 1 − λ i ( A ) ≤ ℓ f ( ℓ ) i − 1 ≤ ℓ f ( ℓ ) k . (4)
which also sharpens the stated O ( 1 / f ( ℓ ) ) O(1/f(\ell)) O ( 1/ f ( ℓ )) spike error.
For spectral degeneration, put t ℓ = ℓ / g ( ℓ ) ≥ 1 t_{\ell}=\ell/g(\ell)\ge1 t ℓ = ℓ / g ( ℓ ) ≥ 1 for all sufficiently large ℓ \ell ℓ . Every bulk row satisfies λ i ( A ) ≤ min { 1 , t ℓ / i } \lambda_i(A)\le\min\{1,t_{\ell}/i\} λ i ( A ) ≤ min { 1 , t ℓ / i } , since a i + i − 1 ≥ i a_i+i-1\ge i a i + i − 1 ≥ i . Splitting the sum at ⌈ t ℓ ⌉ \lceil t_{\ell}\rceil ⌈ t ℓ ⌉ gives ∑ i = 1 ℓ min { 1 , t ℓ 2 / i 2 } ≤ 3 t ℓ \sum_{i=1}^{\ell}\min\{1,t_{\ell}^{2}/i^{2}\}\le3t_{\ell} ∑ i = 1 ℓ min { 1 , t ℓ 2 / i 2 } ≤ 3 t ℓ . Together with the spike estimate, this yields
1 ℓ ∑ i = 1 ℓ ( λ i ( A ) − 1 { i ≤ k } ) 2 ≤ k 3 ℓ 3 f ( ℓ ) 2 + 3 g ( ℓ ) = o ( 1 ) . \frac{1}{\ell}\sum_{i=1}^{\ell}\left(\lambda_i(A)-\mathbf{1}_{\{i\le k\}}\right)^2\le\frac{k^3}{\ell^3f(\ell)^2}+\frac{3}{g(\ell)}=o(1). ℓ 1 i = 1 ∑ ℓ ( λ i ( A ) − 1 { i ≤ k } ) 2 ≤ ℓ 3 f ( ℓ ) 2 k 3 + g ( ℓ ) 3 = o ( 1 ) .
The empirical eigenvalue distribution therefore approaches c ℓ δ 1 + ( 1 − c ℓ ) δ 0 c_{\ell}\delta_1+(1-c_{\ell})\delta_0 c ℓ δ 1 + ( 1 − c ℓ ) δ 0 in mean-square transport distance. If c ℓ → c ∈ [ 0 , 1 ] c_{\ell}\to c\in[0,1] c ℓ → c ∈ [ 0 , 1 ] , it therefore converges weakly to c δ 1 + ( 1 − c ) δ 0 c\delta_1+(1-c)\delta_0 c δ 1 + ( 1 − c ) δ 0 , without any bounded-bulk assumption.
Finally, for every bounded-score bulk row, 1 ≤ a i ≤ C 1\le a_i\le C 1 ≤ a i ≤ C gives 1 / i ≤ λ i ( A ) ≤ C / i 1/i\le\lambda_i(A)\le C/i 1/ i ≤ λ i ( A ) ≤ C / i , proving the harmonic bounds. This bounded-score condition cannot be dropped: if k = 0 k=0 k = 0 and a i = ℓ a_i=\sqrt{\ell} a i = ℓ for every i i i , spectral separation holds with g ( ℓ ) = ℓ g(\ell)=\sqrt{\ell} g ( ℓ ) = ℓ , but λ ℓ ( A ) = ℓ / ( ℓ + ℓ − 1 ) ≍ ℓ − 1 / 2 \lambda_{\ell}(A)=\sqrt{\ell}/(\sqrt{\ell}+\ell-1)\asymp\ell^{-1/2} λ ℓ ( A ) = ℓ / ( ℓ + ℓ − 1 ) ≍ ℓ − 1/2 . Thus the empirical eigenvalue distribution still converges to δ 0 \delta_0 δ 0 , while uniform harmonic decay fails.
Proof of Theorem 4
Proof. Write a i = e s i ≥ 1 a_i=e^{s_i}\ge1 a i = e s i ≥ 1 and z i = ( a i + i − 1 ) − 1 z_i=(a_i+i-1)^{-1} z i = ( a i + i − 1 ) − 1 . Then
A i j = { z i , j < i , a i z i , j = i , 0 , j > i . A_{ij}=
\begin{cases}
z_i, & j<i,\\
a_i z_i, & j=i,\\
0, & j>i.
\end{cases} A ij = ⎩ ⎨ ⎧ z i , a i z i , 0 , j < i , j = i , j > i .
Step 1: universal singular-value bounds. Let H ℓ H_{\ell} H ℓ be the Cesàro matrix, ( H ℓ ) i j = i − 1 1 { j ≤ i } (H_{\ell})_{ij}=i^{-1}\mathbf{1}_{\{j\le i\}} ( H ℓ ) ij = i − 1 1 { j ≤ i } , and define θ i = ( a i − 1 ) / ( a i + i − 1 ) ∈ [ 0 , 1 ] \theta_i=(a_i-1)/(a_i+i-1)\in[0,1] θ i = ( a i − 1 ) / ( a i + i − 1 ) ∈ [ 0 , 1 ] . We have the exact identity
A = diag ( θ i ) + ( I − diag ( θ i ) ) H ℓ . A=\operatorname{diag}(\theta_i)+(I-\operatorname{diag}(\theta_i))H_{\ell}. A = diag ( θ i ) + ( I − diag ( θ i )) H ℓ .
By convexity of the square and the Hardy inequality (Lemma 3),
∥ A x ∥ 2 2 ≤ ∑ i ( θ i ∣ x i ∣ 2 + ( 1 − θ i ) ∣ ( H ℓ x ) i ∣ 2 ) ≤ ∥ x ∥ 2 2 + ∥ H ℓ x ∥ 2 2 ≤ 5 ∥ x ∥ 2 2 . \|Ax\|_2^2\le\sum_i\left(\theta_i|x_i|^2+(1-\theta_i)|(H_{\ell}x)_i|^2\right)\le\|x\|_2^2+\|H_{\ell}x\|_2^2\le5\|x\|_2^2. ∥ A x ∥ 2 2 ≤ i ∑ ( θ i ∣ x i ∣ 2 + ( 1 − θ i ) ∣ ( H ℓ x ) i ∣ 2 ) ≤ ∥ x ∥ 2 2 + ∥ H ℓ x ∥ 2 2 ≤ 5∥ x ∥ 2 2 .
Thus ∥ A ∥ o p ≤ 5 \|A\|_{\mathrm{op}}\le\sqrt{5} ∥ A ∥ op ≤ 5 , even without the bounded-bulk assumption.
For the lower bound, let L r L_r L r be the r × r r\times r r × r lower triangular matrix of ones. For any nonnegative diagonal matrix Λ \Lambda Λ ,
2 x T ( L r + Λ ) x = ∥ x ∥ 2 2 + ( 1 r T x ) 2 + 2 x T Λ x ≥ ∥ x ∥ 2 2 . 2x^{\mathsf T}(L_r+\Lambda)x=\|x\|_2^2+(\mathbf{1}_r^{\mathsf T}x)^2+2x^{\mathsf T}\Lambda x\ge\|x\|_2^2. 2 x T ( L r + Λ ) x = ∥ x ∥ 2 2 + ( 1 r T x ) 2 + 2 x T Λ x ≥ ∥ x ∥ 2 2 .
Cauchy–Schwarz therefore gives
σ min ( L r + Λ ) ≥ 1 2 , ∥ ( L r + Λ ) − 1 ∥ o p ≤ 2. \sigma_{\min}(L_r+\Lambda)\ge\frac{1}{2},\qquad\|(L_r+\Lambda)^{-1}\|_{\mathrm{op}}\le2. σ m i n ( L r + Λ ) ≥ 2 1 , ∥ ( L r + Λ ) − 1 ∥ op ≤ 2.
Fix 1 ≤ j ≤ ℓ 1\le j\le\ell 1 ≤ j ≤ ℓ , and let C j C_j C j be the leading j × j j\times j j × j principal submatrix of A A A . Write
C j = D j − 1 M j , D j = diag ( a i + i − 1 ) i = 1 j , M j = L j + diag ( a i − 1 ) i = 1 j . C_j=D_j^{-1}M_j,\qquad D_j=\operatorname{diag}(a_i+i-1)_{i=1}^{j},\qquad M_j=L_j+\operatorname{diag}(a_i-1)_{i=1}^{j}. C j = D j − 1 M j , D j = diag ( a i + i − 1 ) i = 1 j , M j = L j + diag ( a i − 1 ) i = 1 j .
The preceding coercivity estimate gives ∥ M j − 1 ∥ o p ≤ 2 \|M_j^{-1}\|_{\mathrm{op}}\le2 ∥ M j − 1 ∥ op ≤ 2 . Moreover,
D j − M j = diag ( i − 1 ) i = 1 j − ( L j − I j ) = : K j . D_j-M_j=\operatorname{diag}(i-1)_{i=1}^{j}-(L_j-I_j)=:K_j. D j − M j = diag ( i − 1 ) i = 1 j − ( L j − I j ) =: K j .
The absolute row sums of K j K_j K j are at most 2 ( j − 1 ) 2(j-1) 2 ( j − 1 ) , and every absolute column sum equals j − 1 j-1 j − 1 . Hence the Schur test gives ∥ K j ∥ o p ≤ 2 ( j − 1 ) \|K_j\|_{\mathrm{op}}\le\sqrt{2}(j-1) ∥ K j ∥ op ≤ 2 ( j − 1 ) , and
∥ C j − 1 ∥ o p = ∥ I j + M j − 1 K j ∥ o p ≤ 1 + 2 2 ( j − 1 ) ≤ 3 j . \|C_j^{-1}\|_{\mathrm{op}}=\|I_j+M_j^{-1}K_j\|_{\mathrm{op}}\le1+2\sqrt{2}(j-1)\le3j. ∥ C j − 1 ∥ op = ∥ I j + M j − 1 K j ∥ op ≤ 1 + 2 2 ( j − 1 ) ≤ 3 j .
For vectors supported on the first j j j coordinates, projecting A x Ax A x onto its first j j j coordinates gives C j x C_jx C j x . The max-min principle therefore yields
σ j ( A ) ≥ σ min ( C j ) ≥ 1 3 j , 1 ≤ j ≤ ℓ . \sigma_j(A)\ge\sigma_{\min}(C_j)\ge\frac{1}{3j},\qquad1\le j\le\ell. σ j ( A ) ≥ σ m i n ( C j ) ≥ 3 j 1 , 1 ≤ j ≤ ℓ .
Together with the operator bound, this proves the universal estimates for every sequence length, using only nonnegative scores.
Now assume spectral separation as in Assumption 2. Split after the first k k k rows and columns, interpreting zero-size blocks in the usual way:
A = ( A 11 0 A 21 A 22 ) , A 21 = z ( b ) 1 k T , z ( b ) = ( z k + 1 , … , z ℓ ) T . A = \begin{pmatrix} A_{11} & 0 \\ A_{21} & A_{22} \end{pmatrix}, \qquad A_{21} = z^{(b)}\mathbf{1}_k^{\mathsf{T}}, \qquad z^{(b)} = (z_{k+1}, \ldots, z_{\ell})^{\mathsf{T}}. A = ( A 11 A 21 0 A 22 ) , A 21 = z ( b ) 1 k T , z ( b ) = ( z k + 1 , … , z ℓ ) T .
Deleting the last ℓ − k \ell-k ℓ − k columns leaves a matrix of rank at most k k k . Lemma 4 consequently gives
σ k + m ( A ) ≤ σ m ( A 22 ) , 1 ≤ m ≤ ℓ − k . \sigma_{k+m}(A) \le\sigma_m(A_{22}), \qquad1 \le m \le\ell-k. σ k + m ( A ) ≤ σ m ( A 22 ) , 1 ≤ m ≤ ℓ − k .
Step 2: spike subspace. For k ≥ 1 k \ge1 k ≥ 1 , the spike assumption gives z i ≤ 1 / ( ℓ f ( ℓ ) ) z_i \le1/(\ell f(\ell)) z i ≤ 1/ ( ℓ f ( ℓ )) for i ≤ k i \le k i ≤ k . Thus A 11 = I k + E ℓ A_{11} = I_k + E_{\ell} A 11 = I k + E ℓ , where the absolute row sums and column sums of E ℓ E_{\ell} E ℓ are at most 2 k / ( ℓ f ( ℓ ) ) 2k/(\ell f(\ell)) 2 k / ( ℓ f ( ℓ )) and k / ( ℓ f ( ℓ ) ) k/(\ell f(\ell)) k / ( ℓ f ( ℓ )) , respectively. The Schur test yields
δ ℓ : = ∥ E ℓ ∥ o p = O ( k ℓ f ( ℓ ) ) = o ( 1 ) . \delta_{\ell} := \lVert E_{\ell}\rVert_{\mathrm{op}} = O\left(\frac{k}{\ell f(\ell)}\right) = o(1). δ ℓ := ∥ E ℓ ∥ op = O ( ℓ f ( ℓ ) k ) = o ( 1 ) .
On the ( k − 1 ) (k-1) ( k − 1 ) -dimensional subspace
E s p = { ( u , 0 ) : u ∈ R k , 1 k T u = 0 } , E_{\mathrm{sp}} = \{(u,0) : u \in\mathbb{R}^{k},\ \mathbf{1}_k^{\mathsf{T}}u = 0\}, E sp = {( u , 0 ) : u ∈ R k , 1 k T u = 0 } ,
the coupling block vanishes, so A ( u , 0 ) = ( A 11 u , 0 ) A(u,0) = (A_{11}u,0) A ( u , 0 ) = ( A 11 u , 0 ) . Consequently,
( 1 − δ ℓ ) ∥ u ∥ 2 ≤ ∥ A ( u , 0 ) ∥ 2 ≤ ( 1 + δ ℓ ) ∥ u ∥ 2 . (1-\delta_{\ell})\lVert u\rVert_2 \le\lVert A(u,0)\rVert_2 \le(1+\delta_{\ell})\lVert u\rVert_2. ( 1 − δ ℓ ) ∥ u ∥ 2 ≤ ∥ A ( u , 0 ) ∥ 2 ≤ ( 1 + δ ℓ ) ∥ u ∥ 2 .
This proves the almost-isometry assertion. In fact, on the whole first k k k -coordinate subspace, ∥ A ( u , 0 ) ∥ 2 ≥ ∥ A 11 u ∥ 2 \lVert A(u,0)\rVert_2 \ge\lVert A_{11}u\rVert_2 ∥ A ( u , 0 ) ∥ 2 ≥ ∥ A 11 u ∥ 2 , and hence the max-min principle gives the slightly stronger conclusion
σ k ( A ) ≥ 1 − δ ℓ ( k ≥ 1 ) . \sigma_k(A) \ge1-\delta_{\ell} \qquad(k \ge1). σ k ( A ) ≥ 1 − δ ℓ ( k ≥ 1 ) .
Step 3: singular-value degeneration without bounded bulk. Here we use only spectral separation. Let P k = diag ( I k , 0 ) P_k = \operatorname{diag}(I_k,0) P k = diag ( I k , 0 ) and put t ℓ = ℓ / g ( ℓ ) ≥ 1 t_{\ell} = \ell/g(\ell) \ge1 t ℓ = ℓ / g ( ℓ ) ≥ 1 for all sufficiently large ℓ \ell ℓ . For a bulk row i > k i > k i > k ,
∑ j A i j 2 = i − 1 + a i 2 ( a i + i − 1 ) 2 ≤ 1 i + min { 1 , t ℓ 2 i 2 } . \sum_j A_{ij}^2 = \frac{i-1+a_i^2}{(a_i+i-1)^2} \le\frac{1}{i} + \min\left\{1,\frac{t_{\ell}^2}{i^2}\right\}. j ∑ A ij 2 = ( a i + i − 1 ) 2 i − 1 + a i 2 ≤ i 1 + min { 1 , i 2 t ℓ 2 } .
Splitting the sum at ⌈ t ℓ ⌉ \lceil t_{\ell}\rceil ⌈ t ℓ ⌉ gives
∑ i = 1 ℓ min { 1 , t ℓ 2 i 2 } ≤ 3 t ℓ . \sum_{i=1}^{\ell} \min\left\{1,\frac{t_{\ell}^2}{i^2}\right\} \le3t_{\ell}. i = 1 ∑ ℓ min { 1 , i 2 t ℓ 2 } ≤ 3 t ℓ .
Writing H n = ∑ i = 1 n i − 1 \mathcal{H}_n = \sum_{i=1}^{n} i^{-1} H n = ∑ i = 1 n i − 1 and H 0 = 0 \mathcal{H}_0=0 H 0 = 0 , the block comparison above therefore yields
# { i > k : σ i ( A ) > ε } ≤ H ℓ + 3 ℓ / g ( ℓ ) ε 2 . \#\{i>k : \sigma_i(A)>\varepsilon\} \le\frac{\mathcal{H}_{\ell}+3\ell/g(\ell)}{\varepsilon^2}. # { i > k : σ i ( A ) > ε } ≤ ε 2 H ℓ + 3 ℓ / g ( ℓ ) .
For a spike row i ≤ k i \le k i ≤ k ,
∑ j ∣ ( A − P k ) i j ∣ 2 = i ( i − 1 ) ( a i + i − 1 ) 2 ≤ i 2 ℓ 2 f ( ℓ ) 2 . \sum_j \lvert(A-P_k)_{ij}\rvert^2 = \frac{i(i-1)}{(a_i+i-1)^2} \le\frac{i^2}{\ell^2 f(\ell)^2}. j ∑ ∣( A − P k ) ij ∣ 2 = ( a i + i − 1 ) 2 i ( i − 1 ) ≤ ℓ 2 f ( ℓ ) 2 i 2 .
Consequently,
∥ A − P k ∥ F 2 ℓ ≤ k 3 ℓ 3 f ( ℓ ) 2 + H ℓ ℓ + 3 g ( ℓ ) = o ( 1 ) . \frac{\lVert A-P_k\rVert_{\mathrm{F}}^2}{\ell} \le\frac{k^3}{\ell^3 f(\ell)^2} + \frac{\mathcal{H}_{\ell}}{\ell} + \frac{3}{g(\ell)} = o(1). ℓ ∥ A − P k ∥ F 2 ≤ ℓ 3 f ( ℓ ) 2 k 3 + ℓ H ℓ + g ( ℓ ) 3 = o ( 1 ) .
Mirsky’s singular-value inequality gives
1 ℓ ∑ i = 1 ℓ ( σ i ( A ) − 1 { i ≤ k } ) 2 ≤ ∥ A − P k ∥ F 2 ℓ = o ( 1 ) . \frac{1}{\ell}\sum_{i=1}^{\ell}\left(\sigma_i(A)-\mathbf{1}_{\{i\le k\}}\right)^2 \le\frac{\lVert A-P_k\rVert_{\mathrm{F}}^2}{\ell} = o(1). ℓ 1 i = 1 ∑ ℓ ( σ i ( A ) − 1 { i ≤ k } ) 2 ≤ ℓ ∥ A − P k ∥ F 2 = o ( 1 ) .
The empirical singular-value distribution approaches the moving mixture c ℓ δ 1 + ( 1 − c ℓ ) δ 0 c_{\ell}\delta_1 + (1-c_{\ell})\delta_0 c ℓ δ 1 + ( 1 − c ℓ ) δ 0 in mean-square transport distance. If c ℓ → c ∈ [ 0 , 1 ] c_{\ell}\to c\in[0,1] c ℓ → c ∈ [ 0 , 1 ] , it therefore converges weakly to c δ 1 + ( 1 − c ) δ 0 c\delta_1 + (1-c)\delta_0 c δ 1 + ( 1 − c ) δ 0 .
For Steps 4 and 5, assume 1 ≤ a i ≤ e B 1 \le a_i \le e^B 1 ≤ a i ≤ e B for i > k i > k i > k .
Step 4: Frobenius control of the bulk. The bulk-block row associated with original row i > k i > k i > k has squared norm
i − k − 1 + a i 2 ( a i + i − 1 ) 2 ≤ i + e 2 B i 2 ≤ 1 + e 2 B i . \frac{i-k-1+a_i^2}{(a_i+i-1)^2} \le\frac{i+e^{2B}}{i^2} \le\frac{1+e^{2B}}{i}. ( a i + i − 1 ) 2 i − k − 1 + a i 2 ≤ i 2 i + e 2 B ≤ i 1 + e 2 B .
Consequently, we obtain
∥ A 22 ∥ F 2 ≤ C B ( H ℓ − H k ) ≤ C B log ( ℓ + 1 ) , σ k + m ( A ) ≤ C B H ℓ − H k m . \lVert A_{22}\rVert_F^2 \le C_B(\mathcal{H}_{\ell}-\mathcal{H}_k) \le C_B\log(\ell+1), \qquad\sigma_{k+m}(A) \le C_B\sqrt{\frac{\mathcal{H}_{\ell}-\mathcal{H}_k}{m}}. ∥ A 22 ∥ F 2 ≤ C B ( H ℓ − H k ) ≤ C B log ( ℓ + 1 ) , σ k + m ( A ) ≤ C B m H ℓ − H k .
In particular this proves the logarithmic bound in the theorem. For each ε > 0 \varepsilon> 0 ε > 0 , the same comparison gives
# { i > k : σ i ( A ) > ε } ≤ ∥ A 22 ∥ F 2 ε 2 ≤ C B log ( ℓ + 1 ) ε 2 . \#\{i > k:\sigma_i(A)>\varepsilon\} \le\frac{\lVert A_{22}\rVert_F^2}{\varepsilon^2} \le\frac{C_B\log(\ell+1)}{\varepsilon^2}. # { i > k : σ i ( A ) > ε } ≤ ε 2 ∥ A 22 ∥ F 2 ≤ ε 2 C B log ( ℓ + 1 ) .
The asserted vanishing after m ℓ ≫ log ℓ m_\ell\gg\log\ell m ℓ ≫ log ℓ follows.
Step 5: the lower-right block bound. For j > k j > k j > k , choose the ( ℓ − j + 1 ) (\ell-j+1) ( ℓ − j + 1 ) -dimensional coordinate subspace F j = span { e j , … , e ℓ } F_j=\operatorname{span}\{e_j,\ldots,e_\ell\} F j = span { e j , … , e ℓ } . By triangularity, the restriction to this subspace is the lower-right block T j T_j T j . All its nonzero entries are at most e B / j e^B/j e B / j , so coordinatewise
∣ T j x ∣ ≤ e B j L ℓ − j + 1 ∣ x ∣ . \lvert T_jx\rvert\le\frac{e^B}{j}L_{\ell-j+1}\lvert x\rvert. ∣ T j x ∣ ≤ j e B L ℓ − j + 1 ∣ x ∣ .
Using ∥ L r ∥ o p ≤ ∥ L r ∥ F ≤ r \lVert L_r\rVert_{\mathrm{op}} \le\lVert L_r\rVert_F \le r ∥ L r ∥ op ≤ ∥ L r ∥ F ≤ r and the min-max principle gives
σ j ( A ) ≤ ∥ T j ∥ o p ≤ e B ( ℓ − j + 1 ) j . \sigma_j(A) \le\lVert T_j\rVert_{\mathrm{op}} \le\frac{e^B(\ell-j+1)}{j}. σ j ( A ) ≤ ∥ T j ∥ op ≤ j e B ( ℓ − j + 1 ) .
Combining this with Steps 1 and 4 proves all upper bounds. In particular, if k < ℓ k < \ell k < ℓ , then σ ℓ ( A ) = Θ B ( ℓ − 1 ) \sigma_\ell(A)=\Theta_B(\ell^{-1}) σ ℓ ( A ) = Θ B ( ℓ − 1 ) . The proof already includes k = 0 k=0 k = 0 . If k = ℓ k=\ell k = ℓ , Step 2 applies to the whole matrix, Step 1 remains valid, and there are no bulk assertions.
Remarks on causal singular values
Therefore there exists C ^ B , k > 0 \widehat{C}_{B,k}>0 C B , k > 0 such that
∥ A ∥ F 2 ≥ ∑ p = k + 1 ℓ p − 1 ( e B + p − 1 ) 2 ≥ C ^ B , k ∑ p = k + 1 ℓ 1 p ≥ C ^ B , k 2 log ℓ \lVert A\rVert_{F}^{2}\ge\sum_{p=k+1}^{\ell}\frac{p-1}{(e^{B}+p-1)^{2}}\ge\widehat{C}_{B,k}\sum_{p=k+1}^{\ell}\frac{1}{p}\ge\frac{\widehat{C}_{B,k}}{2}\log\ell ∥ A ∥ F 2 ≥ p = k + 1 ∑ ℓ ( e B + p − 1 ) 2 p − 1 ≥ C B , k p = k + 1 ∑ ℓ p 1 ≥ 2 C B , k log ℓ
for all sufficiently large ℓ \ell ℓ .
On the other hand, if σ i ( A ) ≤ C ^ B , k / i \sigma_i(A)\le\widehat{C}_{B,k}/i σ i ( A ) ≤ C B , k / i held for all i ∈ { k + 1 , … , ℓ } i\in\{k+1,\ldots,\ell\} i ∈ { k + 1 , … , ℓ } , then
∥ A ∥ F 2 = ∑ j = 1 ℓ σ j ( A ) 2 ≤ k ∥ A ∥ o p 2 + C ^ B , k 2 ∑ i = k + 1 ℓ 1 i 2 ≤ C ~ B , k , \lVert A\rVert_{F}^{2}=\sum_{j=1}^{\ell}\sigma_j(A)^{2}\le k\lVert A\rVert_{\mathrm{op}}^{2}+\widehat{C}_{B,k}^{2}\sum_{i=k+1}^{\ell}\frac{1}{i^{2}}\le\widetilde{C}_{B,k}, ∥ A ∥ F 2 = j = 1 ∑ ℓ σ j ( A ) 2 ≤ k ∥ A ∥ op 2 + C B , k 2 i = k + 1 ∑ ℓ i 2 1 ≤ C B , k ,
which contradicts the logarithmic lower bound above. Thus the factor ℓ − i + 1 \ell-i+1 ℓ − i + 1 in the upper bound of Theorem 4 cannot, in general, be removed from the ordered bulk estimate.
Remark 5 (Spike subspace versus ordered singular values). Step 2 does not imply
∣ σ i ( A ) − 1 ∣ = O ( k ℓ f ( ℓ ) ) for 1 ≤ i ≤ k . \lvert\sigma_i(A)-1\rvert=O\left(\frac{k}{\ell f(\ell)}\right)\qquad\text{for }1\le i\le k. ∣ σ i ( A ) − 1 ∣ = O ( ℓ f ( ℓ ) k ) for 1 ≤ i ≤ k .
For k ≥ 1 k\ge1 k ≥ 1 , what Step 2 proves is that the restriction of A A A to the ( k − 1 ) (k-1) ( k − 1 ) -dimensional spike subspace E s p E_{\mathrm{sp}} E sp is almost isometric:
( 1 − O ( k ℓ f ( ℓ ) ) ) ∥ v ∥ 2 ≤ ∥ A v ∥ 2 ≤ ( 1 + O ( k ℓ f ( ℓ ) ) ) ∥ v ∥ 2 , v ∈ E s p . \left(1-O\left(\frac{k}{\ell f(\ell)}\right)\right)\lVert v\rVert_2\le\lVert Av\rVert_2\le\left(1+O\left(\frac{k}{\ell f(\ell)}\right)\right)\lVert v\rVert_2,\qquad v\in E_{\mathrm{sp}}. ( 1 − O ( ℓ f ( ℓ ) k ) ) ∥ v ∥ 2 ≤ ∥ A v ∥ 2 ≤ ( 1 + O ( ℓ f ( ℓ ) k ) ) ∥ v ∥ 2 , v ∈ E sp .
Equivalently, the singular values of the compressed operator A ∣ E s p A|_{E_{\mathrm{sp}}} A ∣ E sp all lie in
[ 1 − O ( k ℓ f ( ℓ ) ) , 1 + O ( k ℓ f ( ℓ ) ) ] . \left[1-O\left(\frac{k}{\ell f(\ell)}\right),1+O\left(\frac{k}{\ell f(\ell)}\right)\right]. [ 1 − O ( ℓ f ( ℓ ) k ) , 1 + O ( ℓ f ( ℓ ) k ) ] .
However, these are not the same as the ordered singular values of the full matrix A A A . The bulk block A 22 A_{22} A 22 can already have operator norm strictly larger than 1; for instance, for fixed k k k and s i = 0 s_i=0 s i = 0 for i > k i>k i > k , it is the shifted Cesàro matrix, whose norm is bounded away from 1. Therefore the top ordered singular values of A A A may exceed 1 by an O ( 1 ) O(1) O ( 1 ) amount. Projecting onto the first k k k output coordinates in Step 2 gives the one-sided estimate
σ k ( A ) ≥ 1 − O ( k ℓ f ( ℓ ) ) , k ≥ 1 , \sigma_k(A)\ge1-O\left(\frac{k}{\ell f(\ell)}\right),\qquad k\ge1, σ k ( A ) ≥ 1 − O ( ℓ f ( ℓ ) k ) , k ≥ 1 ,
together with the universal upper bound σ i ( A ) ≤ ∥ A ∥ o p ≤ 5 \sigma_i(A)\le\lVert A\rVert_{\mathrm{op}}\le\sqrt{5} σ i ( A ) ≤ ∥ A ∥ op ≤ 5 .
Rank proxies for attention and causal attention
This appendix collects the rank-proxy consequences of the singular- and eigenvalue results in the main article. The entropy-based proxies require more care than pointwise spectral collapse: many individually vanishing modes can retain non-negligible total entropy.
Definition 1 (Rank proxies and asymptotic notation). For a nonzero matrix A ∈ R ℓ × ℓ A\in\mathbb{R}^{\ell\times\ell} A ∈ R ℓ × ℓ with singular values σ 1 ( A ) , … , σ ℓ ( A ) \sigma_1(A),\ldots,\sigma_{\ell}(A) σ 1 ( A ) , … , σ ℓ ( A ) , define
srank ( A ) : = ∥ A ∥ F 2 ∥ A ∥ o p 2 , erank 1 ( A ) : = exp { − ∑ j = 1 ℓ q j ( 1 ) log q j ( 1 ) } , q j ( 1 ) = σ j ( A ) ∑ r σ r ( A ) , \operatorname{srank}(A):=\frac{\lVert A\rVert_{F}^{2}}{\lVert A\rVert_{\mathrm{op}}^{2}},\qquad
\operatorname{erank}_{1}(A):=\exp\left\{-\sum_{j=1}^{\ell}q_j^{(1)}\log q_j^{(1)}\right\},\qquad
q_j^{(1)}=\frac{\sigma_j(A)}{\sum_r\sigma_r(A)}, srank ( A ) := ∥ A ∥ op 2 ∥ A ∥ F 2 , erank 1 ( A ) := exp { − j = 1 ∑ ℓ q j ( 1 ) log q j ( 1 ) } , q j ( 1 ) = ∑ r σ r ( A ) σ j ( A ) ,
and
erank 2 ( A ) : = exp { − ∑ j = 1 ℓ q j ( 2 ) log q j ( 2 ) } , q j ( 2 ) = σ j ( A ) 2 ∑ r σ r ( A ) 2 , \operatorname{erank}_{2}(A):=\exp\left\{-\sum_{j=1}^{\ell}q_j^{(2)}\log q_j^{(2)}\right\},\qquad
q_j^{(2)}=\frac{\sigma_j(A)^2}{\sum_r\sigma_r(A)^2}, erank 2 ( A ) := exp { − j = 1 ∑ ℓ q j ( 2 ) log q j ( 2 ) } , q j ( 2 ) = ∑ r σ r ( A ) 2 σ j ( A ) 2 ,
with 0 log 0 = 0 0\log0=0 0 log 0 = 0 . For the zero matrix, set these three singular-value proxies equal to zero. For eigenvalues λ 1 ( A ) , … , λ ℓ ( A ) \lambda_1(A),\ldots,\lambda_{\ell}(A) λ 1 ( A ) , … , λ ℓ ( A ) , define
srank λ ( A ) : = ∑ i ∣ λ i ( A ) ∣ max i ∣ λ i ( A ) ∣ , erank λ ( A ) : = exp { − ∑ i = 1 ℓ p i log p i } , p i = ∣ λ i ( A ) ∣ ∑ j ∣ λ j ( A ) ∣ . \operatorname{srank}_{\lambda}(A):=\frac{\sum_i\lvert\lambda_i(A)\rvert}{\max_i\lvert\lambda_i(A)\rvert},\qquad
\operatorname{erank}_{\lambda}(A):=\exp\left\{-\sum_{i=1}^{\ell}p_i\log p_i\right\},\qquad
p_i=\frac{\lvert\lambda_i(A)\rvert}{\sum_j\lvert\lambda_j(A)\rvert}. srank λ ( A ) := max i ∣ λ i ( A )∣ ∑ i ∣ λ i ( A )∣ , erank λ ( A ) := exp { − i = 1 ∑ ℓ p i log p i } , p i = ∑ j ∣ λ j ( A )∣ ∣ λ i ( A )∣ .
The eigenvalue proxies are used when the spectrum is not identically zero; by convention, both are zero when their normalizing spectral mass vanishes. Thus srank λ \operatorname{srank}_{\lambda} srank λ is the stated ℓ 1 / ℓ ∞ \ell^1/\ell^\infty ℓ 1 / ℓ ∞ eigenvalue participation proxy; it is distinct from the singular-value stable rank srank \operatorname{srank} srank . Throughout, O ( ⋅ ) O(\cdot) O ( ⋅ ) , Ω ( ⋅ ) \Omega(\cdot) Ω ( ⋅ ) , and Θ ( ⋅ ) \Theta(\cdot) Θ ( ⋅ ) refer to ℓ → ∞ \ell\to\infty ℓ → ∞ ; their implicit constants are independent of ℓ \ell ℓ , but may depend on fixed parameters explicitly indicated in the statement.
Bidirectional attention
Pointwise spectral collapse does not by itself control a rank proxy: the relevant bulk mass is quadratic for stable rank and linear for the eigenvalue participation proxy, while entropy ranks additionally depend on bulk entropy moments. The following result separates these requirements.
Theorem 6 (Rank proxies for attention). Under Assumptions 1 and 2, retain A A A , d i d_i d i , c ℓ c_\ell c ℓ , ε ℓ \varepsilon_\ell ε ℓ from Section 3.2. Using α ± \alpha_\pm α ± from Theorem 1, define
a ± ( x ) : = α ± ( x ) 2 , R ℓ = k + 1 − c ℓ . a_\pm(x) := \alpha_\pm(x)^2,\qquad R_\ell= k + 1 - c_\ell. a ± ( x ) := α ± ( x ) 2 , R ℓ = k + 1 − c ℓ . For p ∈ { 1 , 2 } p \in\{1,2\} p ∈ { 1 , 2 } , define the bulk mass and entropy moment
M p , ℓ = ∑ i = k + 1 ℓ d i p , E p , ℓ = ∑ i = k + 1 ℓ d i p log 1 d i , B p , ℓ = M p , ℓ + E p , ℓ , M_{p,\ell} = \sum_{i=k+1}^{\ell} d_i^p,\qquad E_{p,\ell} = \sum_{i=k+1}^{\ell} d_i^p \log\frac{1}{d_i},\qquad B_{p,\ell} = M_{p,\ell} + E_{p,\ell}, M p , ℓ = i = k + 1 ∑ ℓ d i p , E p , ℓ = i = k + 1 ∑ ℓ d i p log d i 1 , B p , ℓ = M p , ℓ + E p , ℓ , where a summand with d i = 0 d_i = 0 d i = 0 is zero. Define also Δ p , ℓ = ∑ i = 1 k ( 1 − d i p ) = O ( k / f ( ℓ ) ) \Delta_{p,\ell} = \sum_{i=1}^{k}(1-d_i^p) = O(k/f(\ell)) Δ p , ℓ = ∑ i = 1 k ( 1 − d i p ) = O ( k / f ( ℓ )) . Then
srank ( A ) = R ℓ + ( 1 − ℓ − 1 ) ( M 2 , ℓ − Δ 2 , ℓ ) a + ( c ℓ ) + O ( ε ℓ ) , (5) \operatorname{srank}(A) = \frac{R_\ell+ (1-\ell^{-1})(M_{2,\ell}-\Delta_{2,\ell})}{a_+(c_\ell)+O(\varepsilon_\ell)}, \tag*{(5)} srank ( A ) = a + ( c ℓ ) + O ( ε ℓ ) R ℓ + ( 1 − ℓ − 1 ) ( M 2 , ℓ − Δ 2 , ℓ ) , (5) srank λ ( A ) = R ℓ + ( 1 − ℓ − 1 ) ( M 1 , ℓ − Δ 1 , ℓ ) . (6) \operatorname{srank}_{\lambda}(A) = R_\ell+ (1-\ell^{-1})(M_{1,\ell}-\Delta_{1,\ell}). \tag*{(6)} srank λ ( A ) = R ℓ + ( 1 − ℓ − 1 ) ( M 1 , ℓ − Δ 1 , ℓ ) . (6) In particular, the following are necessary and sufficient conditions for the corresponding ideal-profile equivalents:
srank ( A ) = R ℓ a + ( c ℓ ) ( 1 + o ( 1 ) ) ⟺ M 2 , ℓ = o ( R ℓ ) , \operatorname{srank}(A) = \frac{R_\ell}{a_+(c_\ell)}(1+o(1)) \Longleftrightarrow M_{2,\ell}=o(R_\ell), srank ( A ) = a + ( c ℓ ) R ℓ ( 1 + o ( 1 )) ⟺ M 2 , ℓ = o ( R ℓ ) , srank λ ( A ) = R ℓ ( 1 + o ( 1 ) ) ⟺ M 1 , ℓ = o ( R ℓ ) , \operatorname{srank}_{\lambda}(A) = R_\ell(1+o(1)) \Longleftrightarrow M_{1,\ell}=o(R_\ell), srank λ ( A ) = R ℓ ( 1 + o ( 1 )) ⟺ M 1 , ℓ = o ( R ℓ ) , erank 2 ( A ) = ( k + 1 ) ( 1 + o ( 1 ) ) ⟺ B 2 , ℓ = o ( R ℓ ) , \operatorname{erank}_2(A) = (k+1)(1+o(1)) \Longleftrightarrow B_{2,\ell}=o(R_\ell), erank 2 ( A ) = ( k + 1 ) ( 1 + o ( 1 )) ⟺ B 2 , ℓ = o ( R ℓ ) , erank 1 ( A ) = ( k + 1 ) ( 1 + o ( 1 ) ) ⟺ B 1 , ℓ = o ( R ℓ ) , \operatorname{erank}_1(A) = (k+1)(1+o(1)) \Longleftrightarrow B_{1,\ell}=o(R_\ell), erank 1 ( A ) = ( k + 1 ) ( 1 + o ( 1 )) ⟺ B 1 , ℓ = o ( R ℓ ) , erank λ ( A ) = ( k + 1 ) ( 1 + o ( 1 ) ) ⟺ B 1 , ℓ = o ( R ℓ ) . \operatorname{erank}_{\lambda}(A) = (k+1)(1+o(1)) \Longleftrightarrow B_{1,\ell}=o(R_\ell). erank λ ( A ) = ( k + 1 ) ( 1 + o ( 1 )) ⟺ B 1 , ℓ = o ( R ℓ ) . Each equivalence is asserted separately; no logarithmic rate condition is needed on the spike defects 1 − d i 1-d_i 1 − d i . Since E p , ℓ ≥ log g ( ℓ ) M p , ℓ E_{p,\ell} \ge\log g(\ell) M_{p,\ell} E p , ℓ ≥ log g ( ℓ ) M p , ℓ for all sufficiently large ℓ \ell ℓ , each entropy condition B p , ℓ = o ( R ℓ ) B_{p,\ell}=o(R_\ell) B p , ℓ = o ( R ℓ ) is equivalently E p , ℓ = o ( R ℓ ) E_{p,\ell}=o(R_\ell) E p , ℓ = o ( R ℓ ) .
For comparison with the deterministic spectral profiles, let N ℓ ⋆ N_\ell^\star N ℓ ⋆ be the nuclear mass of the ideal singular-value profile. Explicitly,
N ℓ ⋆ = { 1 , k = 0 , k − 1 + a + ( c ℓ ) + a − ( c ℓ ) , 1 ≤ k < ℓ , ℓ , k = ℓ . N_\ell^\star=
\begin{cases}
1, & k=0,\\
k-1+\sqrt{a_+(c_\ell)}+\sqrt{a_-(c_\ell)}, & 1\le k<\ell,\\
\ell, & k=\ell.
\end{cases} N ℓ ⋆ = ⎩ ⎨ ⎧ 1 , k − 1 + a + ( c ℓ ) + a − ( c ℓ ) , ℓ , k = 0 , 1 ≤ k < ℓ , k = ℓ . Under B 1 , ℓ = o ( R ℓ ) B_{1,\ell}=o(R_\ell) B 1 , ℓ = o ( R ℓ ) , respectively B 2 , ℓ = o ( R ℓ ) B_{2,\ell}=o(R_\ell) B 2 , ℓ = o ( R ℓ ) , the entropy equivalents can also be written as
erank 1 ( A ) = N ℓ ⋆ exp { − a + ( c ℓ ) log a + ( c ℓ ) + a − ( c ℓ ) log a − ( c ℓ ) N ℓ ⋆ } ( 1 + o ( 1 ) ) , \operatorname{erank}_1(A) = N_\ell^\star\exp\left\{-\frac{\sqrt{a_+(c_\ell)}\log\sqrt{a_+(c_\ell)}+\sqrt{a_-(c_\ell)}\log\sqrt{a_-(c_\ell)}}{N_\ell^\star}\right\}(1+o(1)), erank 1 ( A ) = N ℓ ⋆ exp { − N ℓ ⋆ a + ( c ℓ ) log a + ( c ℓ ) + a − ( c ℓ ) log a − ( c ℓ ) } ( 1 + o ( 1 )) , erank 2 ( A ) = R ℓ exp { − a + ( c ℓ ) log a + ( c ℓ ) + a − ( c ℓ ) log a − ( c ℓ ) R ℓ } ( 1 + o ( 1 ) ) . \operatorname{erank}_2(A) = R_\ell\exp\left\{-\frac{a_+(c_\ell)\log a_+(c_\ell)+a_-(c_\ell)\log a_-(c_\ell)}{R_\ell}\right\}(1+o(1)). erank 2 ( A ) = R ℓ exp { − R ℓ a + ( c ℓ ) log a + ( c ℓ ) + a − ( c ℓ ) log a − ( c ℓ ) } ( 1 + o ( 1 )) . Under B 1 , ℓ = o ( R ℓ ) B_{1,\ell}=o(R_\ell) B 1 , ℓ = o ( R ℓ ) ,
erank λ ( A ) = R ℓ exp { − ( 1 − c ℓ ) log ( 1 − c ℓ ) R ℓ } ( 1 + o ( 1 ) ) . \operatorname{erank}_{\lambda}(A) = R_\ell\exp\left\{-\frac{(1-c_\ell)\log(1-c_\ell)}{R_\ell}\right\}(1+o(1)). erank λ ( A ) = R ℓ exp { − R ℓ ( 1 − c ℓ ) log ( 1 − c ℓ ) } ( 1 + o ( 1 )) .
Here 0 log 0 = 0 0\log0=0 0 log 0 = 0 ; at k = 0 k=0 k = 0 the ideal profile has just one unit singular value and one unit eigenvalue, and at k = ℓ k=\ell k = ℓ all ideal spectral values are one. The displayed expressions agree with these endpoint conventions.
Some useful consequences require only simpler conditions. If all bulk scores are uniformly bounded, then
M 2 , ℓ = O ( ℓ − 1 ) , B 2 , ℓ = O ( ℓ − 1 log ℓ ) , M_{2,\ell}=O(\ell^{-1}),\qquad B_{2,\ell}=O(\ell^{-1}\log\ell), M 2 , ℓ = O ( ℓ − 1 ) , B 2 , ℓ = O ( ℓ − 1 log ℓ ) ,
so the stable-rank and erank2 _2 2 equivalents hold automatically. In particular, if also k = O ( 1 ) k=O(1) k = O ( 1 ) , then
srank ( A ) = k + 1 + o ( 1 ) , erank 2 ( A ) = k + 1 + o ( 1 ) . \operatorname{srank}(A)=k+1+o(1),\qquad\operatorname{erank}_{2}(A)=k+1+o(1). srank ( A ) = k + 1 + o ( 1 ) , erank 2 ( A ) = k + 1 + o ( 1 ) .
For any bulk satisfying Assumption 2,
B p , ℓ = O ( ( ℓ − k ) ( 1 + log g ( ℓ ) ) g ( ℓ ) p ) . B_{p,\ell}=O\left(\frac{(\ell-k)(1+\log g(\ell))}{g(\ell)^p}\right). B p , ℓ = O ( g ( ℓ ) p ( ℓ − k ) ( 1 + log g ( ℓ )) ) .
Thus, if c ℓ → c ∈ ( 0 , 1 ] c_\ell\to c\in(0,1] c ℓ → c ∈ ( 0 , 1 ] , all the tail conditions above hold automatically and
srank ( A ) ℓ ⟶ c a + ( c ) , erank 1 ( A ) ℓ , erank 2 ( A ) ℓ , srank λ ( A ) ℓ , erank λ ( A ) ℓ ⟶ c . \frac{\operatorname{srank}(A)}{\ell}\longrightarrow\frac{c}{a_{+}(c)},\qquad
\frac{\operatorname{erank}_{1}(A)}{\ell},\qquad
\frac{\operatorname{erank}_{2}(A)}{\ell},\qquad
\frac{\operatorname{srank}_{\lambda}(A)}{\ell},\qquad
\frac{\operatorname{erank}_{\lambda}(A)}{\ell}\longrightarrow c. ℓ srank ( A ) ⟶ a + ( c ) c , ℓ erank 1 ( A ) , ℓ erank 2 ( A ) , ℓ srank λ ( A ) , ℓ erank λ ( A ) ⟶ c .
The condition B 1 , ℓ = o ( R ℓ ) B_{1,\ell}=o(R_\ell) B 1 , ℓ = o ( R ℓ ) implies all five equivalents. It is implied by the stronger aggregate-defect condition
τ ℓ log ℓ = o ( R ℓ ) , τ ℓ = ∑ i = 1 k ( 1 − d i ) + ∑ i = k + 1 ℓ d i , \tau_\ell\log\ell=o(R_\ell),\qquad\tau_\ell=\sum_{i=1}^{k}(1-d_i)+\sum_{i=k+1}^{\ell}d_i, τ ℓ log ℓ = o ( R ℓ ) , τ ℓ = i = 1 ∑ k ( 1 − d i ) + i = k + 1 ∑ ℓ d i ,
but does not impose that condition’s logarithmic penalty on the spikes. Here τ ℓ = O ( k / f ( ℓ ) + ( ℓ − k ) / g ( ℓ ) ) \tau_\ell=O(k/f(\ell)+(\ell-k)/g(\ell)) τ ℓ = O ( k / f ( ℓ ) + ( ℓ − k ) / g ( ℓ )) .
Proof. Let J = ℓ − 1 11 T J=\ell^{-1}\mathbf{1}\mathbf{1}^{\mathsf T} J = ℓ − 1 1 1 T and P = I − J P=I-J P = I − J . By (1) ,
A = J + diag ( d 1 , … , d ℓ ) P . A=J+\operatorname{diag}(d_1,\ldots,d_\ell)P. A = J + diag ( d 1 , … , d ℓ ) P .
Together with P J = 0 PJ=0 P J = 0 , this gives
∥ A ∥ F 2 = 1 + ( 1 − ℓ − 1 ) ∑ i = 1 ℓ d i 2 (7) \lVert A\rVert_{\mathrm{F}}^2=1+(1-\ell^{-1})\sum_{i=1}^{\ell}d_i^2 \tag*{(7)} ∥ A ∥ F 2 = 1 + ( 1 − ℓ − 1 ) i = 1 ∑ ℓ d i 2 (7)
tr ( A ) = 1 + ( 1 − ℓ − 1 ) ∑ i = 1 ℓ d i (8) \operatorname{tr}(A)=1+(1-\ell^{-1})\sum_{i=1}^{\ell}d_i \tag*{(8)} tr ( A ) = 1 + ( 1 − ℓ − 1 ) i = 1 ∑ ℓ d i (8)
Indeed, the two matrix terms are Frobenius orthogonal and the diagonal entries of the projection P P P are 1 − ℓ − 1 1-\ell^{-1} 1 − ℓ − 1 . Spectral separation gives, uniformly,
0 ≤ 1 − d i ≤ f ( ℓ ) − 1 ( i ≤ k ) , 0 ≤ d i ≤ g ( ℓ ) − 1 ( i > k ) . 0\le1-d_i\le f(\ell)^{-1}\quad(i\le k),\qquad0\le d_i\le g(\ell)^{-1}\quad(i>k). 0 ≤ 1 − d i ≤ f ( ℓ ) − 1 ( i ≤ k ) , 0 ≤ d i ≤ g ( ℓ ) − 1 ( i > k ) .
If d i ⋆ = 1 { i ≤ k } d_i^\star=\mathbf{1}_{\{i\le k\}} d i ⋆ = 1 { i ≤ k } and A ⋆ = J + diag ( d ⋆ ) P A^\star=J+\operatorname{diag}(d^\star)P A ⋆ = J + diag ( d ⋆ ) P , then
∥ A − A ⋆ ∥ o p ≤ ε ℓ . \lVert A-A^\star\rVert_{\mathrm{op}}\le\varepsilon_\ell. ∥ A − A ⋆ ∥ op ≤ ε ℓ .
The ideal singular profile in Theorem 1 therefore gives ∥ A ∥ o p 2 = a + ( c ℓ ) + O ( ε ℓ ) \lVert A\rVert_{\mathrm{op}}^2=a_{+}(c_\ell)+O(\varepsilon_\ell) ∥ A ∥ op 2 = a + ( c ℓ ) + O ( ε ℓ ) , including k = 0 , ℓ k=0,\ell k = 0 , ℓ . This proves (5) .
By Theorem 2, all eigenvalues are nonnegative and the largest is 1. Hence srank λ ( A ) = tr ( A ) \operatorname{srank}_{\lambda}(A)=\operatorname{tr}(A) srank λ ( A ) = tr ( A ) , proving (6) . Since Δ p , ℓ = o ( R ℓ ) \Delta_{p,\ell}=o(R_\ell) Δ p , ℓ = o ( R ℓ ) , the two stable-rank equivalences follow.
We next control the entropy without multiplying the spike error by log ℓ \log\ell log ℓ . Write D = diag ( d ) D=\operatorname{diag}(d) D = diag ( d ) . By (1) , A A A is a rank-one perturbation of D D D . The singular-value min–max principle and Hermitian interlacing give, respectively,
d j + 1 ≤ σ j ( A ) ≤ d j − 1 ( 2 ≤ j ≤ ℓ ) , d j ≤ λ j ( A ) ≤ d j − 1 ( 2 ≤ j ≤ ℓ ) , (9) d_{j+1}\le\sigma_j(A)\le d_{j-1}\quad(2\le j\le\ell),\qquad
d_j\le\lambda_j(A)\le d_{j-1}\quad(2\le j\le\ell), \tag*{(9)} d j + 1 ≤ σ j ( A ) ≤ d j − 1 ( 2 ≤ j ≤ ℓ ) , d j ≤ λ j ( A ) ≤ d j − 1 ( 2 ≤ j ≤ ℓ ) , (9)
where d ℓ + 1 = 0 d_{\ell+1}=0 d ℓ + 1 = 0 . For example, adding a rank-one matrix can shift a singular-value index by at most one; applying that fact both to A = D + z 1 T A=D+z\mathbf{1}^{\mathsf T} A = D + z 1 T and to D = A − z 1 T D=A-z\mathbf{1}^{\mathsf T} D = A − z 1 T proves the first pair of inequalities. Suppose first that k < ℓ k < \ell k < ℓ , and set h = k + 1 h = k + 1 h = k + 1 . For p ∈ { 1 , 2 } p \in\{1,2\} p ∈ { 1 , 2 } , apply (9) to the tail j > h j > h j > h . Both functions x p x^p x p and x p log ( 1 / x ) x^p \log(1/x) x p log ( 1/ x ) are increasing on a sufficiently small fixed neighborhood of zero. The upper and lower sums obtained by interlacing differ from the bulk sums by at most three endpoint terms, each tending to zero. Consequently,
∑ j > h σ j ( A ) p = M p , ℓ + o ( 1 ) , ∑ j > h σ j ( A ) p log 1 σ j ( A ) = E p , ℓ + o ( 1 ) , ∑ j > h λ j ( A ) = M 1 , ℓ + o ( 1 ) , ∑ j > h λ j ( A ) log 1 λ j ( A ) = E 1 , ℓ + o ( 1 ) . \begin{aligned}
\sum_{j>h}\sigma_j(A)^p &= M_{p,\ell}+o(1), &
\sum_{j>h}\sigma_j(A)^p\log\frac{1}{\sigma_j(A)} &= E_{p,\ell}+o(1),\\
\sum_{j>h}\lambda_j(A) &= M_{1,\ell}+o(1), &
\sum_{j>h}\lambda_j(A)\log\frac{1}{\lambda_j(A)} &= E_{1,\ell}+o(1).
\end{aligned} j > h ∑ σ j ( A ) p j > h ∑ λ j ( A ) = M p , ℓ + o ( 1 ) , = M 1 , ℓ + o ( 1 ) , j > h ∑ σ j ( A ) p log σ j ( A ) 1 j > h ∑ λ j ( A ) log λ j ( A ) 1 = E p , ℓ + o ( 1 ) , = E 1 , ℓ + o ( 1 ) .
These conclusions include k = 0 k=0 k = 0 and k = ℓ − 1 k=\ell-1 k = ℓ − 1 , with empty tail sums interpreted as zero. The displayed o ( 1 ) o(1) o ( 1 ) terms are uniformly dominated by the neighboring bulk endpoint terms; if a bulk endpoint vanishes exactly, the corresponding comparison is exact.
For the head singular values, Weyl’s inequality gives k − 1 k-1 k − 1 values equal to 1 + O ( ε ℓ ) 1+O(\varepsilon_\ell) 1 + O ( ε ℓ ) , together with the two exceptional ideal-profile values perturbed by O ( ε ℓ ) O(\varepsilon_\ell) O ( ε ℓ ) ; when k = 0 k=0 k = 0 , there is only one head value, 1 + O ( ε ℓ ) 1+O(\varepsilon_\ell) 1 + O ( ε ℓ ) . For the head eigenvalues, λ 1 = 1 \lambda_1=1 λ 1 = 1 , the next k − 1 k-1 k − 1 values are 1 + O ( f ( ℓ ) − 1 ) 1+O(f(\ell)^{-1}) 1 + O ( f ( ℓ ) − 1 ) , and the intermediate value is 1 − c ℓ + O ( ε ℓ ) 1-c_\ell+O(\varepsilon_\ell) 1 − c ℓ + O ( ε ℓ ) . Thus the total head mass is R ℓ + o ( R ℓ ) R_\ell+o(R_\ell) R ℓ + o ( R ℓ ) , for each of the three choices of spectral weights
x j = σ j ( A ) , x j = σ j ( A ) 2 , x j = λ j ( A ) . x_j=\sigma_j(A),\qquad x_j=\sigma_j(A)^2,\qquad x_j=\lambda_j(A). x j = σ j ( A ) , x j = σ j ( A ) 2 , x j = λ j ( A ) .
Here the ideal nuclear mass N ℓ ⋆ N_\ell^\star N ℓ ⋆ satisfies N ℓ ⋆ / R ℓ → 1 N_\ell^\star/R_\ell\to1 N ℓ ⋆ / R ℓ → 1 . Also, the head moment ∑ j ≤ h x j log x j \sum_{j\leq h}x_j\log x_j ∑ j ≤ h x j log x j is o ( R ℓ ) o(R_\ell) o ( R ℓ ) : unit-profile coordinates contribute O ( k ε ℓ ) O(k\varepsilon_\ell) O ( k ε ℓ ) , and the exceptional coordinates contribute o ( R ℓ ) o(R_\ell) o ( R ℓ ) . To see the last assertion uniformly, their moments are bounded if k → ∞ k\to\infty k → ∞ , while if k k k is bounded then c ℓ → 0 c_\ell\to0 c ℓ → 0 and every exceptional nonzero head value tends to one. If k = ℓ k=\ell k = ℓ , all spectral values are 1 + o ( 1 ) 1+o(1) 1 + o ( 1 ) uniformly and the same head conclusions hold with an empty tail. These arguments apply along subsequences and hence also when the endpoint regimes vary with ℓ \ell ℓ .
For any of the three weight lists, let p = 1 , 2 , 1 p=1,2,1 p = 1 , 2 , 1 , respectively, and let T ℓ = ∑ j x j \mathcal{T}_\ell=\sum_j x_j T ℓ = ∑ j x j . The preceding estimates state
T ℓ = R ℓ + M p , ℓ + o ( R ℓ ) , ∑ j x j log x j = − p E p , ℓ + o ( R ℓ ) . (10) \mathcal{T}_\ell=R_\ell+M_{p,\ell}+o(R_\ell),\qquad\sum_jx_j\log x_j=-pE_{p,\ell}+o(R_\ell). \tag*{(10)} T ℓ = R ℓ + M p , ℓ + o ( R ℓ ) , j ∑ x j log x j = − p E p , ℓ + o ( R ℓ ) . (10)
For eigenvalue weights the second identity uses p = 1 p=1 p = 1 . The normalization identity for entropy is exact:
H ( x T ℓ ) − log R ℓ = log T ℓ R ℓ − ∑ j x j log x j T ℓ = log T ℓ R ℓ + p E p , ℓ T ℓ + o ( 1 ) . \begin{aligned}
H\left(\frac{x}{\mathcal{T}_\ell}\right)-\log R_\ell
&=\log\frac{\mathcal{T}_\ell}{R_\ell}-\frac{\sum_jx_j\log x_j}{\mathcal{T}_\ell}\\
&=\log\frac{\mathcal{T}_\ell}{R_\ell}+\frac{pE_{p,\ell}}{\mathcal{T}_\ell}+o(1).
\end{aligned} H ( T ℓ x ) − log R ℓ = log R ℓ T ℓ − T ℓ ∑ j x j log x j = log R ℓ T ℓ + T ℓ p E p , ℓ + o ( 1 ) .
This identity accounts for the normalization before taking a limit; in particular it introduces no extra factor log R ℓ \log R_\ell log R ℓ . Since M p , ℓ , E p , ℓ ≥ 0 M_{p,\ell},E_{p,\ell}\geq0 M p , ℓ , E p , ℓ ≥ 0 and T ℓ / R ℓ = 1 + M p , ℓ / R ℓ + o ( 1 ) \mathcal{T}_\ell/R_\ell=1+M_{p,\ell}/R_\ell+o(1) T ℓ / R ℓ = 1 + M p , ℓ / R ℓ + o ( 1 ) , its right-hand side tends to zero if and only if M p , ℓ + E p , ℓ = o ( R ℓ ) M_{p,\ell}+E_{p,\ell}=o(R_\ell) M p , ℓ + E p , ℓ = o ( R ℓ ) . Indeed, convergence forces first M p , ℓ / R ℓ → 0 M_{p,\ell}/R_\ell\to0 M p , ℓ / R ℓ → 0 and then E p , ℓ / R ℓ → 0 E_{p,\ell}/R_\ell\to0 E p , ℓ / R ℓ → 0 . Because R ℓ / ( k + 1 ) → 1 R_\ell/(k+1)\to1 R ℓ / ( k + 1 ) → 1 , this proves all three entropy equivalences.
The ideal singular, squared-singular, and eigenvalue lists have masses N ℓ ⋆ N_\ell^\star N ℓ ⋆ , R ℓ R_\ell R ℓ , R ℓ R_\ell R ℓ , respectively. Substitution in the entropy definition yields the displayed deterministic-profile expressions. Each ideal head entropy moment divided by its mass tends to zero by the same head argument, so these expressions are equivalent to the already proved ( k + 1 ) ( 1 + o ( 1 ) ) (k+1)(1+o(1)) ( k + 1 ) ( 1 + o ( 1 )) formulas.
Finally, bounded bulk scores give d i = O ( ℓ − 1 ) d_i=O(\ell^{-1}) d i = O ( ℓ − 1 ) for i > k i>k i > k , and hence the stated quadratic bounds. More generally, d i ≤ g ( ℓ ) − 1 d_i\leq g(\ell)^{-1} d i ≤ g ( ℓ ) − 1 and monotonicity near zero give the bounds on B p , ℓ B_{p,\ell} B p , ℓ . Since B 2 , ℓ ≤ B 1 , ℓ B_{2,\ell}\leq B_{1,\ell} B 2 , ℓ ≤ B 1 , ℓ , the linear entropy condition implies all the others. To compare with the aggregate-defect condition, write m = M 1 , ℓ m=M_{1,\ell} m = M 1 , ℓ and n = ℓ − k n=\ell-k n = ℓ − k . Concavity gives E 1 , ℓ ≤ m log ( n / m ) E_{1,\ell}\leq m\log(n/m) E 1 , ℓ ≤ m log ( n / m ) when m > 0 m>0 m > 0 and n > 0 n>0 n > 0 . For m > 1 m>1 m > 1 , this is at most m log ℓ m\log\ell m log ℓ . For 0 ≤ m ≤ 1 0\leq m\leq1 0 ≤ m ≤ 1 , it is at most m log ( 1 / m ) + m m\log(1/m)+m m log ( 1/ m ) + m and the latter term is o ( R ℓ ) o(R_\ell) o ( R ℓ ) under τ ℓ log ℓ = o ( R ℓ ) \tau_\ell\log\ell=o(R_\ell) τ ℓ log ℓ = o ( R ℓ ) , either because R ℓ → ∞ R_\ell\to\infty R ℓ → ∞ , or along bounded subsequences, because m → 0 m\to0 m → 0 . This proves the final implication. □
Remark 6 (Why the tail conditions differ). Bounded bulk scores suffice for stable rank and squared-singular-value entropy rank, but not for linear spectral mass or entropy. For example, take k = 0 k=0 k = 0 and s i = b > 0 s_i=b>0 s i = b > 0 for every i i i . Then
A = d I + ( 1 − d ) J , d = e b − 1 e b + ℓ − 1 ∼ e b − 1 ℓ . A=dI+(1-d)J,\qquad d=\frac{e^b-1}{e^b+\ell-1}\sim\frac{e^b-1}{\ell}. A = d I + ( 1 − d ) J , d = e b + ℓ − 1 e b − 1 ∼ ℓ e b − 1 .
Its singular values and eigenvalues are 1 , d , … , d 1,d,\ldots,d 1 , d , … , d , so
srank ( A ) , erank 2 ( A ) ⟶ 1 , srank λ ( A ) ⟶ e b , erank 1 ( A ) = erank λ ( A ) ≍ ℓ 1 − e − b . \operatorname{srank}(A),\ \operatorname{erank}_2(A)\longrightarrow1,\qquad\operatorname{srank}_{\lambda}(A)\longrightarrow e^b,\qquad\operatorname{erank}_1(A)=\operatorname{erank}_{\lambda}(A)\asymp\ell^{1-e^{-b}}. srank ( A ) , erank 2 ( A ) ⟶ 1 , srank λ ( A ) ⟶ e b , erank 1 ( A ) = erank λ ( A ) ≍ ℓ 1 − e − b .
Even vanishing bulk mass does not suffice for entropy convergence. For any fixed t > 0 t>0 t > 0 and p ∈ { 1 , 2 } p\in\{1,2\} p ∈ { 1 , 2 } , take k = 0 k=0 k = 0 and
d i = d ℓ = ( t ℓ log ℓ ) 1 / p , s i = log 1 + ( ℓ − 1 ) d ℓ 1 − d ℓ . d_i=d_\ell=\left(\frac{t}{\ell\log\ell}\right)^{1/p},\qquad s_i=\log\frac{1+(\ell-1)d_\ell}{1-d_\ell}. d i = d ℓ = ( ℓ log ℓ t ) 1/ p , s i = log 1 − d ℓ 1 + ( ℓ − 1 ) d ℓ .
These scores satisfy spectral separation. The same matrix formula gives M p , ℓ = t / log ℓ → 0 M_{p,\ell}=t/\log\ell\to0 M p , ℓ = t / log ℓ → 0 , but p E p , ℓ → t pE_{p,\ell}\to t p E p , ℓ → t , and therefore
erank p ( A ) ⟶ e t . \operatorname{erank}_p(A)\longrightarrow e^t. erank p ( A ) ⟶ e t .
For p = 1 p=1 p = 1 , srank λ ( A ) → 1 \operatorname{srank}_{\lambda}(A)\to1 srank λ ( A ) → 1 ; for p = 2 p=2 p = 2 , srank ( A ) → 1 \operatorname{srank}(A)\to1 srank ( A ) → 1 . Thus the logarithmic bulk entropy moments cannot in general be replaced by the corresponding masses alone.
Causal attention
For causal attention, bounded bulk scores produce harmonic squared-singular mass. The stable ranks and the squared-singular-value entropy rank have the same order, whereas the singular-value and eigenvalue entropy ranks can be larger. Throughout this subsection the causal mask is
M i j = { − ∞ , j > i , 0 , j ≤ i . M_{ij}=
\begin{cases}
-\infty, & j>i,\\
0, & j\le i.
\end{cases} M ij = { − ∞ , 0 , j > i , j ≤ i .
Lemma 1 (Schatten bounds for the finite Hardy matrix). Let H n = ( i − 1 1 { j ≤ i } ) i , j = 1 n H_n=(i^{-1}\mathbf{1}_{\{j\le i\}})_{i,j=1}^n H n = ( i − 1 1 { j ≤ i } ) i , j = 1 n . For every fixed p > 1 p>1 p > 1 ,
∑ j = 1 n σ j ( H n ) p ≤ C p log ( n + 1 ) , ∥ H n ∥ ∗ = Θ ( ( log ( n + 1 ) ) 2 ) , \sum_{j=1}^{n}\sigma_j(H_n)^p\le C_p\log(n+1),\qquad\lVert H_n\rVert_*=\Theta\bigl((\log(n+1))^2\bigr), j = 1 ∑ n σ j ( H n ) p ≤ C p log ( n + 1 ) , ∥ H n ∥ ∗ = Θ ( ( log ( n + 1 ) ) 2 ) , where ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ denotes the nuclear norm.
Proof. Partition { 1 , … , n } \{1,\ldots,n\} { 1 , … , n } into the nonempty dyadic intervals I r = { 2 r , … , min ( 2 r + 1 − 1 , n ) } I_r=\{2^r,\ldots,\min(2^{r+1}-1,n)\} I r = { 2 r , … , min ( 2 r + 1 − 1 , n )} , 0 ≤ r ≤ R = ⌊ log 2 n ⌋ 0\le r\le R=\lfloor\log_2 n\rfloor 0 ≤ r ≤ R = ⌊ log 2 n ⌋ . Let B d B_d B d retain exactly the blocks I r × I r − d I_r\times I_{r-d} I r × I r − d , d ≤ r ≤ R d\le r\le R d ≤ r ≤ R , of H n H_n H n , with all other entries zero. Then H n = ∑ d = 0 R B d H_n=\sum_{d=0}^R B_d H n = ∑ d = 0 R B d . For d ≥ 1 d\ge1 d ≥ 1 , each retained block has rank one and singular value at most
( ∑ i ∈ I r i − 2 ) 1 / 2 ∣ I r − d ∣ 1 / 2 ≤ 2 − d / 2 . \left(\sum_{i\in I_r}i^{-2}\right)^{1/2}|I_{r-d}|^{1/2}\le2^{-d/2}. ( i ∈ I r ∑ i − 2 ) 1/2 ∣ I r − d ∣ 1/2 ≤ 2 − d /2 . The retained blocks have disjoint row and column supports. Consequently, with ∥ X ∥ S p = ( ∑ j σ j ( X ) p ) 1 / p \lVert X\rVert_{S_p}=(\sum_j\sigma_j(X)^p)^{1/p} ∥ X ∥ S p = ( ∑ j σ j ( X ) p ) 1/ p ,
∥ B d ∥ S p ≤ ( R + 1 ) 1 / p 2 − d / 2 . \lVert B_d\rVert_{S_p}\le(R+1)^{1/p}2^{-d/2}. ∥ B d ∥ S p ≤ ( R + 1 ) 1/ p 2 − d /2 . For d = 0 d=0 d = 0 , reindexing within a dyadic block of size m = ∣ I r ∣ m=|I_r| m = ∣ I r ∣ gives D r L m D_rL_m D r L m , where D r = diag ( i − 1 : i ∈ I r ) D_r=\operatorname{diag}(i^{-1}:i\in I_r) D r = diag ( i − 1 : i ∈ I r ) and L m = ( 1 { q ≤ u } ) u , q = 1 m L_m=(\mathbf{1}_{\{q\le u\}})_{u,q=1}^m L m = ( 1 { q ≤ u } ) u , q = 1 m . The cumulative matrix has singular values
σ j ( L m ) = 1 2 sin ( ( 2 j − 1 ) π / ( 4 m + 2 ) ) ≍ m j , 1 ≤ j ≤ m . \sigma_j(L_m)=\frac{1}{2\sin((2j-1)\pi/(4m+2))}\asymp\frac{m}{j},\qquad1\le j\le m. σ j ( L m ) = 2 sin (( 2 j − 1 ) π / ( 4 m + 2 )) 1 ≍ j m , 1 ≤ j ≤ m . Indeed, L m − 1 L_m^{-1} L m − 1 is lower bidiagonal with diagonal entries one and subdiagonal entries minus one. Its Gram matrix has diagonal ( 2 , … , 2 , 1 ) (2,\ldots,2,1) ( 2 , … , 2 , 1 ) , off-diagonal entries minus one, and eigenvalues 4 sin 2 ( ( 2 j − 1 ) π / ( 4 m + 2 ) ) 4\sin^2((2j-1)\pi/(4m+2)) 4 sin 2 (( 2 j − 1 ) π / ( 4 m + 2 )) , as follows by substituting the vectors with coordinates sin ( u ( 2 j − 1 ) π / ( 2 m + 1 ) ) \sin(u(2j-1)\pi/(2m+1)) sin ( u ( 2 j − 1 ) π / ( 2 m + 1 )) . Since ∥ D r ∥ o p ≤ 2 − r \lVert D_r\rVert_{\mathrm{op}}\le2^{-r} ∥ D r ∥ op ≤ 2 − r and m ≤ 2 r m\le2^r m ≤ 2 r , ideal-norm monotonicity gives ∥ D r L m ∥ S p p ≤ C p ∑ j ≥ 1 j − p < ∞ \lVert D_rL_m\rVert_{S_p}^p\le C_p\sum_{j\ge1}j^{-p}<\infty ∥ D r L m ∥ S p p ≤ C p ∑ j ≥ 1 j − p < ∞ uniformly in r r r . Thus ∥ B 0 ∥ S p ≤ C p ( R + 1 ) 1 / p \lVert B_0\rVert_{S_p}\le C_p(R+1)^{1/p} ∥ B 0 ∥ S p ≤ C p ( R + 1 ) 1/ p . The Schatten triangle inequality and the convergent series ∑ d ≥ 1 2 − d / 2 \sum_{d\ge1}2^{-d/2} ∑ d ≥ 1 2 − d /2 prove the first assertion.
For p = 1 p=1 p = 1 , the same argument gives ∥ B 0 ∥ ∗ ≤ C ∑ r = 0 R ( r + 1 ) \lVert B_0\rVert_* \le C\sum_{r=0}^{R}(r+1) ∥ B 0 ∥ ∗ ≤ C ∑ r = 0 R ( r + 1 ) and ∑ d ≥ 1 ∥ B d ∥ ∗ ≤ C ( R + 1 ) \sum_{d\ge1}\lVert B_d\rVert_* \le C(R+1) ∑ d ≥ 1 ∥ B d ∥ ∗ ≤ C ( R + 1 ) , proving the upper bound. Pinching onto the diagonal blocks contracts the nuclear norm: the pinched matrix is the average of the conjugations by all block-diagonal sign matrices. Every full dyadic block has nuclear norm at least c ( r + 1 ) c(r+1) c ( r + 1 ) , because its row multipliers are at least 2 − r − 1 2^{-r-1} 2 − r − 1 and ∥ L 2 r ∥ ∗ ≥ c 2 r ( r + 1 ) \lVert L_{2^r}\rVert_* \ge c2^r(r+1) ∥ L 2 r ∥ ∗ ≥ c 2 r ( r + 1 ) . Summing the full blocks r < R r<R r < R , and adjusting constants for bounded n n n , proves the lower bound.
Theorem 7 (Rank proxies for causal attention). Under Assumptions 1 and 2 , let A = softmax ( Σ + M ) A=\operatorname{softmax}(\Sigma+M) A = softmax ( Σ + M ) , and suppose that s i ≤ B s_i\le B s i ≤ B for all i > k i>k i > k , where B B B is independent of ℓ \ell ℓ . Define
L ℓ , k = ∑ i = k + 1 ℓ 1 i , J ℓ , k = ∑ i = k + 1 ℓ log i i , T ℓ , k = k + L ℓ , k , L_{\ell,k}=\sum_{i=k+1}^{\ell}\frac{1}{i},\qquad J_{\ell,k}=\sum_{i=k+1}^{\ell}\frac{\log i}{i},\qquad T_{\ell,k}=k+L_{\ell,k}, L ℓ , k = i = k + 1 ∑ ℓ i 1 , J ℓ , k = i = k + 1 ∑ ℓ i log i , T ℓ , k = k + L ℓ , k , with empty sums equal to zero, and set
ρ i = exp ( 2 s i ) + i − 1 ( exp ( s i ) + i − 1 ) 2 , W ℓ = ∑ i = 1 ℓ ρ i = ∥ A ∥ F 2 , N ℓ = ∥ A ∥ ∗ . \rho_i=\frac{\exp(2s_i)+i-1}{(\exp(s_i)+i-1)^2},\qquad W_\ell=\sum_{i=1}^{\ell}\rho_i=\lVert A\rVert_F^2,\qquad N_\ell=\lVert A\rVert_*. ρ i = ( exp ( s i ) + i − 1 ) 2 exp ( 2 s i ) + i − 1 , W ℓ = i = 1 ∑ ℓ ρ i = ∥ A ∥ F 2 , N ℓ = ∥ A ∥ ∗ . Then
W ℓ = T ℓ , k + O B ( 1 + k 2 ℓ f ( ℓ ) ) = ( 1 + o ( 1 ) ) T ℓ , k , 1 ≤ ∥ A ∥ o p ≤ 5 , W_\ell=T_{\ell,k}+O_B\left(1+\frac{k^2}{\ell f(\ell)}\right)=(1+o(1))T_{\ell,k},\qquad1\le\lVert A\rVert_{\mathrm{op}}\le\sqrt{5}, W ℓ = T ℓ , k + O B ( 1 + ℓ f ( ℓ ) k 2 ) = ( 1 + o ( 1 )) T ℓ , k , 1 ≤ ∥ A ∥ op ≤ 5 , and
srank ( A ) , erank 2 ( A ) = Θ B ( T ℓ , k ) , N ℓ = Θ B ( k + ( log ( ℓ + 1 ) ) 2 ) . \operatorname{srank}(A),\ \operatorname{erank}_2(A)=\Theta_B(T_{\ell,k}),\qquad N_\ell=\Theta_B\left(k+(\log(\ell+1))^2\right). srank ( A ) , erank 2 ( A ) = Θ B ( T ℓ , k ) , N ℓ = Θ B ( k + ( log ( ℓ + 1 ) ) 2 ) . In particular, the singular-value entropy proxies satisfy
erank 1 ( A ) ≥ N ℓ 2 W ℓ ≥ c B ( k + ( log ( ℓ + 1 ) ) 2 ) 2 T ℓ , k , erank 2 ( A ) ≤ erank 1 ( A ) ≤ ℓ . \operatorname{erank}_1(A)\ge\frac{N_\ell^2}{W_\ell}\ge c_B\frac{\left(k+(\log(\ell+1))^2\right)^2}{T_{\ell,k}},\qquad\operatorname{erank}_2(A)\le\operatorname{erank}_1(A)\le\ell. erank 1 ( A ) ≥ W ℓ N ℓ 2 ≥ c B T ℓ , k ( k + ( log ( ℓ + 1 ) ) 2 ) 2 , erank 2 ( A ) ≤ erank 1 ( A ) ≤ ℓ . An additional explicit upper bound is
erank 2 ( A ) ≤ W ℓ exp { K ℓ W ℓ } , K ℓ : = ∑ i = 1 ℓ ρ i log 1 ρ i = J ℓ , k + O B ( 1 + k 2 ℓ f ( ℓ ) ) . \operatorname{erank}_2(A)\le W_\ell\exp\left\{\frac{K_\ell}{W_\ell}\right\},\qquad K_\ell:=\sum_{i=1}^{\ell}\rho_i\log\frac{1}{\rho_i}=J_{\ell,k}+O_B\left(1+\frac{k^2}{\ell f(\ell)}\right). erank 2 ( A ) ≤ W ℓ exp { W ℓ K ℓ } , K ℓ := i = 1 ∑ ℓ ρ i log ρ i 1 = J ℓ , k + O B ( 1 + ℓ f ( ℓ ) k 2 ) . For the eigenvalue proxies, set
U ℓ , k = ∑ i = k + 1 ℓ exp ( s i ) i , V ℓ , k = ∑ i = k + 1 ℓ exp ( s i ) i log ( i exp ( s i ) ) , U_{\ell,k}=\sum_{i=k+1}^{\ell}\frac{\exp(s_i)}{i},\qquad V_{\ell,k}=\sum_{i=k+1}^{\ell}\frac{\exp(s_i)}{i}\log\left(\frac{i}{\exp(s_i)}\right), U ℓ , k = i = k + 1 ∑ ℓ i exp ( s i ) , V ℓ , k = i = k + 1 ∑ ℓ i exp ( s i ) log ( exp ( s i ) i ) , and define the exact eigenvalue mass and log-moment
Λ ℓ = ∑ i = 1 ℓ exp ( s i ) exp ( s i ) + i − 1 , Q ℓ = ∑ i = 1 ℓ exp ( s i ) exp ( s i ) + i − 1 log ( 1 + i − 1 exp ( s i ) ) . \Lambda_\ell=\sum_{i=1}^{\ell}\frac{\exp(s_i)}{\exp(s_i)+i-1},\qquad Q_\ell=\sum_{i=1}^{\ell}\frac{\exp(s_i)}{\exp(s_i)+i-1}\log\left(1+\frac{i-1}{\exp(s_i)}\right). Λ ℓ = i = 1 ∑ ℓ exp ( s i ) + i − 1 exp ( s i ) , Q ℓ = i = 1 ∑ ℓ exp ( s i ) + i − 1 exp ( s i ) log ( 1 + exp ( s i ) i − 1 ) . They satisfy
Λ ℓ = k + U ℓ , k + O B ( 1 + k 2 ℓ f ( ℓ ) ) , Q ℓ = V ℓ , k + O B ( 1 + k 2 ℓ f ( ℓ ) ) , \Lambda_\ell=k+U_{\ell,k}+O_B\left(1+\frac{k^2}{\ell f(\ell)}\right),\qquad Q_\ell=V_{\ell,k}+O_B\left(1+\frac{k^2}{\ell f(\ell)}\right), Λ ℓ = k + U ℓ , k + O B ( 1 + ℓ f ( ℓ ) k 2 ) , Q ℓ = V ℓ , k + O B ( 1 + ℓ f ( ℓ ) k 2 ) , and
srank λ ( A ) = Λ ℓ = Θ B ( T ℓ , k ) , erank λ ( A ) = Λ ℓ exp { Q ℓ Λ ℓ } . \operatorname{srank}_{\lambda}(A)=\Lambda_\ell=\Theta_B(T_{\ell,k}),\qquad\operatorname{erank}_{\lambda}(A)=\Lambda_\ell\exp\left\{\frac{Q_\ell}{\Lambda_\ell}\right\}. srank λ ( A ) = Λ ℓ = Θ B ( T ℓ , k ) , erank λ ( A ) = Λ ℓ exp { Λ ℓ Q ℓ } . Consequently,
srank ( A ) , srank λ ( A ) , erank 2 ( A ) = Θ B ( k + log ℓ + 1 k + 1 ) . \operatorname{srank}(A),\ \operatorname{srank}_{\lambda}(A),\ \operatorname{erank}_2(A)=\Theta_B\left(k+\log\frac{\ell+1}{k+1}\right). srank ( A ) , srank λ ( A ) , erank 2 ( A ) = Θ B ( k + log k + 1 ℓ + 1 ) . If J ℓ , k ≤ C 0 T ℓ , k J_{\ell,k}\le C_0T_{\ell,k} J ℓ , k ≤ C 0 T ℓ , k for a fixed constant C 0 C_0 C 0 , then erank λ ( A ) = Θ B , C 0 ( T ℓ , k ) \operatorname{erank}_{\lambda}(A)=\Theta_{B,C_0}(T_{\ell,k}) erank λ ( A ) = Θ B , C 0 ( T ℓ , k ) . Such a constant exists whenever k = Ω ( ( log ℓ ) 2 ) k=\Omega((\log\ell)^2) k = Ω (( log ℓ ) 2 ) . The sharper singular-value equivalents are
erank 2 ( A ) = k ( 1 + o ( 1 ) ) if log ( ℓ + 1 ) = o ( k ) , \operatorname{erank}_2(A)=k(1+o(1))\qquad\text{if }\log(\ell+1)=o(k), erank 2 ( A ) = k ( 1 + o ( 1 )) if log ( ℓ + 1 ) = o ( k ) , erank 1 ( A ) = k ( 1 + o ( 1 ) ) if ( log ( ℓ + 1 ) ) 3 = o ( k ) . \operatorname{erank}_1(A)=k(1+o(1))\qquad\text{if }(\log(\ell+1))^3=o(k). erank 1 ( A ) = k ( 1 + o ( 1 )) if ( log ( ℓ + 1 ) ) 3 = o ( k ) .
At the other extreme, when k = 0 k = 0 k = 0 and s i = 0 s_i = 0 s i = 0 for every i i i ,
srank ( A ) ∼ 1 4 log ℓ , srank λ ( A ) = H ℓ ∼ log ℓ , erank λ ( A ) ∼ e − γ / 2 ℓ log ℓ , \operatorname{srank}(A) \sim\frac{1}{4}\log\ell,\qquad\operatorname{srank}_{\lambda}(A) = \mathcal{H}_{\ell} \sim\log\ell,\qquad\operatorname{erank}_{\lambda}(A) \sim e^{-\gamma/2}\sqrt{\ell\log\ell}, srank ( A ) ∼ 4 1 log ℓ , srank λ ( A ) = H ℓ ∼ log ℓ , erank λ ( A ) ∼ e − γ /2 ℓ log ℓ ,
where H ℓ = ∑ i = 1 ℓ i − 1 \mathcal{H}_{\ell} = \sum_{i=1}^{\ell} i^{-1} H ℓ = ∑ i = 1 ℓ i − 1 and γ \gamma γ is the Euler–Mascheroni constant. In this case erank 2 ( A ) = Θ ( log ℓ ) \operatorname{erank}_{2}(A) = \Theta(\log\ell) erank 2 ( A ) = Θ ( log ℓ ) and erank 1 ( A ) ≥ c ( log ℓ ) 3 \operatorname{erank}_{1}(A) \ge c(\log\ell)^3 erank 1 ( A ) ≥ c ( log ℓ ) 3 .
Proof. Write a i = exp ( s i ) a_i = \exp(s_i) a i = exp ( s i ) and z i = ( a i + i − 1 ) − 1 z_i = (a_i + i - 1)^{-1} z i = ( a i + i − 1 ) − 1 . The i i i -th causal row has i − 1 i - 1 i − 1 entries equal to z i z_i z i , diagonal entry a i z i a_i z_i a i z i , and zeros thereafter, giving the stated ρ i \rho_i ρ i . For a bulk row, direct subtraction gives
ρ i − 1 i = ( i − 1 ) ( a i − 1 ) 2 i ( a i + i − 1 ) 2 , 0 ≤ ρ i − 1 i ≤ C B i 2 . \rho_i - \frac{1}{i} = \frac{(i-1)(a_i-1)^2}{i(a_i+i-1)^2},\qquad0 \le\rho_i - \frac{1}{i} \le\frac{C_B}{i^2}. ρ i − i 1 = i ( a i + i − 1 ) 2 ( i − 1 ) ( a i − 1 ) 2 , 0 ≤ ρ i − i 1 ≤ i 2 C B .
For a spike row,
0 ≤ 1 − ρ i = ( i − 1 ) ( 2 a i + i − 2 ) ( a i + i − 1 ) 2 ≤ C i ℓ f ( ℓ ) . 0 \le1-\rho_i = \frac{(i-1)(2a_i+i-2)}{(a_i+i-1)^2} \le\frac{Ci}{\ell f(\ell)}. 0 ≤ 1 − ρ i = ( a i + i − 1 ) 2 ( i − 1 ) ( 2 a i + i − 2 ) ≤ ℓ f ( ℓ ) C i .
Summation proves the expansion of W ℓ W_\ell W ℓ . Its error is o ( T ℓ , k ) o(T_{\ell,k}) o ( T ℓ , k ) : the spike error divided by T ℓ , k ≥ k T_{\ell,k} \ge k T ℓ , k ≥ k is at most C k / ( ℓ f ( ℓ ) ) = o ( 1 ) Ck/(\ell f(\ell)) = o(1) C k / ( ℓ f ( ℓ )) = o ( 1 ) , and T ℓ , k → ∞ T_{\ell,k} \to\infty T ℓ , k → ∞ , making the bounded remainder negligible.
The exact decomposition
A = D H ℓ + C , D i i = i a i + i − 1 , C i i = a i − 1 a i + i − 1 , D + C = I A = DH_\ell+ C,\qquad D_{ii} = \frac{i}{a_i+i-1},\qquad C_{ii} = \frac{a_i-1}{a_i+i-1},\qquad D+C=I A = D H ℓ + C , D ii = a i + i − 1 i , C ii = a i + i − 1 a i − 1 , D + C = I
has diagonal matrices 0 ≤ D , C ≤ I 0 \le D,C \le I 0 ≤ D , C ≤ I . Step 1 of Appendix C.4 gives ∥ A ∥ o p ≤ 5 \lVert A\rVert_{\mathrm{op}} \le\sqrt{5} ∥ A ∥ op ≤ 5 . Since A 1 = 1 A\mathbf{1}=\mathbf{1} A 1 = 1 , its operator norm is at least one. This proves the stable-rank bounds.
For h ( x ) = x log ( 1 / x ) h(x)=x\log(1/x) h ( x ) = x log ( 1/ x ) , with h ( 0 ) = 0 h(0)=0 h ( 0 ) = 0 , the row estimates and the mean-value theorem give
h ( ρ i ) = log i i + O B ( log ( i + 1 ) i 2 ) ( i > k ) . h(\rho_i)=\frac{\log i}{i}+O_B\left(\frac{\log(i+1)}{i^2}\right)\qquad(i>k). h ( ρ i ) = i log i + O B ( i 2 log ( i + 1 ) ) ( i > k ) .
For spikes, 0 ≤ h ( 1 − ε ) ≤ ε 0\le h(1-\varepsilon)\le\varepsilon 0 ≤ h ( 1 − ε ) ≤ ε for 0 ≤ ε ≤ 1 0\le\varepsilon\le1 0 ≤ ε ≤ 1 , so
0 ≤ ∑ i ≤ k h ( ρ i ) ≤ ∑ i ≤ k ( 1 − ρ i ) = O ( k 2 ℓ f ( ℓ ) ) . 0\le\sum_{i\le k}h(\rho_i)\le\sum_{i\le k}(1-\rho_i)=O\left(\frac{k^2}{\ell f(\ell)}\right). 0 ≤ i ≤ k ∑ h ( ρ i ) ≤ i ≤ k ∑ ( 1 − ρ i ) = O ( ℓ f ( ℓ ) k 2 ) .
Their sum is the claimed expansion of K ℓ K_\ell K ℓ . The normalized squared singular values are the eigenvalues of A A T / W ℓ AA^{\mathsf T}/W_\ell A A T / W ℓ , whose diagonal is ( ρ i / W ℓ ) i (\rho_i/W_\ell)_i ( ρ i / W ℓ ) i . By Schur–Horn and Schur concavity of entropy,
log erank 2 ( A ) ≤ log W ℓ + K ℓ / W ℓ . \log\operatorname{erank}_2(A)\le\log W_\ell+K_\ell/W_\ell. log erank 2 ( A ) ≤ log W ℓ + K ℓ / W ℓ .
The sharper entropy order follows from the Schatten bound. Fix p ∈ ( 1 , 2 ) p\in(1,2) p ∈ ( 1 , 2 ) . Bounded bulk scores imply
∥ C ∥ S p p ≤ k + ( e B − 1 ) p ∑ i = k + 1 ℓ i − p ≤ k + C B , p . \lVert C\rVert_{\mathcal{S}_p}^p\le k+(e^B-1)^p\sum_{i=k+1}^{\ell}i^{-p}\le k+C_{B,p}. ∥ C ∥ S p p ≤ k + ( e B − 1 ) p i = k + 1 ∑ ℓ i − p ≤ k + C B , p .
The Schatten triangle inequality and Lemma 1 therefore give
∑ j σ j ( A ) p ≤ C B , p ( k + log ( ℓ + 1 ) ) ≤ C B , p T ℓ , k , \sum_j \sigma_j(A)^p\le C_{B,p}(k+\log(\ell+1))\le C_{B,p}T_{\ell,k}, j ∑ σ j ( A ) p ≤ C B , p ( k + log ( ℓ + 1 )) ≤ C B , p T ℓ , k ,
where log ( ℓ + 1 ) ≤ log ( k + 1 ) + L ℓ , k ≤ T ℓ , k \log(\ell+1)\le\log(k+1)+L_{\ell,k}\le T_{\ell,k} log ( ℓ + 1 ) ≤ log ( k + 1 ) + L ℓ , k ≤ T ℓ , k . For 0 < x ≤ 1 0<x\le1 0 < x ≤ 1 , x 2 log ( 1 / x 2 ) ≤ C p x p x^2\log(1/x^2)\le C_p x^p x 2 log ( 1/ x 2 ) ≤ C p x p ; for x > 1 x>1 x > 1 the left-hand side is nonpositive. Thus
log erank 2 ( A ) = log W ℓ + 1 W ℓ ∑ j σ j ( A ) 2 log 1 σ j ( A ) 2 ≤ log W ℓ + C B . \log\operatorname{erank}_2(A)=\log W_\ell+\frac{1}{W_\ell}\sum_j\sigma_j(A)^2\log\frac{1}{\sigma_j(A)^2}\le\log W_\ell+C_B. log erank 2 ( A ) = log W ℓ + W ℓ 1 j ∑ σ j ( A ) 2 log σ j ( A ) 2 1 ≤ log W ℓ + C B .
Conversely, Shannon entropy is at least minus the logarithm of the largest probability, so erank 2 ( A ) ≥ srank ( A ) \operatorname{erank}_2(A)\ge\operatorname{srank}(A) erank 2 ( A ) ≥ srank ( A ) . This proves erank 2 ( A ) = Θ B ( T ℓ , k ) \operatorname{erank}_2(A)=\Theta_B(T_{\ell,k}) erank 2 ( A ) = Θ B ( T ℓ , k ) .
We next estimate N ℓ N_\ell N ℓ . Let P P P project onto the bulk coordinates. On these coordinates D i i ≥ e − B D_{ii}\ge e^{-B} D ii ≥ e − B ; hence
∥ P D H ℓ ∥ ∗ ≥ e − B ∥ P H ℓ ∥ ∗ ≥ e − B ( ∥ H ℓ ∥ ∗ − 2 k ) . \lVert PDH_\ell\rVert_*\ge e^{-B}\lVert PH_\ell\rVert_*\ge e^{-B}(\lVert H_\ell\rVert_*-2k). ∥ P D H ℓ ∥ ∗ ≥ e − B ∥ P H ℓ ∥ ∗ ≥ e − B (∥ H ℓ ∥ ∗ − 2 k ) .
The last inequality uses that ( I − P ) H ℓ (I-P)H_\ell ( I − P ) H ℓ has rank at most k k k and operator norm at most two. Also ∥ P C ∥ ∗ ≤ ( e B − 1 ) H ℓ \|PC\|_* \le(e^B-1)H_\ell ∥ P C ∥ ∗ ≤ ( e B − 1 ) H ℓ . The reverse triangle inequality therefore gives
N ℓ ≥ ∥ P A ∥ ∗ ≥ c B ( log ( ℓ + 1 ) ) 2 − C B k − C B log ( ℓ + 1 ) . N_\ell\ge\|PA\|_* \ge c_B(\log(\ell+1))^2-C_Bk-C_B\log(\ell+1). N ℓ ≥ ∥ P A ∥ ∗ ≥ c B ( log ( ℓ + 1 ) ) 2 − C B k − C B log ( ℓ + 1 ) .
Meanwhile, N ℓ ≥ ∣ tr A ∣ ≥ k / 2 N_\ell\ge|\operatorname{tr} A| \ge k/2 N ℓ ≥ ∣ tr A ∣ ≥ k /2 for all sufficiently large ℓ \ell ℓ , since spike diagonal entries tend uniformly to one. Use the first bound when k k k is a sufficiently small multiple of ( log ( ℓ + 1 ) ) 2 (\log(\ell+1))^2 ( log ( ℓ + 1 ) ) 2 , and the second otherwise, to obtain N ℓ ≥ c B ( k + ( log ( ℓ + 1 ) ) 2 ) N_\ell\ge c_B(k+(\log(\ell+1))^2) N ℓ ≥ c B ( k + ( log ( ℓ + 1 ) ) 2 ) . The reverse bound follows from
N ℓ ≤ ∥ H ℓ ∥ ∗ + ∥ C ∥ ∗ ≤ C B ( k + ( log ( ℓ + 1 ) ) 2 ) . N_\ell\le\|H_\ell\|_*+\|C\|_* \le C_B\left(k+(\log(\ell+1))^2\right). N ℓ ≤ ∥ H ℓ ∥ ∗ + ∥ C ∥ ∗ ≤ C B ( k + ( log ( ℓ + 1 ) ) 2 ) .
For q j ( 1 ) = σ j ( A ) / N ℓ q_j^{(1)}=\sigma_j(A)/N_\ell q j ( 1 ) = σ j ( A ) / N ℓ , Shannon entropy dominates the order-two Renyi entropy, yielding erank 1 ( A ) ≥ 1 / ∑ j ( q j ( 1 ) ) 2 = N ℓ 2 / W ℓ \operatorname{erank}_1(A)\ge1/\sum_j(q_j^{(1)})^2=N_\ell^2/W_\ell erank 1 ( A ) ≥ 1/ ∑ j ( q j ( 1 ) ) 2 = N ℓ 2 / W ℓ . Finally, for q j ( α ) = σ j ( A ) α / ∑ r σ r ( A ) α q_j^{(\alpha)}=\sigma_j(A)^\alpha/\sum_r\sigma_r(A)^\alpha q j ( α ) = σ j ( A ) α / ∑ r σ r ( A ) α ,
d d α H ( q ( α ) ) = − α Var q ( α ) ( log σ j ( A ) ) ≤ 0. \frac{\mathrm{d}}{\mathrm{d}\alpha}H(q^{(\alpha)})=-\alpha\operatorname{Var}_{q^{(\alpha)}}(\log\sigma_j(A))\le0. d α d H ( q ( α ) ) = − α Var q ( α ) ( log σ j ( A )) ≤ 0.
Taking α = 1 , 2 \alpha=1,2 α = 1 , 2 proves erank 2 ( A ) ≤ erank 1 ( A ) \operatorname{erank}_2(A)\le\operatorname{erank}_1(A) erank 2 ( A ) ≤ erank 1 ( A ) , and erank 1 ( A ) ≤ ℓ \operatorname{erank}_1(A)\le\ell erank 1 ( A ) ≤ ℓ follows from the maximum entropy bound.
For the eigenvalue proxies, triangularity gives
λ i ( A ) = a i a i + i − 1 > 0 , max i λ i ( A ) = λ 1 ( A ) = 1. \lambda_i(A)=\frac{a_i}{a_i+i-1}>0,\qquad\max_i\lambda_i(A)=\lambda_1(A)=1. λ i ( A ) = a i + i − 1 a i > 0 , i max λ i ( A ) = λ 1 ( A ) = 1.
The exact expressions for srank λ \operatorname{srank}_\lambda srank λ and erank λ \operatorname{erank}_\lambda erank λ follow immediately. For i > k i>k i > k ,
λ i ( A ) = a i i + O B ( i − 2 ) , h ( λ i ( A ) ) = a i i log i a i + O B ( log ( i + 1 ) i 2 ) . \lambda_i(A)=\frac{a_i}{i}+O_B(i^{-2}),\qquad h(\lambda_i(A))=\frac{a_i}{i}\log\frac{i}{a_i}+O_B\left(\frac{\log(i+1)}{i^2}\right). λ i ( A ) = i a i + O B ( i − 2 ) , h ( λ i ( A )) = i a i log a i i + O B ( i 2 log ( i + 1 ) ) .
For i ≤ k i\le k i ≤ k ,
0 ≤ h ( λ i ( A ) ) ≤ 1 − λ i ( A ) ≤ i ℓ f ( ℓ ) . 0\le h(\lambda_i(A))\le1-\lambda_i(A)\le\frac{i}{\ell f(\ell)}. 0 ≤ h ( λ i ( A )) ≤ 1 − λ i ( A ) ≤ ℓ f ( ℓ ) i .
Summation proves the expansions of Λ ℓ \Lambda_\ell Λ ℓ and Q ℓ Q_\ell Q ℓ . Since 1 ≤ a i ≤ e B 1\le a_i\le e^B 1 ≤ a i ≤ e B in the bulk,
U ℓ , k = Θ B ( L ℓ , k ) , ∣ V ℓ , k ∣ ≤ e B ( J ℓ , k + B L ℓ , k ) . U_{\ell,k}=\Theta_B(L_{\ell,k}),\qquad|V_{\ell,k}|\le e^B(J_{\ell,k}+BL_{\ell,k}). U ℓ , k = Θ B ( L ℓ , k ) , ∣ V ℓ , k ∣ ≤ e B ( J ℓ , k + B L ℓ , k ) .
This gives Λ ℓ = Θ B ( T ℓ , k ) \Lambda_\ell=\Theta_B(T_{\ell,k}) Λ ℓ = Θ B ( T ℓ , k ) and, when J ℓ , k ≤ C 0 T ℓ , k J_{\ell,k}\le C_0T_{\ell,k} J ℓ , k ≤ C 0 T ℓ , k , gives Q ℓ / Λ ℓ = O B , C 0 ( 1 ) Q_\ell/\Lambda_\ell=O_{B,C_0}(1) Q ℓ / Λ ℓ = O B , C 0 ( 1 ) . The harmonic-sum estimate
L ℓ , k ≍ log ℓ + 1 k + 1 L_{\ell,k}\asymp\log\frac{\ell+1}{k+1} L ℓ , k ≍ log k + 1 ℓ + 1
includes the empty-sum endpoint. Also J ℓ , k ≤ log ℓ L ℓ , k = O ( ( log ℓ ) 2 ) J_{\ell,k}\le\log\ell\,L_{\ell,k}=O((\log\ell)^2) J ℓ , k ≤ log ℓ L ℓ , k = O (( log ℓ ) 2 ) , so k = Ω ( ( log ℓ ) 2 ) k=\Omega((\log\ell)^2) k = Ω (( log ℓ ) 2 ) implies the stated condition on J J J .
It remains to prove the two singular-value equivalents. The spike principal block is I k + E ℓ I_k+E_\ell I k + E ℓ , where the Schur test gives
δ ℓ : = ∥ E ℓ ∥ o p = O ( k ℓ f ( ℓ ) ) = o ( 1 ) . \delta_\ell:=\|E_\ell\|_{\mathrm{op}}=O\left(\frac{k}{\ell f(\ell)}\right)=o(1). δ ℓ := ∥ E ℓ ∥ op = O ( ℓ f ( ℓ ) k ) = o ( 1 ) .
The max-min principle on the first k k k coordinates gives σ j ( A ) ≥ 1 − δ ℓ \sigma_j(A)\ge1-\delta_\ell σ j ( A ) ≥ 1 − δ ℓ for j ≤ k j\le k j ≤ k . The best rank-k k k approximation property of Schatten norms, applied to ( I − P ) A (I-P)A ( I − P ) A , gives
∑ j > k σ j ( A ) p ≤ ∥ P A ∥ S p p ≤ C B , p log ( ℓ + 1 ) , 1 < p < 2 , \sum_{j>k}\sigma_j(A)^p\le\|PA\|_{\mathcal{S}_p}^p\le C_{B,p}\log(\ell+1),\qquad1<p<2, j > k ∑ σ j ( A ) p ≤ ∥ P A ∥ S p p ≤ C B , p log ( ℓ + 1 ) , 1 < p < 2 ,
and
τ ℓ : = ∑ j > k σ j ( A ) ≤ ∥ P A ∥ ∗ ≤ C B ( log ( ℓ + 1 ) ) 2 . \tau_\ell:=\sum_{j>k}\sigma_j(A)\le\|PA\|_*\le C_B(\log(\ell+1))^2. τ ℓ := j > k ∑ σ j ( A ) ≤ ∥ P A ∥ ∗ ≤ C B ( log ( ℓ + 1 ) ) 2 .
Here both estimates follow from P A = P D H ℓ + P C PA=PDH_\ell+PC P A = P D H ℓ + P C and the Hardy Schatten bounds, since ∥ P C ∥ S p ≤ C B , p \|PC\|_{\mathcal{S}_p}\le C_{B,p} ∥ P C ∥ S p ≤ C B , p .
If log ( ℓ + 1 ) = o ( k ) \log(\ell+1)=o(k) log ( ℓ + 1 ) = o ( k ) , then W ℓ = k + o ( k ) W_\ell=k+o(k) W ℓ = k + o ( k ) . Put
S 1 = ∑ j ≤ k σ j ( A ) 2 , R = W ℓ − S 1 , Δ − = ∑ j ≤ k ( 1 − σ j ( A ) 2 ) + . S_1=\sum_{j\le k}\sigma_j(A)^2,\qquad R=W_\ell-S_1,\qquad\Delta_-=\sum_{j\le k}(1-\sigma_j(A)^2)_+. S 1 = j ≤ k ∑ σ j ( A ) 2 , R = W ℓ − S 1 , Δ − = j ≤ k ∑ ( 1 − σ j ( A ) 2 ) + .
The spike lower bound gives S 1 ≥ k ( 1 − 2 δ ℓ ) S_1 \ge k(1-2\delta_\ell) S 1 ≥ k ( 1 − 2 δ ℓ ) and Δ − ≤ 2 k δ ℓ \Delta_- \le2k\delta_\ell Δ − ≤ 2 k δ ℓ , while R ≤ W ℓ − k + 2 k δ ℓ = o ( k ) R \le W_\ell-k+2k\delta_\ell=o(k) R ≤ W ℓ − k + 2 k δ ℓ = o ( k ) . If Δ + = ∑ j ≤ k ( σ j ( A ) 2 − 1 ) + \Delta_+=\sum_{j\le k}(\sigma_j(A)^2-1)_+ Δ + = ∑ j ≤ k ( σ j ( A ) 2 − 1 ) + , then the identity
Δ + = ( S 1 − k ) + Δ − = ( W ℓ − k ) − R + Δ − \Delta_+=(S_1-k)+\Delta_-=(W_\ell-k)-R+\Delta_- Δ + = ( S 1 − k ) + Δ − = ( W ℓ − k ) − R + Δ −
shows that Δ + = o ( k ) \Delta_+=o(k) Δ + = o ( k ) . Thus the total squared mass above one, Δ + + ∑ j > k ( σ j ( A ) 2 − 1 ) + \Delta_++\sum_{j>k}(\sigma_j(A)^2-1)_+ Δ + + ∑ j > k ( σ j ( A ) 2 − 1 ) + , is o ( k ) o(k) o ( k ) . The positive part of the entropy moment (the terms with σ j ≤ 1 \sigma_j\le1 σ j ≤ 1 ) is bounded by C k δ ℓ + C B log ( ℓ + 1 ) = o ( k ) Ck\delta_\ell+C_B\log(\ell+1)=o(k) C k δ ℓ + C B log ( ℓ + 1 ) = o ( k ) , using the fixed-p p p Schatten estimate above. Since all singular values are at most 5 \sqrt{5} 5 and x log x ≤ C ( x − 1 ) x\log x\le C(x-1) x log x ≤ C ( x − 1 ) on [ 1 , 5 ] [1,5] [ 1 , 5 ] , the negative part is also o ( k ) o(k) o ( k ) . Substituting into the entropy formula proves erank 2 ( A ) = k ( 1 + o ( 1 ) ) \operatorname{erank}_2(A)=k(1+o(1)) erank 2 ( A ) = k ( 1 + o ( 1 )) .
For erank 1 \operatorname{erank}_1 erank 1 , the row decomposition and the spike block bound give
k ( 1 − δ ℓ ) ≤ N ℓ ≤ k ( 1 + δ ℓ ) + C B ( log ( ℓ + 1 ) ) 2 k(1-\delta_\ell)\le N_\ell\le k(1+\delta_\ell)+C_B(\log(\ell+1))^2 k ( 1 − δ ℓ ) ≤ N ℓ ≤ k ( 1 + δ ℓ ) + C B ( log ( ℓ + 1 ) ) 2
The first k k k terms obey h ( σ j ( A ) ) ≤ C δ ℓ h(\sigma_j(A))\le C\delta_\ell h ( σ j ( A )) ≤ C δ ℓ . For the tail, concavity of h h h yields
∑ j > k h ( σ j ( A ) ) ≤ τ ℓ log ℓ − k τ ℓ ≤ τ ℓ log ℓ + 1 e ≤ C B ( log ( ℓ + 1 ) ) 3 , \sum_{j>k}h(\sigma_j(A))\le\tau_\ell\log\frac{\ell-k}{\tau_\ell}\le\tau_\ell\log\ell+\frac{1}{e}\le C_B(\log(\ell+1))^3, j > k ∑ h ( σ j ( A )) ≤ τ ℓ log τ ℓ ℓ − k ≤ τ ℓ log ℓ + e 1 ≤ C B ( log ( ℓ + 1 ) ) 3 ,
with a zero tail when k = ℓ k=\ell k = ℓ . If ( log ( ℓ + 1 ) ) 3 = o ( k ) (\log(\ell+1))^3=o(k) ( log ( ℓ + 1 ) ) 3 = o ( k ) , then
log erank 1 ( A ) ≤ log N ℓ + C k δ ℓ + C B ( log ( ℓ + 1 ) ) 3 N ℓ = log k + o ( 1 ) . \log\operatorname{erank}_1(A)\le\log N_\ell+\frac{Ck\delta_\ell+C_B(\log(\ell+1))^3}{N_\ell}=\log k+o(1). log erank 1 ( A ) ≤ log N ℓ + N ℓ C k δ ℓ + C B ( log ( ℓ + 1 ) ) 3 = log k + o ( 1 ) .
The lower bound erank 1 ( A ) ≥ N ℓ 2 / W ℓ = k ( 1 + o ( 1 ) ) \operatorname{erank}_1(A)\ge N_\ell^2/W_\ell=k(1+o(1)) erank 1 ( A ) ≥ N ℓ 2 / W ℓ = k ( 1 + o ( 1 )) proves the equivalent.
Finally, in the zero-score case A = H ℓ A=H_\ell A = H ℓ and W ℓ = H ℓ W_\ell=H_\ell W ℓ = H ℓ . For the test vector x j = j − 1 / 2 x_j=j^{-1/2} x j = j − 1/2 ,
∥ x ∥ 2 2 = H ℓ , ( A x ) i = 1 i ∑ j = 1 i 1 j = 2 i + O ( i − 1 ) \lVert x\rVert_2^2=H_\ell,\qquad(Ax)_i=\frac{1}{i}\sum_{j=1}^{i}\frac{1}{\sqrt{j}}=\frac{2}{\sqrt{i}}+O(i^{-1}) ∥ x ∥ 2 2 = H ℓ , ( A x ) i = i 1 j = 1 ∑ i j 1 = i 2 + O ( i − 1 )
so ∥ A x ∥ 2 2 = 4 H ℓ + O ( 1 ) \lVert Ax\rVert_2^2=4H_\ell+O(1) ∥ A x ∥ 2 2 = 4 H ℓ + O ( 1 ) . Together with ∥ H ℓ ∥ o p ≤ 2 \lVert H_\ell\rVert_{\mathrm{op}}\le2 ∥ H ℓ ∥ op ≤ 2 , this proves ∥ A ∥ o p → 2 \lVert A\rVert_{\mathrm{op}}\to2 ∥ A ∥ op → 2 and the stated stable-rank equivalent. The eigenvalues are exactly 1 / i 1/i 1/ i , whence
Λ ℓ = H ℓ = log ℓ + γ + o ( 1 ) , Q ℓ = ∑ i = 1 ℓ log i i = 1 2 ( log ℓ ) 2 + O ( 1 ) . \Lambda_\ell=H_\ell=\log\ell+\gamma+o(1),\qquad Q_\ell=\sum_{i=1}^{\ell}\frac{\log i}{i}=\frac{1}{2}(\log\ell)^2+O(1). Λ ℓ = H ℓ = log ℓ + γ + o ( 1 ) , Q ℓ = i = 1 ∑ ℓ i log i = 2 1 ( log ℓ ) 2 + O ( 1 ) .
Thus Q ℓ / Λ ℓ = 1 2 log ℓ − γ / 2 + o ( 1 ) Q_\ell/\Lambda_\ell=\frac{1}{2}\log\ell-\gamma/2+o(1) Q ℓ / Λ ℓ = 2 1 log ℓ − γ /2 + o ( 1 ) , proving the final eigenvalue-entropy equivalent. □
Remark 7 (Weaker assumptions for the order bounds). The estimates
srank ( A ) , srank λ ( A ) , erank 2 ( A ) = Θ B , η ( k + log ℓ + 1 k + 1 ) \operatorname{srank}(A),\ \operatorname{srank}_{\lambda}(A),\ \operatorname{erank}_2(A)=\Theta_{B,\eta}\left(k+\log\frac{\ell+1}{k+1}\right) srank ( A ) , srank λ ( A ) , erank 2 ( A ) = Θ B , η ( k + log k + 1 ℓ + 1 )
need only a i = e s i ≥ 1 a_i=e^{s_i}\ge1 a i = e s i ≥ 1 , a i ≤ e B a_i\le e^B a i ≤ e B for i > k i>k i > k , and a i ≥ η i a_i\ge\eta i a i ≥ η i for i ≤ k i\le k i ≤ k , where η > 0 \eta>0 η > 0 is fixed. Indeed, the spike diagonal entries are then at least η / ( 1 + η ) \eta/(1+\eta) η / ( 1 + η ) , so W ℓ ≍ B , η T ℓ , k W_\ell\asymp_{B,\eta}T_{\ell,k} W ℓ ≍ B , η T ℓ , k ; the operator and Schatten bounds above remain valid. The triangular eigenvalue formulas likewise give Λ ℓ ≍ B , η T ℓ , k \Lambda_\ell\asymp_{B,\eta}T_{\ell,k} Λ ℓ ≍ B , η T ℓ , k . The nuclear-norm order and the resulting erank 1 \operatorname{erank}_1 erank 1 lower bound also hold, with constants depending on B , η B,\eta B , η . These order statements require neither ordering of the scores nor asymptotically diverging spike separation. The stronger equivalents in the theorem use the vanishing spike deficits supplied by Assumption 2.
Remark 8 (Polynomial entropy rank of the Hardy matrix). For zero scores, the two singular-value entropy ranks have different growth scales:
erank 2 ( H ℓ ) = Θ ( log ℓ ) , log erank 1 ( H ℓ ) = Θ ( log ℓ ) . \operatorname{erank}_2(H_\ell)=\Theta(\log\ell),\qquad\log\operatorname{erank}_1(H_\ell)=\Theta(\log\ell). erank 2 ( H ℓ ) = Θ ( log ℓ ) , log erank 1 ( H ℓ ) = Θ ( log ℓ ) .
Thus erank 1 ( H ℓ ) = ℓ Θ ( 1 ) \operatorname{erank}_1(H_\ell)=\ell^{\Theta(1)} erank 1 ( H ℓ ) = ℓ Θ ( 1 ) , strengthening the polylogarithmic lower bound in Theorem 7.
To see this, use the dyadic blocks in Lemma 1 and let B 0 B_0 B 0 retain only the diagonal blocks, with R = ⌈ log 2 ℓ ⌉ R=\lceil\log_2\ell\rceil R = ⌈ log 2 ℓ ⌉ . In each row block, the entries strictly before its diagonal block form a rank-one matrix; hence rank ( H ℓ − B 0 ) ≤ R \operatorname{rank}(H_\ell-B_0)\le R rank ( H ℓ − B 0 ) ≤ R . Both matrices have operator norm at most two. Singular-value rank interlacing therefore implies that their counting functions n X ( t ) = # { j : σ j ( X ) > t } n_X(t)=\#\{j:\sigma_j(X)>t\} n X ( t ) = # { j : σ j ( X ) > t } differ by at most R R R . For h ( t ) = t log ( 1 / t ) h(t)=t\log(1/t) h ( t ) = t log ( 1/ t ) , with h ( 0 ) = 0 h(0)=0 h ( 0 ) = 0 , it follows that
∣ ∑ j h ( σ j ( H ℓ ) ) − ∑ j h ( σ j ( B 0 ) ) ∣ ≤ R ∫ 0 2 ∣ h ′ ( t ) ∣ d t = O ( log ℓ ) . \left|\sum_j h(\sigma_j(H_\ell))-\sum_j h(\sigma_j(B_0))\right|\le R\int_0^2 |h'(t)|\,\mathrm{d}t=O(\log\ell). j ∑ h ( σ j ( H ℓ )) − j ∑ h ( σ j ( B 0 )) ≤ R ∫ 0 2 ∣ h ′ ( t ) ∣ d t = O ( log ℓ ) .
Here we used ∑ j h ( σ j ( X ) ) = ∫ 0 2 h ′ ( t ) n X ( t ) d t \sum_j h(\sigma_j(X))=\int_0^2 h'(t)n_X(t)\,\mathrm{d}t ∑ j h ( σ j ( X )) = ∫ 0 2 h ′ ( t ) n X ( t ) d t . Every full dyadic block of size m = 2 r m=2^r m = 2 r has c / j ≤ σ j ≤ C / j c/j\le\sigma_j\le C/j c / j ≤ σ j ≤ C / j , by the formula for σ j ( L m ) \sigma_j(L_m) σ j ( L m ) and the bounds on its row multipliers. For all sufficiently large j j j , its entropy summand is consequently comparable to ( log j ) / j (\log j)/j ( log j ) / j ; the remaining finitely many summands are bounded. Summing first over j j j and then over the full blocks r < R r<R r < R gives
∑ j h ( σ j ( B 0 ) ) = Θ ( ( log ℓ ) 3 ) . \sum_j h(\sigma_j(B_0))=\Theta((\log\ell)^3). j ∑ h ( σ j ( B 0 )) = Θ (( log ℓ ) 3 ) .
The possibly truncated final block obeys the same upper singular-value bound, contributes at most O ( ( log ℓ ) 2 ) O((\log\ell)^2) O (( log ℓ ) 2 ) , and has a negative part bounded by a constant. Since N ℓ = ∥ H ℓ ∥ ∗ = Θ ( ( log ℓ ) 2 ) N_\ell=\|H_\ell\|_*=\Theta((\log\ell)^2) N ℓ = ∥ H ℓ ∥ ∗ = Θ (( log ℓ ) 2 ) , the entropy identity
log erank 1 ( H ℓ ) = log N ℓ + ∑ j h ( σ j ( H ℓ ) ) N ℓ \log\operatorname{erank}_1(H_\ell)=\log N_\ell+\frac{\sum_j h(\sigma_j(H_\ell))}{N_\ell} log erank 1 ( H ℓ ) = log N ℓ + N ℓ ∑ j h ( σ j ( H ℓ ))
proves the claim. Determining an exact exponent or a sharp equivalent requires finer spectral information.
Auxiliary matrix norm bound
Lemma 2 (Schur test). Let A = ( a p q ) 1 ≤ p , q ≤ m A=(a_{pq})_{1\le p,q\le m} A = ( a pq ) 1 ≤ p , q ≤ m be a real m × m m\times m m × m matrix. Assume that
sup 1 ≤ p ≤ m ∑ q = 1 m ∣ a p q ∣ ≤ R , sup 1 ≤ q ≤ m ∑ p = 1 m ∣ a p q ∣ ≤ C . \sup_{1\le p\le m}\sum_{q=1}^m|a_{pq}|\le R,\qquad\sup_{1\le q\le m}\sum_{p=1}^m|a_{pq}|\le C. 1 ≤ p ≤ m sup q = 1 ∑ m ∣ a pq ∣ ≤ R , 1 ≤ q ≤ m sup p = 1 ∑ m ∣ a pq ∣ ≤ C . Then
∥ A ∥ o p ≤ R C . \|A\|_{\mathrm{op}}\le\sqrt{RC}. ∥ A ∥ op ≤ R C . In particular, if every row sum and every column sum of ∣ A ∣ |A| ∣ A ∣ are bounded by M M M , then
∥ A ∥ o p ≤ M . \|A\|_{\mathrm{op}}\le M. ∥ A ∥ op ≤ M .
Lemma 3 (Hardy inequality on ℓ 2 \ell^2 ℓ 2 ). For m ≥ 1 m\ge1 m ≥ 1 and u ∈ R m u\in\mathbb{R}^m u ∈ R m , define
( H m u ) p = 1 p ∑ q = 1 p u q , 1 ≤ p ≤ m . (H_mu)_p=\frac{1}{p}\sum_{q=1}^p u_q,\qquad1\le p\le m. ( H m u ) p = p 1 q = 1 ∑ p u q , 1 ≤ p ≤ m . Then
∑ p = 1 m ( 1 p ∑ q = 1 p ∣ u q ∣ ) 2 ≤ 4 ∑ q = 1 m ∣ u q ∣ 2 , equivalently ∥ H m ∣ u ∣ ∥ 2 ≤ 2 ∥ u ∥ 2 . \sum_{p=1}^m\left(\frac{1}{p}\sum_{q=1}^p|u_q|\right)^2\le4\sum_{q=1}^m|u_q|^2,\qquad\text{equivalently}\qquad\|H_m|u|\|_2\le2\|u\|_2. p = 1 ∑ m ( p 1 q = 1 ∑ p ∣ u q ∣ ) 2 ≤ 4 q = 1 ∑ m ∣ u q ∣ 2 , equivalently ∥ H m ∣ u ∣ ∥ 2 ≤ 2∥ u ∥ 2 . Moreover, for any integer k ≥ 1 k\ge1 k ≥ 1 , the shifted Hardy operator
( H m , k s h i f t u ) p = 1 k + p − 1 ∑ q = 1 p u q , 1 ≤ p ≤ m , (H^{\mathrm{shift}}_{m,k}u)_p=\frac{1}{k+p-1}\sum_{q=1}^p u_q,\qquad1\le p\le m, ( H m , k shift u ) p = k + p − 1 1 q = 1 ∑ p u q , 1 ≤ p ≤ m , satisfies
∥ H m , k s h i f t ∣ u ∣ ∥ 2 ≤ 2 ∥ u ∥ 2 . \|H^{\mathrm{shift}}_{m,k}|u|\|_2\le2\|u\|_2. ∥ H m , k shift ∣ u ∣ ∥ 2 ≤ 2∥ u ∥ 2 . Consequently, if a linear map M : R m → R m M:\mathbb{R}^m\to\mathbb{R}^m M : R m → R m satisfies
∣ ( M v ) p ∣ ≤ C ( H m , k s h i f t ∣ v ∣ ) p for all 1 ≤ p ≤ m , |(Mv)_p|\le C(H^{\mathrm{shift}}_{m,k}|v|)_p\qquad\text{for all }1\le p\le m, ∣ ( M v ) p ∣ ≤ C ( H m , k shift ∣ v ∣ ) p for all 1 ≤ p ≤ m , then ∥ M ∥ o p ≤ 2 C \|M\|_{\mathrm{op}}\le2C ∥ M ∥ op ≤ 2 C .
Lemma 4 (Rank perturbation and singular values). Let M = N + R M=N+R M = N + R be real ℓ × ℓ \ell\times\ell ℓ × ℓ matrices, and suppose rank ( N ) ≤ k \operatorname{rank}(N)\le k rank ( N ) ≤ k . Then, for every 1 ≤ m ≤ ℓ − k 1\le m\le\ell-k 1 ≤ m ≤ ℓ − k ,
σ k + m ( M ) ≤ σ m ( R ) . \sigma_{k+m}(M)\le\sigma_m(R). σ k + m ( M ) ≤ σ m ( R ) .
Additional experimentation details
The hidden-state angular diagnostics and GPT-2 score-rank summaries (Figure 3 , 7, and 4) were performed on an AMD EPYC 9554 dual-socket CPU server with 64 cores per socket and 126 GB of RAM, without GPU acceleration, requiring approximately 24 hours of wall-clock time in total. The real-versus-canonical comparisons and score-level angular and spectral diagnostics (Sections 4.3 and 4.4) were performed on a MacBook Pro with an M1 Pro chip and 16 GB of unified memory, requiring approximately two hours in total. The reported timings are totals for each experiment group.
The analysis code and saved figure inputs are released at https://github.com/Mengda-Li/attention, with the authors’ code under the MIT license. The repository README and docs/REPRODUCING_PAPER.md provide environment setup, data preparation, and a figure-to-script guide; paper_figures/README.md documents the publication figures and their inputs and output definitions. The guides distinguish rendering from supplied CSV or archived figure inputs from collecting model measurements and record the available collection workflows and their limitations.
Licenses for existing assets
All existing assets are used only for non-commercial research evaluation and spectral diagnostics, and we do not redistribute the underlying datasets or model weights. The datasets used in the experiments are Imagenette [Howard, 2019], a subset of ImageNet [3 ] , and Wikitext-103 [Merity et al., 2017]. ImageNet states that it does not own the copyright of the images and provides access under terms permitting non-commercial research and educational use; these terms are available at https://image-net.org/accessagreement. The Wikitext-103 data were accessed through the Hugging Face dataset repository https://huggingface.co/datasets/Salesforce/wikitext, whose repository metadata lists the licenses as cc-by-sa-3.0 and gfdl.
For language-model experiments, we use GPT-2 [Radford et al., 2019] through the Hugging Face Transformers library, which is licensed under Apache-2.0 https://github.com/huggingface/transformers. The GPT-2 weights are loaded from openai-community/gpt2 on Hugging Face https://huggingface.co/openai-community/gpt2, whose model card lists the license as MIT. For vision experiments, we use the ViT-B/16 implementation from TorchVision https://github.com/pytorch/vision, which is licensed under BSD-3-Clause. The IMAGENET1K_SWAG_E2E_V1 weights are documented by TorchVision as SWAG-derived weights, and the SWAG repository releases its model weights under CC-BY-NC 4.0 https://github.com/facebookresearch/SWAG/blob/main/LICENSE.
Additional experimental figures
Figure 7 and 4 provide supplementary hidden-state diagnostics and score-rank summaries.
Additional experiment details
The direct attention comparison uses 12 WikiText-103 validation documents at nested lengths 64, 128, 256, 384, 512, 768, 1024, with all 12 GPT-2 layers and 12 heads, giving 12,096 matched comparisons. The ViT comparison uses 20 class-balanced Imagenette images, all 12 layers and 12 heads, and sequence length 577, giving 2,880 comparisons. Sampling uses seed 233. The images are evaluated at resolution 384 × 384 384 \times384 384 × 384 with IMAGENET1K_SWAG_E2E_V1 weights. All matrices are analyzed per sample, layer, and head before pooling the resulting statistics.
Attention singular values are computed by dense full SVD in float64. Score singular values are obtained from compact QR factorizations of Q h Q_h Q h and K h K_h K h followed by an SVD of R Q h R K h T / d q k R_{Q_h}R_{K_h}^{\mathsf{T}}/\sqrt{d_{qk}} R Q h R K h T / d q k , with structural zeros appended. The core has dimension at most 64; this reduction preserves the core singular values. The W 1 W_1 W 1 comparison assigns mass 1 / ℓ 1/\ell 1/ ℓ to each singular value; the singular values themselves are unnormalized. Quantile bands summarize variation across observations, including dependent heads and nested text prefixes.
These comparisons concern empirical spectral laws. Vanishing W 1 W_1 W 1 would imply a common weak limit whenever either limit exists, while the measured finite-length decrease provides evidence consistent with this behavior. The equally weighted distance can be small even when a few leading modes differ, and does not establish operator-norm or rank-proxy agreement. For example, at GPT-2 length 1024, the pooled stable-rank medians are 1.1823 for real attention and 27.5050 for canonical attention; the corresponding singular-value entropy-rank medians are 314.0596 and 95.1198. The fixed-length ViT comparison describes layerwise agreement.
Angular diagnostics
For the original hidden-state diagnostics, KL divergence is computed from 50 equal-width angle bins, with 10 − 10 10^{-10} 1 0 − 10 added to each bin probability before renormalization. Kolmogorov distance compares the empirical angle CDF with the continuous uniform CDF. The reported angular Wasserstein statistic is the ordinary one-dimensional W 1 W_1 W 1 between the observed angles and an independent uniform sample of the same size on [ − π , π ] [-\pi,\pi] [ − π , π ] :
W ^ 1 = 1 ℓ ∑ j = 1 ℓ ∣ θ ( j ) − u ( j ) ∣ , u j ∼ i i d Unif [ − π , π ] , \widehat{W}_1 = \frac{1}{\ell}\sum_{j=1}^{\ell}\left|\theta_{(j)}-u_{(j)}\right|,\qquad u_j \mathrel{\overset{\mathrm{iid}}{\sim}} \operatorname{Unif}[-\pi,\pi], W 1 = ℓ 1 j = 1 ∑ ℓ θ ( j ) − u ( j ) , u j ∼ iid Unif [ − π , π ] ,
where parentheses denote ascending order. This uses the fixed angle interval, without optimizing over a circular shift. It includes reference sampling noise; finite-size effects in these diagnostics mean that a decreasing discrepancy alone does not quantify increasing Haar isotropy.
The score-orientation experiment covers 1,646 GPT-2 sequences of lengths 17–471 and 3,925 ViT images, producing 237,024 and 565,200 individual score decompositions, respectively. The angular means reported in § 4.4 pool scalar discrepancies over inputs, layers, and heads; the GPT-2 length bands 17–64 and 385–471 contain 254 and 8 sequences, respectively. Figure 8 and Figure 9 report eigen-angle discrepancies for the square orthogonal SVD factors. Their interpretation depends on the numerical SVD convention, including signs and nullspace completion. Uniform eigen-angles are a necessary asymptotic consequence of Haar behavior, rather than a characterization of the Haar law.
Representative score spectra
Figure 10 and Figure 11 use one 1,024-token GPT-2 input and one 577-token ViT input. Each layer pools nonzero singular values from 12 independently decomposed heads; structural zeros are omitted. The color boundary is placed at the largest adjacent ratio in the upper half of the layer-median per-head spectrum. This empirical split illustrates the spectrum without testing the asymptotic thresholds in Assumption 2. Small rank proxies can also arise from smooth spectral decay. The representative plots and rank summaries therefore motivate bulk–spike modeling but do not measure how often the quantitative separation or bounded-bulk conditions hold. The two pretrained models also differ in architecture and training, so their comparison does not isolate the effect of causal masking.
Multi-head attention experiments
To examine how spectral structure varies across heads, we retain the layer and head identities in the real-attention rank proxies and the real-versus-canonical W 1 W_1 W 1 comparison. Figure 12 shows each head’s median over the same 12 GPT-2 documents at length 1,024 and 20 ViT images at length 577. Within each layer, the ratio of the largest to the smallest head-specific median stable rank measures the spread across heads. Its median across layers is 4.22 for GPT-2 and 4.73 for ViT; the corresponding ratios for W 1 W_1 W 1 are 3.19 and 1.49 (Table 1 ).
Figure 12. Head-resolved spectral diagnostics for GPT-2 (top) and ViT (bottom). Each cell is the median over inputs for one layer and head: real-attention stable rank (left), real-attention erank1 _1 1 (middle), and real-versus-canonical spectral W 1 W_1 W 1 (right). Rank colors use logarithmic scales and W 1 W_1 W 1 colors a linear scale, shared across models within each column. Head indices follow each layer’s native ordering and do not indicate a correspondence across layers or models.
Table 1. Median across layers of the within-layer maximum-to-minimum ratio of head-specific median diagnostics.
The head-resolved view reveals substantial variation in spectral concentration within both architectures. It motivates studying how the dominant token subspaces of different heads combine through the value and output projections. Appendix A formulates this question at the level of the multi-head output matrix.
Head-resolved spectral summaries
Figure 12 reaggregates the saved per-head measurements at GPT-2 length 1,024 and ViT length 577, using the same 12 documents and 20 images described above. Each model has 144 layer–head cells, with 12 or 20 observations per cell. With x x x , l l l , h h h indexing inputs, layers, and heads, define for a diagnostic q q q
m l , h ( q ) = median x q x , l , h , R l ( q ) = max h m l , h ( q ) min h m l , h ( q ) . m_{l,h}^{(q)} = \operatorname{median}_{x} q_{x,l,h}, \qquad R_l^{(q)} = \frac{\max_{h} m_{l,h}^{(q)}}{\min_{h} m_{l,h}^{(q)}}. m l , h ( q ) = median x q x , l , h , R l ( q ) = min h m l , h ( q ) max h m l , h ( q ) .
Here q q q is the real-attention stable rank, real-attention erank 1 \operatorname{erank}_{1} erank 1 , or the paired real-versus-canonical W 1 W_{1} W 1 . The heatmaps display m ‾ l , h ( q ) \overline{m}_{l,h}^{(q)} m l , h ( q ) , and Table 1 reports median l R l ( q ) \operatorname{median}_{l} R_{l}^{(q)} median l R l ( q ) . All denominators are positive in these data. Aggregation occurs after computing the diagnostics of each sample–layer–head matrix; no attention matrices or singular vectors are pooled across heads. These are descriptive summaries of the selected inputs and layers. The analysis uses existing measurements and requires no additional model inference.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: The abstract and Introduction describe the canonical diagonal score model, spike–bulk separation, spectral degeneration, and the stable-rank contrast between bidirectional and causal attention. These claims are supported by Assumptions 1–2, Theorems 1–4, and the rank-proxy results in Appendix D. Section 4 provides empirical diagnostics on pretrained ViT/Imagenette and GPT-2/Wikitext-103.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix A discusses the canonical model’s scope, orientation dependence, causal ordering, and the interpretation of the empirical diagnostics. The theoretical assumptions are explicit, and Section 4 and Appendix F describe the empirical scope of the pretrained ViT and GPT-2 experiments on Imagenette and Wikitext-103.
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions 1 and 2 specify the canonical model and separation conditions; additional conditions, including bounded bulk scores, are stated in the relevant theorems. The numbered results are cross-referenced, with complete spectral proofs in Appendices C.1–C.4, rank-proxy proofs in Appendix D, and auxiliary matrix-norm proofs in Appendix E.
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 4 specifies the models, datasets, validation splits, sequence lengths, spectral metrics, and rank definitions. Appendix F provides sampling, SVD, angular-distance, and aggregation details, including sample counts, the seed for the matched attention comparison, and the interpretation of quantile bands.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The experiments use public Imagenette and Wikitext-103 data and pretrained ViT and GPT-2 models. The repository at https://github.com/Mengda-Li/attention provides analysis code, saved figure inputs, environment and data-preparation instructions, and figure-to-script mappings. Its reproduction guide distinguishes rendering from supplied CSV or archived figure inputs from collecting model measurements and records the available collection workflows and their limitations; Appendix § 12 links these materials.
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: The paper evaluates pretrained ViT-B/16 (SWAG) on Imagenette validation images and GPT-2 on Wikitext-103 validation text, without training new models. Section 4 and Appendix § 12 specify the exact checkpoints, input dimensions, sequence lengths, sampling and seed, float64 attention SVD, score factorization, angular distances, and stable/effective-rank computations.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Figure 3 , Figure 4 , and Figure 5 report means or medians with quantile bands; Figure 7 and Figure 6 use layer-wise box plots. The captions and Appendix § 12 .3 define these summaries. They describe empirical variability across sampled observations, including dependent heads and nested text prefixes in the matched comparison, rather than confidence intervals.
Figure 5. GPT-2 real-versus-canonical attention spectral discrepancy across sequence lengths, pooled over all layers (left) and restricted to the last layer (right). Each length uses nested prefixes of the same 12 documents, with 12 heads per layer. Curves show median W 1 W_1 W 1 ; IQR and 10–90% bands summarize variation across the paired observations.
Figure 6. ViT real-versus-canonical attention spectral discrepancy across layers at sequence length 577, with 20 images and 12 heads per layer. Boxes show medians and IQRs, with whiskers extending to 1.5 times the IQR. All layer medians are below 0.104.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Appendix § 12 identifies the compute used for each experiment group: hidden-state angular diagnostics and GPT-2 score-rank summaries used an AMD EPYC 9554 dual-socket CPU server with 64 cores per socket and 126 GB RAM, without GPU acceleration, for approximately 24 hours in total. Real-versus-canonical comparisons and score-level angular and spectral diagnostics used a MacBook Pro with an M1 Pro chip and 16 GB unified memory for approximately two hours in total. The reported timings are totals for each group.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes] Justification: The work is theoretical and diagnostic, uses public benchmark data and pretrained models, and does not involve human subjects, private data, or deployed systems. The paper does not identify any confidential or sensitive data sources.
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [N/A]
Justification: The paper is foundational theory and empirical diagnosis of attention spectra, and it does not introduce a deployable model, dataset, or capability with a direct path to societal deployment. Any downstream impacts would be indirect through broader understanding of Transformer mechanisms.
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A]
Justification: The paper does not release a new high-risk model, image generator, language model, or scraped dataset. It releases analysis/plotting code for reproducing figures, so additional model-release safeguards are not applicable.
Guidelines:
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Appendix F.1 identifies the existing datasets, code libraries, and model weights used in the experiments and lists the corresponding licenses or access terms, including ImageNet/Imagenette, Wikitext-103, Hugging Face Transformers, GPT-2 weights, TorchVision, and SWAG-derived ViT weights. The paper cites the creators or source repositories and uses these assets only for non-commercial research evaluation and spectral diagnostics.
Guidelines:
The answer [N/A] means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The released analysis and plotting code is documented alongside the code with its inputs, dependencies, execution commands, output definitions, and reproduction limitations. The authors’ code is released under the MIT license; existing data and models retain their original terms. Appendix F links the repository and reproduction guides. No new dataset, benchmark, or trained model is introduced.
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: The work does not involve crowdsourcing, participant studies, data annotation by human subjects, or user-facing experiments.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: No human-subject research is conducted, so IRB approval or equivalent human-subject review is not applicable.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLM assistance was used in revising the manuscript and checking mathematical arguments. The resulting statements are accompanied by explicit assumptions and proofs; LLM output is not treated as mathematical evidence.
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.