betaFeedback

WorksComputer Science

Attention Sinks as Spectral Spikes: A Mechanism Analysis of Gated Attention

Abstract

Transformer attention exhibits attention sinks, where a small subset of source tokens absorbs disproportionate attention mass across layers. Recent work shows that post-SDPA output gating strongly reduces attention-sink behavior, while separate studies link attention sinks to gradient sinks and massive activations through backward training dynamics. Yet how the same gate affects forward sink structure, backward gradient concentration, and training-time representation signatures remains unclear. We address this gap by formulating attention sinks as sink-induced low-rank modes in token-space covariance, with directions aligned to high-mass attention columns. Under this view, post-SDPA gating acts as a mode-dependent contraction: it attenuates sink-induced spectral energy in the forward pass and locally reduces value-path gradients routed through sink columns in the backward pass. Empirically, random-matrix diagnostics show that gating reduces outlier mass, weakens sink-subspace alignment, and increases effective rank, while backward analyses show active-gate attenuation of first-token value gradients. Training-time diagnostics further show that gating suppresses the co-emergence of attention concentration, value-gradient amplification, and representation dominance.

Introduction

Transformers have become the backbone of modern large language models, with self-attention enabling flexible token interactions across long contexts [1]. Yet recent studies have shown that this same mechanism can give rise to attention sinks: a small number of tokens that absorb disproportionate attention mass across layers, often independently of semantic relevance [2, 3]. Such sink behavior is not merely an artifact of attention visualization: it has been linked to distorted representation geometry, massive activations, and training instability [4, 5]. Understanding and mitigating attention sinks has therefore become an important step toward more stable Transformer architectures [6, 7, 8, 9].

Recently, a strikingly simple mitigation was proposed: applying a sigmoid gate after the scaled dot-product attention (SDPA) output [10]. In its most effective form, this gate is head-specific and elementwise, and it nearly eliminates attention sinks, reducing first-token attention from 46.7% to 4.8%, while substantially improving training stability. This mechanism has since been adopted in production-scale architectures such as Qwen3-Next [11], underscoring the practical relevance of understanding why it works. The authors attribute this effect to query-dependent sparsity, arguing that sparse gating filters out context irrelevant to the current query. However, this explanation remains largely phenomenological. It leaves open a central question: how can a standard attention-output gate selectively suppress sink-aligned structure, rather than merely shrink the attention output uniformly?

This question is especially important because post-SDPA gating is not merely a post-hoc correction to a trained attention map. It is used throughout training, shaping both the forward representations and the gradients by which they are learned. Its effect therefore appears at multiple coupled levels: it changes the forward representation geometry associated with sink tokens, attenuates sink-directed value-path gradients in the backward dynamics, and suppresses training-time sink signatures. Recent work [12] explains one part of this picture, showing that attention sinks can induce value-path gradient sinks under causal masking and that massive activations may arise as an RMSNorm-mediated response to this localized gradient pressure. However, this backward account has not yet been connected to the standard post-SDPA gate.

We bridge this gap by recasting an attention sink as a sink-induced low-rank mode in token-space covariance. Under this view, a sink is not merely a token that receives large attention mass; rather, a high-mass attention column can induce a structured token-space direction in the post-attention representation. Post-SDPA gating then has a precise structural interpretation: it induces an effective contraction of this sink-induced mode. Under an idealized isotropic reference model, this contraction predicts that a supercritical spike can be pushed below the Baik–Ben Arous–Péché (BBP) threshold and merge back into the Marchenko–Pastur bulk. In the backward pass, the gate locally attenuates value-path gradients routed through sink columns, replacing original sink-column statistics with gated effective counterparts. Thus, post-SDPA gating is not simply a generic damping operation; it attenuates sink-induced structure in both forward spectral geometry and backward gradient flow.

Taken together, we argue that post-SDPA gating goes beyond generic sparsity or uniform attenuation: it weakens sink-induced token-space structure in the forward pass and locally reduces value-path gradient flow through high-mass sink columns. We make the following contributions:

  • Spectral characterization of attention sinks. We formulate attention sinks as sink-induced low-rank modes in token-space covariance, with directions aligned to high-mass attention columns, connecting attention concentration to spectral outliers and providing a random-matrix perspective on sink emergence.

  • Dual contraction mechanism of post-SDPA gating. We show that post-SDPA gating suppresses sink-induced structure in two coupled ways: it attenuates forward spectral energy and it locally contracts value-path gradient aggregation through sink columns.

  • Training-time sink signatures. We show that models without gating develop first-token attention concentration and first-token representation dominance during training, whereas post-SDPA gated models substantially attenuate both signatures.

From Attention Sinks to Spectral Spikes

This section recasts attention sinks as spectral objects in the post-attention representation. We first connect high-mass attention columns to token-space covariance directions, then introduce a sink-induced spiked model and the post-SDPA gate as a row-wise diagonal contraction.

Attention Sinks as Column Imbalance

Transformer attention computes token interactions through non-negative, row-stochastic attention weights. For a single attention head, let Q,K,V∈Rn×dQ,K,V \in\mathbb{R}^{n \times d} denote the query, key, and value matrices. Under a causal mask, the attention weights AA and corresponding output YY are defined as follows:

Aij={exp⁡(qiTkj/d)∑ℓ≤iexp⁡(qiTkℓ/d),j≤i,0,j>i,yi=∑j≤iAijvj,Y=AV∈Rn×d.(1)A_{ij} = \begin{cases} \frac{\exp(q_i^{\mathsf{T}} k_j / \sqrt{d})}{\sum_{\ell\le i} \exp(q_i^{\mathsf{T}} k_\ell/ \sqrt{d})}, & j \le i,\\ 0, & j > i, \end{cases} \qquad y_i = \sum_{j \le i} A_{ij} v_j,\qquad Y = AV \in\mathbb{R}^{n \times d}. \tag*{(1)}

Thus Aij≥0A_{ij} \ge0 and ∑jAij=1\sum_j A_{ij} = 1. Due to the non-negativity and row-wise normalization of the softmax, attention mass must be assigned to some previous token even when no semantically relevant context is needed. As a result, certain source tokens can accumulate attention mass across many query positions and layers. This phenomenon is known as an attention sink.

Empirically, early-position tokens, especially the first token, often act as sinks, receiving disproportionately large attention weights independent of their semantic relevance. Formally, we measure the incoming attention mass of a source token jj by its causal column mass, the average attention it receives over all nn query positions:

mj=1n∑i=1nAij.(2)m_j = \frac{1}{n}\sum_{i=1}^{n} A_{ij}. \tag*{(2)}

A token jj is referred to as an attention sink when mjm_j is significantly larger than the typical non-sink scale. Large values of mjm_j indicate a column-wise imbalance in the attention matrix, reflecting persistent aggregation of attention toward a small number of source positions. While mjm_j identifies where attention mass concentrates in the map AA, our goal is to understand how this column imbalance appears in the representation Y=AVY = AV.

From Sink Columns to Token-Space Covariance

The standard definition of an attention sink is based on column-wise concentration in the attention matrix. To study how this phenomenon affects representations, we shift attention from the map AA to the token-space covariance of the post-attention output:

CY:=1dYY⊤=1dAVV⊤A⊤,Y=AV∈Rn×d.(3)C_Y := \frac{1}{d}YY^{\top} = \frac{1}{d}AVV^{\top}A^{\top}, \qquad Y = AV \in\mathbb{R}^{n \times d}. \tag*{(3)}

Because CYC_Y is a token-space covariance, its leading eigenvectors describe token-position modes of concentrated representation energy. Under an isotropic value approximation, E[VV⊤/d∣A]=σV2I\mathbb{E}[VV^{\top}/d \mid A] = \sigma_V^2 I, and therefore

E[CY∣A]=1dAE[VV⊤∣A]A⊤=σV2AA⊤=σV2∑jA:,jA:,j⊤.(4)\mathbb{E}[C_Y \mid A] = \frac{1}{d}A\mathbb{E}[VV^{\top} \mid A]A^{\top} = \sigma_V^2 AA^{\top} = \sigma_V^2 \sum_j A_{:,j}A_{:,j}^{\top}. \tag*{(4)}

This column-wise decomposition identifies the directions through which attention sinks can enter the token-space covariance. When a source token jj has high incoming attention mass, its attention column contributes the rank-one term A:,jA:,j⊤A_{:,j}A_{:,j}^{\top}, whose direction is the normalized column aˉj:=A:,j/∥A:,j∥2\bar{a}_j := A_{:,j}/\lVert A_{:,j}\rVert_2. This observation provides the bridge from attention-map concentration to representation geometry: a persistent sink column can appear as a dominant token-space mode in CYC_Y.

Spectral Spikes and Sink-Induced Representations

The covariance view above reduces sink structure to the question of whether CYC_Y contains a dominant token-space direction aligned with high-mass attention columns. Random matrix theory provides a baseline for separating such structured directions from high-dimensional noise. When the representation is purely isotropic noise, we write Y=σZY = \sigma Z, where Z∈Rn×dZ \in\mathbb{R}^{n \times d} has i.i.d. standard Gaussian entries. Then the spectrum of CYC_Y converges, in the high-dimensional limit n,d→∞n,d \to\infty with n/d→cn/d \to c, to the Marchenko–Pastur (MP) law [13], whose support is given by

λ±=σ2(1±c)2,c=nd.(5)\lambda_{\pm} = \sigma^2(1 \pm\sqrt{c})^2, \qquad c = \frac{n}{d}. \tag*{(5)}

Eigenvalues within this bulk are consistent with isotropic noise, whereas eigenvalues separated above λ+\lambda_+ indicate excess structured variance. A standard way to represent such structure is a rank-one spiked model [14, 15, 16],

Y=dasu⊤+σZ,(6)Y = \sqrt{d}a s u^{\top} + \sigma Z, \tag*{(6)}

where s∈Rns \in\mathbb{R}^n and u∈Rdu \in\mathbb{R}^d are unit vectors, and a>0a > 0 is the spike amplitude. Here ss specifies a token-space direction and uu specifies its feature-space direction. Under the Baik–Ben Arous–Péché (BBP) transition [17, 18], the rank-one component produces a separated spectral outlier when a2>ca^2 > \sqrt{c}. Thus, the MP edge serves as a noise threshold, and a spectral outlier signals a structured low-rank mode in the token representations.

We now specialize this generic spiked model to attention sinks. From the column-covariance relation in Section 2.2, high-mass sink columns induce token-space directions aˉj\bar{a}_j. We therefore identify sink-induced spikes by the alignment of the spike direction ss with the subspace spanned by these directions.

Definition 2.1 (Sink-induced spiked representation model). Consider a rank-one spiked post-attention representation

Y=dasu⊤+σZ,Y = \sqrt{d}a s u^{\top} + \sigma Z,

where s∈Rns \in\mathbb{R}^{n} and u∈Rdu \in\mathbb{R}^{d} are unit vectors, a>0a > 0 is the spike amplitude, and ZZ is isotropic noise. For sink tokens JsinkJ_{\mathrm{sink}}, let Ssink=span⁡{aˉj:j∈Jsink}\mathcal{S}_{\mathrm{sink}} = \operatorname{span}\{\bar{a}_{j}: j \in J_{\mathrm{sink}}\}, and let PsinkP_{\mathrm{sink}} be the orthogonal projector onto this subspace. We call the spike sink-induced if

∥Psinks∥22≥1−ε.\left\|P_{\mathrm{sink}}s\right\|_{2}^{2} \ge1-\varepsilon.

For a single first-token sink, this becomes ⟨s,aˉ1⟩2≥1−ε\langle s,\bar{a}_{1}\rangle^{2} \ge1-\varepsilon.

This model connects attention-map concentration with representation geometry: a persistent sink column can manifest as a low-rank token-space mode in CYC_Y. When this mode is supercritical, it appears as a spectral outlier above the MP bulk.

Post-SDPA Gating

A recent line of work [10] mitigates attention sinks by applying a query-dependent multiplicative gate to the output of scaled dot-product attention. We describe the post-SDPA gated attention variant for a single attention head and write its head dimension as dd. The attention output is transformed as

y~i=Gi⊙yi,Gi=σ(xiWg),Gi∈[0,1]d,Y~=Y⊙G,(7)\widetilde{y}_i = G_i \odot y_i,\qquad G_i = \sigma(x_iW_g),\qquad G_i \in[0,1]^d,\qquad\widetilde{Y} = Y \odot G, \tag*{(7)}

where xix_i denotes the hidden state used to compute the gate, WgW_g is a learned gating projection, σ\sigma is the sigmoid function, ⊙\odot denotes element-wise multiplication, and G∈[0,1]n×dG \in[0,1]^{n \times d} collects the gate values row-wise. Equivalently, with Γi:=diag⁡(Gi)\Gamma_i := \operatorname{diag}(G_i), we have

y~i=Γiyi,0⪯Γi⪯I,(8)\widetilde{y}_i = \Gamma_i y_i,\qquad0 \preceq\Gamma_i \preceq I, \tag*{(8)}

so the gate acts on each token row as a diagonal contraction. The post-SDPA gate is therefore not a scalar shrinkage of YY, but a token- and feature-dependent diagonal contraction. Its effect on a sink-induced spike depends on how the spike’s token positions and feature direction interact with the row-wise operators Γi\Gamma_i.

Forward Pass: Spectral Suppression

Theory: Sink-mode Contraction under Post-SDPA Gating

We analyze the effect of post-SDPA gating on a sink-induced spectral mode. Let Ysig=dasuTY_{\mathrm{sig}} = \sqrt{d}asu^{\mathsf{T}} denote a sink-induced signal component of the SDPA output, where s∈Rns \in\mathbb{R}^{n} is aligned with the sink-column subspace in Definition 2.1 and u∈Rdu \in\mathbb{R}^{d} is the corresponding feature direction. The analysis is deterministic and makes no distributional assumption on the learned gate.

Using the row-wise gate operators Γi\Gamma_i defined in Section 2.4, define the effective gate on the feature direction uu by

gi:=∥Γiu∥2,D:=diag⁡(g1,…,gn).(9)g_i := \left\|\Gamma_i u\right\|_2,\qquad D := \operatorname{diag}(g_1,\ldots,g_n). \tag*{(9)}

Since ∥u∥2=1\|u\|_2 = 1 and 0⪯Γi⪯I0 \preceq\Gamma_i \preceq I, we have 0≤gi≤10 \le g_i \le1.

Theorem 3.1 (Sink-induced mode contraction under post-SDPA gating). For the sink-induced signal Ysig=dasuTY_{\mathrm{sig}} = \sqrt{d}asu^{\mathsf{T}}, the gated signal satisfies

1d∥Y~sig∥F2=a2∥Ds∥22=a2αs≤a2,(10)\frac{1}{d}\left\|\widetilde{Y}_{\mathrm{sig}}\right\|_{F}^{2} = a^{2}\left\|Ds\right\|_{2}^{2} = a^{2}\alpha_s \le a^{2}, \tag*{(10)}

where

αs:=∥Ds∥22=∑i=1nsi2∥Γiu∥22.(11)\alpha_s := \left\|Ds\right\|_{2}^{2} = \sum_{i=1}^{n}s_i^{2}\left\|\Gamma_i u\right\|_{2}^{2}. \tag*{(11)}

Moreover, the gated signal covariance C~sig:=1dY~sigY~sigT\widetilde{C}_{\mathrm{sig}} := \frac{1}{d}\widetilde{Y}_{\mathrm{sig}}\widetilde{Y}_{\mathrm{sig}}^{\mathsf{T}} satisfies

λmax⁡(C~sig)≤tr⁡(C~sig)=a2αs.(12)\lambda_{\max}\left(\widetilde{C}_{\mathrm{sig}}\right) \le\operatorname{tr}\left(\widetilde{C}_{\mathrm{sig}}\right) = a^{2}\alpha_s. \tag*{(12)}

Theorem 3.1 shows that αs\alpha_s quantifies how much of the sink-induced mode is preserved by the gate. Thus, gating suppresses a sink-induced spectral spike when it contracts the query positions and feature direction participating in the corresponding attention-column mode.

To connect this contraction to spectral outlier behavior, we use the classical spiked-covariance model as an idealized reference. The deterministic contraction above applies to learned gates, but a general learned gate also transforms the noise component and therefore does not preserve the exact assumptions of a standard spiked Wishart model. We therefore isolate a simplified isotropic thinning setting in which the same contraction mechanism can be related exactly to the BBP phase transition.

Proposition 3.2 (BBP suppression under isotropic thinning). Assume the spiked representation model in Eq. (6). Let PI∈{0,1}n′×nP_I \in\{0,1\}^{n'\times n} be a row-selection matrix independent of ZZ, where n′=∣I∣n'=|I|, and define αs:=∥PIs∥22\alpha_s := \lVert P_I s\rVert_2^2, c′:=n′dc' := \frac{n'}{d}. If αs>0\alpha_s>0, then

PIY=daαssˉuT+σZ′,sˉ:=PIs∥PIs∥2,(13)P_IY = \sqrt{d}a\sqrt{\alpha_s}\bar{s}u^{\mathsf{T}}+\sigma Z', \qquad\bar{s}:=\frac{P_Is}{\lVert P_Is\rVert_2}, \tag*{(13)}

where Z′∈Rn′×dZ' \in\mathbb{R}^{n'\times d} has i.i.d. standard Gaussian entries. Hence the thinned covariance

C~=1d(PIY)(PIY)T(14)\widetilde{C}=\frac{1}{d}(P_IY)(P_IY)^{\mathsf{T}} \tag*{(14)}

is a rank-one spiked Wishart model with effective spike strength a2αsa^2\alpha_s. Consequently, the sink-induced spectral outlier disappears when

a2αs≤σ2c′.(15)a^2\alpha_s \leq\sigma^2\sqrt{c'}. \tag*{(15)}

When a2αs>σ2c′a^2\alpha_s>\sigma^2\sqrt{c'}, a spectral outlier persists, but its effective spike strength is reduced to a2αsa^2\alpha_s.

Proposition 3.2 should be read as an idealized spectral reference rather than an exact theorem for learned gates. For a general post-SDPA gate, the sink-induced signal component is contracted as in Theorem 3.1, but the noise component is also modulated by the row-wise feature gates Γi\Gamma_i. The resulting covariance is generally anisotropic and need not be a standard spiked Wishart matrix. Nevertheless, the BBP reference captures the mechanism-level prediction: when gating reduces the effective strength of a sink-induced mode, the associated spectral outlier should weaken, move closer to the MP bulk, or disappear. We empirically demonstrate that this happens in the learned post-SDPA gate setting. Proofs of Theorem 3.1 and Proposition 3.2 can be found in Appendix § 9.

Empirical: Spectral Suppression and Sink-Subspace Alignment

We empirically test the forward-pass prediction of Theorem 3.1: if post-SDPA gating contracts structured token-space modes, then the post-gate representation should exhibit weaker spectral concentration. In addition, following Definition 2.1, we ask whether the suppressed spectral structure is aligned with sink-induced attention-column subspaces.

Analysis setting. We evaluate the post-SDPA gated Qwen-1B model following previous work [10] and record both the pre-gate output YrawY_{\mathrm{raw}} and the post-gate output YgateY_{\mathrm{gate}} at each layer. All spectral statistics are computed layer-wise and averaged across sixteen samples of sequence length 512 from WikiText dataset [19].

For each layer, we compute the token-space covariance CY=1DYˉYˉTC_Y=\frac{1}{D}\bar{Y}\bar{Y}^{\mathsf{T}}, where Yˉ∈RT×D\bar{Y}\in\mathbb{R}^{T\times D} is obtained by centering and normalizing each feature dimension across tokens. This removes trivial feature-scale effects and focuses the diagnostic on token-space correlation structure. We use the theoretical MP edge λ^+=(1+T/D)2\hat{\lambda}_+=(1+\sqrt{T/D})^2 of the unit-variance isotropic null as a fixed reference and summarize each spectrum by

OutMass⁡(CY)=∑λk>λ^+λk∑iλi,λ1(CY)λ^+,reff(CY)=exp⁡(−∑ipilog⁡pi),(16)\operatorname{OutMass}(C_Y)=\frac{\sum_{\lambda_k>\hat{\lambda}_+}\lambda_k}{\sum_i\lambda_i}, \qquad\frac{\lambda_1(C_Y)}{\hat{\lambda}_+}, \qquad r_{\mathrm{eff}}(C_Y)=\exp\left(-\sum_i p_i\log p_i\right), \tag*{(16)}

where pi=λi/∑jλjp_i=\lambda_i/\sum_j\lambda_j. These metrics measure out-of-bulk spectral mass, leading-mode dominance, and the spread of variance across token-space directions. To connect spectral suppression to attention sinks, we construct a sink-column subspace from the attention matrix. For each layer and sample, let JsinkJ_{\mathrm{sink}} denote the source positions whose head-averaged column mass Mˉj=1T∑iAij\bar{M}_j=\frac{1}{T}\sum_i A_{ij} is at least half its maximum (at most k=8k=8). For each j∈Jsinkj\in J_{\mathrm{sink}}, define the normalized column mode aˉj=Aˉ:,j/∥Aˉ:,j∥2\bar{a}_j=\bar{A}_{:,j}/\lVert\bar{A}_{:,j}\rVert_2 and let PsinkP_{\mathrm{sink}} be the orthogonal projector onto span⁡{aˉj:j∈Jsink}\operatorname{span}\{\bar{a}_j:j\in J_{\mathrm{sink}}\}. If q1(CY)q_1(C_Y) denotes the leading eigenvector of CYC_Y, we measure sink-subspace alignment by

Align⁡sink(CY)=∥Psinkq1(CY)∥22.(17)\operatorname{Align}_{\mathrm{sink}}(C_Y)=\lVert P_{\mathrm{sink}}q_1(C_Y)\rVert_2^2. \tag*{(17)}

The random baseline for a kk-dimensional subspace is approximately k/Tk/T where T=512T=512.

Gating suppresses dominant token-space spectral structure. Figure 1(a-Figure 1c) shows that post-SDPA gating consistently reduces spectral concentration. The gated output has lower outlier mass and a smaller leading eigenvalue relative to the MP edge, while its effective rank increases. Thus, after gating, the representation is less dominated by a small number of token-space directions. Since the covariance is computed after feature-wise standardization, which removes any per-feature scaling shared across tokens, this reduction is not a consequence of uniform feature-scale shrinkage; rather, the token-dependent variation of the learned gate changes the token-space correlation structure. Robustness checks using non-standardized covariance and scalar gating are reported in Appendix D.1.

Four-panel plots showing outlier mass, maximum eigenvalue versus MP edge, effective rank, and sink alignment

Figure 1. Post-SDPA gating suppresses token-space spectral structure. (a) Outlier mass above the MP edge decreases. (b) The leading eigenvalue moves closer to the MP edge. (c) Effective rank increases. (d) Sink-subspace alignment Align⁡sink(C)\operatorname{Align}_{\mathrm{sink}}(C) decreases after gating; the dashed line shows the random-subspace baseline (layer-wise mean of k/Tk/T).

Gating reduces sink-subspace alignment. Figure 1(d) further shows that the leading pre-gate spectral mode is often aligned with the sink-column subspace, substantially above the random k/Tk/T baseline. After gating, this alignment is strongly reduced across layers. This supports the sink-induced spike formulation in Definition 2.1: high-mass attention columns are not only attention-map artifacts, but can correspond to structured low-rank directions in the post-attention representation. Post-SDPA gating attenuates these sink-aligned directions together with the overall spectral outlier mass. Additional sink-subspace counterfactuals, which further probe the role of sink-induced modes, are reported in Appendix D.

Gating weakens sink-aligned covariance structure. We further examine how post-SDPA gating affects spectral directions with different degrees of sink alignment. As shown in Table 1, highly sink-aligned directions retain substantially less covariance energy than weakly aligned directions. This gap largely disappears when the elementwise post-SDPA gate is replaced by its feature-averaged scalar counterpart, while raw-energy retention remains similar across the two groups. These results indicate that post-SDPA gating preferentially weakens sink-aligned covariance structure beyond uniform magnitude reduction, consistent with the spectral suppression observed in Figure 1. Further details are provided in Appendix D.2.

MetricHigh align.Low align.
Raw energy (αk\alpha_k)0.0470.044
Std. covariance (RkR_k)0.7400.945
Scalar gate (RkR_k)0.9570.990

Table 1. Sink-aligned mode retention. Retention of highly and weakly sink-aligned directions in Qwen-1B. The scalar control uses feature-averaged gates.

Backward Pass: Gradient Valve

Having shown that gating attenuates sink-induced structure in the forward geometry, we next analyze its effect on backward dynamics. Building on the gradient-sink view [12], we show that under fixed upstream signals, post-SDPA gating replaces sink-column gradient statistics with gated effective counterparts, acting as an implicit value-path gradient valve.

Theory: A Gated Extension of the Gradient-Sink Bound

Consider a single causal attention head, yt=∑j≤tatjvjy_t = \sum_{j \le t} a_{tj}v_j, where atja_{tj} is the attention weight from query position tt to source position jj, and vj∈Rdv_j \in\mathbb{R}^{d} is the value vector. Since post-SDPA gate Γt\Gamma_t is diagonal and contractive, define κt:=∥Γt∥op≤1\kappa_t := \lVert\Gamma_t\rVert_{\mathrm{op}} \le1. For a candidate sink token ss, define the sink column statistics without gate and corresponding gated effective statistics as

Ms≔∑t=sTats,Ss≔∑t=sTats2,Mseff≔∑t=sTatsκt,Sseff≔∑t=sTats2κt2.(18)M_s \coloneqq\sum_{t=s}^{T} a_{ts}, \qquad S_s \coloneqq\sum_{t=s}^{T} a_{ts}^{2}, \qquad M_s^{\mathrm{eff}} \coloneqq\sum_{t=s}^{T} a_{ts}\kappa_t, \qquad S_s^{\mathrm{eff}} \coloneqq\sum_{t=s}^{T} a_{ts}^{2}\kappa_t^{2}. \tag*{(18)}

By construction,

Mseff≤Ms,Sseff≤Ss.(19)M_s^{\mathrm{eff}} \le M_s, \qquad S_s^{\mathrm{eff}} \le S_s. \tag*{(19)}

Theorem 4.1 (Gated value-path gradient bound). Let δt≔∇y~tL\delta_t \coloneqq\nabla_{\widetilde{y}_t}\mathcal{L} be the upstream gradient at the post-SDPA gated attention output. Then the gradient with respect to the source value vsv_s is

∇vsL=∑t=sTatsΓtTδt.(20)\nabla_{v_s}\mathcal{L} = \sum_{t=s}^{T} a_{ts}\Gamma_t^{\mathsf{T}}\delta_t. \tag*{(20)}

Consequently,

∥∇vsL∥≤∑t=sTatsκt∥δt∥.(21)\left\|\nabla_{v_s}\mathcal{L}\right\| \le\sum_{t=s}^{T} a_{ts}\kappa_t\left\|\delta_t\right\|. \tag*{(21)}

In particular, if ∥δt∥≤B\left\|\delta_t\right\| \le B for all tt, then

∥∇vsL∥≤BMseff≤BMs.(22)\left\|\nabla_{v_s}\mathcal{L}\right\| \le B M_s^{\mathrm{eff}} \le B M_s. \tag*{(22)}

The coherent bound above controls the worst-case aligned contribution of future-query gradients. A complementary second-moment statement applies when the upstream gradients behave incoherently.

Corollary 4.2 (Second-moment attenuation). Conditioned on the attention weights and gates, assume that the upstream gradients satisfy

E[δtδrT]=0(t≠r),E∥δt∥2≤B2.(23)\mathbb{E}[\delta_t\delta_r^{\mathsf{T}}] = 0 \quad(t \ne r), \qquad\mathbb{E}\left\|\delta_t\right\|^2 \le B^2. \tag*{(23)}

Then

E∥∇vsL∥2≤B2Sseff≤B2Ss.(24)\mathbb{E}\left\|\nabla_{v_s}\mathcal{L}\right\|^2 \le B^2 S_s^{\mathrm{eff}} \le B^2 S_s. \tag*{(24)}

Theorem 4.1 and Corollary 4.2 show that post-SDPA gating attenuates value-path gradient aggregation under both coherent and incoherent regimes. Importantly, this contraction does not require changing the attention matrix itself. Even if the same attention weights atsa_{ts} are fixed, each backpropagated contribution from query tt to source value vsv_s is filtered by the gate. Thus, the backward strength of a sink column is governed by MseffM_s^{\mathrm{eff}} and SseffS_s^{\mathrm{eff}}, not only by MsM_s and SsS_s. Proofs for Theorem 4.1 and Corollary 4.2 can be found in Appendix § 10.

Empirical: Gradient Sink under Post-SDPA Gating

Practical proxies. Theorem 4.1 is a local Jacobian statement: conditioned on fixed attention weights and upstream gradients, post-SDPA gating contracts value-path contributions routed through a sink column. We evaluate this prediction using two complementary diagnostics. First, we compare Qwen-1B models with and without post-SDPA gating under their actual forward-backward passes. This measures the outcome of training with post-SDPA gates. Second, within post-SDPA gated model, we isolate the local value-path effect of the gate by keeping the attention weights and upstream gradients fixed, and recomputing the first-token value-gradient contribution with and without the gate. This fixed-upstream-gradient counterfactual directly tests the valve effect predicted by Theorem 4.1.

We focus on the first-token sink, which is the primary sink position in this setting. We measure value-gradient amplification at source position s=1s=1 by

ρ1=∥∇v1L∥(T−1)−1∑j≠1∥∇vjL∥.(25)\rho_1 = \frac{\left\|\nabla_{v_1}\mathcal{L}\right\|}{(T-1)^{-1}\sum_{j\ne1}\left\|\nabla_{v_j}\mathcal{L}\right\|}. \tag*{(25)}

To isolate the local value-path effect of the active gate, we cache δt≔∇y~tL\delta_t \coloneqq\nabla_{\widetilde{y}_t}\mathcal{L} from the actual gated backward pass and compute

gv1actual=∑t=1Tat1ΓtTδt,gv1cf=∑t=1Tat1δt,A1local=∥gv1actual∥2∥gv1cf∥2.(26)g_{v_1}^{\mathrm{actual}} = \sum_{t=1}^{T} a_{t1}\Gamma_t^{\mathsf{T}}\delta_t, \qquad g_{v_1}^{\mathrm{cf}} = \sum_{t=1}^{T} a_{t1}\delta_t, \qquad A_1^{\mathrm{local}} = \frac{\left\|g_{v_1}^{\mathrm{actual}}\right\|_2}{\left\|g_{v_1}^{\mathrm{cf}}\right\|_2}. \tag*{(26)}
Three-panel plots showing actual gradient amplification, local gradient attenuation, and attenuation versus effective mass

Figure 2. Post-SDPA gating attenuates first-token value-path gradients. (a) Post-SDPA gate suppresses first-token value-gradient amplification. (b) Local attenuation tracks the RMS effective-mass proxy R1rmsR^{\mathrm{rms}}_1. (c) Their layer-wise correlation supports the fixed-upstream-gradient valve mechanism.

As a simple gate-mass proxy, we also compute the RMS effective-mass ratio

M~1rms=∑t=1Tat1rt,R1rms=M~1rmsM1,M1=∑t=1Tat1,rt=(1d∑c=1dGt,c2)1/2.(27)\widetilde{M}^{\mathrm{rms}}_1 = \sum_{t=1}^{T} a_{t1}r_t,\qquad R^{\mathrm{rms}}_1 = \frac{\widetilde{M}^{\mathrm{rms}}_1}{M_1},\qquad M_1 = \sum_{t=1}^{T} a_{t1},\qquad r_t = \left(\frac{1}{d}\sum_{c=1}^{d}G_{t,c}^2\right)^{1/2}. \tag*{(27)}

This RMS quantity is an isotropic-gradient proxy for average feature-wise attenuation.

Results. Figure 2(a) reports the trained-model outcome. Baseline (without post-SDPA gating) develops large first-token value-gradient amplification in mid-to-late layers, whereas the gated model remains close to the non-sink scale. The counterfactual curve in the model with post-SDPA gating (Gated (G ≡\equiv 1)) shows that removing the gate increases the first-token gradient even without changing the trained weights, indicating that the active gate provides an additional attenuation mechanism beyond the learned representation itself. Figures 2(b,c) isolate this local valve effect. Holding the attention weights and upstream gradients fixed, we recompute the first-token value-gradient contribution with and without the gate. The resulting attenuation ratio A1localA^{\mathrm{local}}_1 is far below one across layers and closely tracks the RMS effective-mass proxy R1rmsR^{\mathrm{rms}}_1. The layer-wise scatter strongly correlates these quantities, supporting the Jacobian prediction of Theorem 4.1 that post-SDPA gating attenuates value gradients according to the effective gated mass through the first-token sink column.

Implication for the Attention-Sink–Gradient-Sink–Massive-Activation Pathway

The forward attention sink can create a backward gradient sink because many future query positions route their value-path gradients through the same source token. Previous work [12] suggests that this localized gradient pressure mediates the emergence of massive activations during training. Our analysis shows that post-SDPA gating weakens precisely this value-path link. The gate does not need to remove the attention sink column; it attenuates each future-query contribution to the sink-token value gradient, reducing the effective sink-column mass even with fixed attention weights and upstream gradients. Thus, standard gated attention implements an implicit gradient valve using the same learned gate that suppresses forward spectral structure. This provides the backward counterpart to the spectral suppression mechanism in § 3, connecting the attenuation of sink-induced representation modes with reduced sink-directed gradient flow.

The quantitative diagnostics in Table 2 provide complementary evidence: gating reduces attention concentration, value-gradient amplification, and massive activations in Pythia-410M. These signatures are most pronounced in the middle-to-late layers of the baseline model, while gating substantially suppresses their emergence during training. Qwen-1B further exhibits strong massive-activation suppression, supporting the consistency of this effect across models. Together, these findings are consistent with attenuation along the AS–GS–MA pathway. Further layer-wise and training-time analyses and details of metrics are provided in Appendix § 13.§ 2.

MetricBaselineGated
AS (m1m_1)0.0840.013
GS (ρ1\rho_1)25.405.67
MA (M∞M_\infty)320.067.4

Table 2. AS–GS–MA suppression. Final-checkpoint diagnostics for baseline and post-SDPA gated models on Pythia-410M. Detailed measurements are provided in Appendix § 13.§ 2.

Training-Time Analyses of Sink Signatures

Training-Time Sink Emergence

In this section, we examine whether the sink signatures analyzed in § 3 and § 4 also emerge during training. Using checkpoints of baseline and post-SDPA gated models that we train with the Pythia-410M architecture [20], we track first-token attention concentration and representation dominance across layers. Details and additional results are in Appendices § 11 and § 13.

Diagnostics. For each checkpoint and layer, we track the canonical first-token sink via the causal column-mass diagnostic from § 2.1. Since this corresponds to s=1s = 1, we report

m1(ℓ)=1T∑t=1TAt,1(ℓ),D1(ℓ)=∥h1(ℓ)∥21T−1∑j≠1∥hj(ℓ)∥2.(28)m_1^{(\ell)} = \frac{1}{T}\sum_{t=1}^{T} A_{t,1}^{(\ell)}, \qquad D_1^{(\ell)} = \frac{\lVert h_1^{(\ell)} \rVert_2}{\frac{1}{T-1}\sum_{j\ne1}\lVert h_j^{(\ell)} \rVert_2}. \tag*{(28)}

The first quantity measures first-token attention concentration, and the second measures whether the corresponding hidden state becomes unusually large relative to other positions.

Results. Figure 3(a,Figure 3b) tracks first-token attention mass across training checkpoints. In the baseline, first-token mass increases sharply in middle-to-late layers, indicating the emergence of a strong first-token attention sink. The post-SDPA gated model suppresses this growth and maintains substantially lower first-token mass throughout training. Figure 3(c) shows the corresponding representation-level signature at the final checkpoint: the baseline model develops pronounced first-token hidden-state dominance in the same layer range, whereas the gated model remains closer to the non-first-token scale. Thus, both attention- and representation-level sink signatures emerge during training and are substantially attenuated by post-SDPA gating. The additional analyses in Appendix § 12.§ 4 show reduced heavy-tail dominance while the bulk spacing ratio remains stable. Layer-wise linear probing in Appendix § 13.§ 3 further shows comparable next-token cross-entropy loss for the baseline and gated models once the probes are trained to convergence, with a similar result after excluding the initial source position.

Six-panel plots showing training-time sink and gradient signatures

Figure 3. Training-time sink and gradient signatures. Curves in (a, b, d–f) correspond to training checkpoints. (a, b) First-token attention mass: Baseline develops a middle-to-late-layer sink, whereas the post-SDPA gated model suppresses it. (c) First-token representation dominance at the final checkpoint, which is large in the baseline but attenuated by gating. (d, e) First-token value-gradient amplification ρ1\rho_1: Baseline develops strong amplification in middle-to-late layers, whereas gating suppresses it. (f) Gated effective-mass ratio R1rmsR^{\mathrm{rms}}_1 remains well below one across training.

Training-Time Gradient Amplification

We next examine whether the backward-pass signature analyzed in § 4 also emerges during training. Because many future queries route their value-path gradients through the same high-mass source token, a forward attention sink can induce a backward gradient sink (§ 4.3). Using the same checkpoints, we track the first-token value-gradient amplification ρ1\rho_1 in Eq. (25) and, for the gated model, the effective-mass ratio R1rmsR_1^{\mathrm{rms}} in Eq. (27). As shown in Figure 3(d,Figure 3e), the baseline develops large first-token value-gradient amplification in middle-to-late layers, the same range in which the first-token attention sink emerges, whereas the gated model keeps this amplification substantially lower in these layers. The effective-mass ratio remains well below one across much of training (Figure 3(f)), indicating that the active gate attenuates value-path contributions routed through the sink column, consistent with the local gradient valve in § 4.1. Since R1rmsR_1^{\mathrm{rms}} depends only on the attention weights and gate values, this attenuation applies even where the gated model still assigns non-negligible attention to the first token. Together with the results above, these findings show that attention concentration, value-gradient amplification, and representation dominance co-emerge in the baseline and are jointly attenuated by post-SDPA gating (additional diagnostics in Appendix § 13).

Related Work

Attention sink phenomenon. Attention sinks describe the tendency of autoregressive language models to route disproportionate attention mass to a small set of tokens, often early positions, regardless of semantic relevance. Prior work shows that this phenomenon is widespread across models and inputs, emerges during pretraining, and is affected by optimization, data distribution, loss design, and attention normalization [3, 21]. Recent studies further clarify its relation to massive activations [5, 22]: sinks and activation spikes often co-occur, but they are geometrically and functionally distinct [4]. Another study [12] complements this view from the backward pass, showing that attention sinks can induce value-path gradient concentration under causal masking.

Gated and normalization-based sink mitigation. Recent work has explored architectural interventions for reducing attention sinks, massive activations, or their co-occurrence. Post-SDPA sigmoid gating improves performance and stability while strongly reducing first-token attention concentration, attributing the effect to nonlinearity and query-dependent sparsity [10]. Alternative approaches modify attention normalization, as in Softpick’s rectified softmax and Elastic-Softmax’s learnable filtering of softmax weights [23, 24], or directly gate value states, as in Value-State Gated Attention (VGA) [6]. Other methods modify normalization or activation dynamics, including DyT [7], or introduce normalization and attention-bias variants that shift, suppress, or decouple sinks and massive activations [4, 12, 25]. Despite these advances, the mechanisms underlying sink suppression by post-SDPA gating remain unclear. We investigate them through spectral and gradient analyses.

Spectral analysis on neural representations. Spectral methods and random matrix theory have been widely used to analyze neural network weights, activations, and learned representations, separating noise-like bulk spectra from structured low-rank components [26, 27, 13, 17]. Related studies of representation collapse also emphasize the role of low-rank structure in deep features [28, 29]. Complementary work uses spectral or graph-signal viewpoints to design and interpret sequence models and representation dynamics [30, 31, 32]. In contrast, we use spectral outliers to characterize sink-aligned structure and its suppression in token-space attention representations.

Conclusion

We presented a spectral and gradient-based analysis of how post-SDPA gating mitigates attention sinks. High-mass attention columns can induce low-rank token-space modes, which learned gates attenuate beyond uniform shrinkage. In the backward pass, the gate locally reduces value-path gradient aggregation through sink columns, a gradient-valve counterpart to the forward spectral contraction. Empirically, gating reduces spectral outliers and sink-subspace alignment, attenuates first-token value-gradient amplification, and suppresses the training-time emergence of first-token attention concentration and representation dominance. Together, these results provide a unified forward–backward account of sink suppression by post-SDPA gating.

Acknowledgments

This research was partly supported by the MSIT (Ministry of Science and ICT), Korea, under the Development and Demonstration of Agentic AI Technology for Autonomous Physics Simulation Based on MCP Standard with Multi-Agent Collaboration (RS-2026-25587486), Global Research Support Program in the Digital Field program (RS-2024-00436680), AI Research Hub Project (RS-2024-00457882), and LG AI STAR Talent Development Program for Leading Large-Scale Generative AI Models in the Physical AI Domain (RS-2025-25442149), supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation). This research was partly supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. RS-2026-25607520, Center for Specialized Orchestration and Training of AI for Frontier Mathematics (SOTA-MATH)). Research for this paper was also conducted under the KICT Strategic Research Program(SRP)(project no. 20260437-001, Development of a Construction Site-Specific Vertical AI Model and Sustainable Platform) funded by the Ministry of Science and ICT. This project is supported by Microsoft Research Asia.

Proofs for Spectral Sink Suppression

Proof of Theorem 3.1

Proof. The ii-th row of the sink-induced signal component is

(Ysig)i=dasiuT.(Y_{\mathrm{sig}})_i = \sqrt{d}as_i u^{\mathsf{T}}.

After applying the post-SDPA gate, it becomes

(Y~sig)i=dasi(Γiu)T.(\widetilde{Y}_{\mathrm{sig}})_i = \sqrt{d}as_i(\Gamma_i u)^{\mathsf{T}}.

Therefore,

1d∥Y~sig∥F2=a2∑i=1nsi2∥Γiu∥22.\frac{1}{d}\|\widetilde{Y}_{\mathrm{sig}}\|_{\mathrm{F}}^2 = a^2\sum_{i=1}^{n}s_i^2\|\Gamma_i u\|_2^2.

By the definitions gi=∥Γiu∥2g_i=\|\Gamma_i u\|_2, D=diag⁡(g1,…,gn)D=\operatorname{diag}(g_1,\ldots,g_n), and

αs=∥Ds∥22=∑i=1ngi2si2,\alpha_s=\|Ds\|_2^2=\sum_{i=1}^{n}g_i^2s_i^2,

we obtain

1d∥Y~sig∥F2=a2∥Ds∥22=a2αs.\frac{1}{d}\|\widetilde{Y}_{\mathrm{sig}}\|_{\mathrm{F}}^2=a^2\|Ds\|_2^2=a^2\alpha_s.

Since 0⪯Γi⪯I0\preceq\Gamma_i\preceq I and ∥u∥2=1\|u\|_2=1, we have 0≤gi≤10\leq g_i\leq1. Hence

αs=∑i=1ngi2si2≤∑i=1nsi2=1,\alpha_s=\sum_{i=1}^{n}g_i^2s_i^2\leq\sum_{i=1}^{n}s_i^2=1,

which gives

1d∥Y~sig∥F2=a2αs≤a2.\frac{1}{d}\|\widetilde{Y}_{\mathrm{sig}}\|_{\mathrm{F}}^2=a^2\alpha_s\leq a^2.

It remains to prove the covariance-level spectral bound. Define

C~sig:=1dY~sigY~sigT.\widetilde{C}_{\mathrm{sig}}:=\frac{1}{d}\widetilde{Y}_{\mathrm{sig}}\widetilde{Y}_{\mathrm{sig}}^{\mathsf{T}}.

Since C~sig\widetilde{C}_{\mathrm{sig}} is positive semidefinite,

λmax⁡(C~sig)≤tr⁡(C~sig).\lambda_{\max}(\widetilde{C}_{\mathrm{sig}})\leq\operatorname{tr}(\widetilde{C}_{\mathrm{sig}}).

Moreover,

tr⁡(C~sig)=1dtr⁡(Y~sigY~sigT)=1d∥Y~sig∥F2=a2αs.\operatorname{tr}(\widetilde{C}_{\mathrm{sig}})=\frac{1}{d}\operatorname{tr}\left(\widetilde{Y}_{\mathrm{sig}}\widetilde{Y}_{\mathrm{sig}}^{\mathsf{T}}\right)=\frac{1}{d}\|\widetilde{Y}_{\mathrm{sig}}\|_{\mathrm{F}}^2=a^2\alpha_s.

Therefore,

λmax⁡(C~sig)≤a2αs.\lambda_{\max}(\widetilde{C}_{\mathrm{sig}})\leq a^2\alpha_s.

This proves the claim. □

Proof of Proposition 3.2

Proof. Let PI∈{0,1}n′×nP_{\mathcal{I}}\in\{0,1\}^{n'\times n} be a row-selection matrix that extracts the rows indexed by I\mathcal{I}, and suppose PIP_{\mathcal{I}} is independent of ZZ. Applying PIP_{\mathcal{I}} to the spiked representation gives

PIY=d a(PIs)uT+σPIZ.P_{\mathcal{I}}Y=\sqrt{d}\,a\left(P_{\mathcal{I}}s\right)u^{\mathsf{T}}+\sigma P_{\mathcal{I}}Z.

Define

αs:=∥PIs∥22.\alpha_s:=\|P_{\mathcal{I}}s\|_2^2.

If αs>0\alpha_s>0, let

sˉ:=PIs∥PIs∥2.\bar{s}:=\frac{P_{\mathcal{I}}s}{\|P_{\mathcal{I}}s\|_2}.

Then PIS=αssˉP_{\mathcal{I}_S}=\sqrt{\alpha_s}\bar{s}, and hence

PIY=d aαs sˉu⊤+σZ′,P_{\mathcal{I}}Y=\sqrt{d}\,a\sqrt{\alpha_s}\,\bar{s}u^{\top}+\sigma Z',

where Z′:=PIZZ':=P_{\mathcal{I}}Z. Since ZZ has i.i.d. standard Gaussian entries and PIP_{\mathcal{I}} selects rows independently of ZZ, the matrix Z′∈Rn′×dZ'\in\mathbb{R}^{n'\times d} also has i.i.d. standard Gaussian entries.

Therefore,

C~=1d(PIY)(PIY)⊤\widetilde{C}=\frac{1}{d}(P_{\mathcal{I}}Y)(P_{\mathcal{I}}Y)^{\top}

is a standard rank-one spiked Wishart covariance matrix with aspect ratio c′=n′/dc'=n'/d and effective spike strength aeff2=a2αsa_{\mathrm{eff}}^2=a^2\alpha_s. By the classical BBP phase transition, the leading eigenvalue separates from the Marchenko–Pastur bulk only when

a2αs>σ2c′.a^2\alpha_s>\sigma^2\sqrt{c'}.

Equivalently, if

a2αs≤σ2c′,a^2\alpha_s\leq\sigma^2\sqrt{c'},

the spike is subcritical and the associated spectral outlier disappears, with the top eigenvalue converging to

λ+=σ2(1+c′)2.\lambda_+=\sigma^2(1+\sqrt{c'})^2.

Discussion: post-SDPA Gates and the Scope of the BBP statement

Theorem 3.1 is deterministic and applies to learned post-SDPA gates. It only uses the transformation of the sink-induced signal row

dasiu⊤↦dasi(Γiu)⊤.\sqrt{d}as_i u^{\top}\mapsto\sqrt{d}as_i(\Gamma_i u)^{\top}.

Thus the contraction factor

αs=∑isi2∥Γiu∥22\alpha_s=\sum_i s_i^2\lVert\Gamma_i u\rVert_2^2

does not rely on any randomness assumption about the gate or on coordinate localization of ss. However, for the full representation Y=dasu⊤+σZY=\sqrt{d}asu^{\top}+\sigma Z, the learned gate also transforms the noise component. The ii-th noise row is mapped from σzi\sigma z_i to σΓizi\sigma\Gamma_i z_i. Unless the matrices Γi\Gamma_i satisfy additional isotropy assumptions, this noise is feature-dependent and token-dependent. Consequently, the gated covariance is not exactly a standard spiked Wishart matrix, and the classical BBP transition cannot be applied as an exact theorem for learned gates.

Proposition 3.2 isolates an idealized setting in which the same contraction mechanism can be connected exactly to BBP. In the row-selection model, the retained noise matrix remains i.i.d. Gaussian, so the thinned covariance is again a standard spiked Wishart model. Learned gates are not random row selectors, but the deterministic contraction result still applies. Therefore, the BBP condition should be read as a mechanism-level reference: when the contracted sink-induced signal becomes weak relative to the noise bulk, the associated spectral outlier is expected to weaken or disappear. This motivates the empirical diagnostics used in § 3.2.

Discussion: Higher-Rank Sink Subspaces and Contraction

Our empirical sink object is the multi-column subspace

Ssink=span⁡{aˉj:j∈Jsink},(29)S_{\mathrm{sink}}=\operatorname{span}\{\bar{a}_j:j\in\mathcal{J}_{\mathrm{sink}}\}, \tag*{(29)}

where Jsink\mathcal{J}_{\mathrm{sink}} may contain multiple positions, including positions other than the first. The rank-one component analyzed in the main text isolates one elementary sink contribution; it is not an assumption that all empirical sinks are literally rank one. For any combined sink component YsinkY_{\mathrm{sink}}, an elementwise post-SDPA gate with entries in [0,1][0,1] satisfies

∥G⊙Ysink∥F2≤∥Ysink∥F2.(30)\lVert G\odot Y_{\mathrm{sink}}\rVert_F^2\leq\lVert Y_{\mathrm{sink}}\rVert_F^2. \tag*{(30)}

This global inequality does not imply independent scalar contraction of every spectral spike: gating may change the relative mode strengths and rotate the resulting eigenspace. We therefore evaluate the full sink-subspace projector. The contraction applies to the combined higher-rank sink contribution, but it does not imply that a rank-rr sink subspace decomposes into rr independently contracted eigenmodes. This distinction motivates the empirical subspace-level tests in Appendix D.8.

Proofs for the Gated Gradient-Sink Bound

Proof of Theorem 4.1

Proof. For each query position tt, the post-SDPA gated attention output is y~t=Gt⊙yt\tilde{y}_t = G_t \odot y_t. Let δt=∇y~tL\delta_t = \nabla_{\tilde{y}_t}\mathcal{L}. By the chain rule,

∇ytL=Gt⊙δt.\nabla_{y_t}\mathcal{L} = G_t \odot\delta_t.

Since yt=∑j≤tatjvjy_t = \sum_{j \le t} a_{tj}v_j, the Jacobian of yty_t with respect to vsv_s is

∂yt∂vs={atsI,s≤t,0,s>t.\frac{\partial y_t}{\partial v_s} = \begin{cases} a_{ts}I, & s \le t, \\ 0, & s > t. \end{cases}

Therefore,

∇vsL=∑t=sTats∇ytL=∑t=sTats(Gt⊙δt).\nabla_{v_s}\mathcal{L} = \sum_{t=s}^{T} a_{ts}\nabla_{y_t}\mathcal{L} = \sum_{t=s}^{T} a_{ts}(G_t \odot\delta_t).

Taking norms and viewing GtG_t as a diagonal linear map gives

∥∇vsL∥≤∑t=sTats∥Gt⊙δt∥≤∑t=sTats∥Gt∥op⁡∥δt∥.\lVert\nabla_{v_s}\mathcal{L}\rVert\le\sum_{t=s}^{T} a_{ts}\lVert G_t \odot\delta_t\rVert\le\sum_{t=s}^{T} a_{ts}\lVert G_t\rVert_{\operatorname{op}}\lVert\delta_t\rVert.

If ∥δt∥≤B\lVert\delta_t\rVert\le B for every tt, then

∥∇vsL∥≤B∑t=sTats∥Gt∥op⁡=BMseff⁡.\lVert\nabla_{v_s}\mathcal{L}\rVert\le B\sum_{t=s}^{T}a_{ts}\lVert G_t\rVert_{\operatorname{op}} = B M_s^{\operatorname{eff}}.

Since Gt∈[0,1]dG_t \in[0,1]^d, we have ∥Gt∥op⁡≤1\lVert G_t\rVert_{\operatorname{op}} \le1, and hence Mseff⁡≤MsM_s^{\operatorname{eff}} \le M_s.

Proof of Corollary 4.2

Proof. From Theorem 4.1,

∇vsL=∑t=sTats(Gt⊙δt).\nabla_{v_s}\mathcal{L} = \sum_{t=s}^{T}a_{ts}(G_t \odot\delta_t).

Expanding the squared norm and using the cross-position uncorrelatedness assumption, the cross terms vanish:

E∥∇vsL∥2=∑t=sTats2E∥Gt⊙δt∥2.\mathbb{E}\lVert\nabla_{v_s}\mathcal{L}\rVert^2 = \sum_{t=s}^{T}a_{ts}^2\mathbb{E}\lVert G_t \odot\delta_t\rVert^2.

Since GtG_t acts as a diagonal linear map,

∥Gt⊙δt∥2≤∥Gt∥op⁡2∥δt∥2.\lVert G_t \odot\delta_t\rVert^2 \le\lVert G_t\rVert_{\operatorname{op}}^2\lVert\delta_t\rVert^2.

Therefore,

E∥∇vsL∥2≤B2∑t=sTats2∥Gt∥op⁡2=B2Sseff⁡.\mathbb{E}\lVert\nabla_{v_s}\mathcal{L}\rVert^2 \le B^2\sum_{t=s}^{T}a_{ts}^2\lVert G_t\rVert_{\operatorname{op}}^2 = B^2 S_s^{\operatorname{eff}}.

Finally, Gt∈[0,1]dG_t \in[0,1]^d implies ∥Gt∥op⁡≤1\lVert G_t\rVert_{\operatorname{op}} \le1, and hence Sseff⁡≤SsS_s^{\operatorname{eff}} \le S_s.

Local Value-path Attenuation versus Global Training Dynamics

Our gradient results are local statements about the value-path Jacobian. Conditioned on fixed attention weights and upstream gradients δt=∇ytL\delta_t = \nabla_{y_t}\mathcal{L}, post-SDPA gating filters each contribution from query position tt to source value vsv_s. Equivalently, the raw sink-column statistics MsM_s and SsS_s are replaced by their gated counterparts MseffM_s^{\mathrm{eff}} and SseffS_s^{\mathrm{eff}}. This does not imply that all gradients in a trained model with post-SDPA gating are uniformly smaller than in a model without post-SDPA gating, since gating also changes the forward computation, upstream gradients, and learned representations. Rather, the structural claim is that for a fixed upstream signal, the value-path map through a sink column is contractive. The deterministic and second-moment bounds cover coherent and incoherent aggregation regimes, respectively, and in both cases

Mseff≤Ms,Sseff≤Ss.M_s^{\mathrm{eff}} \le M_s,\qquad S_s^{\mathrm{eff}} \le S_s.

Thus, post-SDPA gating weakens the attention sink to gradient sink link without necessarily removing the attention sink itself: a sink column may still receive attention, but each value-path contribution is filtered before reaching the source value. This provides a local mechanism for reducing gradient pressure in the attention-sink–gradient-sink–massive-activation pathway, without claiming a full theory of global training dynamics.

Experimental Setting

Hardware. All experiments were conducted on a server equipped with two AMD EPYC 9654 96-core CPUs (192 physical cores / 384 threads in total, 1.5–3.7 GHz, 768 MiB L3 cache) and 1 TB of system memory. The server is configured with eight NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation GPUs (96 GB GDDR7 each, driver 580.95.05). All code was implemented in PyTorch 2.9.1 built against CUDA 12.8 (toolkit nvcc 12.8.93) on Python 3.13.5.

Model checkpoints. For the Qwen analyses, we use the public checkpoints released by a previous work [10] in the QwQZh/gated_attention HuggingFace repository.3 Specifically, we analyze 1B_baseline and 1B_gate_elementwise. All models are evaluated in inference mode, with no additional training or parameter updates.

Training setup for training-time diagnostics. For the training-time diagnostics, we train Pythia-410M baseline and post-SDPA gated variants using the HuggingFace training framework, following the language-modeling training protocol of [33]. The two models are trained under matched data, optimization, and checkpointing conditions. The baseline model uses 24 Transformer layers, hidden size 1024, 16 attention heads, and MLP intermediate size 4096. The elementwise gated variant uses the same configuration except for the post-SDPA elementwise gate and a reduced MLP intermediate size of 3584, which compensates for the additional gate parameters. Both models use the EleutherAI/gpt-neox-20b tokenizer and are trained on 15B tokens from FineWeb-Edu [34] with causal language modeling, sequence length 4096, AdamW optimization, and peak learning rate 4×10−44 \times10^{-4}. We save checkpoints throughout training and evaluate without further parameter updates.

To examine robustness beyond the original setting, we also analyze matched baseline and post-SDPA elementwise-gated Pythia-2.8B models; matched Pythia-410M models trained with an 8K context length and evaluated at 4K or 8K; and matched Pythia-410M models trained with document-boundary EOS separators. The plain sequence convention uses add_bos_token = False and add_eos_token = False; the EOS-separated convention uses add_bos_token = False and add_eos_token = True. Protocol-specific evaluation conditions and measurements are provided in Appendices F.1–F.3.

Language modeling and zero-shot commonsense reasoning. Following prior works [33, 30], we evaluate baseline and post-SDPA gated variants across two evaluation categories: WikiText (Wiki.; [19]) perplexity and zero-shot commonsense reasoning tasks. The commonsense tasks include LAMBADA (LMB.; [35]), PIQA [36], HellaSwag (Hella.; [37]), WinoGrande (Wino.; [38]), ARC-easy (ARC-e) and ARC-challenge (ARC-c) [39], BoolQ [40], and OpenbookQA (OBQA.; [41]).

Additional RMT Diagnostics

This appendix provides additional diagnostics supporting the RMT analysis in § 3.2. The main text focuses on spectral suppression and sink-subspace alignment. We report robustness checks, gate statistics, sink-subspace counterfactuals, contraction diagnostics, and bulk-level spectral analyses.

Robustness to Covariance Normalization and Feature-Averaged Gate Control

In the main text, we compute token-space covariance after feature-wise standardization across tokens. This removes trivial feature-scale effects and focuses the diagnostics on token-space correlation structure. To verify that the observed suppression is not an artifact of this normalization, we repeat the analysis using the non-standardized covariance CYraw=1DYYTC_Y^{\mathrm{raw}} = \frac{1}{D}YY^{\mathsf{T}}. Since the overall scale of CYrawC_Y^{\mathrm{raw}} is not fixed, we rescale the MP edge by tr⁡(CYraw)/T\operatorname{tr}(C_Y^{\mathrm{raw}})/T so that the outlier mass is invariant to global rescaling.

We also compare the learned elementwise gate with a feature-averaged gate control. For each token, we replace the feature-wise gate vector Gi∈[0,1]DG_i \in[0,1]^D by its feature average Gˉi=1D∑c=1DGi,c\bar{G}_i = \frac{1}{D}\sum_{c=1}^{D}G_{i,c}, and apply the scalar gate GˉiI\bar{G}_i I to the SDPA output. This control preserves token-wise gate strength, so it can still suppress particular token positions, but removes feature-direction-specific selectivity.

Figure 4 shows that the post-SDPA gate still suppresses dominant spectral structure without feature-wise standardization. In particular, compared with Pre-Gate, Post-Gate (elem) has lower outlier mass and higher effective rank in the non-standardized covariance across most layers. By contrast, Post-Gate (scalar) does not reproduce the same suppression pattern: its outlier mass and effective rank remain close to those of Pre-Gate, indicating that average token-wise shrinkage alone is not sufficient. Figure 4(c) further shows that the reduction in leading-mode concentration λ1/tr⁡(C)\lambda_1/\operatorname{tr}(C) from Pre-Gate to Post-Gate (elem) is at least as strong without standardization as with it, so feature-wise standardization does not create or inflate the observed suppression. These results indicate that the spectral suppression is not merely a covariance-normalization artifact or a consequence of average token-wise shrinkage. Rather, the feature-selective action of the learned post-SDPA gate is important for suppressing dominant token-space spectral structure. The following sections further examine where this selective action appears in gate statistics and how it relates to sink-induced subspaces.

Figure 4. Robustness to standardization and gate control. Without standardization, the post-SDPA gate also reduces outlier mass and increases effective rank. A feature-averaged scalar gate preserves token-wise gate strength but removes feature-direction-specific selectivity, and does not reproduce the same spectral suppression.

Learned Selectivity Beyond Generic Mode Contraction

The contraction result in Theorem 3.1 applies to any individual mode; it does not, by itself, establish preferential suppression of sink-aligned directions. To make this distinction explicit, consider a rank-one pre-gate component Yk=σkskukTY_k = \sigma_k s_k u_k^{\mathsf{T}}, where ∥sk∥2=∥uk∥2=1\lVert s_k\rVert_2 = \lVert u_k\rVert_2 = 1. For the elementwise gate GG, with Γi\Gamma_i denoting the diagonal matrix of gate values at token ii, its raw-energy retention is

αk=∥G⊙Yk∥F2∥Yk∥F2=∑isk,i2∥Γiuk∥22≤1.(31)\alpha_k = \frac{\lVert G \mathbin{\odot} Y_k\rVert_F^2}{\lVert Y_k\rVert_F^2} = \sum_i s_{k,i}^2 \lVert\Gamma_i u_k\rVert_2^2 \le1. \tag*{(31)}

This inequality does not order sink- and non-sink-aligned modes by their retention. We evaluate this distinction using the 64 leading pre-gate modes of the elementwise-gated Qwen-1B model, comparing the top and bottom terciles of sink alignment ∥Psinksk∥22\lVert P_{\mathrm{sink}}s_k\rVert_2^2 (Table 3).

Retention metricHigh alignmentLow alignmentDifference
Raw activation energy (αk\alpha_k)0.0470.044+0.003
Standardized covariance (RkR_k)0.7400.945−0.205
Scalar gate (RkR_k)0.9570.990−0.033

Table 3. Mode retention for strongly vs. weakly sink-aligned pre-gate directions in Qwen-1B (Gated). Raw-energy retention is computed on individual rank-one components; the other rows use feature-standardized token-space covariance.

We therefore additionally examine retention in the feature-standardized token-space covariance geometry used for the main spectral analysis. For each pre-gate covariance eigenvector qkq_k, define

Rk=qkTCpostqkqkTCpreqk,(32)R_k = \frac{q_k^\mathsf{T} C_{\mathrm{post}} q_k}{q_k^\mathsf{T} C_{\mathrm{pre}} q_k}, \tag*{(32)}

where CpreC_{\mathrm{pre}} and CpostC_{\mathrm{post}} are the pre- and post-gate feature-standardized token-space covariance matrices. This standardized geometry reveals a directional difference that is absent from raw-energy retention. The negative association between sink alignment and retention persists after controlling for pre-gate spectral energy, so it cannot be explained merely by larger initial eigenvalues. As an ablation, we replace each token’s featurewise gate values by their feature average, removing within-token feature selectivity. The resulting retention gap becomes substantially smaller, as shown in Table 3. Thus, the gate is nearly non-selective in raw activation energy but selectively reshapes the feature-standardized covariance through its learned feature-dependent structure. This empirical selectivity is additional to, rather than a consequence of, the unconditional mode-wise contraction bound.

Post-SDPA Gate-score Statistics

We also inspect the post-SDPA gate scores directly. While average gate values do not by themselves determine spectral suppression, they provide an interpretable view of where the model applies stronger filtering. We compute gate statistics across layers, tokens, and heads, and compare the gate values at sink-related positions with those at non-sink positions. Figure 5 shows that gate scores vary substantially across depth and heads. In layers where spectral spikes are pronounced, the gate tends to apply stronger filtering to sink-related positions and hot-spot heads. This is consistent with the contraction analysis in § 3: suppression depends not only on the average gate magnitude, but on how the learned gate acts on the dominant representation modes.

Gate at sink vs. non-sink positionsToken-wise gate heatmapHead-wise sink/non-sink gate ratio

Figure 5. Gate-score statistics. The post-SDPA gate is not spatially uniform: its strength varies across depth and is often concentrated around sink-related positions and hotspot heads. These statistics provide an interpretable view of the learned filtering pattern.

Structural and Bulk Diagnostics

We report a set of complementary diagnostics that probe different facets of the post-SDPA spectrum, separating outlier-level structure from bulk behavior. Where the main analysis (§ 3.2) used the data-adaptive sink subspace PsinkP_{\mathrm{sink}}, here we additionally measure the alignment of the leading eigenvector with the simpler first-token reference direction e0e_0. Second, we compute head-wise token-space spectra to identify whether dominant modes are distributed uniformly across heads or concentrated in a small number of hotspot heads. Third, we examine the eigenvalue tail using the complementary cumulative distribution function (CCDF), P(λ≥x)P(\lambda\ge x), which summarizes how much spectral mass remains in large eigenvalues. Finally, after removing eigenvalues above the Marchenko–Pastur upper edge λ+\lambda_+ (i.e., the bulk only), we compute the adjacent spacing ratio

Structural and bulk diagnostics

Figure 6. Structural and bulk diagnostics. (a) Leading-eigenvector alignment with the first-token direction is reduced where such alignment is present. (b) Head-wise λ1\lambda_1 ratios show that localized hotspot heads with large leading eigenvalues tend to be suppressed by gating. (c) The eigenvalue-tail CCDF P(λ≥x)P(\lambda\ge x) shows reduced heavy-tail dominance after gating. (d) Bulk spacing ratios remain stable, indicating that gating suppresses outlier structure while largely preserving the spectral bulk.

ri=min⁡(Δi,Δi+1)max⁡(Δi,Δi+1),Δi=λi+1−λi.r_i = \frac{\min(\Delta_i,\Delta_{i+1})}{\max(\Delta_i,\Delta_{i+1})}, \qquad\Delta_i = \lambda_{i+1} - \lambda_i.

and report its layer-wise mean. This statistic tests whether gating disrupts the local bulk spectrum, with typical reference values near the GOE and Poisson limits shown in the figure.

Figure 6 summarizes these diagnostics. Leading eigenspace alignment decreases in layers where sink-related alignment is elevated, and head-wise spectra show that large leading eigenvalues are localized to a small number of heads. Gating reduces these localized spectral hotspots. The tail CCDF shows reduced heavy-tail dominance after gating, while the bulk spacing ratio remains stable. Together, these results support the view that post-SDPA gating primarily suppresses outlier and heavy-tail components associated with sink structure, rather than broadly disrupting the spectral bulk.

Sink-subspace Counterfactuals and Contraction Diagnostics

We next examine whether the spectral suppression observed in § 3.2 is attributable to the sink-induced subspace. Using the data-adaptive sink projector PsinkP_{\mathrm{sink}}, we compare three quantities per layer: the pre-gate leading eigenvalue λ1(CY)\lambda_1(C_Y), the same quantity after explicitly removing the sink subspace, λ1((I−Psink)CY(I−Psink))\lambda_1((I-P_{\mathrm{sink}})C_Y(I-P_{\mathrm{sink}})), and the post-gate value λ1(CZ)\lambda_1(C_Z). As shown in Figure 7, projecting out the sink subspace lowers λ1\lambda_1 toward the Marchenko–Pastur edge λ+\lambda_+ in most layers, indicating that sink-induced directions account for part of the dominant pre-gate spectral component. The post-gate value λ1(CZ)\lambda_1(C_Z) lies below this sink-ablated value in most layers, suggesting that the gate also attenuates leading structure outside the selected sink subspace.

Sink-subspace diagnostic

Figure 7. Sink-subspace diagnostic. Removing PsinkP_{\mathrm{sink}} lowers λ1\lambda_1.

Attention–Value Pairing Permutation Control

The alignment of the dominant token-space direction with a sink subspace could, in principle, arise from a special value state associated with a sink position rather than from the high-mass attention-column geometry. To test this alternative, we preserve the entire attention matrix AA and empirical value matrix VV while randomly permuting their positionwise pairing:

Yπ=AΠV=∑jA:,jVπ(j),:T,(33)Y_{\pi} = A\Pi V = \sum_j A_{:,j}V_{\pi(j),:}^{\mathsf{T}}, \tag*{(33)}

where Π\Pi is a permutation matrix. The original sink-subspace projector is held fixed, and the alignment is recomputed using the same feature-standardized diagnostic as in the main spectral analysis. We average over 20 random permutations for each of the same 16 sequences used in the original analysis packed with an EOS separator.

RepresentationOriginalPermutedLayers with increase
Baseline0.0620.196100%
Gated (pre-gate)0.2200.51096%

Table 4. Sink-subspace alignment under attention–value pairing permutation. The attention matrix, empirical value matrix, and original sink projector are preserved; only the column–value pairing is randomized. Results are averaged over 20 permutations per sequence and 16 sequences. Alignment is the feature-standardized sink alignment of the pre-gate output, as in the main spectral analysis.

Table 4 shows that sink alignment increases rather than decreases when the original attention-column–value pairing is permuted. Therefore, the sink-aligned leading direction does not require a privileged value vector attached to the original sink position.

Robustness Beyond the Isotropic-Value Approximation

The isotropic-value approximation is an interpretable reference rather than a literal assumption about trained values. For an attention output Y=AVY = AV, write

BV=1dVV⊤=σ^V2I+ΔV.(34)B_V = \frac{1}{d}VV^\top= \hat{\sigma}_V^2 I + \Delta_V. \tag*{(34)}

Then the exact raw token-space covariance is

CYraw=1dYY⊤=ABVA⊤=σ^V2AA⊤+AΔVA⊤.(35)C_Y^{\mathrm{raw}} = \frac{1}{d}YY^\top= AB_VA^\top= \hat{\sigma}_V^2 AA^\top+ A\Delta_VA^\top. \tag*{(35)}

The first term isolates the attention-column reference; the second captures learned value anisotropy, which can strengthen, weaken, or rotate the resulting spectral directions.

ModelCross-token corr.Norm. feature rankAlign. (α=0\alpha= 0)Align. (α=1\alpha= 1)Random align.
Baseline4.90.440.970.64≈0.003\approx0.003
Gated14.40.330.790.32≈0.009\approx0.009

Table 5. Value anisotropy and sink alignment. Alignment at the endpoints α=0\alpha= 0 (isotropic) and α=1\alpha= 1 (learned values) of the value-Gram interpolation in Qwen-1B, computed on the non-standardized pre-gate covariance CαC_\alpha and thus not directly comparable to Table 4. An isotropic Gaussian null gives cross-token correlation ≈1\approx1 and normalized feature rank ≈0.88\approx0.88.

On the Qwen-1B checkpoints and sequences used for the original spectral analysis, learned values are strongly anisotropic. Table 5 reports the dimension-normalized cross-token correlations and normalized feature effective ranks, alongside a matched isotropic Gaussian reference. These diagnostics show substantial learned-value anisotropy in both checkpoints. To assess sensitivity to this anisotropy while holding AA fixed, we interpolate the value Gram matrix in the positive semidefinite cone:

Bα=(1−α)σ^V2I+αBV,Cα=ABαA⊤,α∈[0,1].(36)B_\alpha= (1-\alpha)\hat{\sigma}_V^2 I + \alpha B_V,\qquad C_\alpha= AB_\alpha A^\top,\qquad\alpha\in[0,1]. \tag*{(36)}

The endpoint α=0\alpha= 0 is the isotropic attention-column reference, whereas α=1\alpha= 1 uses the empirical learned-value Gram matrix. As Table 5 shows, moving from the isotropic reference to the empirical value geometry reduces alignment in both models, but the empirical endpoint remains far above the corresponding random-subspace reference in every evaluated layer and sample. The isotropic approximation is therefore useful as a structural prediction for sink-aligned spectral modes. Importantly, the deterministic gate contraction theorem does not rely on value isotropy; isotropy enters only the interpretable AA⊤AA^\top reference and the idealized BBP analysis.

Naturally Occurring Non-First-Position Sinks

We test whether the sink-subspace spectral signature persists beyond the first position and beyond a literal rank-one object. The general higher-rank contraction argument is discussed in Appendix § 9.§ 4. We exclude position 0 when selecting sink columns in normally trained checkpoints and evaluate their contribution to the dominant token-space covariance direction. Table 6 reports the observed alignment and the effect of projecting out the selected sink subspace. Across the two model families, the selected sink sets contain approximately 6.7 non-first columns on average. Their leading-eigenvector alignment is roughly 20× the random-subspace reference, and removing these subspaces reduces λ1\lambda_1 by about 19%. Approximately 43–45% of the selected non-first column instances correspond to structural delimiters, including periods, commas, and newlines. Within the gated models, post-SDPA gating reduces the corresponding alignment by 33–41%. These observations show that the forward spectral signature extends to multi-column, delimiter-rich sink structure.

MetricQwen-1BPythia-410M
Align. (pre → post)0.280 → 0.1860.262 → 0.154
Pre/random21×20×
Align. drop33.4%41.2%
λ1\lambda_1 drop (proj.)19.4%19.2%

Table 6. Non-first-position sink structure. The first position is excluded from the sink subspace. Alignment is compared before and after gating, with pre-gate alignment normalized by the random-subspace baseline. Projection measures the reduction in the leading covariance eigenvalue after removing the selected sink subspace.

Additional Training-Time Diagnostics

Training-Time Sink and Gradient Diagnostics

We provide additional training-time diagnostics that complement the final-checkpoint analyses in the main text. These diagnostics track how attention concentration, token-space spectral structure, and value-gradient amplification evolve across checkpoints in the baseline and post-SDPA gated models. To verify that the gated variant remains competitive under the matched 15B-token training setup, we also report standard language-modeling and downstream evaluation results in Table 7. Unless otherwise stated, all diagnostics are evaluated on sixteen 512-token WikiText-103 validation sequences, with an EOS token between passages. Table 8 reports their layer means at the final checkpoint, and Figure 8 shows them layer-wise across checkpoints.

Training-time sink and spectral diagnostics

Figure 8. Training-time sink and spectral diagnostics. Each curve corresponds to a training checkpoint. (a,b) Top-kk sink mass is the mean M‾j\overline{M}_j over the sink columns selected as in § 3.2, while (c,d) outlier mass measures the fraction of token-space covariance spectrum above the MP edge. Compared with the baseline, post-SDPA gating keeps both high-mass sink columns and out-of-bulk spectral concentration substantially weaker across training.

ModelWiki. ppl↓\downarrowLMB. ppl↓\downarrowLMB. acc↑\uparrowPIQA acc↑\uparrowHella acc_n↑\uparrowWino acc↑\uparrowARC-e acc↑\uparrowARC-c acc_n↑\uparrowOBQA acc_n↑\uparrowBoolQ acc↑\uparrowAvg.↑\uparrow
Baseline29.5355.2129.9465.8336.7651.7056.0226.5430.8062.1144.96
Gated27.9940.5232.4766.4337.9352.4957.7927.6532.0061.7146.06

Table 7. Downstream evaluation of Pythia-410M. Baseline and post-SDPA gated variants trained on 15B FineWeb-Edu tokens. All variants use matched data and optimization settings, with gated models adjusting MLP size to compensate for gate parameters. The best results are highlighted in bold. Avg. denotes the average of accuracies and normalized accuracies over 8 tasks.

Variantm1m_1mtopkm_{\mathrm{top}k}λ1/λ+\lambda_1/\lambda_+OutMassreffr_{\mathrm{eff}}ρ1\rho_1
Baseline0.0840.12518.420.67989.1425.40
Gated0.0130.01312.490.545153.205.67

Table 8. Final-checkpoint diagnostics. Layer-mean values for Pythia-410M at the final checkpoint. Post-SDPA gating reduces attention concentration, weakens dominant spectral structure, reduces out-of-bulk mass, increases effective rank, and substantially lowers first-token value-gradient amplification. Spectral metrics for the gated model are computed on the post-gate output.

Table 8 summarizes the same trend observed in the layer-wise diagnostics. Post-SDPA gating reduces first-token and top-kk sink mass, lowers the leading-eigenvalue ratio, reduces outlier mass, increases effective rank, and substantially reduces first-token value-gradient amplification. Thus, the gated model shows weaker attention concentration, weaker dominant spectral structure, and weaker sink-directed gradient amplification at the final checkpoint.

Figure 8 tracks the emergence of sink and spectral signatures during training. In the baseline, top-kk sink mass grows in middle-to-late layers, indicating the formation of high-mass attention columns. The same layers also show stronger out-of-bulk spectral mass, consistent with the sink-induced spectral-spike view. In contrast, the post-SDPA gated model keeps both quantities lower across checkpoints, suggesting that gating suppresses the co-emergence of attention concentration and token-space spectral outliers.

Direct Measurements of Massive Activations and Their Relation to Sinks

To measure massive activations directly, rather than relying solely on relative first-token dominance, we record residual-stream hidden states hth_t after each Transformer block. Let V\mathcal{V} denote the set of non-padding token positions in each sample. We measure the following massive-activation (MA) diagnostics:

M2=max⁡t∈V∥ht∥2,M_2 = \max_{t \in\mathcal{V}} \lVert h_t \rVert_2,

(maximum token hidden-state norm), {#eq:37}

M∞=max⁡t∈V, c∣ht,c∣,M_\infty= \max_{t \in\mathcal{V},\,c} \lvert h_{t,c} \rvert,

(maximum coordinate magnitude). {#eq:38}

For Pythia-410M and Qwen-1B, we evaluate both MA diagnostics on the same sixteen 512-token WikiText-103 validation blocks without padding using model-specific tokenization. Each metric is computed per sample and layer, then averaged across samples and layers for Table 9.

ModelDiagnosticBaselineGatedBaseline/gated
Pythia-410MM2M_2 (MA)379.2108.43.50×
M∞M_\infty (MA)320.067.44.75×
Qwen-1BM2M_2 (MA)11,854.5146.381.1×
M∞M_\infty (MA)11,839.587.2135.7×

Table 9. Massive-activation (MA) diagnostics. Entries are sample- and layer-mean values on sixteen 512-token WikiText-103 validation blocks without padding. The last column reports baseline-to-gated ratios.

In Pythia-410M, post-SDPA gating jointly suppresses AS, GS, and MA (Table 8 and Table 9). Their baseline peaks occur in middle-to-late layers (17–21), while the layerwise maxima of M2M_2 and M∞M_\infty decrease from 726.7 to 186.6 and from 643.0 to 154.7, respectively. Across 56 training checkpoints, the baseline’s layer-mean M∞M_\infty grows from 8.7 to 320.0, whereas the gated model never exceeds 69.4. Qwen-1B further confirms cross-model MA suppression, with both M2M_2 and M∞M_\infty reduced by over an order of magnitude (Table 9). The maximum observed coordinate magnitude is 17,536 in the baseline (first token, layer 8), compared with 378 in the gated model across the evaluated samples and layers. Together, these measurements show lower attention concentration, value-gradient amplification, and massive-activation magnitudes in the gated Pythia-410M model, while the Qwen-1B results confirm massive-activation suppression. Combined with the fixed-upstream-gradient counterfactual in § 4.2, these findings are consistent with attenuation along the AS–GS–MA pathway.

Additional Next-Token Probing Diagnostic

We additionally examine whether the sink-related changes observed under gating are accompanied by changes in linearly accessible next-token information. Conceptually, this asks whether target-related information remains recoverable from hidden states across layers, especially around layers where sink signatures are pronounced.

Practical proxy. Since the conditional uncertainty of the next token given a hidden state is difficult to estimate directly, we use layer-wise linear probing as a practical proxy for next-token decodability. For each layer ℓ\ell, we train a linear probe to predict the next token from the hidden state HℓH_{\ell}, and report cross-entropy loss on the test set. Lower cross-entropy indicates that next-token information is more linearly recoverable from the hidden representation.

Experimental setting. We use the Qwen-1B models with and without post-SDPA gating and the WikiText-2 dataset with maximum sequence length 128. For each model and layer, we train a linear probe and evaluate it on the test split, where layer 0 denotes the embedding output. Each probe is trained on raw or per-dimension standardized hidden states, either for one epoch with learning rate 10−310^{-3} or for five epochs with learning rate 10−410^{-4}. We report results both with and without the initial source position to check whether the observed trend is driven solely by the canonical sink position.

Experimental results. Under the original protocol (Figure 9(a)), baseline probes are much worse than gated probes in layers 8–28, where massive activations emerge in the baseline (Appendix § 13.§ 2). Massive activations place a few hidden dimensions at a far larger scale than the rest, which makes a linear probe difficult to optimize under a fixed training budget; in these layers, the baseline probes are even worse than a uniform prediction over the vocabulary. Gated probes, whose hidden states have much smaller activations, are far less affected. To test whether this gap reflects missing next-token information or only this optimization difficulty, we standardize each hidden dimension before probing, which leaves the class of linear probes unchanged. Under the same training budget, the large gap in deep layers mostly disappears (Figure 9(b)). A smaller gap remains in layers 8–17, where the first-token massive activation appears, and it vanishes once the initial source position is excluded. With a lower learning rate and longer training, this remaining gap also largely closes (Figure 9(c)), and the two models reach comparable probing loss across depth, with or without the initial source position. These results indicate that the gap under the original protocol arises from the difficulty of training linear probes on hidden states with massive activations, rather than from a lack of next-token information. Post-SDPA gating suppresses massive activations and thereby makes next-token information easier to read out with a linear probe, while both models retain comparable linearly recoverable next-token information.

Three plots of layer-wise probing cross-entropy loss for baseline and gated models

Figure 9. Layer-wise probing cross-entropy loss. Baseline and gated models are evaluated with and without the initial source position. Probes are trained for one epoch on (a) raw and (b) per-dimension standardized hidden states, and (c) for up to five epochs with a lower learning rate on standardized hidden states. Lower is better.

Robustness Across Models and Settings

We examine whether the attention, spectral, and value-gradient signatures extend beyond the original model scale, context length, and sequence-formatting convention. The following analyses compare matched baseline and post-SDPA elementwise-gated models. These experiments use separately constructed WikiText-103 test inputs, so absolute values differ slightly from Table 8.

Robustness Across Model Scale

We trained matched ungated and post-SDPA elementwise-gated Pythia-2.8B models and repeated the attention, value-gradient, and token-space spectral diagnostics. Table 10 reports absolute layer means for the 2.8B models and gated-to-baseline ratios for both 2.8B and the corresponding 410M comparison. At 2.8B, gating reduces first-token attention mass by approximately 13×, top-kk attention mass by approximately 15×, and first-token and top-kk value-gradient amplification by 6.6× and 7.5×, respectively. The leading-outlier ratio decreases from 24.21 to 15.54, while effective rank increases from 111.2 to 185.0. All diagnostics change in the same qualitative direction at 2.8B and 410M. The attention- and gradient-related reductions are at least as strong at 2.8B, and the leading-outlier and effective-rank ratios are similar at the two scales. Thus, the attenuation of sink-related attention, gradient, and spectral signatures by post-SDPA gating is not specific to the 410M scale.

Modelm1m_1mtopkm_{\mathrm{top}k}ρ1\rho_1ρtopk\rho_{\mathrm{top}k}OutMassλ1/λ+\lambda_1/\lambda_+Eff. rankSink alignment
Baseline 2.8B0.1470.15628.2427.040.68424.21111.20.174
Gated 2.8B0.0110.0104.293.610.56615.54185.00.156
2.8B ratio0.0770.0670.1520.1330.8270.6421.6640.895
410M ratio0.1290.1010.1950.1600.7960.6581.7170.657

Table 10. Model-scale robustness. The first two rows are absolute layer-mean values at 2.8B; the last two rows report gated/baseline ratios. Ratios below 1 indicate decreases under gating.

Robustness to Longer Training and Evaluation Contexts

To test context-length robustness beyond the original 4K setting, we trained matched baseline and elementwise-gated Pythia-410M models with an 8K context window. We evaluate the 8K-trained checkpoints at both 4K and 8K, and compare them to 4K-trained checkpoints evaluated at 4K. All use the original plain sequence convention, in which the first token is an ordinary content token. Table 11 reports baseline-to-gated values. Effective rank and sink alignment are normalized by min⁡(T,D)\min(T,D) and k/Tk/T, respectively, for comparison across sequence lengths.

Train/evalModelm1m_1mtopkm_{\mathrm{top}k}Sink columnsλ1/λ+\lambda_1/\lambda_+Norm. eff. rankAlignment/randomρtopk\rho_{\mathrm{top}k}
4K/4KBaseline0.0250.0712.544.10.159180.458.4
Gated0.0020.0036.529.10.28858.56.2
8K/4KBaseline0.0150.0583.353.00.118203.854.7
Gated0.0020.0035.735.40.23750.77.5
8K/8KBaseline0.0080.0583.159.00.134349.9—
Gated0.0010.0025.842.20.26268.7—

Table 11. Robustness across training and evaluation context lengths. Baseline and elementwise-gated models are compared under matched context-length settings. Top-kk value-gradient amplification is omitted for the 8K/8K setting because the backward pass exceeded available device memory.

At both evaluation lengths, gating reduces the first-position and adaptive top-kk sink masses, the normalized leading spectral outlier, and sink-subspace alignment, while increasing normalized effective rank. Within the 8K-trained baseline, extending evaluation from 4K to 8K reduces first-position mass from 0.015 to 0.008, even though mtopkm_{\mathrm{top}k} stays at 0.058. This suggests that the longer-context sink pattern is less exclusively concentrated at position 0 and is better captured by adaptive, potentially non-first sink columns. The 8K-trained model also exhibits strong value-gradient amplification at 4K evaluation, reduced by gating from 54.7 to 7.5. We do not report an 8K/8K backward measurement because retaining the attention states for backpropagation scales quadratically with sequence length and exceeded device memory. Consequently, forward attention and spectral robustness are established at 8K evaluation, while the gradient result for the 8K-trained checkpoint is directly verified at 4K evaluation.

Robustness to Sequence Formatting and Special Tokens

Sequence formatting conventions. We additionally trained matched baseline and post-SDPA gated Pythia-410M models under two sequence conventions: (i) plain, with add_bos_token = False and add_eos_token = False; and (ii) EOS-separated, with EOS appended at document boundaries (add_eos_token = True, add_bos_token = False). Table 12 shows gated/baseline ratios, not absolute metric values. Values below 1 indicate attenuation, except that an effective-rank ratio greater than 1 indicates increased effective rank. Across both conventions, gating decreases attention-sink mass, value-gradient amplification, outlier mass, the leading-outlier ratio, and sink alignment, while increasing effective rank. The direct comparison before and after the gate within each gated model also reproduces the main spectral pattern: λ1/λ+\lambda_1/\lambda_+ falls to 0.608×0.608\times under both conventions, effective rank increases by 1.811.81–1.89×1.89\times, and sink alignment decreases to 0.510.51–0.53×0.53\times.

Conventionm1m_1mtopkm_{\mathrm{top}k}ρ1\rho_1ρtopk\rho_{\mathrm{top}k}OutMassλ1/λ+\lambda_1/\lambda_+Eff. rankSink alignment
Plain0.1350.1080.2060.1750.7970.6491.7230.660
EOS-separated0.1830.2260.2530.3200.7860.6441.7350.385

Table 12. Robustness across natural sequence-formatting conventions. All entries are gated/baseline ratios.

Controlled EOS-separator sinks. We also train matched baseline and elementwise-gated Pythia-410M models with EOS tokens inserted at document boundaries, evaluating each model under the same convention in Table 12. Position 0 is excluded when constructing the sink subspace. The selected non-first sink sets contain an average of 2.64 columns in the baseline and 3.54 in the gated model, with EOS tokens accounting for 58% and 71% of these selected columns, respectively. Their pre-gate or ungated alignments with the leading covariance direction are 0.184 (baseline) and 0.190 (gated, before the gate), corresponding to 28–36×\times their random-subspace references. Projecting out these sink subspaces reduces λ1\lambda_1 by 12–14%. Within the gated model, the alignment falls from 0.190 to 0.099 after gating, a factor of 0.52. Repeating the analysis with the column-energy selector ∥A:,j∥22\lVert A_{:,j}\rVert_2^2 yields a reduction from 0.182 to 0.069. Thus, learned EOS-related sinks can occupy multiple non-first columns and remain attenuated by gating.

Position-Controlled EOS Counterfactual

To isolate the role of token identity from absolute position, we evaluate otherwise identical content after replacing one token with EOS at the specified relative position. The strongest sink is selected by availability-normalized causal column mass, i.e., the column mass divided by the number of queries that can attend to each position. Unlike the other analyses, we use this normalization here so that sink strength is comparable across insertion positions. Since it divides by only a few queries near the end of the sequence, we restrict candidates to positions with at least T/8T/8 future queries. For the no-EOS condition, the target is position 0. Table 13 reports baseline and gated results. A first-position sink arises even without EOS. When EOS is inserted, the strongest attention sink largely follows it in both models: the inserted EOS is the strongest sink in 88–95% of ungated and 71–99% of gated evaluations. Gating therefore leaves the sink location largely unchanged but reduces its strength: target attention mass decreases by 2.8–5.4× and target gradient amplification by 2.5–3.8×. When EOS appears away from the start, position 0 is no longer the dominant attention sink but retains a residual value-gradient signature, with amplification of 24.47–28.84 reduced to 6.00–6.18 by gating. Thus, sink location is affected by both sequence position and special-token identity, whereas gate-induced attenuation persists under these changes.

Input conditionModelP(target strongest)P(\text{target strongest})Target attn. massTarget spectral alignmentTarget gradient amplification
No EOS; position 0Baseline0.5830.1300.20431.39
No EOS; position 0Gated0.1740.0150.0316.38
EOS at 0Baseline0.9300.2000.22940.97
EOS at 0Gated0.7060.0370.05610.77
EOS at 0.25T0.25TBaseline0.8780.1810.04228.64
EOS at 0.25T0.25TGated0.7530.0410.02310.20
EOS at 0.50T0.50TBaseline0.9110.1990.13125.30
EOS at 0.50T0.50TGated0.8850.0550.03110.03
EOS at 0.75T0.75TBaseline0.9530.2370.14020.88
EOS at 0.75T0.75TGated0.9900.0860.0528.33

Table 13. Position-controlled EOS counterfactual. Baseline and post-SDPA gated models are compared across EOS insertion positions. Without EOS, the target is position 0; otherwise, the target is the inserted EOS token. The strongest sink is selected among positions with at least T/8T/8 future queries. Spectral alignment for the gated model uses the post-gate output.

Extensions Beyond Post-SDPA Gating

We use the spectral and gradient diagnostics to characterize intervention classes beyond post-SDPA gating. We distinguish direct empirical comparisons of attention operators from theoretical comparisons of local gradient pathways and gate placements.

Softpick: An Attention-Operator Comparison

The raw token-space spectral identity

CYraw=1dAVV⊤A⊤(37)C_Y^{\mathrm{raw}} = \frac{1}{d} A V V^{\top} A^{\top} \tag*{(37)}

applies to any attention operator AA, including Softpick [23]. The same token-space spectral diagnostics can therefore characterize normalization-based sink mitigations even though their mechanisms need not match post-SDPA gating. In contrast, the deterministic mode contraction result in Theorem 3.1 concerns an already-formed attention output acted on by a downstream gate. Here we separate the effect of changing the normalization operator at fixed learned scores from that of training a model using the alternative operator.

Protocols and metrics. In a frozen-operator swap, we hold Qwen-1B baseline scores S=QK⊤/dS = QK^{\top}/\sqrt{d} and values VV fixed, changing only Softmax to the Softpick operator. In a separate trained-model comparison, we analyze released Vanilla and Softpick-340M checkpoints.4 Sink-bearing layers are selected independently of spectral outcomes using only the Vanilla raw-score geometry: the first position must be the raw-logit argmax for more than 50% of queries. This yields layers 8–28 for Qwen-1B and 8–24 for the matched 340M models.

Since Softpick need not produce attention rows summing to 1, we report both absolute top-kk column mass and a normalized fraction of routed mass. With

Mj=∑t≥jAtj,Jtopk=TopK⁡j(Mj),(38)M_j = \sum_{t \ge j} A_{tj}, \qquad J_{\mathrm{top}k} = \operatorname{TopK}_j(M_j), \tag*{(38)}
mtopk=1kT∑j∈JtopkMj,Rtopk=∑j∈JtopkMj∑jMj.(39)m_{\mathrm{top}k} = \frac{1}{kT} \sum_{j \in J_{\mathrm{top}k}} M_j, \qquad R_{\mathrm{top}k} = \frac{\sum_{j \in J_{\mathrm{top}k}} M_j}{\sum_j M_j}. \tag*{(39)}

Both statistics use the same top-kk columns per head (k=8k = 8), ranked by the column mass MjM_j. This is the quantity used for the sink subspace in Section 3.2 up to the constant factor 1/T1/T, but with a fixed kk instead of the adaptive threshold. Ranking by MjM_j avoids selecting late positions whose few valid queries make availability-normalized mass unreliable. The leading-outlier ratios below use the feature-standardized covariance and MP edge λ+\lambda_+ employed in the main spectral analysis.

Operator Swap (Qwen-1B, layers 8–28)Trained models (340M, layers 8–24)
MetricSoftmaxSoftpickEffectMetricVanillaSoftpickEffect
mtopkm_{\mathrm{topk}}0.07670.07541.7%mtopkm_{\mathrm{topk}}0.07860.003224.3×
λ1/λ+\lambda_1/\lambda_+19.8619.690.9%λ1/λ+\lambda_1/\lambda_+15.3913.741.12×
––––RtopkR_{\mathrm{topk}}0.6290.1993.2×

Table 14. Comparison of Softmax and Softpick under frozen-operator and trained-model settings. The frozen control uses Qwen-1B without retraining, while the trained comparison uses released Vanilla/Softpick-340M checkpoints. Effects indicate percentage decreases (frozen) or Vanilla-to-Softpick ratios (trained).

Results. In the frozen swap, all reported diagnostics change by less than 2%. The first token is already the largest raw score for 79–94% of queries in the selected Qwen layers, so replacing Softmax after training largely preserves the same dominant source. In contrast, the Softpick-trained model selects the first token as raw-logit argmax for fewer than 1% of queries and exhibits a negative first-token logit margin. Its top-kk mass and routed-mass concentration decrease by 24.3× and 3.2×, respectively, relative to the trained Vanilla model. The leading-outlier ratio changes more modestly, from 15.39 to 13.74. Thus, Softpick substantially reduces sink concentration, although structured spectral modes in the representation largely remain. The strong trained effect is primarily associated with the score geometry learned under Softpick, rather than a large post-hoc effect of swapping normalizers on frozen Softmax-trained scores. These observations distinguish upstream operator-mediated prevention of sink formation from downstream contraction of an already-formed sink contribution. The algebraic covariance framework also accommodates other attention operators, including Elastic-Softmax [24], but the empirical comparison here is limited to Softpick.

Gradient-Valve Interpretation of Value-State Gated Attention

The local value-gradient interpretation can also be applied to Value-State Gated Attention (VGA) [6], while distinguishing source-conditioned and query-conditioned gates. Using column-vector notation, the local value Jacobian of a post-SDPA gate for source token ss and query token tt is

Jt,spost=∂y~t∂vs=atsΓt,∇vsL=∑t≥satsΓtTδt,δt:=∇y~tL.(40)J^{\mathrm{post}}_{t,s} = \frac{\partial\widetilde{y}_t}{\partial v_s} = a_{ts}\Gamma_t,\qquad\nabla_{v_s}\mathcal{L} = \sum_{t\ge s} a_{ts}\Gamma_t^{\mathsf{T}}\delta_t,\qquad\delta_t := \nabla_{\widetilde{y}_t}\mathcal{L}. \tag*{(40)}

Here Γt\Gamma_t is the diagonal, query-dependent sigmoid gate. Since ∥Γt∥op≤1\lVert\Gamma_t\rVert_{\mathrm{op}} \le1, each individual value-path contribution is directly contracted at fixed upstream gradient.

For a simplified scalar VGA gate, let

ϕ(vs)=gsvs,gs=sigmoid⁡(wgTvs).(41)\phi(v_s) = g_s v_s,\qquad g_s = \operatorname{sigmoid}(w_g^{\mathsf{T}}v_s). \tag*{(41)}

Because the gate depends on its own value state, ∇vsgs=gs(1−gs)wg\nabla_{v_s}g_s = g_s(1-g_s)w_g, so its Jacobian is

Jϕ(vs)=∂ϕ(vs)∂vs=gsI+gs(1−gs)vswgT.(42)J_{\phi}(v_s) = \frac{\partial\phi(v_s)}{\partial v_s} = g_s I + g_s(1-g_s)v_s w_g^{\mathsf{T}}. \tag*{(42)}

Consequently,

Jt,sVGA=atsJϕ(vs),∇vsL=∑t≥satsJϕ(vs)Tδt.(43)J^{\mathrm{VGA}}_{t,s} = a_{ts}J_{\phi}(v_s),\qquad\nabla_{v_s}\mathcal{L} = \sum_{t\ge s} a_{ts}J_{\phi}(v_s)^{\mathsf{T}}\delta_t. \tag*{(43)}

The first term, gsIg_s I, is the gated content path; the second, rank-one term, gs(1−gs)vswgTg_s(1-g_s)v_s w_g^{\mathsf{T}}, is a self-regulatory path because the source value also controls its gate. Both Jacobians therefore share the local form

∇vsL=∑t≥satsBt,sTδt,Bt,s={Γt,post-SDPA gating,Jϕ(vs),VGA.(44)\nabla_{v_s}\mathcal{L} = \sum_{t\ge s} a_{ts}B_{t,s}^{\mathsf{T}}\delta_t,\qquad B_{t,s} = \begin{cases} \Gamma_t, & \text{post-SDPA gating},\\ J_{\phi}(v_s), & \text{VGA}. \end{cases} \tag*{(44)}

High attention-column mass does not necessarily imply proportionally high value-gradient mass when these local Jacobians suppress the corresponding contributions.

The conditioning and guarantees differ. Post-SDPA gating acts separately at each receiving query after aggregation and has the unconditional local operator-norm bound ∥Γt∥op≤1\lVert\Gamma_t\rVert_{\mathrm{op}} \le1 for its value path. VGA gates each source value before aggregation and includes an additional derivative term. In general,

∥Jϕ(vs)∥op≤gs+gs(1−gs)∥vs∥2∥wg∥2,(45)\lVert J_{\phi}(v_s)\rVert_{\mathrm{op}} \le g_s + g_s(1-g_s)\lVert v_s\rVert_2\lVert w_g\rVert_2, \tag*{(45)}

so gs<1g_s < 1 alone does not guarantee operator-norm contraction at intermediate gate values. In the closed-gate limit gs→0g_s \to0, with the remaining factors bounded,

gsI→0,gs(1−gs)vswgT→0,Jϕ(vs)→0.(46)g_s I \to0,\qquad g_s(1-g_s)v_s w_g^{\mathsf{T}} \to0,\qquad J_{\phi}(v_s) \to0. \tag*{(46)}

Thus, complete severing is a limiting case of local Jacobian suppression, not a requirement for substantial sink-gradient attenuation. Our fixed-upstream-gradient counterfactual already shows considerable attenuation under ordinary sigmoid post-SDPA gates. This comparison is mechanistic; it is not an experimental claim that VGA and post-SDPA gating have identical training dynamics.

Mechanistic Comparison of Gate Placements

The five gate placements studied in gated-attention architectures intervene at different points along the sink pathway. For one head, let yt=∑jatjvjy_t = \sum_j a_{tj}v_j; multi-head outputs are mixed as ot=∑hWO(h)yt(h)o_t = \sum_h W_O^{(h)}y_t^{(h)}. Denote multiplicative gates at the query, key, value, attention-output, and mixed-output positions by ΓQ\Gamma^Q, ΓK\Gamma^K, ΓV\Gamma^V, ΓY\Gamma^Y, and ΓO\Gamma^O:

G4 (query):qt↦ΓtQqt,G3 (key):kj↦ΓjKkj,G2 (value):vj↦ΓjVvj,G1 (post-SDPA):yt↦ΓtYyt,G5 (post-output):ot↦ΓtOot.(47)\begin{aligned} \mathrm{G4\ (query)}:\quad&q_t \mapsto\Gamma_t^Q q_t,\\ \mathrm{G3\ (key)}:\quad&k_j \mapsto\Gamma_j^K k_j,\\ \mathrm{G2\ (value)}:\quad&v_j \mapsto\Gamma_j^V v_j,\\ \mathrm{G1\ (post\text{-}SDPA)}:\quad&y_t \mapsto\Gamma_t^Y y_t,\\ \mathrm{G5\ (post\text{-}output)}:\quad&o_t \mapsto\Gamma_t^O o_t. \tag*{(47)} \end{aligned}

For sink source ss, the attention column creates a rank-one forward contribution Ys=A:,svsTY_s = A_{:,s}v_s^{\mathsf{T}}, while the same attention weights accumulate value-path gradients across receiving queries.

Query/key gating (G4/G3) intervenes before the attention matrix is formed; the coupled changes to logits and normalization mean that attenuating queries or keys does not guarantee monotone attenuation of a particular sink column. Value gating (G2) directly transforms the source contribution

YsG2=A:,s(ΓsVvs)T,(48)Y_s^{\mathrm{G2}} = A_{:,s}(\Gamma_s^V v_s)^{\mathsf{T}}, \tag*{(48)}

which, for a fixed attention matrix, retains the same token-space column direction A:,sA_{:,s}. It also uses a single source-side gate for all queries receiving that value.

Post-SDPA gating (G1) instead transforms the sink contribution at each receiving query:

yt,sG1=atsΓtYvs,∇vsL=∑tats(ΓtY)Tδt.(49)y_{t,s}^{\mathrm{G1}} = a_{ts}\Gamma_t^Y v_s,\qquad\nabla_{v_s}\mathcal{L} = \sum_t a_{ts}(\Gamma_t^Y)^{\mathsf{T}}\delta_t. \tag*{(49)}

It can therefore suppress the same source differently across receiving queries, after the sink contribution is formed but before head mixing. The contraction theorem directly describes its forward attenuation, and its local value Jacobian attenuates query-to-source contributions before they accumulate at a sink value. G5 gates only after the head-specific outputs have been mixed by WOW_O; contraction in that mixed output space does not guarantee contraction of each individual sink-carrying head or of the corresponding projected value path.

Limitations and Future Work

Our analysis is intended as a mechanism-level account rather than a complete theory of gated training dynamics. The BBP interpretation relies on an idealized isotropic reference model; learned feature-wise gates also reshape the noise component, so the resulting covariance need not be a standard spiked Wishart model. Similarly, our gradient result is local: it shows that gating contracts value-path gradients for fixed attention weights and upstream gradients, but does not imply that all gradients in a trained gated model are uniformly smaller. Empirically, gating reduces sink-subspace alignment and dominant spectral outliers, but not all spectral suppression is explained by sink-induced modes. Learned gates may also affect non-sink structured directions, head-specific outliers, and anisotropic noise. Thus, our findings provide a mechanism-level account within the evaluated settings, while broader generalization remains to be established.

Broader Impact

This work contributes to the mechanistic understanding of gated attention and attention-sink mitigation in Transformer language models. By connecting sink-induced spectral structure with value-gradient attenuation, our analysis may help develop more stable, interpretable, and efficient architectures, particularly for large-scale and long-context training. The societal impact of this work is indirect. We do not introduce a new deployed system or task-specific capability, but architectural insights that improve training stability and scalability may contribute to stronger foundation models. Such improvements can support beneficial applications, but may also amplify risks associated with misuse or unsafe deployment. These risks should be addressed through appropriate evaluation, safety, and governance practices alongside architectural research.

References

  1. [1]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03762
  2. [2]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, volume 2024, pages 21875–21895, 2024.arxiv.org/abs/2309.17453
  3. [3]Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2410.10781
  4. [4]Shangwen Sun, Alfredo Canziani, Yann LeCun, and Jiachen Zhu. The spike, the sparse and the sink: Anatomy of massive activations and attention sinks. arXiv preprint arXiv:2603.05498, 2026.arxiv.org/abs/2603.05498
  5. [5]Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. In First Conference on Language Modeling, 2024.arxiv.org/abs/2402.17762
  6. [6]Rui Bu, Haofeng Zhong, Wenzheng Chen, and Yangyan Li. Value-state gated attention for mitigating extreme-token phenomena in transformers. arXiv preprint arXiv:2510.09017, 2025.arxiv.org/abs/2510.09017
  7. [7]Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization. In Proceedings of the computer vision and pattern recognition conference, pages 14901–14911, 2025.arxiv.org/abs/2503.10622
  8. [8]Hoyoon Byun, Youngjun Choi, Taero Kim, Sungrae Park, and Kyungwoo Song. Bounded hyperbolic tangent: A stable and efficient alternative to pre-layer normalization in large language models. arXiv preprint arXiv:2601.09719, 2025.arxiv.org/abs/2601.09719
  9. [9]Xingyue Huang, Xueying Ding, Mingxuan Ju, Yozen Liu, Neil Shah, and Tong Zhao. Threshold differential attention for sink-free, ultra-sparse, and non-dispersive language modeling. arXiv preprint arXiv:2601.12145, 2026.arxiv.org/abs/2601.12145
  10. [10]Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2505.06708
  11. [11]Qwen Team. Qwen3-next: Towards ultimate training & inference efficiency. https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd, September 2025. Blog post. Accessed: 2026-11-06.
  12. [12]Yihong Chen and Quanming Yao. Attention sinks induce gradient sinks. arXiv preprint arXiv:2603.17771, 2026.arxiv.org/abs/2603.17771
  13. [13]Vladimir A Marčenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1(4):457–483, 1967.DOI
  14. [14]Iain M Johnstone. On the distribution of the largest eigenvalue in principal components analysis. The Annals of statistics, 29(2):295–327, 2001.DOI
  15. [15]Debashis Paul. Asymptotics of sample eigenstructure for a large dimensional spiked covariance model. Statistica Sinica, pages 1617–1642, 2007.
  16. [16]Iain M Johnstone and Arthur Yu Lu. On consistency and sparsity for principal components analysis in high dimensions. Journal of the American Statistical Association, 104(486):682–693, 2009.pubmed.ncbi.nlm.nih.gov/20617121
  17. [17]Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. The Annals of Probability, 33(5):1643–1697, 2005.DOI
  18. [18]Florent Benaych-Georges and Raj Rao Nadakuditi. The singular values and vectors of low rank perturbations of large rectangular random matrices. Journal of Multivariate Analysis, 111:120–135, 2012.
  19. [19]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.arxiv.org/abs/1609.07843
  20. [20]Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Af Kham, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International conference on machine learning, pages 2397–2430. PMLR, 2023.arxiv.org/abs/2304.01373
  21. [21]Runyu Peng, Ruixiao Li, Mingshu Chen, Yunhua Zhou, Qipeng Guo, and Xipeng Qiu. How attention sinks emerge in large language models: An interpretability perspective. arXiv preprint arXiv:2603.06591, 2026.
  22. [22]Mengxia Yu, De Wang, Qi Shan, Colorado J Reed, and Alvin Wan. The super weight in large language models. arXiv preprint arXiv:2411.07919, 2024.
  23. [23]Zayd Muhammad Kawakibi Zuhri, Erland Hilman Fuadi, and Alham Fikri Aji. Softpick: No attention sink, no massive activations with rectified softmax. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Findings of the Association for Computational Linguistics: ACL 2026, pages 18179–18213, San Diego, California, United States, July 2026. Association for Computational Linguistics.arxiv.org/abs/2504.20966
  24. [24]Zichuan Fu, Wentao Song, Guojing Li, Yejing Wang, Xian Wu, Yimin Deng, Hanyu Yan, Yefeng Zheng, and Xiangyu Zhao. Attention needs to focus: A unified perspective on attention allocation. arXiv preprint arXiv:2601.00919, 2026.arxiv.org/abs/2601.00919
  25. [25]Jason Ramapuram, Federico Danieli, Eeshan Gunesh Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, et al. Theory, analysis, and best practices for sigmoid self-attention. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2409.04431
  26. [26]Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1–73, 2021.
  27. [27]Cosme Louart, Zhenyu Liao, and Romain Couillet. A random matrix approach to neural networks. The Annals of Applied Probability, 28(2):1190–1248, 2018.arxiv.org/abs/1702.05419
  28. [28]Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020.DOI
  29. [29]Enrique Queipo de Llano, Alvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael M. Bronstein, Yann LeCun, and Ravid Shwartz-Ziv. Attention sinks and compression valleys in LLMs are two sides of the same coin. In The Fourteenth International Conference on Learning Representations, 2026.
  30. [30]Yehjin Shin, Seojin Kim, and Noseong Park. Graph signal processing meets mamba2: Adaptive filter bank via delta modulation. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2603.22333
  31. [31]Jeongwhan Choi, Hyowon Wi, Jayoung Kim, Yehjin Shin, Kookjin Lee, Nathaniel Trask, and Noseong Park. Graph convolutions enrich the self-attention in transformers! arXiv preprint arXiv:2312.04234, 2023.arxiv.org/abs/2312.04234
  32. [32]Hyowon Wi, Jeongwhan Choi, and Noseong Park. Learning advanced self-attention for linear transformers in the singular value domain. arXiv preprint arXiv:2505.08516, 2025.arxiv.org/abs/2505.08516
  33. [33]Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663, 2024.arxiv.org/abs/2501.00663
  34. [34]Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024.arxiv.org/abs/2406.17557
  35. [35]Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534, Berlin, Germany, August 2016. Association for Computational Linguistics.arxiv.org/abs/1606.06031
  36. [36]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.arxiv.org/abs/1911.11641
  37. [37]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics.arxiv.org/abs/1905.07830
  38. [38]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.arxiv.org/abs/1907.10641
  39. [39]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
  40. [40]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, 2019.arxiv.org/abs/1905.10044
  41. [41]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, 2018.

Paper details

vlnc:W_kjzx8qqv
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents