WorksComputer Science
Attention Sinks as Spectral Spikes: A Mechanism Analysis of Gated Attention
Abstract
Transformer attention exhibits attention sinks, where a small subset of source tokens absorbs disproportionate attention mass across layers. Recent work shows that post-SDPA output gating strongly reduces attention-sink behavior, while separate studies link attention sinks to gradient sinks and massive activations through backward training dynamics. Yet how the same gate affects forward sink structure, backward gradient concentration, and training-time representation signatures remains unclear. We address this gap by formulating attention sinks as sink-induced low-rank modes in token-space covariance, with directions aligned to high-mass attention columns. Under this view, post-SDPA gating acts as a mode-dependent contraction: it attenuates sink-induced spectral energy in the forward pass and locally reduces value-path gradients routed through sink columns in the backward pass. Empirically, random-matrix diagnostics show that gating reduces outlier mass, weakens sink-subspace alignment, and increases effective rank, while backward analyses show active-gate attenuation of first-token value gradients. Training-time diagnostics further show that gating suppresses the co-emergence of attention concentration, value-gradient amplification, and representation dominance.
Introduction
Transformers have become the backbone of modern large language models, with self-attention enabling flexible token interactions across long contexts [1]. Yet recent studies have shown that this same mechanism can give rise to attention sinks: a small number of tokens that absorb disproportionate attention mass across layers, often independently of semantic relevance [2, 3]. Such sink behavior is not merely an artifact of attention visualization: it has been linked to distorted representation geometry, massive activations, and training instability [4, 5]. Understanding and mitigating attention sinks has therefore become an important step toward more stable Transformer architectures [6, 7, 8, 9].
Recently, a strikingly simple mitigation was proposed: applying a sigmoid gate after the scaled dot-product attention (SDPA) output [10]. In its most effective form, this gate is head-specific and elementwise, and it nearly eliminates attention sinks, reducing first-token attention from 46.7% to 4.8%, while substantially improving training stability. This mechanism has since been adopted in production-scale architectures such as Qwen3-Next [11], underscoring the practical relevance of understanding why it works. The authors attribute this effect to query-dependent sparsity, arguing that sparse gating filters out context irrelevant to the current query. However, this explanation remains largely phenomenological. It leaves open a central question: how can a standard attention-output gate selectively suppress sink-aligned structure, rather than merely shrink the attention output uniformly?
This question is especially important because post-SDPA gating is not merely a post-hoc correction to a trained attention map. It is used throughout training, shaping both the forward representations and the gradients by which they are learned. Its effect therefore appears at multiple coupled levels: it changes the forward representation geometry associated with sink tokens, attenuates sink-directed value-path gradients in the backward dynamics, and suppresses training-time sink signatures. Recent work [12] explains one part of this picture, showing that attention sinks can induce value-path gradient sinks under causal masking and that massive activations may arise as an RMSNorm-mediated response to this localized gradient pressure. However, this backward account has not yet been connected to the standard post-SDPA gate.
We bridge this gap by recasting an attention sink as a sink-induced low-rank mode in token-space covariance. Under this view, a sink is not merely a token that receives large attention mass; rather, a high-mass attention column can induce a structured token-space direction in the post-attention representation. Post-SDPA gating then has a precise structural interpretation: it induces an effective contraction of this sink-induced mode. Under an idealized isotropic reference model, this contraction predicts that a supercritical spike can be pushed below the Baik–Ben Arous–Péché (BBP) threshold and merge back into the Marchenko–Pastur bulk. In the backward pass, the gate locally attenuates value-path gradients routed through sink columns, replacing original sink-column statistics with gated effective counterparts. Thus, post-SDPA gating is not simply a generic damping operation; it attenuates sink-induced structure in both forward spectral geometry and backward gradient flow.
Taken together, we argue that post-SDPA gating goes beyond generic sparsity or uniform attenuation: it weakens sink-induced token-space structure in the forward pass and locally reduces value-path gradient flow through high-mass sink columns. We make the following contributions:
Spectral characterization of attention sinks. We formulate attention sinks as sink-induced low-rank modes in token-space covariance, with directions aligned to high-mass attention columns, connecting attention concentration to spectral outliers and providing a random-matrix perspective on sink emergence.
Dual contraction mechanism of post-SDPA gating. We show that post-SDPA gating suppresses sink-induced structure in two coupled ways: it attenuates forward spectral energy and it locally contracts value-path gradient aggregation through sink columns.
Training-time sink signatures. We show that models without gating develop first-token attention concentration and first-token representation dominance during training, whereas post-SDPA gated models substantially attenuate both signatures.
From Attention Sinks to Spectral Spikes
This section recasts attention sinks as spectral objects in the post-attention representation. We first connect high-mass attention columns to token-space covariance directions, then introduce a sink-induced spiked model and the post-SDPA gate as a row-wise diagonal contraction.
Attention Sinks as Column Imbalance
Transformer attention computes token interactions through non-negative, row-stochastic attention weights. For a single attention head, let denote the query, key, and value matrices. Under a causal mask, the attention weights and corresponding output are defined as follows:
Thus and . Due to the non-negativity and row-wise normalization of the softmax, attention mass must be assigned to some previous token even when no semantically relevant context is needed. As a result, certain source tokens can accumulate attention mass across many query positions and layers. This phenomenon is known as an attention sink.
Empirically, early-position tokens, especially the first token, often act as sinks, receiving disproportionately large attention weights independent of their semantic relevance. Formally, we measure the incoming attention mass of a source token by its causal column mass, the average attention it receives over all query positions:
A token is referred to as an attention sink when is significantly larger than the typical non-sink scale. Large values of indicate a column-wise imbalance in the attention matrix, reflecting persistent aggregation of attention toward a small number of source positions. While identifies where attention mass concentrates in the map , our goal is to understand how this column imbalance appears in the representation .
From Sink Columns to Token-Space Covariance
The standard definition of an attention sink is based on column-wise concentration in the attention matrix. To study how this phenomenon affects representations, we shift attention from the map to the token-space covariance of the post-attention output:
Because is a token-space covariance, its leading eigenvectors describe token-position modes of concentrated representation energy. Under an isotropic value approximation, , and therefore
This column-wise decomposition identifies the directions through which attention sinks can enter the token-space covariance. When a source token has high incoming attention mass, its attention column contributes the rank-one term , whose direction is the normalized column . This observation provides the bridge from attention-map concentration to representation geometry: a persistent sink column can appear as a dominant token-space mode in .
Spectral Spikes and Sink-Induced Representations
The covariance view above reduces sink structure to the question of whether contains a dominant token-space direction aligned with high-mass attention columns. Random matrix theory provides a baseline for separating such structured directions from high-dimensional noise. When the representation is purely isotropic noise, we write , where has i.i.d. standard Gaussian entries. Then the spectrum of converges, in the high-dimensional limit with , to the Marchenko–Pastur (MP) law [13], whose support is given by
Eigenvalues within this bulk are consistent with isotropic noise, whereas eigenvalues separated above indicate excess structured variance. A standard way to represent such structure is a rank-one spiked model [14, 15, 16],
where and are unit vectors, and is the spike amplitude. Here specifies a token-space direction and specifies its feature-space direction. Under the Baik–Ben Arous–Péché (BBP) transition [17, 18], the rank-one component produces a separated spectral outlier when . Thus, the MP edge serves as a noise threshold, and a spectral outlier signals a structured low-rank mode in the token representations.
We now specialize this generic spiked model to attention sinks. From the column-covariance relation in Section 2.2, high-mass sink columns induce token-space directions . We therefore identify sink-induced spikes by the alignment of the spike direction with the subspace spanned by these directions.
Definition 2.1 (Sink-induced spiked representation model). Consider a rank-one spiked post-attention representation
where and are unit vectors, is the spike amplitude, and is isotropic noise. For sink tokens , let , and let be the orthogonal projector onto this subspace. We call the spike sink-induced if
For a single first-token sink, this becomes .
This model connects attention-map concentration with representation geometry: a persistent sink column can manifest as a low-rank token-space mode in . When this mode is supercritical, it appears as a spectral outlier above the MP bulk.
Post-SDPA Gating
A recent line of work [10] mitigates attention sinks by applying a query-dependent multiplicative gate to the output of scaled dot-product attention. We describe the post-SDPA gated attention variant for a single attention head and write its head dimension as . The attention output is transformed as
where denotes the hidden state used to compute the gate, is a learned gating projection, is the sigmoid function, denotes element-wise multiplication, and collects the gate values row-wise. Equivalently, with , we have
so the gate acts on each token row as a diagonal contraction. The post-SDPA gate is therefore not a scalar shrinkage of , but a token- and feature-dependent diagonal contraction. Its effect on a sink-induced spike depends on how the spike’s token positions and feature direction interact with the row-wise operators .
Forward Pass: Spectral Suppression
Theory: Sink-mode Contraction under Post-SDPA Gating
We analyze the effect of post-SDPA gating on a sink-induced spectral mode. Let denote a sink-induced signal component of the SDPA output, where is aligned with the sink-column subspace in Definition 2.1 and is the corresponding feature direction. The analysis is deterministic and makes no distributional assumption on the learned gate.
Using the row-wise gate operators defined in Section 2.4, define the effective gate on the feature direction by
Since and , we have .
Theorem 3.1 (Sink-induced mode contraction under post-SDPA gating). For the sink-induced signal , the gated signal satisfies
where
Moreover, the gated signal covariance satisfies
Theorem 3.1 shows that quantifies how much of the sink-induced mode is preserved by the gate. Thus, gating suppresses a sink-induced spectral spike when it contracts the query positions and feature direction participating in the corresponding attention-column mode.
To connect this contraction to spectral outlier behavior, we use the classical spiked-covariance model as an idealized reference. The deterministic contraction above applies to learned gates, but a general learned gate also transforms the noise component and therefore does not preserve the exact assumptions of a standard spiked Wishart model. We therefore isolate a simplified isotropic thinning setting in which the same contraction mechanism can be related exactly to the BBP phase transition.
Proposition 3.2 (BBP suppression under isotropic thinning). Assume the spiked representation model in Eq. (6). Let be a row-selection matrix independent of , where , and define , . If , then
where has i.i.d. standard Gaussian entries. Hence the thinned covariance
is a rank-one spiked Wishart model with effective spike strength . Consequently, the sink-induced spectral outlier disappears when
When , a spectral outlier persists, but its effective spike strength is reduced to .
Proposition 3.2 should be read as an idealized spectral reference rather than an exact theorem for learned gates. For a general post-SDPA gate, the sink-induced signal component is contracted as in Theorem 3.1, but the noise component is also modulated by the row-wise feature gates . The resulting covariance is generally anisotropic and need not be a standard spiked Wishart matrix. Nevertheless, the BBP reference captures the mechanism-level prediction: when gating reduces the effective strength of a sink-induced mode, the associated spectral outlier should weaken, move closer to the MP bulk, or disappear. We empirically demonstrate that this happens in the learned post-SDPA gate setting. Proofs of Theorem 3.1 and Proposition 3.2 can be found in Appendix § 9.
Empirical: Spectral Suppression and Sink-Subspace Alignment
We empirically test the forward-pass prediction of Theorem 3.1: if post-SDPA gating contracts structured token-space modes, then the post-gate representation should exhibit weaker spectral concentration. In addition, following Definition 2.1, we ask whether the suppressed spectral structure is aligned with sink-induced attention-column subspaces.
Analysis setting. We evaluate the post-SDPA gated Qwen-1B model following previous work [10] and record both the pre-gate output and the post-gate output at each layer. All spectral statistics are computed layer-wise and averaged across sixteen samples of sequence length 512 from WikiText dataset [19].
For each layer, we compute the token-space covariance , where is obtained by centering and normalizing each feature dimension across tokens. This removes trivial feature-scale effects and focuses the diagnostic on token-space correlation structure. We use the theoretical MP edge of the unit-variance isotropic null as a fixed reference and summarize each spectrum by
where . These metrics measure out-of-bulk spectral mass, leading-mode dominance, and the spread of variance across token-space directions. To connect spectral suppression to attention sinks, we construct a sink-column subspace from the attention matrix. For each layer and sample, let denote the source positions whose head-averaged column mass is at least half its maximum (at most ). For each , define the normalized column mode and let be the orthogonal projector onto . If denotes the leading eigenvector of , we measure sink-subspace alignment by
The random baseline for a -dimensional subspace is approximately where .
Gating suppresses dominant token-space spectral structure. Figure 1(a-Figure 1c) shows that post-SDPA gating consistently reduces spectral concentration. The gated output has lower outlier mass and a smaller leading eigenvalue relative to the MP edge, while its effective rank increases. Thus, after gating, the representation is less dominated by a small number of token-space directions. Since the covariance is computed after feature-wise standardization, which removes any per-feature scaling shared across tokens, this reduction is not a consequence of uniform feature-scale shrinkage; rather, the token-dependent variation of the learned gate changes the token-space correlation structure. Robustness checks using non-standardized covariance and scalar gating are reported in Appendix D.1.

Figure 1. Post-SDPA gating suppresses token-space spectral structure. (a) Outlier mass above the MP edge decreases. (b) The leading eigenvalue moves closer to the MP edge. (c) Effective rank increases. (d) Sink-subspace alignment decreases after gating; the dashed line shows the random-subspace baseline (layer-wise mean of ).
Gating reduces sink-subspace alignment. Figure 1(d) further shows that the leading pre-gate spectral mode is often aligned with the sink-column subspace, substantially above the random baseline. After gating, this alignment is strongly reduced across layers. This supports the sink-induced spike formulation in Definition 2.1: high-mass attention columns are not only attention-map artifacts, but can correspond to structured low-rank directions in the post-attention representation. Post-SDPA gating attenuates these sink-aligned directions together with the overall spectral outlier mass. Additional sink-subspace counterfactuals, which further probe the role of sink-induced modes, are reported in Appendix D.
Gating weakens sink-aligned covariance structure. We further examine how post-SDPA gating affects spectral directions with different degrees of sink alignment. As shown in Table 1, highly sink-aligned directions retain substantially less covariance energy than weakly aligned directions. This gap largely disappears when the elementwise post-SDPA gate is replaced by its feature-averaged scalar counterpart, while raw-energy retention remains similar across the two groups. These results indicate that post-SDPA gating preferentially weakens sink-aligned covariance structure beyond uniform magnitude reduction, consistent with the spectral suppression observed in Figure 1. Further details are provided in Appendix D.2.
| Metric | High align. | Low align. |
| Raw energy () | 0.047 | 0.044 |
| Std. covariance () | 0.740 | 0.945 |
| Scalar gate () | 0.957 | 0.990 |
Table 1. Sink-aligned mode retention. Retention of highly and weakly sink-aligned directions in Qwen-1B. The scalar control uses feature-averaged gates.
Backward Pass: Gradient Valve
Having shown that gating attenuates sink-induced structure in the forward geometry, we next analyze its effect on backward dynamics. Building on the gradient-sink view [12], we show that under fixed upstream signals, post-SDPA gating replaces sink-column gradient statistics with gated effective counterparts, acting as an implicit value-path gradient valve.
Theory: A Gated Extension of the Gradient-Sink Bound
Consider a single causal attention head, , where is the attention weight from query position to source position , and is the value vector. Since post-SDPA gate is diagonal and contractive, define . For a candidate sink token , define the sink column statistics without gate and corresponding gated effective statistics as
By construction,
Theorem 4.1 (Gated value-path gradient bound). Let be the upstream gradient at the post-SDPA gated attention output. Then the gradient with respect to the source value is
Consequently,
In particular, if for all , then
The coherent bound above controls the worst-case aligned contribution of future-query gradients. A complementary second-moment statement applies when the upstream gradients behave incoherently.
Corollary 4.2 (Second-moment attenuation). Conditioned on the attention weights and gates, assume that the upstream gradients satisfy
Then
Theorem 4.1 and Corollary 4.2 show that post-SDPA gating attenuates value-path gradient aggregation under both coherent and incoherent regimes. Importantly, this contraction does not require changing the attention matrix itself. Even if the same attention weights are fixed, each backpropagated contribution from query to source value is filtered by the gate. Thus, the backward strength of a sink column is governed by and , not only by and . Proofs for Theorem 4.1 and Corollary 4.2 can be found in Appendix § 10.
Empirical: Gradient Sink under Post-SDPA Gating
Practical proxies. Theorem 4.1 is a local Jacobian statement: conditioned on fixed attention weights and upstream gradients, post-SDPA gating contracts value-path contributions routed through a sink column. We evaluate this prediction using two complementary diagnostics. First, we compare Qwen-1B models with and without post-SDPA gating under their actual forward-backward passes. This measures the outcome of training with post-SDPA gates. Second, within post-SDPA gated model, we isolate the local value-path effect of the gate by keeping the attention weights and upstream gradients fixed, and recomputing the first-token value-gradient contribution with and without the gate. This fixed-upstream-gradient counterfactual directly tests the valve effect predicted by Theorem 4.1.
We focus on the first-token sink, which is the primary sink position in this setting. We measure value-gradient amplification at source position by
To isolate the local value-path effect of the active gate, we cache from the actual gated backward pass and compute

Figure 2. Post-SDPA gating attenuates first-token value-path gradients. (a) Post-SDPA gate suppresses first-token value-gradient amplification. (b) Local attenuation tracks the RMS effective-mass proxy . (c) Their layer-wise correlation supports the fixed-upstream-gradient valve mechanism.
As a simple gate-mass proxy, we also compute the RMS effective-mass ratio
This RMS quantity is an isotropic-gradient proxy for average feature-wise attenuation.
Results. Figure 2(a) reports the trained-model outcome. Baseline (without post-SDPA gating) develops large first-token value-gradient amplification in mid-to-late layers, whereas the gated model remains close to the non-sink scale. The counterfactual curve in the model with post-SDPA gating (Gated (G 1)) shows that removing the gate increases the first-token gradient even without changing the trained weights, indicating that the active gate provides an additional attenuation mechanism beyond the learned representation itself. Figures 2(b,c) isolate this local valve effect. Holding the attention weights and upstream gradients fixed, we recompute the first-token value-gradient contribution with and without the gate. The resulting attenuation ratio is far below one across layers and closely tracks the RMS effective-mass proxy . The layer-wise scatter strongly correlates these quantities, supporting the Jacobian prediction of Theorem 4.1 that post-SDPA gating attenuates value gradients according to the effective gated mass through the first-token sink column.
Implication for the Attention-Sink–Gradient-Sink–Massive-Activation Pathway
The forward attention sink can create a backward gradient sink because many future query positions route their value-path gradients through the same source token. Previous work [12] suggests that this localized gradient pressure mediates the emergence of massive activations during training. Our analysis shows that post-SDPA gating weakens precisely this value-path link. The gate does not need to remove the attention sink column; it attenuates each future-query contribution to the sink-token value gradient, reducing the effective sink-column mass even with fixed attention weights and upstream gradients. Thus, standard gated attention implements an implicit gradient valve using the same learned gate that suppresses forward spectral structure. This provides the backward counterpart to the spectral suppression mechanism in § 3, connecting the attenuation of sink-induced representation modes with reduced sink-directed gradient flow.
The quantitative diagnostics in Table 2 provide complementary evidence: gating reduces attention concentration, value-gradient amplification, and massive activations in Pythia-410M. These signatures are most pronounced in the middle-to-late layers of the baseline model, while gating substantially suppresses their emergence during training. Qwen-1B further exhibits strong massive-activation suppression, supporting the consistency of this effect across models. Together, these findings are consistent with attenuation along the AS–GS–MA pathway. Further layer-wise and training-time analyses and details of metrics are provided in Appendix § 13.§ 2.
| Metric | Baseline | Gated |
| AS () | 0.084 | 0.013 |
| GS () | 25.40 | 5.67 |
| MA () | 320.0 | 67.4 |
Table 2. AS–GS–MA suppression. Final-checkpoint diagnostics for baseline and post-SDPA gated models on Pythia-410M. Detailed measurements are provided in Appendix § 13.§ 2.
Training-Time Analyses of Sink Signatures
Training-Time Sink Emergence
In this section, we examine whether the sink signatures analyzed in § 3 and § 4 also emerge during training. Using checkpoints of baseline and post-SDPA gated models that we train with the Pythia-410M architecture [20], we track first-token attention concentration and representation dominance across layers. Details and additional results are in Appendices § 11 and § 13.
Diagnostics. For each checkpoint and layer, we track the canonical first-token sink via the causal column-mass diagnostic from § 2.1. Since this corresponds to , we report
The first quantity measures first-token attention concentration, and the second measures whether the corresponding hidden state becomes unusually large relative to other positions.
Results. Figure 3(a,Figure 3b) tracks first-token attention mass across training checkpoints. In the baseline, first-token mass increases sharply in middle-to-late layers, indicating the emergence of a strong first-token attention sink. The post-SDPA gated model suppresses this growth and maintains substantially lower first-token mass throughout training. Figure 3(c) shows the corresponding representation-level signature at the final checkpoint: the baseline model develops pronounced first-token hidden-state dominance in the same layer range, whereas the gated model remains closer to the non-first-token scale. Thus, both attention- and representation-level sink signatures emerge during training and are substantially attenuated by post-SDPA gating. The additional analyses in Appendix § 12.§ 4 show reduced heavy-tail dominance while the bulk spacing ratio remains stable. Layer-wise linear probing in Appendix § 13.§ 3 further shows comparable next-token cross-entropy loss for the baseline and gated models once the probes are trained to convergence, with a similar result after excluding the initial source position.

Figure 3. Training-time sink and gradient signatures. Curves in (a, b, d–f) correspond to training checkpoints. (a, b) First-token attention mass: Baseline develops a middle-to-late-layer sink, whereas the post-SDPA gated model suppresses it. (c) First-token representation dominance at the final checkpoint, which is large in the baseline but attenuated by gating. (d, e) First-token value-gradient amplification : Baseline develops strong amplification in middle-to-late layers, whereas gating suppresses it. (f) Gated effective-mass ratio remains well below one across training.
Training-Time Gradient Amplification
We next examine whether the backward-pass signature analyzed in § 4 also emerges during training. Because many future queries route their value-path gradients through the same high-mass source token, a forward attention sink can induce a backward gradient sink (§ 4.3). Using the same checkpoints, we track the first-token value-gradient amplification in Eq. (25) and, for the gated model, the effective-mass ratio in Eq. (27). As shown in Figure 3(d,Figure 3e), the baseline develops large first-token value-gradient amplification in middle-to-late layers, the same range in which the first-token attention sink emerges, whereas the gated model keeps this amplification substantially lower in these layers. The effective-mass ratio remains well below one across much of training (Figure 3(f)), indicating that the active gate attenuates value-path contributions routed through the sink column, consistent with the local gradient valve in § 4.1. Since depends only on the attention weights and gate values, this attenuation applies even where the gated model still assigns non-negligible attention to the first token. Together with the results above, these findings show that attention concentration, value-gradient amplification, and representation dominance co-emerge in the baseline and are jointly attenuated by post-SDPA gating (additional diagnostics in Appendix § 13).
Related Work
Attention sink phenomenon. Attention sinks describe the tendency of autoregressive language models to route disproportionate attention mass to a small set of tokens, often early positions, regardless of semantic relevance. Prior work shows that this phenomenon is widespread across models and inputs, emerges during pretraining, and is affected by optimization, data distribution, loss design, and attention normalization [3, 21]. Recent studies further clarify its relation to massive activations [5, 22]: sinks and activation spikes often co-occur, but they are geometrically and functionally distinct [4]. Another study [12] complements this view from the backward pass, showing that attention sinks can induce value-path gradient concentration under causal masking.
Gated and normalization-based sink mitigation. Recent work has explored architectural interventions for reducing attention sinks, massive activations, or their co-occurrence. Post-SDPA sigmoid gating improves performance and stability while strongly reducing first-token attention concentration, attributing the effect to nonlinearity and query-dependent sparsity [10]. Alternative approaches modify attention normalization, as in Softpick’s rectified softmax and Elastic-Softmax’s learnable filtering of softmax weights [23, 24], or directly gate value states, as in Value-State Gated Attention (VGA) [6]. Other methods modify normalization or activation dynamics, including DyT [7], or introduce normalization and attention-bias variants that shift, suppress, or decouple sinks and massive activations [4, 12, 25]. Despite these advances, the mechanisms underlying sink suppression by post-SDPA gating remain unclear. We investigate them through spectral and gradient analyses.
Spectral analysis on neural representations. Spectral methods and random matrix theory have been widely used to analyze neural network weights, activations, and learned representations, separating noise-like bulk spectra from structured low-rank components [26, 27, 13, 17]. Related studies of representation collapse also emphasize the role of low-rank structure in deep features [28, 29]. Complementary work uses spectral or graph-signal viewpoints to design and interpret sequence models and representation dynamics [30, 31, 32]. In contrast, we use spectral outliers to characterize sink-aligned structure and its suppression in token-space attention representations.
Conclusion
We presented a spectral and gradient-based analysis of how post-SDPA gating mitigates attention sinks. High-mass attention columns can induce low-rank token-space modes, which learned gates attenuate beyond uniform shrinkage. In the backward pass, the gate locally reduces value-path gradient aggregation through sink columns, a gradient-valve counterpart to the forward spectral contraction. Empirically, gating reduces spectral outliers and sink-subspace alignment, attenuates first-token value-gradient amplification, and suppresses the training-time emergence of first-token attention concentration and representation dominance. Together, these results provide a unified forward–backward account of sink suppression by post-SDPA gating.
Acknowledgments
This research was partly supported by the MSIT (Ministry of Science and ICT), Korea, under the Development and Demonstration of Agentic AI Technology for Autonomous Physics Simulation Based on MCP Standard with Multi-Agent Collaboration (RS-2026-25587486), Global Research Support Program in the Digital Field program (RS-2024-00436680), AI Research Hub Project (RS-2024-00457882), and LG AI STAR Talent Development Program for Leading Large-Scale Generative AI Models in the Physical AI Domain (RS-2025-25442149), supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation). This research was partly supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. RS-2026-25607520, Center for Specialized Orchestration and Training of AI for Frontier Mathematics (SOTA-MATH)). Research for this paper was also conducted under the KICT Strategic Research Program(SRP)(project no. 20260437-001, Development of a Construction Site-Specific Vertical AI Model and Sustainable Platform) funded by the Ministry of Science and ICT. This project is supported by Microsoft Research Asia.
Proofs for Spectral Sink Suppression
Proof of Theorem 3.1
Proof. The -th row of the sink-induced signal component is
After applying the post-SDPA gate, it becomes
Therefore,
By the definitions , , and
we obtain
Since and , we have . Hence
which gives
It remains to prove the covariance-level spectral bound. Define
Since is positive semidefinite,
Moreover,
Therefore,
This proves the claim. □
Proof of Proposition 3.2
Proof. Let be a row-selection matrix that extracts the rows indexed by , and suppose is independent of . Applying to the spiked representation gives
Define
If , let
Then , and hence
where . Since has i.i.d. standard Gaussian entries and selects rows independently of , the matrix also has i.i.d. standard Gaussian entries.
Therefore,
is a standard rank-one spiked Wishart covariance matrix with aspect ratio and effective spike strength . By the classical BBP phase transition, the leading eigenvalue separates from the Marchenko–Pastur bulk only when
Equivalently, if
the spike is subcritical and the associated spectral outlier disappears, with the top eigenvalue converging to
Discussion: post-SDPA Gates and the Scope of the BBP statement
Theorem 3.1 is deterministic and applies to learned post-SDPA gates. It only uses the transformation of the sink-induced signal row
Thus the contraction factor
does not rely on any randomness assumption about the gate or on coordinate localization of . However, for the full representation , the learned gate also transforms the noise component. The -th noise row is mapped from to . Unless the matrices satisfy additional isotropy assumptions, this noise is feature-dependent and token-dependent. Consequently, the gated covariance is not exactly a standard spiked Wishart matrix, and the classical BBP transition cannot be applied as an exact theorem for learned gates.
Proposition 3.2 isolates an idealized setting in which the same contraction mechanism can be connected exactly to BBP. In the row-selection model, the retained noise matrix remains i.i.d. Gaussian, so the thinned covariance is again a standard spiked Wishart model. Learned gates are not random row selectors, but the deterministic contraction result still applies. Therefore, the BBP condition should be read as a mechanism-level reference: when the contracted sink-induced signal becomes weak relative to the noise bulk, the associated spectral outlier is expected to weaken or disappear. This motivates the empirical diagnostics used in § 3.2.
Discussion: Higher-Rank Sink Subspaces and Contraction
Our empirical sink object is the multi-column subspace
where may contain multiple positions, including positions other than the first. The rank-one component analyzed in the main text isolates one elementary sink contribution; it is not an assumption that all empirical sinks are literally rank one. For any combined sink component , an elementwise post-SDPA gate with entries in satisfies
This global inequality does not imply independent scalar contraction of every spectral spike: gating may change the relative mode strengths and rotate the resulting eigenspace. We therefore evaluate the full sink-subspace projector. The contraction applies to the combined higher-rank sink contribution, but it does not imply that a rank- sink subspace decomposes into independently contracted eigenmodes. This distinction motivates the empirical subspace-level tests in Appendix D.8.
Proofs for the Gated Gradient-Sink Bound
Proof of Theorem 4.1
Proof. For each query position , the post-SDPA gated attention output is . Let . By the chain rule,
Since , the Jacobian of with respect to is
Therefore,
Taking norms and viewing as a diagonal linear map gives
If for every , then
Since , we have , and hence .
Proof of Corollary 4.2
Proof. From Theorem 4.1,
Expanding the squared norm and using the cross-position uncorrelatedness assumption, the cross terms vanish:
Since acts as a diagonal linear map,
Therefore,
Finally, implies , and hence .
Local Value-path Attenuation versus Global Training Dynamics
Our gradient results are local statements about the value-path Jacobian. Conditioned on fixed attention weights and upstream gradients , post-SDPA gating filters each contribution from query position to source value . Equivalently, the raw sink-column statistics and are replaced by their gated counterparts and . This does not imply that all gradients in a trained model with post-SDPA gating are uniformly smaller than in a model without post-SDPA gating, since gating also changes the forward computation, upstream gradients, and learned representations. Rather, the structural claim is that for a fixed upstream signal, the value-path map through a sink column is contractive. The deterministic and second-moment bounds cover coherent and incoherent aggregation regimes, respectively, and in both cases
Thus, post-SDPA gating weakens the attention sink to gradient sink link without necessarily removing the attention sink itself: a sink column may still receive attention, but each value-path contribution is filtered before reaching the source value. This provides a local mechanism for reducing gradient pressure in the attention-sink–gradient-sink–massive-activation pathway, without claiming a full theory of global training dynamics.
Experimental Setting
Hardware. All experiments were conducted on a server equipped with two AMD EPYC 9654 96-core CPUs (192 physical cores / 384 threads in total, 1.5–3.7 GHz, 768 MiB L3 cache) and 1 TB of system memory. The server is configured with eight NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation GPUs (96 GB GDDR7 each, driver 580.95.05). All code was implemented in PyTorch 2.9.1 built against CUDA 12.8 (toolkit nvcc 12.8.93) on Python 3.13.5.
Model checkpoints. For the Qwen analyses, we use the public checkpoints released by a previous work [10] in the QwQZh/gated_attention HuggingFace repository.3 Specifically, we analyze 1B_baseline and 1B_gate_elementwise. All models are evaluated in inference mode, with no additional training or parameter updates.
Training setup for training-time diagnostics. For the training-time diagnostics, we train Pythia-410M baseline and post-SDPA gated variants using the HuggingFace training framework, following the language-modeling training protocol of [33]. The two models are trained under matched data, optimization, and checkpointing conditions. The baseline model uses 24 Transformer layers, hidden size 1024, 16 attention heads, and MLP intermediate size 4096. The elementwise gated variant uses the same configuration except for the post-SDPA elementwise gate and a reduced MLP intermediate size of 3584, which compensates for the additional gate parameters. Both models use the EleutherAI/gpt-neox-20b tokenizer and are trained on 15B tokens from FineWeb-Edu [34] with causal language modeling, sequence length 4096, AdamW optimization, and peak learning rate . We save checkpoints throughout training and evaluate without further parameter updates.
To examine robustness beyond the original setting, we also analyze matched baseline and post-SDPA elementwise-gated Pythia-2.8B models; matched Pythia-410M models trained with an 8K context length and evaluated at 4K or 8K; and matched Pythia-410M models trained with document-boundary EOS separators. The plain sequence convention uses add_bos_token = False and add_eos_token = False; the EOS-separated convention uses add_bos_token = False and add_eos_token = True. Protocol-specific evaluation conditions and measurements are provided in Appendices F.1–F.3.
Language modeling and zero-shot commonsense reasoning. Following prior works [33, 30], we evaluate baseline and post-SDPA gated variants across two evaluation categories: WikiText (Wiki.; [19]) perplexity and zero-shot commonsense reasoning tasks. The commonsense tasks include LAMBADA (LMB.; [35]), PIQA [36], HellaSwag (Hella.; [37]), WinoGrande (Wino.; [38]), ARC-easy (ARC-e) and ARC-challenge (ARC-c) [39], BoolQ [40], and OpenbookQA (OBQA.; [41]).
Additional RMT Diagnostics
This appendix provides additional diagnostics supporting the RMT analysis in § 3.2. The main text focuses on spectral suppression and sink-subspace alignment. We report robustness checks, gate statistics, sink-subspace counterfactuals, contraction diagnostics, and bulk-level spectral analyses.
Robustness to Covariance Normalization and Feature-Averaged Gate Control
In the main text, we compute token-space covariance after feature-wise standardization across tokens. This removes trivial feature-scale effects and focuses the diagnostics on token-space correlation structure. To verify that the observed suppression is not an artifact of this normalization, we repeat the analysis using the non-standardized covariance . Since the overall scale of is not fixed, we rescale the MP edge by so that the outlier mass is invariant to global rescaling.
We also compare the learned elementwise gate with a feature-averaged gate control. For each token, we replace the feature-wise gate vector by its feature average , and apply the scalar gate to the SDPA output. This control preserves token-wise gate strength, so it can still suppress particular token positions, but removes feature-direction-specific selectivity.
Figure 4 shows that the post-SDPA gate still suppresses dominant spectral structure without feature-wise standardization. In particular, compared with Pre-Gate, Post-Gate (elem) has lower outlier mass and higher effective rank in the non-standardized covariance across most layers. By contrast, Post-Gate (scalar) does not reproduce the same suppression pattern: its outlier mass and effective rank remain close to those of Pre-Gate, indicating that average token-wise shrinkage alone is not sufficient. Figure 4(c) further shows that the reduction in leading-mode concentration from Pre-Gate to Post-Gate (elem) is at least as strong without standardization as with it, so feature-wise standardization does not create or inflate the observed suppression. These results indicate that the spectral suppression is not merely a covariance-normalization artifact or a consequence of average token-wise shrinkage. Rather, the feature-selective action of the learned post-SDPA gate is important for suppressing dominant token-space spectral structure. The following sections further examine where this selective action appears in gate statistics and how it relates to sink-induced subspaces.
Figure 4. Robustness to standardization and gate control. Without standardization, the post-SDPA gate also reduces outlier mass and increases effective rank. A feature-averaged scalar gate preserves token-wise gate strength but removes feature-direction-specific selectivity, and does not reproduce the same spectral suppression.
Learned Selectivity Beyond Generic Mode Contraction
The contraction result in Theorem 3.1 applies to any individual mode; it does not, by itself, establish preferential suppression of sink-aligned directions. To make this distinction explicit, consider a rank-one pre-gate component , where . For the elementwise gate , with denoting the diagonal matrix of gate values at token , its raw-energy retention is
This inequality does not order sink- and non-sink-aligned modes by their retention. We evaluate this distinction using the 64 leading pre-gate modes of the elementwise-gated Qwen-1B model, comparing the top and bottom terciles of sink alignment (Table 3).
| Retention metric | High alignment | Low alignment | Difference |
| Raw activation energy () | 0.047 | 0.044 | +0.003 |
| Standardized covariance () | 0.740 | 0.945 | −0.205 |
| Scalar gate () | 0.957 | 0.990 | −0.033 |
Table 3. Mode retention for strongly vs. weakly sink-aligned pre-gate directions in Qwen-1B (Gated). Raw-energy retention is computed on individual rank-one components; the other rows use feature-standardized token-space covariance.
We therefore additionally examine retention in the feature-standardized token-space covariance geometry used for the main spectral analysis. For each pre-gate covariance eigenvector , define
where and are the pre- and post-gate feature-standardized token-space covariance matrices. This standardized geometry reveals a directional difference that is absent from raw-energy retention. The negative association between sink alignment and retention persists after controlling for pre-gate spectral energy, so it cannot be explained merely by larger initial eigenvalues. As an ablation, we replace each token’s featurewise gate values by their feature average, removing within-token feature selectivity. The resulting retention gap becomes substantially smaller, as shown in Table 3. Thus, the gate is nearly non-selective in raw activation energy but selectively reshapes the feature-standardized covariance through its learned feature-dependent structure. This empirical selectivity is additional to, rather than a consequence of, the unconditional mode-wise contraction bound.
Post-SDPA Gate-score Statistics
We also inspect the post-SDPA gate scores directly. While average gate values do not by themselves determine spectral suppression, they provide an interpretable view of where the model applies stronger filtering. We compute gate statistics across layers, tokens, and heads, and compare the gate values at sink-related positions with those at non-sink positions. Figure 5 shows that gate scores vary substantially across depth and heads. In layers where spectral spikes are pronounced, the gate tends to apply stronger filtering to sink-related positions and hot-spot heads. This is consistent with the contraction analysis in § 3: suppression depends not only on the average gate magnitude, but on how the learned gate acts on the dominant representation modes.



Figure 5. Gate-score statistics. The post-SDPA gate is not spatially uniform: its strength varies across depth and is often concentrated around sink-related positions and hotspot heads. These statistics provide an interpretable view of the learned filtering pattern.
Structural and Bulk Diagnostics
We report a set of complementary diagnostics that probe different facets of the post-SDPA spectrum, separating outlier-level structure from bulk behavior. Where the main analysis (§ 3.2) used the data-adaptive sink subspace , here we additionally measure the alignment of the leading eigenvector with the simpler first-token reference direction . Second, we compute head-wise token-space spectra to identify whether dominant modes are distributed uniformly across heads or concentrated in a small number of hotspot heads. Third, we examine the eigenvalue tail using the complementary cumulative distribution function (CCDF), , which summarizes how much spectral mass remains in large eigenvalues. Finally, after removing eigenvalues above the Marchenko–Pastur upper edge (i.e., the bulk only), we compute the adjacent spacing ratio

Figure 6. Structural and bulk diagnostics. (a) Leading-eigenvector alignment with the first-token direction is reduced where such alignment is present. (b) Head-wise ratios show that localized hotspot heads with large leading eigenvalues tend to be suppressed by gating. (c) The eigenvalue-tail CCDF shows reduced heavy-tail dominance after gating. (d) Bulk spacing ratios remain stable, indicating that gating suppresses outlier structure while largely preserving the spectral bulk.
and report its layer-wise mean. This statistic tests whether gating disrupts the local bulk spectrum, with typical reference values near the GOE and Poisson limits shown in the figure.
Figure 6 summarizes these diagnostics. Leading eigenspace alignment decreases in layers where sink-related alignment is elevated, and head-wise spectra show that large leading eigenvalues are localized to a small number of heads. Gating reduces these localized spectral hotspots. The tail CCDF shows reduced heavy-tail dominance after gating, while the bulk spacing ratio remains stable. Together, these results support the view that post-SDPA gating primarily suppresses outlier and heavy-tail components associated with sink structure, rather than broadly disrupting the spectral bulk.
Sink-subspace Counterfactuals and Contraction Diagnostics
We next examine whether the spectral suppression observed in § 3.2 is attributable to the sink-induced subspace. Using the data-adaptive sink projector , we compare three quantities per layer: the pre-gate leading eigenvalue , the same quantity after explicitly removing the sink subspace, , and the post-gate value . As shown in Figure 7, projecting out the sink subspace lowers toward the Marchenko–Pastur edge in most layers, indicating that sink-induced directions account for part of the dominant pre-gate spectral component. The post-gate value lies below this sink-ablated value in most layers, suggesting that the gate also attenuates leading structure outside the selected sink subspace.

Figure 7. Sink-subspace diagnostic. Removing lowers .
Attention–Value Pairing Permutation Control
The alignment of the dominant token-space direction with a sink subspace could, in principle, arise from a special value state associated with a sink position rather than from the high-mass attention-column geometry. To test this alternative, we preserve the entire attention matrix and empirical value matrix while randomly permuting their positionwise pairing:
where is a permutation matrix. The original sink-subspace projector is held fixed, and the alignment is recomputed using the same feature-standardized diagnostic as in the main spectral analysis. We average over 20 random permutations for each of the same 16 sequences used in the original analysis packed with an EOS separator.
| Representation | Original | Permuted | Layers with increase |
| Baseline | 0.062 | 0.196 | 100% |
| Gated (pre-gate) | 0.220 | 0.510 | 96% |
Table 4. Sink-subspace alignment under attention–value pairing permutation. The attention matrix, empirical value matrix, and original sink projector are preserved; only the column–value pairing is randomized. Results are averaged over 20 permutations per sequence and 16 sequences. Alignment is the feature-standardized sink alignment of the pre-gate output, as in the main spectral analysis.
Table 4 shows that sink alignment increases rather than decreases when the original attention-column–value pairing is permuted. Therefore, the sink-aligned leading direction does not require a privileged value vector attached to the original sink position.
Robustness Beyond the Isotropic-Value Approximation
The isotropic-value approximation is an interpretable reference rather than a literal assumption about trained values. For an attention output , write
Then the exact raw token-space covariance is
The first term isolates the attention-column reference; the second captures learned value anisotropy, which can strengthen, weaken, or rotate the resulting spectral directions.
| Model | Cross-token corr. | Norm. feature rank | Align. () | Align. () | Random align. |
| Baseline | 4.9 | 0.44 | 0.97 | 0.64 | |
| Gated | 14.4 | 0.33 | 0.79 | 0.32 |
Table 5. Value anisotropy and sink alignment. Alignment at the endpoints (isotropic) and (learned values) of the value-Gram interpolation in Qwen-1B, computed on the non-standardized pre-gate covariance and thus not directly comparable to Table 4. An isotropic Gaussian null gives cross-token correlation and normalized feature rank .
On the Qwen-1B checkpoints and sequences used for the original spectral analysis, learned values are strongly anisotropic. Table 5 reports the dimension-normalized cross-token correlations and normalized feature effective ranks, alongside a matched isotropic Gaussian reference. These diagnostics show substantial learned-value anisotropy in both checkpoints. To assess sensitivity to this anisotropy while holding fixed, we interpolate the value Gram matrix in the positive semidefinite cone:
The endpoint is the isotropic attention-column reference, whereas uses the empirical learned-value Gram matrix. As Table 5 shows, moving from the isotropic reference to the empirical value geometry reduces alignment in both models, but the empirical endpoint remains far above the corresponding random-subspace reference in every evaluated layer and sample. The isotropic approximation is therefore useful as a structural prediction for sink-aligned spectral modes. Importantly, the deterministic gate contraction theorem does not rely on value isotropy; isotropy enters only the interpretable reference and the idealized BBP analysis.
Naturally Occurring Non-First-Position Sinks
We test whether the sink-subspace spectral signature persists beyond the first position and beyond a literal rank-one object. The general higher-rank contraction argument is discussed in Appendix § 9.§ 4. We exclude position 0 when selecting sink columns in normally trained checkpoints and evaluate their contribution to the dominant token-space covariance direction. Table 6 reports the observed alignment and the effect of projecting out the selected sink subspace. Across the two model families, the selected sink sets contain approximately 6.7 non-first columns on average. Their leading-eigenvector alignment is roughly 20× the random-subspace reference, and removing these subspaces reduces by about 19%. Approximately 43–45% of the selected non-first column instances correspond to structural delimiters, including periods, commas, and newlines. Within the gated models, post-SDPA gating reduces the corresponding alignment by 33–41%. These observations show that the forward spectral signature extends to multi-column, delimiter-rich sink structure.
| Metric | Qwen-1B | Pythia-410M |
| Align. (pre → post) | 0.280 → 0.186 | 0.262 → 0.154 |
| Pre/random | 21× | 20× |
| Align. drop | 33.4% | 41.2% |
| drop (proj.) | 19.4% | 19.2% |
Table 6. Non-first-position sink structure. The first position is excluded from the sink subspace. Alignment is compared before and after gating, with pre-gate alignment normalized by the random-subspace baseline. Projection measures the reduction in the leading covariance eigenvalue after removing the selected sink subspace.
Additional Training-Time Diagnostics
Training-Time Sink and Gradient Diagnostics
We provide additional training-time diagnostics that complement the final-checkpoint analyses in the main text. These diagnostics track how attention concentration, token-space spectral structure, and value-gradient amplification evolve across checkpoints in the baseline and post-SDPA gated models. To verify that the gated variant remains competitive under the matched 15B-token training setup, we also report standard language-modeling and downstream evaluation results in Table 7. Unless otherwise stated, all diagnostics are evaluated on sixteen 512-token WikiText-103 validation sequences, with an EOS token between passages. Table 8 reports their layer means at the final checkpoint, and Figure 8 shows them layer-wise across checkpoints.

Figure 8. Training-time sink and spectral diagnostics. Each curve corresponds to a training checkpoint. (a,b) Top- sink mass is the mean over the sink columns selected as in § 3.2, while (c,d) outlier mass measures the fraction of token-space covariance spectrum above the MP edge. Compared with the baseline, post-SDPA gating keeps both high-mass sink columns and out-of-bulk spectral concentration substantially weaker across training.
| Model | Wiki. ppl | LMB. ppl | LMB. acc | PIQA acc | Hella acc_n | Wino acc | ARC-e acc | ARC-c acc_n | OBQA acc_n | BoolQ acc | Avg. |
| Baseline | 29.53 | 55.21 | 29.94 | 65.83 | 36.76 | 51.70 | 56.02 | 26.54 | 30.80 | 62.11 | 44.96 |
| Gated | 27.99 | 40.52 | 32.47 | 66.43 | 37.93 | 52.49 | 57.79 | 27.65 | 32.00 | 61.71 | 46.06 |
Table 7. Downstream evaluation of Pythia-410M. Baseline and post-SDPA gated variants trained on 15B FineWeb-Edu tokens. All variants use matched data and optimization settings, with gated models adjusting MLP size to compensate for gate parameters. The best results are highlighted in bold. Avg. denotes the average of accuracies and normalized accuracies over 8 tasks.
| Variant | OutMass | |||||
| Baseline | 0.084 | 0.125 | 18.42 | 0.679 | 89.14 | 25.40 |
| Gated | 0.013 | 0.013 | 12.49 | 0.545 | 153.20 | 5.67 |
Table 8. Final-checkpoint diagnostics. Layer-mean values for Pythia-410M at the final checkpoint. Post-SDPA gating reduces attention concentration, weakens dominant spectral structure, reduces out-of-bulk mass, increases effective rank, and substantially lowers first-token value-gradient amplification. Spectral metrics for the gated model are computed on the post-gate output.
Table 8 summarizes the same trend observed in the layer-wise diagnostics. Post-SDPA gating reduces first-token and top- sink mass, lowers the leading-eigenvalue ratio, reduces outlier mass, increases effective rank, and substantially reduces first-token value-gradient amplification. Thus, the gated model shows weaker attention concentration, weaker dominant spectral structure, and weaker sink-directed gradient amplification at the final checkpoint.
Figure 8 tracks the emergence of sink and spectral signatures during training. In the baseline, top- sink mass grows in middle-to-late layers, indicating the formation of high-mass attention columns. The same layers also show stronger out-of-bulk spectral mass, consistent with the sink-induced spectral-spike view. In contrast, the post-SDPA gated model keeps both quantities lower across checkpoints, suggesting that gating suppresses the co-emergence of attention concentration and token-space spectral outliers.
Direct Measurements of Massive Activations and Their Relation to Sinks
To measure massive activations directly, rather than relying solely on relative first-token dominance, we record residual-stream hidden states after each Transformer block. Let denote the set of non-padding token positions in each sample. We measure the following massive-activation (MA) diagnostics:
(maximum token hidden-state norm), {#eq:37}
(maximum coordinate magnitude). {#eq:38}
For Pythia-410M and Qwen-1B, we evaluate both MA diagnostics on the same sixteen 512-token WikiText-103 validation blocks without padding using model-specific tokenization. Each metric is computed per sample and layer, then averaged across samples and layers for Table 9.
| Model | Diagnostic | Baseline | Gated | Baseline/gated |
| Pythia-410M | (MA) | 379.2 | 108.4 | 3.50× |
| (MA) | 320.0 | 67.4 | 4.75× | |
| Qwen-1B | (MA) | 11,854.5 | 146.3 | 81.1× |
| (MA) | 11,839.5 | 87.2 | 135.7× |
Table 9. Massive-activation (MA) diagnostics. Entries are sample- and layer-mean values on sixteen 512-token WikiText-103 validation blocks without padding. The last column reports baseline-to-gated ratios.
In Pythia-410M, post-SDPA gating jointly suppresses AS, GS, and MA (Table 8 and Table 9). Their baseline peaks occur in middle-to-late layers (17–21), while the layerwise maxima of and decrease from 726.7 to 186.6 and from 643.0 to 154.7, respectively. Across 56 training checkpoints, the baseline’s layer-mean grows from 8.7 to 320.0, whereas the gated model never exceeds 69.4. Qwen-1B further confirms cross-model MA suppression, with both and reduced by over an order of magnitude (Table 9). The maximum observed coordinate magnitude is 17,536 in the baseline (first token, layer 8), compared with 378 in the gated model across the evaluated samples and layers. Together, these measurements show lower attention concentration, value-gradient amplification, and massive-activation magnitudes in the gated Pythia-410M model, while the Qwen-1B results confirm massive-activation suppression. Combined with the fixed-upstream-gradient counterfactual in § 4.2, these findings are consistent with attenuation along the AS–GS–MA pathway.
Additional Next-Token Probing Diagnostic
We additionally examine whether the sink-related changes observed under gating are accompanied by changes in linearly accessible next-token information. Conceptually, this asks whether target-related information remains recoverable from hidden states across layers, especially around layers where sink signatures are pronounced.
Practical proxy. Since the conditional uncertainty of the next token given a hidden state is difficult to estimate directly, we use layer-wise linear probing as a practical proxy for next-token decodability. For each layer , we train a linear probe to predict the next token from the hidden state , and report cross-entropy loss on the test set. Lower cross-entropy indicates that next-token information is more linearly recoverable from the hidden representation.
Experimental setting. We use the Qwen-1B models with and without post-SDPA gating and the WikiText-2 dataset with maximum sequence length 128. For each model and layer, we train a linear probe and evaluate it on the test split, where layer 0 denotes the embedding output. Each probe is trained on raw or per-dimension standardized hidden states, either for one epoch with learning rate or for five epochs with learning rate . We report results both with and without the initial source position to check whether the observed trend is driven solely by the canonical sink position.
Experimental results. Under the original protocol (Figure 9(a)), baseline probes are much worse than gated probes in layers 8–28, where massive activations emerge in the baseline (Appendix § 13.§ 2). Massive activations place a few hidden dimensions at a far larger scale than the rest, which makes a linear probe difficult to optimize under a fixed training budget; in these layers, the baseline probes are even worse than a uniform prediction over the vocabulary. Gated probes, whose hidden states have much smaller activations, are far less affected. To test whether this gap reflects missing next-token information or only this optimization difficulty, we standardize each hidden dimension before probing, which leaves the class of linear probes unchanged. Under the same training budget, the large gap in deep layers mostly disappears (Figure 9(b)). A smaller gap remains in layers 8–17, where the first-token massive activation appears, and it vanishes once the initial source position is excluded. With a lower learning rate and longer training, this remaining gap also largely closes (Figure 9(c)), and the two models reach comparable probing loss across depth, with or without the initial source position. These results indicate that the gap under the original protocol arises from the difficulty of training linear probes on hidden states with massive activations, rather than from a lack of next-token information. Post-SDPA gating suppresses massive activations and thereby makes next-token information easier to read out with a linear probe, while both models retain comparable linearly recoverable next-token information.

Figure 9. Layer-wise probing cross-entropy loss. Baseline and gated models are evaluated with and without the initial source position. Probes are trained for one epoch on (a) raw and (b) per-dimension standardized hidden states, and (c) for up to five epochs with a lower learning rate on standardized hidden states. Lower is better.
Robustness Across Models and Settings
We examine whether the attention, spectral, and value-gradient signatures extend beyond the original model scale, context length, and sequence-formatting convention. The following analyses compare matched baseline and post-SDPA elementwise-gated models. These experiments use separately constructed WikiText-103 test inputs, so absolute values differ slightly from Table 8.
Robustness Across Model Scale
We trained matched ungated and post-SDPA elementwise-gated Pythia-2.8B models and repeated the attention, value-gradient, and token-space spectral diagnostics. Table 10 reports absolute layer means for the 2.8B models and gated-to-baseline ratios for both 2.8B and the corresponding 410M comparison. At 2.8B, gating reduces first-token attention mass by approximately 13×, top- attention mass by approximately 15×, and first-token and top- value-gradient amplification by 6.6× and 7.5×, respectively. The leading-outlier ratio decreases from 24.21 to 15.54, while effective rank increases from 111.2 to 185.0. All diagnostics change in the same qualitative direction at 2.8B and 410M. The attention- and gradient-related reductions are at least as strong at 2.8B, and the leading-outlier and effective-rank ratios are similar at the two scales. Thus, the attenuation of sink-related attention, gradient, and spectral signatures by post-SDPA gating is not specific to the 410M scale.
| Model | OutMass | Eff. rank | Sink alignment | |||||
| Baseline 2.8B | 0.147 | 0.156 | 28.24 | 27.04 | 0.684 | 24.21 | 111.2 | 0.174 |
| Gated 2.8B | 0.011 | 0.010 | 4.29 | 3.61 | 0.566 | 15.54 | 185.0 | 0.156 |
| 2.8B ratio | 0.077 | 0.067 | 0.152 | 0.133 | 0.827 | 0.642 | 1.664 | 0.895 |
| 410M ratio | 0.129 | 0.101 | 0.195 | 0.160 | 0.796 | 0.658 | 1.717 | 0.657 |
Table 10. Model-scale robustness. The first two rows are absolute layer-mean values at 2.8B; the last two rows report gated/baseline ratios. Ratios below 1 indicate decreases under gating.
Robustness to Longer Training and Evaluation Contexts
To test context-length robustness beyond the original 4K setting, we trained matched baseline and elementwise-gated Pythia-410M models with an 8K context window. We evaluate the 8K-trained checkpoints at both 4K and 8K, and compare them to 4K-trained checkpoints evaluated at 4K. All use the original plain sequence convention, in which the first token is an ordinary content token. Table 11 reports baseline-to-gated values. Effective rank and sink alignment are normalized by and , respectively, for comparison across sequence lengths.
| Train/eval | Model | Sink columns | Norm. eff. rank | Alignment/random | ||||
| 4K/4K | Baseline | 0.025 | 0.071 | 2.5 | 44.1 | 0.159 | 180.4 | 58.4 |
| Gated | 0.002 | 0.003 | 6.5 | 29.1 | 0.288 | 58.5 | 6.2 | |
| 8K/4K | Baseline | 0.015 | 0.058 | 3.3 | 53.0 | 0.118 | 203.8 | 54.7 |
| Gated | 0.002 | 0.003 | 5.7 | 35.4 | 0.237 | 50.7 | 7.5 | |
| 8K/8K | Baseline | 0.008 | 0.058 | 3.1 | 59.0 | 0.134 | 349.9 | — |
| Gated | 0.001 | 0.002 | 5.8 | 42.2 | 0.262 | 68.7 | — |
Table 11. Robustness across training and evaluation context lengths. Baseline and elementwise-gated models are compared under matched context-length settings. Top- value-gradient amplification is omitted for the 8K/8K setting because the backward pass exceeded available device memory.
At both evaluation lengths, gating reduces the first-position and adaptive top- sink masses, the normalized leading spectral outlier, and sink-subspace alignment, while increasing normalized effective rank. Within the 8K-trained baseline, extending evaluation from 4K to 8K reduces first-position mass from 0.015 to 0.008, even though stays at 0.058. This suggests that the longer-context sink pattern is less exclusively concentrated at position 0 and is better captured by adaptive, potentially non-first sink columns. The 8K-trained model also exhibits strong value-gradient amplification at 4K evaluation, reduced by gating from 54.7 to 7.5. We do not report an 8K/8K backward measurement because retaining the attention states for backpropagation scales quadratically with sequence length and exceeded device memory. Consequently, forward attention and spectral robustness are established at 8K evaluation, while the gradient result for the 8K-trained checkpoint is directly verified at 4K evaluation.
Robustness to Sequence Formatting and Special Tokens
Sequence formatting conventions. We additionally trained matched baseline and post-SDPA gated Pythia-410M models under two sequence conventions: (i) plain, with add_bos_token = False and add_eos_token = False; and (ii) EOS-separated, with EOS appended at document boundaries (add_eos_token = True, add_bos_token = False). Table 12 shows gated/baseline ratios, not absolute metric values. Values below 1 indicate attenuation, except that an effective-rank ratio greater than 1 indicates increased effective rank. Across both conventions, gating decreases attention-sink mass, value-gradient amplification, outlier mass, the leading-outlier ratio, and sink alignment, while increasing effective rank. The direct comparison before and after the gate within each gated model also reproduces the main spectral pattern: falls to under both conventions, effective rank increases by –, and sink alignment decreases to –.
| Convention | OutMass | Eff. rank | Sink alignment | |||||
| Plain | 0.135 | 0.108 | 0.206 | 0.175 | 0.797 | 0.649 | 1.723 | 0.660 |
| EOS-separated | 0.183 | 0.226 | 0.253 | 0.320 | 0.786 | 0.644 | 1.735 | 0.385 |
Table 12. Robustness across natural sequence-formatting conventions. All entries are gated/baseline ratios.
Controlled EOS-separator sinks. We also train matched baseline and elementwise-gated Pythia-410M models with EOS tokens inserted at document boundaries, evaluating each model under the same convention in Table 12. Position 0 is excluded when constructing the sink subspace. The selected non-first sink sets contain an average of 2.64 columns in the baseline and 3.54 in the gated model, with EOS tokens accounting for 58% and 71% of these selected columns, respectively. Their pre-gate or ungated alignments with the leading covariance direction are 0.184 (baseline) and 0.190 (gated, before the gate), corresponding to 28–36 their random-subspace references. Projecting out these sink subspaces reduces by 12–14%. Within the gated model, the alignment falls from 0.190 to 0.099 after gating, a factor of 0.52. Repeating the analysis with the column-energy selector yields a reduction from 0.182 to 0.069. Thus, learned EOS-related sinks can occupy multiple non-first columns and remain attenuated by gating.
Position-Controlled EOS Counterfactual
To isolate the role of token identity from absolute position, we evaluate otherwise identical content after replacing one token with EOS at the specified relative position. The strongest sink is selected by availability-normalized causal column mass, i.e., the column mass divided by the number of queries that can attend to each position. Unlike the other analyses, we use this normalization here so that sink strength is comparable across insertion positions. Since it divides by only a few queries near the end of the sequence, we restrict candidates to positions with at least future queries. For the no-EOS condition, the target is position 0. Table 13 reports baseline and gated results. A first-position sink arises even without EOS. When EOS is inserted, the strongest attention sink largely follows it in both models: the inserted EOS is the strongest sink in 88–95% of ungated and 71–99% of gated evaluations. Gating therefore leaves the sink location largely unchanged but reduces its strength: target attention mass decreases by 2.8–5.4× and target gradient amplification by 2.5–3.8×. When EOS appears away from the start, position 0 is no longer the dominant attention sink but retains a residual value-gradient signature, with amplification of 24.47–28.84 reduced to 6.00–6.18 by gating. Thus, sink location is affected by both sequence position and special-token identity, whereas gate-induced attenuation persists under these changes.
| Input condition | Model | Target attn. mass | Target spectral alignment | Target gradient amplification | |
| No EOS; position 0 | Baseline | 0.583 | 0.130 | 0.204 | 31.39 |
| No EOS; position 0 | Gated | 0.174 | 0.015 | 0.031 | 6.38 |
| EOS at 0 | Baseline | 0.930 | 0.200 | 0.229 | 40.97 |
| EOS at 0 | Gated | 0.706 | 0.037 | 0.056 | 10.77 |
| EOS at | Baseline | 0.878 | 0.181 | 0.042 | 28.64 |
| EOS at | Gated | 0.753 | 0.041 | 0.023 | 10.20 |
| EOS at | Baseline | 0.911 | 0.199 | 0.131 | 25.30 |
| EOS at | Gated | 0.885 | 0.055 | 0.031 | 10.03 |
| EOS at | Baseline | 0.953 | 0.237 | 0.140 | 20.88 |
| EOS at | Gated | 0.990 | 0.086 | 0.052 | 8.33 |
Table 13. Position-controlled EOS counterfactual. Baseline and post-SDPA gated models are compared across EOS insertion positions. Without EOS, the target is position 0; otherwise, the target is the inserted EOS token. The strongest sink is selected among positions with at least future queries. Spectral alignment for the gated model uses the post-gate output.
Extensions Beyond Post-SDPA Gating
We use the spectral and gradient diagnostics to characterize intervention classes beyond post-SDPA gating. We distinguish direct empirical comparisons of attention operators from theoretical comparisons of local gradient pathways and gate placements.
Softpick: An Attention-Operator Comparison
The raw token-space spectral identity
applies to any attention operator , including Softpick [23]. The same token-space spectral diagnostics can therefore characterize normalization-based sink mitigations even though their mechanisms need not match post-SDPA gating. In contrast, the deterministic mode contraction result in Theorem 3.1 concerns an already-formed attention output acted on by a downstream gate. Here we separate the effect of changing the normalization operator at fixed learned scores from that of training a model using the alternative operator.
Protocols and metrics. In a frozen-operator swap, we hold Qwen-1B baseline scores and values fixed, changing only Softmax to the Softpick operator. In a separate trained-model comparison, we analyze released Vanilla and Softpick-340M checkpoints.4 Sink-bearing layers are selected independently of spectral outcomes using only the Vanilla raw-score geometry: the first position must be the raw-logit argmax for more than 50% of queries. This yields layers 8–28 for Qwen-1B and 8–24 for the matched 340M models.
Since Softpick need not produce attention rows summing to 1, we report both absolute top- column mass and a normalized fraction of routed mass. With
Both statistics use the same top- columns per head (), ranked by the column mass . This is the quantity used for the sink subspace in Section 3.2 up to the constant factor , but with a fixed instead of the adaptive threshold. Ranking by avoids selecting late positions whose few valid queries make availability-normalized mass unreliable. The leading-outlier ratios below use the feature-standardized covariance and MP edge employed in the main spectral analysis.
| Operator Swap (Qwen-1B, layers 8–28) | Trained models (340M, layers 8–24) | ||||||
| Metric | Softmax | Softpick | Effect | Metric | Vanilla | Softpick | Effect |
| 0.0767 | 0.0754 | 1.7% | 0.0786 | 0.0032 | 24.3× | ||
| 19.86 | 19.69 | 0.9% | 15.39 | 13.74 | 1.12× | ||
| – | – | – | – | 0.629 | 0.199 | 3.2× |
Table 14. Comparison of Softmax and Softpick under frozen-operator and trained-model settings. The frozen control uses Qwen-1B without retraining, while the trained comparison uses released Vanilla/Softpick-340M checkpoints. Effects indicate percentage decreases (frozen) or Vanilla-to-Softpick ratios (trained).
Results. In the frozen swap, all reported diagnostics change by less than 2%. The first token is already the largest raw score for 79–94% of queries in the selected Qwen layers, so replacing Softmax after training largely preserves the same dominant source. In contrast, the Softpick-trained model selects the first token as raw-logit argmax for fewer than 1% of queries and exhibits a negative first-token logit margin. Its top- mass and routed-mass concentration decrease by 24.3× and 3.2×, respectively, relative to the trained Vanilla model. The leading-outlier ratio changes more modestly, from 15.39 to 13.74. Thus, Softpick substantially reduces sink concentration, although structured spectral modes in the representation largely remain. The strong trained effect is primarily associated with the score geometry learned under Softpick, rather than a large post-hoc effect of swapping normalizers on frozen Softmax-trained scores. These observations distinguish upstream operator-mediated prevention of sink formation from downstream contraction of an already-formed sink contribution. The algebraic covariance framework also accommodates other attention operators, including Elastic-Softmax [24], but the empirical comparison here is limited to Softpick.
Gradient-Valve Interpretation of Value-State Gated Attention
The local value-gradient interpretation can also be applied to Value-State Gated Attention (VGA) [6], while distinguishing source-conditioned and query-conditioned gates. Using column-vector notation, the local value Jacobian of a post-SDPA gate for source token and query token is
Here is the diagonal, query-dependent sigmoid gate. Since , each individual value-path contribution is directly contracted at fixed upstream gradient.
For a simplified scalar VGA gate, let
Because the gate depends on its own value state, , so its Jacobian is
Consequently,
The first term, , is the gated content path; the second, rank-one term, , is a self-regulatory path because the source value also controls its gate. Both Jacobians therefore share the local form
High attention-column mass does not necessarily imply proportionally high value-gradient mass when these local Jacobians suppress the corresponding contributions.
The conditioning and guarantees differ. Post-SDPA gating acts separately at each receiving query after aggregation and has the unconditional local operator-norm bound for its value path. VGA gates each source value before aggregation and includes an additional derivative term. In general,
so alone does not guarantee operator-norm contraction at intermediate gate values. In the closed-gate limit , with the remaining factors bounded,
Thus, complete severing is a limiting case of local Jacobian suppression, not a requirement for substantial sink-gradient attenuation. Our fixed-upstream-gradient counterfactual already shows considerable attenuation under ordinary sigmoid post-SDPA gates. This comparison is mechanistic; it is not an experimental claim that VGA and post-SDPA gating have identical training dynamics.
Mechanistic Comparison of Gate Placements
The five gate placements studied in gated-attention architectures intervene at different points along the sink pathway. For one head, let ; multi-head outputs are mixed as . Denote multiplicative gates at the query, key, value, attention-output, and mixed-output positions by , , , , and :
For sink source , the attention column creates a rank-one forward contribution , while the same attention weights accumulate value-path gradients across receiving queries.
Query/key gating (G4/G3) intervenes before the attention matrix is formed; the coupled changes to logits and normalization mean that attenuating queries or keys does not guarantee monotone attenuation of a particular sink column. Value gating (G2) directly transforms the source contribution
which, for a fixed attention matrix, retains the same token-space column direction . It also uses a single source-side gate for all queries receiving that value.
Post-SDPA gating (G1) instead transforms the sink contribution at each receiving query:
It can therefore suppress the same source differently across receiving queries, after the sink contribution is formed but before head mixing. The contraction theorem directly describes its forward attenuation, and its local value Jacobian attenuates query-to-source contributions before they accumulate at a sink value. G5 gates only after the head-specific outputs have been mixed by ; contraction in that mixed output space does not guarantee contraction of each individual sink-carrying head or of the corresponding projected value path.
Limitations and Future Work
Our analysis is intended as a mechanism-level account rather than a complete theory of gated training dynamics. The BBP interpretation relies on an idealized isotropic reference model; learned feature-wise gates also reshape the noise component, so the resulting covariance need not be a standard spiked Wishart model. Similarly, our gradient result is local: it shows that gating contracts value-path gradients for fixed attention weights and upstream gradients, but does not imply that all gradients in a trained gated model are uniformly smaller. Empirically, gating reduces sink-subspace alignment and dominant spectral outliers, but not all spectral suppression is explained by sink-induced modes. Learned gates may also affect non-sink structured directions, head-specific outliers, and anisotropic noise. Thus, our findings provide a mechanism-level account within the evaluated settings, while broader generalization remains to be established.
Broader Impact
This work contributes to the mechanistic understanding of gated attention and attention-sink mitigation in Transformer language models. By connecting sink-induced spectral structure with value-gradient attenuation, our analysis may help develop more stable, interpretable, and efficient architectures, particularly for large-scale and long-context training. The societal impact of this work is indirect. We do not introduce a new deployed system or task-specific capability, but architectural insights that improve training stability and scalability may contribute to stronger foundation models. Such improvements can support beneficial applications, but may also amplify risks associated with misuse or unsafe deployment. These risks should be addressed through appropriate evaluation, safety, and governance practices alongside architectural research.
References
- [1]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03762
- [2]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, volume 2024, pages 21875–21895, 2024.arxiv.org/abs/2309.17453
- [3]Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2410.10781
- [4]Shangwen Sun, Alfredo Canziani, Yann LeCun, and Jiachen Zhu. The spike, the sparse and the sink: Anatomy of massive activations and attention sinks. arXiv preprint arXiv:2603.05498, 2026.arxiv.org/abs/2603.05498
- [5]Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. In First Conference on Language Modeling, 2024.arxiv.org/abs/2402.17762
- [6]Rui Bu, Haofeng Zhong, Wenzheng Chen, and Yangyan Li. Value-state gated attention for mitigating extreme-token phenomena in transformers. arXiv preprint arXiv:2510.09017, 2025.arxiv.org/abs/2510.09017
- [7]Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization. In Proceedings of the computer vision and pattern recognition conference, pages 14901–14911, 2025.arxiv.org/abs/2503.10622
- [8]Hoyoon Byun, Youngjun Choi, Taero Kim, Sungrae Park, and Kyungwoo Song. Bounded hyperbolic tangent: A stable and efficient alternative to pre-layer normalization in large language models. arXiv preprint arXiv:2601.09719, 2025.arxiv.org/abs/2601.09719
- [9]Xingyue Huang, Xueying Ding, Mingxuan Ju, Yozen Liu, Neil Shah, and Tong Zhao. Threshold differential attention for sink-free, ultra-sparse, and non-dispersive language modeling. arXiv preprint arXiv:2601.12145, 2026.arxiv.org/abs/2601.12145
- [10]Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2505.06708
- [11]Qwen Team. Qwen3-next: Towards ultimate training & inference efficiency. https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd, September 2025. Blog post. Accessed: 2026-11-06.
- [12]Yihong Chen and Quanming Yao. Attention sinks induce gradient sinks. arXiv preprint arXiv:2603.17771, 2026.arxiv.org/abs/2603.17771
- [13]Vladimir A Marčenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1(4):457–483, 1967.DOI
- [14]Iain M Johnstone. On the distribution of the largest eigenvalue in principal components analysis. The Annals of statistics, 29(2):295–327, 2001.DOI
- [15]Debashis Paul. Asymptotics of sample eigenstructure for a large dimensional spiked covariance model. Statistica Sinica, pages 1617–1642, 2007.
- [16]Iain M Johnstone and Arthur Yu Lu. On consistency and sparsity for principal components analysis in high dimensions. Journal of the American Statistical Association, 104(486):682–693, 2009.pubmed.ncbi.nlm.nih.gov/20617121
- [17]Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. The Annals of Probability, 33(5):1643–1697, 2005.DOI
- [18]Florent Benaych-Georges and Raj Rao Nadakuditi. The singular values and vectors of low rank perturbations of large rectangular random matrices. Journal of Multivariate Analysis, 111:120–135, 2012.
- [19]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.arxiv.org/abs/1609.07843
- [20]Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Af Kham, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International conference on machine learning, pages 2397–2430. PMLR, 2023.arxiv.org/abs/2304.01373
- [21]Runyu Peng, Ruixiao Li, Mingshu Chen, Yunhua Zhou, Qipeng Guo, and Xipeng Qiu. How attention sinks emerge in large language models: An interpretability perspective. arXiv preprint arXiv:2603.06591, 2026.
- [22]Mengxia Yu, De Wang, Qi Shan, Colorado J Reed, and Alvin Wan. The super weight in large language models. arXiv preprint arXiv:2411.07919, 2024.
- [23]Zayd Muhammad Kawakibi Zuhri, Erland Hilman Fuadi, and Alham Fikri Aji. Softpick: No attention sink, no massive activations with rectified softmax. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Findings of the Association for Computational Linguistics: ACL 2026, pages 18179–18213, San Diego, California, United States, July 2026. Association for Computational Linguistics.arxiv.org/abs/2504.20966
- [24]Zichuan Fu, Wentao Song, Guojing Li, Yejing Wang, Xian Wu, Yimin Deng, Hanyu Yan, Yefeng Zheng, and Xiangyu Zhao. Attention needs to focus: A unified perspective on attention allocation. arXiv preprint arXiv:2601.00919, 2026.arxiv.org/abs/2601.00919
- [25]Jason Ramapuram, Federico Danieli, Eeshan Gunesh Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, et al. Theory, analysis, and best practices for sigmoid self-attention. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2409.04431
- [26]Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1–73, 2021.
- [27]Cosme Louart, Zhenyu Liao, and Romain Couillet. A random matrix approach to neural networks. The Annals of Applied Probability, 28(2):1190–1248, 2018.arxiv.org/abs/1702.05419
- [28]Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020.DOI
- [29]Enrique Queipo de Llano, Alvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael M. Bronstein, Yann LeCun, and Ravid Shwartz-Ziv. Attention sinks and compression valleys in LLMs are two sides of the same coin. In The Fourteenth International Conference on Learning Representations, 2026.
- [30]Yehjin Shin, Seojin Kim, and Noseong Park. Graph signal processing meets mamba2: Adaptive filter bank via delta modulation. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2603.22333
- [31]Jeongwhan Choi, Hyowon Wi, Jayoung Kim, Yehjin Shin, Kookjin Lee, Nathaniel Trask, and Noseong Park. Graph convolutions enrich the self-attention in transformers! arXiv preprint arXiv:2312.04234, 2023.arxiv.org/abs/2312.04234
- [32]Hyowon Wi, Jeongwhan Choi, and Noseong Park. Learning advanced self-attention for linear transformers in the singular value domain. arXiv preprint arXiv:2505.08516, 2025.arxiv.org/abs/2505.08516
- [33]Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663, 2024.arxiv.org/abs/2501.00663
- [34]Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024.arxiv.org/abs/2406.17557
- [35]Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534, Berlin, Germany, August 2016. Association for Computational Linguistics.arxiv.org/abs/1606.06031
- [36]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.arxiv.org/abs/1911.11641
- [37]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics.arxiv.org/abs/1905.07830
- [38]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.arxiv.org/abs/1907.10641
- [39]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
- [40]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, 2019.arxiv.org/abs/1905.10044
- [41]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, 2018.
