WorksComputer Science
HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs
Abstract
Attention in large language models (LLMs) scores every query against every key with a full dot product. We investigate whether this dot product must be computed completely for attention to be effective. We propose Hidden Sparse Attention (HiSA), which keeps the query and key projections intact but only computes a partial dot product, following a static, parameter-free rule set by the token positions: attention becomes sparse in the hidden (feature) dimension, an axis much less explored than token sparsity, rather than over tokens as in almost all sparse attention methods. Based on the structure of rotary position embeddings (RoPE), we decompose keys and queries into parts which can be omitted when computing attention. By training LLMs from scratch with HiSA, we find that this partial dot product does not hurt but improves performance compared with the complete product while computing only about a third of the query-key products. During inference HiSA improves decoding speed and shrinks memory footprint. We also extend HiSA to FlashAttention-3 kernels for training and inference. Finally, we motivate our approach by showing that dense attention learns to suppress the attention score parts that HiSA drops. Code is available at https://github.com/ambroise-b/hisa.
Introduction
The attention layer (Vaswani et al., [30]) of a large language model compares every query with every key via a dot product over the full head dimension. Most work on making attention cheaper keeps this dot product intact and instead reduces the set of keys a query looks at. Strategies to do so include fixed patterns (Child et al., [6]), sliding windows (Jiang et al., [16]), KV-cache eviction (Xiao et al., [33]; Zhang et al. [41]), and query-dependent token selection, training-free (Tang et al., [27]) or learned (Yuan et al., [39]). In this work, we investigate an orthogonal question: whether attention requires computing the complete query–key dot product.
Rotary position embeddings (RoPE) (Su et al., 2024), used by many modern LLMs (Grattafiori et al., [14]; Yang et al. [37]; Team Olmo [28]; DeepSeek-AI [9]; Gemma Team [12]), give this question a natural structure. RoPE rotates each two-dimensional frequency pair of the query and the key by an angle proportional to its position. Because rotations compose, the two rotations combine into a single one that encodes the relative distance between the query and the key. High-frequency pairs complete many turns within a few tokens, whereas low-frequency pairs barely rotate over the whole context.
The score parts of fast pairs are therefore highly sensitive to the exact distance between the query and the key, whereas language does not follow an exact structure over distant words or tokens, and sequences can be long. This raises the question of whether every pair needs to be computed at every relative distance. We show that high-frequency pairs are not necessary for the model when distance between tokens is large, and that removing them is harmless or even beneficial.
Our Hidden Sparse Attention (HiSA) method turns this hypothesis into an attention rule: the dot product of each frequency pair between a query and a key is kept only when the rotation induced by RoPE is below an angle budget , a hyperparameter that we set to 90° by default. The query and key projections are unchanged and what becomes sparse is the hidden dimension of the dot product. The sparsity pattern follows from the RoPE frequencies and the positions alone: it has no learned parameter and needs no selection step, unlike sparse attention methods that score, route or select keys at runtime. For a given query, fast pairs only see nearby keys, slow pairs see far.
We propose HiSA, a new sparse attention architecture that is sparse in the hidden dimension rather than over tokens. Our contributions can be summarized as follows:
Based on the structure of RoPE, we develop HiSA, a sparse attention in the hidden dimension that computes a partial query–key dot product. Its sparsity pat
Figure 1. Dense attention computes every part (left); token-sparse attention drops whole keys (centre). HiSA keeps a part only while its rotation induced by RoPE is below an angle budget : nearby keys are compared on all pairs, farther keys only on the slow ones, so attention becomes sparse in the hidden dimension rather than over tokens (right).
term is static and parameter-free. We derive its compute, reach, and KV-cache footprint (§ 3).
We evaluate HiSA in pretraining from scratch and in long-context extension, implement it in FlashAttention for training and inference, and compare it with dense attention on validation loss, training speed, KV-cache memory, and decoding throughput (§ 4.1 to § 4.4).
For existing models, we show that a pretrained dense model can be converted to HiSA by continued training and matches dense continued training (§ 4.5).
From an interpretability perspective, we show that the score parts that HiSA drops correspond to those that dense attention learns to suppress during training (§ 5): the rule is not only motivated by the geometry of RoPE, but measured in what dense models learn.
Related Work
Attention and RoPE. Attention (Vaswani et al. [30]) with RoPE (Su et al. (2024)) encodes relative positions by rotating query and key pairs at spaced frequencies . Barbero et al. [2] show that Gemma 7B uses the highest frequencies for positional attention patterns and suggest that the lowest carry semantic information; Men et al. [20] and Liu et al. [18] relate the RoPE base to the usable context length, and Wang et al. [31] learn per-head frequencies. Exact attention is made fast by the IO-aware FlashAttention kernels (Dao et al. [8]; Dao [7]; Shah et al. [24]), which we extend.
Sparse attention over tokens. Static patterns and sliding windows ([6, 3, 40, 16]) restrict each query to a subset of keys. For inference, KV-cache eviction keeps attention sinks and recent tokens (Xiao et al. [33]) or heavy hitters (Zhang et al. [41]), and Quest (Tang et al. [27]) selects cache pages per query. Recent methods are trained with the sparsity in place, using block-sparse kernels: NSA (Yuan et al. [39]) combines compressed, selected and sliding-window branches, and MoBA (Lu et al. (2025)) routes each query to the top-scoring blocks of keys.
Sparsity in the hidden dimension. Several inference methods exploit the feature dimension, but mostly as a proxy for selecting tokens. SparQ (Ribar et al. [23]) and Loki (Singhania et al. [25]) approximate the scores with a few query components or a low-rank key projection, select the top keys, and then compute the full dot product over them; Double Sparsity (Yang et al. [38]) calibrates important channels offline and FASA (Wang et al. [32]) uses a few dominant RoPE frequency chunks for the same purpose. The final attention of these methods is sparse over tokens, not in the hidden dimension. Think (Xu et al. (2025)) does compute the final score on fewer features, by pruning key-cache channels, but chooses them from the content and applies the same pruning to all but the most recent tokens. All of these methods are applied after training. Sparse Feature Attention (Xie et al. [34]) trains models whose queries and keys keep only their largest-magnitude features, chosen per vector from the content; HiSA instead fixes the kept parts by the positions alone, through the RoPE rotation angles.
Two concurrent works also restrict each RoPE pair to a window proportional to its wavelength. ATFlash (Hayashi et al. [15]) applies such windows to pretrained models without training (two periods by default) and ports them to FlashAttention-4 prefill and FlashInfer decode, with speedups that grow with context length. WavePrune (Du et al. (2026)) restricts each pair to one rotation period, both. training-free and when pretraining GPT-2 models from scratch, with FlashAttention-based forward and backward kernels. WavePrune explains the window by the aliasing that fast pairs cause in the attention maps of pretrained models. Both works skip computation and key reads, but neither reduces the KV cache: every key and value stays in memory. HiSA differs in three ways. It trains with much tighter budgets (a quarter period at 90°), where the partial product clearly improves the loss at the training length. It turns the rule into an actual KV-cache reduction, for keys per chunk and for values beyond the reach. Moreover, an interpretability analysis of dense models explains why the dropped pairs are not needed: dense training itself learns to suppress them.
Long-context extension. RoPE models are extended by interpolating positions [5, 21] or raising the base [35], followed by continued pretraining on long documents mixed with short data [35, 11]. We follow this recipe to study HiSA beyond its pretraining length.
Hidden Sparse Attention
The RoPE score as a sum over frequency pairs
Consider one attention head with dimension and RoPE pairs. Let define a pair consisting of a query at position and a key at position before rotation, and be its frequency, with RoPE base . RoPE rotates and by the angle and , respectively. Since rotations preserve norms and only the difference of the two rotations remains in the inner product, the attention score decomposes exactly as
where and is the angle between and . The content of the query and the key enters through the norms and through , whereas their position enters only through the rotation . We can also define each attention score part as
so that .
Consider a fixed query and a key far from it. For a fast pair, moving the key by a single token, a negligible change relative to the distance , already rotates the pair by (up to one radian), so its score part (Equation (2)) changes drastically. Such a pair can only express exact long-range
positions. In other words, even a minor change in sentence structure or word order in the distant context of the key may have a dramatic impact on the score. Language models, however, use distant context differently from nearby context: recurrent language models rely on word order only within the last few dozen tokens and treat more distant history as a coarse, order-insensitive context [17]. The relevant context far from the query is thus rarely tied to an exact offset.
We therefore hypothesise that models do not need precise positional patterns at long range, which is exactly what fast pairs encode there. Slow pairs, on the other hand, rotate little over long distances and can still capture long-range dependencies, so they should be kept up to much larger distances. To test this hypothesis we define the HiSA rule: a new attention formulation which omits the score parts when is large.
The HiSA rule
HiSA defines sparsity using a simple criterion: the score part of a pair is kept only while the rotation induced by RoPE between the query and the key stays below a fixed angle. The projections that produce the queries and keys are left untouched and only the dot product between them becomes partial.
We keep the notation of (1). A head of dimension consists of RoPE pairs, ordered from the fastest () to the slowest (). Pair of the query at position and of the key at position is denoted and before rotation, and rotates at the frequency for a RoPE base . The distance between a query and a key is and a pair is thus rotated by . Given an angle budget , HiSA keeps only the pairs whose rotation is still within the budget:
where denotes the rotation matrix by an angle from RoPE and the indicator function. Each summand is exactly the corresponding score part of the dense score (Equation (2)): the indicator is the only change, and it removes a score part as soon as its rotation exceeds the budget. With the default budget , a pair contributes as long as RoPE has turned it by less than a quarter of a turn.
Since the criterion depends on the distance alone, it gives each pair its own window: pair contributes to keys at most positions away and is ignored beyond. The rotation speed decreases with and so the windows grow. Figure 2 shows that the fastest pairs see only a few tokens and the slowest several thousand. A nearby key is therefore compared with the query on all pairs, a more distant one only on the slower pairs. A key leaves the softmax entirely only once no pair is active, that is, beyond the reach
Figure 2. Which score parts HiSA keeps (, ). Pair contributes only while its RoPE rotation is below the angle budget, i.e. up to the distance (shaded for ). Fast pairs see a few tokens, slow pairs thousands; raising the RoPE base stretches every window.
set by the slowest pair. The attention output is then computed as usual, .
Static, parameter-free sparsity. Token-sparse attention removes entries of the score matrix, while HiSA keeps the entries and removes, for each of them, some of the score parts that make up the score. The pattern is static: it depends only on the distance and on the RoPE frequencies, not on the content of the queries and keys. HiSA therefore adds no parameter, no router and no top- selection; the only new quantity is the hyperparameter , and the projections, the values and the softmax are unchanged. Two consequences matter in practice. First, the model is trained with exactly the pattern it uses at inference. Second, which parts are needed is known before any token is seen, so kernels skip them without a selection pass and the KV cache drops each part of a key once it leaves the window (§ 4.4). With , HiSA is dense attention.
Angle budget and RoPE base. The angle budget and the RoPE base jointly fix every window: . Raising stretches all windows and the reach (Equation (4)), which is how HiSA adapts to long contexts (§ 4.3); lowering shortens them. For instance, with , and the fastest pair sees 2 tokens and the slowest about 11.8K; with the reach becomes about 23K tokens.
Compute and memory. The fraction of the dense causal dot product that HiSA computes over a sequence of length is
Appendix § 8 gives for other budgets, bases and lengths. During decoding, pair of a cached key can be dropped once the key is older than , and the whole token once it is older than .
Experiments
Setup. We train decoder-only models with the SmolLM-135M and SmolLM-360M architectures ([4]) (head dimension , i.e. 32 RoPE pairs, grouped-query attention [1]) from scratch on OpenWebText ([13]), for about 20 tokens per parameter, with AdamW, a cosine schedule and a peak learning rate selected for the dense model. Dense and HiSA runs share every setting and differ only in the attention rule. We write HiSA for the exact rule (Equation (3)), computed by our Triton ([29]) kernels, and Flash-HiSA for its FlashAttention-3 implementation, which uses a block-level version of the rule (§ 4.2). Appendix § 9 gives the full recipe and Appendix § 11 gives additional details on the kernels.
HiSA vs. dense attention
Table 1 compares HiSA () with dense attention at 4K context size. HiSA reaches a lower validation loss than the complete dot product while computing 29–34% of the dense dot products, and the gap is the same at every position of the predicted token in the context window. It also holds at 360M parameters with a Chinchilla budget of 7.24B tokens.
| Model | Val. loss | Loss by position | Loss by position | Loss by position | (%) |
| 0–512 | 512–2K | 2K–4K | |||
| 135M, Dense | 2.906 | 3.170 | 2.920 | 2.830 | 100.0 |
| 135M, HiSA | 2.880 | 3.147 | 2.893 | 2.804 | 29.3 |
| 135M, Dense | 2.908 | 3.177 | 2.923 | 2.829 | 100.0 |
| 135M, HiSA | 2.869 | 3.136 | 2.883 | 2.791 | 34.2 |
| 360M, Dense | 2.587 | 2.823 | 2.598 | 2.519 | 100.0 |
| 360M, HiSA | 2.575 | 2.810 | 2.585 | 2.508 | 29.3 |
Table 1. Dense attention vs. HiSA () at 4K context: validation loss, loss by position, and percentage of the dot product computed. 135M: 2.7B tokens; 360M: 7.24B tokens. Best in bold.
The angle budget and the other choices of this recipe come from the ablations of Appendix § 10: every finite budget from 45° to 360° beats dense attention and 90° is a compromise between compute and loss. § 5 relates the dropped score parts to what a dense model learns.
Flash-HiSA: HiSA in FlashAttention
When using the exact rule, each pair’s window ends at a different distance, so a pair is kept for some entries of a FlashAttention tile and not for others which is not efficient for GPU’s vectorized operations. Flash-HiSA follows the exact rule but applies it block by block instead. Queries and keys are cut along the sequence into blocks of tokens, and the features into chunks of 16 (8 pairs, one matrix-multiply step on Hopper GPUs), so is a grid of tiles. For a query block and a key block, the kept pairs are those that the exact rule keeps at the closest distance between the two blocks. Each tile of is then multiplied in full or skipped, and the one chunk cut by the rule is multiplied with its dropped pairs set to zero in (Figure 3). This block rule keeps every score part of the exact rule plus up to tokens more per pair: an approximation of HiSA that fits the tiling of FlashAttention, and a different model (training and inference must use the same ). We implement it in FlashAttention-3 (Shah et al., [24]), forward and backward (Appendix § 11).
Figure 3. Exact rule and block rule, for one query (block). Rows: keys, nearest first; columns: RoPE pairs, fast to slow. The exact rule (left) drops pairs key by key. The block rule (right) groups keys into blocks (128 tokens) and pairs into chunks (8 pairs, 16 features); each block keeps the pairs of its nearest key, so a (block, chunk) tile of is either skipped or multiplied, with the dropped pairs of the boundary chunk zeroed in .
Table 2 compares the implementations. The exact rule has the best loss but trains 1.7 times slower than dense attention. Flash-HiSA computes slightly more of the dot product (35% instead of 29%), keeps about half of the gain (−0.012 against dense FlashAttention-3), and trains at the speed of dense FlashAttention-3.
| Attention | Kernel | Val. loss | Tokens/s | (%) |
| Dense | SDPA (FlashAtt. 2) | 2.906 | 205k | 100.0 |
| Dense | FlashAtt. 3 | 2.904 | 236k | 100.0 |
| HiSA | Triton (exact rule) | 2.880 | 119k | 29.3 |
| HiSA block rule | Triton | 2.888 | 191k | 34.8 |
| Flash-HiSA | FlashAtt. 3 | 2.892 | 238k | 34.8 |
Table 2. Implementations at 4K context (, , , 135M parameters): validation loss, training throughput on one H200, and percentage of the dot product computed. Best in bold.
Long-context extension
We pretrain dense and HiSA models at 4K context size with a RoPE base , which gives HiSA a reach of 23K tokens, then extend them to 16K with 540M tokens (20% of pretraining) of PG19 books (Rae et al., [22]) and OpenWebText, following Xiong et al. [35]; Gao et al. [11] (details in Appendix § 9). After the extension, HiSA keeps its advantage on books and web text while computing a fifth of the dot product (Table 3).
| PG19 val. loss | OWT | |||||
| all | 0–4K | 4K–8K | 8K–16K | val. loss | (%) | |
| Dense | 3.205 | 3.262 | 3.209 | 3.174 | 3.111 | 100.0 |
| HiSA | 3.170 | 3.226 | 3.174 | 3.140 | 3.071 | 20.2 |
Table 3. After the 16K extension (), exact rule: validation loss on PG19 (overall and by position) and OpenWebText, and percentage of the dot product computed. Best in bold.
Flash-HiSA, pretrained and extended with the same recipe, also keeps its advantage, with a smaller gap (Table 4). It trains at 94% of the speed of dense FlashAttention-3 although it computes only 22% of the dot product.
| PG19 val. loss | OWT | Tok/s | |||||
| all | 0–4K | 4K–8K | 8K–16K | val. loss | (%) | ||
| Dense (FA3) | 3.212 | 3.270 | 3.216 | 3.181 | 3.115 | 100.0 | 143k |
| Flash-HiSA | 3.190 | 3.248 | 3.194 | 3.159 | 3.099 | 21.7 | 135k |
Table 4. After the 16K extension with FlashAttention-3 (): validation loss, percentage of the dot product computed, and training throughput on one H200. Best in bold.
KV cache and decoding
A cached key needs pair only for tokens and its value only within the reach. Flash-HiSA stores each 16-feature key chunk in its own ring buffer, as long as the window of its slowest pair, and its decoding kernel reads only the chunks still in use (Appendix § 11). The cache is a third smaller (Table 5) than the dense one at 4K and 16K, and decoding is faster than dense FlashAttention-3, increasingly so with the context: +42% tokens per second at 16K and batch 128. Part of this comes from our kernel’s smaller query tiles; with the same kernel and no pair dropped, Flash-HiSA is still 13% faster.
| Attention | Batch 32 | Batch 128 | |||
| Tok/s | KV (GiB) | Tok/s | KV (GiB) | ||
| 4K | Dense (FA3) | 7.3k | 2.8 | 20.4k | 11.2 |
| Flash-HiSA | 7.8k | 1.9 | 23.2k | 7.7 | |
| 16K | Dense (FA3) | 4.7k | 11.2 | 9.2k | 45.0 |
| Flash-HiSA | 5.6k | 7.3 | 13.0k | 29.1 |
Table 5.
Table 5. Decoding with FlashAttention-3 at full context: throughput (tokens/s, one H200) and KV-cache memory, for batch sizes 32 and 128. 4K: ; 16K: , after the extension. Best in bold.
Converting a trained dense model
As existing models rely mostly on dense attention, we also investigate whether a model pre-trained with dense attention can be converted to HiSA and match the performance of dense attention under the same training budget. To test this scenario, we take the dense 135M model of Table 1 (, 4K context), switch its attention to HiSA without changing any weight, and continue training for 1.35B tokens. A dense control continues with the same recipe. After continued training (Table 6), the converted model matches the dense one while computing 29% of the dot products.
| Model | Extra tokens | Val. loss | (%) |
| Dense (start) | – | 2.906 | 100.0 |
| Dense → HiSA, no training | 0 | 2.937 | 29.3 |
| Dense, continued | 1.35B | 2.838 | 100.0 |
| Dense → HiSA, continued | 1.35B | 2.836 | 29.3 |
Table 6.
Table 6. Converting the dense 135M model (, 4K context) to HiSA: validation loss before and after continued training, and percentage of the dot product computed. Best in bold.
Dense attention learns to suppress the score parts HiSA drops
To provide an interpretability perspective on our work, we test whether the attention score parts are being used or ignored by dense models when is large. For every frequency pair and distance, we measure how much the pair’s dot product moves the attention output compared with an uninformative score part of the same rotation. Its effect can be above this noise level, at it, or below it, which indicates that the model actively suppresses it. We follow this measure in our dense 4K model during training and in the pretrained SmolLM-135M.
Measure. For a head , a pair , and a band of distances , let be the attention score part of pair (Equation (2), pre-softmax) with query pair and key pair and the values. The bands split the query–key distance on a logarithmic scale: , , , , , , twelve bands in total. A band thus selects, for every query, the keys at a given range of distances, and the measure tells how much pair matters at that range. For a query , the head output is the attention-weighted average of the values,
with the attention score of Equation (2). To measure what pair contributes at the distances of a band , we compute the same output without its score part for the keys in that band:
and
The quantity of interest is the change of the output,
(it depends on and , as do , and ; we omit them from the notation for clarity), which can be written as (derivation in Appendix § 12)
where
is the change of the attention weight of key before renormalization.
To measure this change at the output of the attention layer rather than inside each head, we map it through , the output projection of head . As a reference, we compute the same change with replaced by , the score part of pair computed with pair of the key of another document at the same position (same rotation, unrelated content), while , and stay those of the true document; we write for this change.
Both changes are computed on OpenWebText validation documents of at least 2048 tokens: the model reads the first 2048 tokens of each document, and every token serves as a query (details in Appendix § 12). We accumulate the squared changes over the documents, the heads of layer and all queries,
where is Equation (9) for head and document ; is the same sum with . We report the ratio .
A ratio above 1 means that the true score part moves the output more than an uninformative one (content-specific signal); below 1, it moves the output less than an unrelated key would, so the model has shaped its queries and keys to damp it.
Results. At initialization, the heatmaps are uniformly close to 1 (Figure 4) and start to shift during training. During the first steps the model learns to use the content of the pair scores at small rotation angles, which turn red, left of the 90° line. Right of the line, the scores are slowly turning blue. While this requires more training steps, some layers suppress scores at large distances aggressively. This is particularly visible in SmolLM-135M (all heatmaps and details are in Appendix § 12).
Figure 4. in three layers of the dense 135M model at initialisation, after 1B tokens and at the end of training (2.7B tokens), and of the pretrained SmolLM-135M. In each panel, rows are frequency pairs (fast at the bottom) and columns distance bands (1 to 2048 tokens, log scale). Red: the true score part moves the head output more than one computed with an unrelated key; blue: less. Black line: (right of it, HiSA drops the score part). All layers in Appendix § 12.
Figure 5 summarises all layers for the fast pairs (). At initialisation, most cells lie at the reference. The model quickly learns to use the scores at small angles: after 1B tokens, only 2–5% of the cells below 90° are under the reference. Suppressing the scores at large angles takes much longer: their share under the reference grows slowly over training (e.g. from 33% to 48% at 180–360°) and reaches 87–95% beyond 180° in SmolLM-135M, trained on about 600B tokens.
Figure 5. Share of the (layer, distance band) cells of the fast pairs () below the uninformative reference (), by rotation angle, at initialisation, after 1B tokens and at the end of training of the dense 135M model, and for the pretrained SmolLM-135M (all 30 layers).
Their relative influence on the head outputs, which needs no reference, decreases accordingly: beyond 720° it falls from 4.6% of the influence of the parts below 45° after 1B tokens to 2.1% at the end of training and 0.5% in SmolLM-135M (Appendix § 12).
A dense model thus spends part of its training suppressing the score parts with a large rotation, and this suppression is still progressing at the end of our runs. HiSA builds it into the architecture, which is consistent with its lower loss and with the small cost of converting a dense model (§ 4.5).
Conclusion
We asked whether attention needs the complete query–key dot product. With RoPE, the score splits into one part per frequency pair, and HiSA keeps each part only while its rotation stays within an angle budget. This sparsity needs no learned selection: it is fixed by the positions, so it is known before any token is seen. In our preliminary experiments with 135M- and 360M-parameter models, this partial product gives a lower validation loss than the complete one while computing only part of it. We also developed Flash-HiSA, a FlashAttention-3 implementation of HiSA, which trains at a similar speed to dense FlashAttention-3 while increasing decoding speed and reducing memory footprint during inference. The advantage persists when using long-context extension and a model trained with dense attention can be converted to HiSA by continued training without losing quality. An analysis of the head outputs suggests why: during training, a dense model learns to suppress the score pairs with large rotations, which HiSA removes by construction. Open questions such as scaling to bigger models, downstream tasks benchmarking and better kernel design are left for future work.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning, in particular the efficiency of attention in large language models. Cheaper training and inference lower the cost and energy of running such models; we see no societal consequences specific to this method beyond those of language models in general.
Compute and memory of HiSA
Fraction of the dot product computed. With integer windows (pair is kept iff ), pair contributes to of the causal query–key entries, with , which gives of Equation (5) (Table 7). The windows grow with the pair index, , so pairs whose window exceeds contribute fully and the others contribute little: is close to the share of pairs with , which falls by pairs, about 7.5 percentage points at , per doubling of (37%, 29% and 14% at 2K, 4K and 16K). Doubling the budget or the base has the same effect in the other direction. The block rule of Flash-HiSA (Figure 3) uses the minimum distance between blocks instead of and computes a bit more (35% at 4K and 16% at 16K, instead of 29% and 14%).
| 45° | 90° | 180° | 360° | 45° | 90° | 180° | 360° | 45° | 90° | 180° | 360° | reach (90°) | |
| 29.3 | 36.9 | 44.4 | 51.9 | 21.8 | 29.3 | 36.8 | 44.4 | 8.1 | 14.3 | 21.8 | 29.3 | 11.8K | |
| 34.2 | 41.2 | 48.2 | 55.1 | 27.2 | 34.2 | 41.2 | 48.1 | 13.2 | 20.2 | 27.2 | 34.2 | 23.1K | |
| 39.6 | 46.0 | 52.4 | 58.8 | 33.2 | 39.6 | 46.0 | 52.4 | 20.4 | 26.8 | 33.2 | 39.6 | 56.0K | |
| 50.0 | 55.2 | 60.5 | 65.8 | 44.6 | 49.9 | 55.2 | 60.5 | 34.1 | 39.4 | 44.6 | 49.9 | 521.2K |
Table 7. Percentage of the dense query–key dot product computed by HiSA, (Equation (5)), for angle budgets , RoPE bases and context lengths (, causal attention), and the reach at 90° (Equation (4)).
Training versus decoding. counts the entries of the causal score matrix, where distance occurs times: it is the fraction computed in a full forward pass (training, prefill), dominated by short distances. One decoding step at context compares the new query with keys, one at each distance , so the fraction it computes is
which weighs all distances equally and is smaller than (Table 8). The two are consistent: a full pass is the sum of its decoding steps, .
| 36.9 | 31.5 | 65.7 | 29.3 | 23.9 | 61.9 | 14.3 | 9.0 | 40.4 | |
| 41.2 | 36.1 | 68.1 | 34.2 | 29.1 | 64.5 | 20.2 | 15.1 | 57.6 | |
| 46.0 | 41.4 | 70.7 | 39.6 | 35.0 | 67.5 | 26.8 | 22.1 | 61.1 | |
| 55.2 | 51.4 | 75.7 | 49.9 | 46.1 | 73.1 | 39.4 | 35.5 | 67.8 |
Table 8. Percentage of the dense query–key product computed in a full causal pass, , and in one decoding step at context , , and KV-cache size relative to a dense cache, (per pair), at ().
Which work is removed. counts the multiply–adds of the query–key product. The other parts of attention, the softmax and the product with the values (and, in training, most of the backward pass), scale with the number of query–key entries, which stays the same as long as every key is within the reach. They only shrink for keys beyond the reach, i.e. for a fraction of the entries when . This is why the compute saving appears in decoding, where reading the cache dominates, before it appears in training (§ 4.3).
KV-cache memory. During decoding at context , pair of a cached key is needed for the last tokens and the value for the last . Keys and values have the same size, so the cache, relative to a dense one, is
While every value is kept, so the saving is at most 50%; beyond the reach, keys and values are both evicted. Our caches store keys per 16-feature chunk, with a ring as long as the window of the chunk’s slowest pair (rounded up to 128-token blocks for Flash-HiSA), which keeps slightly more than per-pair storage: 66.5% of a dense cache for HiSA and 68.4% for Flash-HiSA at 4K (), against 62% per pair; 64.6% for Flash-HiSA at 16K ().
Experimental details
Models. Table 9 lists the two architectures, which follow SmolLM ([4]): LLaMA-style decoder blocks with RMSNorm, SwiGLU MLPs, tied input and output embeddings (vocabulary of 49,152 tokens) and RoPE with base unless stated otherwise. All weights are initialised randomly; only the architecture is taken from SmolLM.
| Model | Params | Layers | Width | MLP | Heads (q / kv) | Tokens | |
| 135M | 134.5M | 30 | 576 | 1536 | 9 / 3 | 64 | 2.7B |
| 360M | 361.8M | 32 | 960 | 2560 | 15 / 5 | 64 | 7.24B |
Table 9. Model architectures and training budgets (about 20 tokens per parameter).
Optimisation. Table 10 gives the training recipe, shared by every dense and HiSA run. The peak learning rate is selected for the dense model only, with short runs at several values (Table 11); we keep the value with the lowest validation loss, or the lower of the two best when they are within 0.02.
| Optimizer | AdamW, , weight decay 0.1 on matrices, gradient clipping 1.0 |
| Batch | 524,288 tokens (128 sequences of 4K, 256 of 2K, 32 of 16K) |
| Schedule | linear warmup over 1% of the steps (2% for the 16K extension), cosine decay to 10% of the peak |
| Peak learning rate | (135M), (360M), (16K extension) |
| Precision | bf16 autocast, fp32 master weights, torch.compile |
| Seeds | 1337 (plus 1338 and 1339 for the 2K seed study) |
| Hardware | one NVIDIA H200 GPU per run |
Table 10. Training hyperparameters.
| Setting | Tokens | Peak LR: validation loss (selected in bold) |
| 135M, 2K, packed | 300M | : 4.782, : 4.718, : 4.464 |
| 135M, 4K, documents | 300M | : 4.666, : 4.430, : 4.622, : 4.745 |
| 360M, 4K, documents | 300M | : 4.140, : 3.958, : 3.949, : 4.143 |
| 135M, 16K extension | 100M | : 3.404, : 3.359, : 3.373 |
Table 11. Learning-rate selection on the dense model: validation loss after the given number of tokens, with the same number of warmup steps as the full run. For the 16K extension, the mean of the PG19 and OpenWebText validation losses.
Data. OpenWebText is tokenized with the SmolLM tokenizer; 0.5% of the documents (40,069) are held out for validation. The 2K runs cut the token stream (documents separated by an end-of-sequence token) into windows of 2,049 tokens in a seeded random order. The 4K runs use only windows that lie inside a single document of at least 4,097 tokens, cut at a random offset at every epoch, so that the context is never filled with unrelated documents. The 16K extension mixes, in fixed 50/50 proportions, single-book windows of PG19 ([22]) (28,602 training books, 3.0B tokens) and packed OpenWebText windows. The continued-training runs (dense → HiSA) use the 4K recipe with a new data seed.
Evaluation. The validation loss is the mean next-token cross-entropy over a fixed set of validation windows (3,200 windows at 2K, 1,600 at 4K, and 200 per source at 16K), also reported by position inside the window. Training throughput is the mean over the last 200 steps of the run. Decoding is measured on the first validation windows: one token at a time with teacher forcing, each step (all layers) captured as a CUDA graph, the KV cache allocated for the full context; throughput is divided by the time of the steps, and the decoded loss is checked against the loss of the full forward pass.
Choosing the recipe: angle budget, RoPE base, seeds and score normalisation
These ablations led to the recipe of the main text (, windows from the model’s own RoPE base).
Angle budget. Table 12 varies at 2K context (one seed per budget). Every finite budget beats dense attention, even 45°, which computes under a third of the dot product. 45°, 90° and 180° are within 0.012 of each other, about the spread between seeds (below), while a full rotation period (360°, the window of Du et al., 2026) keeps less of the gain. The data do not single out one budget; we use 90° as a compromise between compute and loss. Removing the score parts with the largest rotation helps, consistent with the analysis of § 5, where a dense model learns to suppress them.
| Val. loss | (%) | Reach | |
| 45° | 3.022 | 29.3 | 5.9K |
| 90° | 3.020 | 36.9 | 11.8K |
| 180° | 3.010 | 44.4 | 23.6K |
| 360° | 3.039 | 51.9 | 47.1K |
| (dense) | 3.055 | 100.0 | – |
Table 12. Angle budget at 2K context (, 135M): validation loss, percentage of the dot product computed, and reach. Best in bold.
Seeds. At 2K context (135M, , ), three seeds give 3.024 ± 0.004 for HiSA and 3.052 ± 0.011 for dense attention; HiSA is better on each seed.
RoPE base. Table 13 pairs each HiSA model with a dense model at the same base. The gain holds at bases and . With a rotation base of but windows still computed at base (†), the dense model itself improves and the gap shrinks.
| Dense | HiSA | Reach | ||
| 2K | 3.055 | 3.020 | 11.8K | |
| 2K | † | 3.019 | 3.010 | 11.8K |
| 4K | 2.906 | 2.880 | 11.8K | |
| 4K | 2.908 | 2.869 | 23.1K |
Table 13. RoPE base (, 135M): validation loss of dense attention and HiSA at the same base, and reach. †Rotation base , HiSA windows computed with base . Best in bold.
Score normalisation. HiSA keeps the scale of Equation (3) while dropping score parts, so the score of a distant key is a sum over fewer pairs and its typical magnitude shrinks with the distance. The gain over dense attention could therefore come from this implicit, content-independent down-weighting of distant keys rather than from removing the score parts themselves. To test this, we train a variant that normalises each score by the number of pairs it keeps,
so that every score has the scale of a dense dot product over its features (4K context, 135M, , one seed). It reaches a validation loss of 2.879, on par with HiSA (2.880) and well below dense attention (2.906): the gain does not come from the shrinking scale of distant scores. The variant is however slower (87k instead of 119k tokens/s with our Triton kernels, since the scale changes with every query–key distance) and worse on a synthetic copy task at long range (repeating a 32-token span seen 2560 tokens earlier: 33% token accuracy, against 77% for HiSA and 60% for dense attention). We therefore keep the scale.
Kernels
Exact rule (Triton). The exact rule is computed by FlashAttention-2-style fused kernels for the forward and the backward pass, written in Triton. The query and key features are permuted so that each 16-feature chunk holds 8 RoPE pairs (the two features of each pair together). For a tile of query and key positions, the pairs whose window covers the whole tile, the pairs that are dropped on the whole tile, and the “mixed” pairs whose window ends inside the tile are separated, and the mixed pairs are masked element-wise; with integer thresholds the result equals the reference rule exactly. At 4K context they train 1.7 times slower than dense attention with SDPA (Table 2).
Block rule in FlashAttention-3. With the block rule (Figure 3) and a rule block equal to the key tile (), the set of kept pairs is the same for every entry of a tile and depends only on the block distance (Figure 6a). Since windows grow with the pair index, the dropped pairs are always the fastest ones: a tile at block distance drops the first pairs, i.e. the chunks below and pairs of chunk . We add this rule to the FlashAttention-3 forward and backward kernels as a separate kernel instance. The query–key product of a tile runs the tensor-core matrix multiply only over the chunks , and the dropped pairs of chunk are zeroed in the shared-memory copy of the query tile (Figure 6b). The key blocks are visited from near to far, so only grows along the loop: the loop is split into one straight segment per value of , without a runtime branch inside the asynchronous matrix-multiply pipeline, and the zeroing is cumulative. A small table (pairs dropped per block distance, the four distances where a whole chunk drops, and the thresholds) is the only input that depends on the angle budget and the RoPE base. Blocks beyond the reach are skipped entirely through the kernel’s sliding-window bounds. In the backward pass the recomputation of the scores skips the dropped chunks in the same way; the other products (with , and the score gradients) still run at full width, which is why the backward pass does not yet benefit from the sparsity (§ 4.3). Applying RoPE directly in the chunked feature layout avoids permuting the queries and keys.
Figure 6. The FlashAttention implementation of HiSA (block rule, , , ). (a) Number of 16-feature chunks computed for each tile of the causal score matrix (2K context): near the diagonal all four, far from it only the slowest chunk. (b) One tile of the training kernel: the matrix multiply runs only over the kept chunks, and the dropped pairs of the boundary chunk are zeroed in the query tile. (c) The decoding cache at 4K context: each key chunk has its own ring buffer, as long as the window of its slowest pair; the values are kept within the reach.
Decoding. During decoding there is one query per sequence. The query heads that share a KV head are packed as the rows of one query tile of 64 rows (FlashAttention-3 uses 192 rows for , which would be almost entirely padding), and the cache is split across thread blocks whose partial results are merged at the end. The keys are stored as four ring buffers, one per 16-feature chunk, each as long as the window of its slowest pair rounded up to whole 128-token blocks, and the values in a ring as long as the reach (Figure 6c). The token at position lives in slot of ring , of length ; since every ring holds whole blocks, a key tile never wraps around. The producer threads load the cache with asynchronous copies; each thread always loads the same chunk, and skips the load when its chunk is dropped for the current key block, so the dropped chunks are never read from memory.
Interpretability: details and all layers
Derivation of Equation (10). Fix a query , a pair and a band . Without the score part of pair on the band, the scores are for and otherwise, so
Writing with and using of Equation (11), the new normaliser and the new weighted sum of the values are
since and . cancels in their ratio:
and subtracting gives Equation (10).
Numerics. Equation (10) cancels when a band holds almost all the attention of a query and a positive score part is removed . We therefore evaluate in fp64 in the equivalent form
which has only positive summands, and take . We check it against recomputing the softmax with the score part removed (relative difference below , also with a dominant pair). In SmolLM-135M, a few cells have very large effects (23 of 11 520 with a ratio above 50): removing a dominant score part moves the attention to other keys. We therefore report the ratio per cell and shares of cells rather than pooled sums.
Setup. The analysis of § 5 uses 256 OpenWebText validation documents of at least 2048 tokens (the first 2048 tokens of each, the same documents for every model) and the distance bands , , . and are summed over the heads of a layer (Equation (12)) before taking the ratio. The null key of a window is the key at the same position in the next window of the batch. The 4K model is analysed on 2048-token windows so that all models see the same inputs; the initialisation is rebuilt from the seed of the run, and the intermediate models are weight snapshots saved during training.
Observations. Figure 8 to Figure 12 show all layers. The first layer is an exception in every model: its score parts are above the reference at every distance, already at initialisation, plausibly because its keys depend on token identity only and a document repeats its own tokens. At initialisation, the fastest pairs at the longest distances are slightly below the reference in most layers.
Figure 8. per layer, pair and distance band, dense 135M model (4K, ) at initialisation. Black line: .
Figure 12. per layer, pair and distance band, pretrained SmolLM-135M. Black line: .
Relative influence. The ratio compares each score part with an uninformative one, but does not show how large its effect is. We therefore also report its relative influence: the change of the head outputs when the score part is removed, relative to the size of the head outputs themselves,
which needs no reference. Both sums run over the same queries and heads as Equation (12); the ratio is taken between the sums (not averaged over queries), so that queries with a nearly zero output do not dominate.
Figure 7 and Figure 13 to Figure 17 show where the influence of each pair lies along the distance axis: every pair (row) is divided by its maximum over the distance bands, so the figures compare the shape of the influence across distances, not its magnitude.
Figure 7. Relative influence of each pair (rows of a panel) at each distance band (columns) in three layers of the dense 135M model during training and of SmolLM-135M, each pair divided by its maximum over the distance bands. Black line: .
Figure 13. Relative influence (Equation (20)) per layer, pair and distance band, dense 135M model (4K, ) at initialisation; each pair divided by its maximum over the distance bands. Black line: .
Figure 17. Relative influence (Equation (20)) per layer, pair and distance band, pretrained SmolLM-135M; each pair divided by its maximum over the distance bands. Black line: .
Figure 9. per layer, pair and distance band, dense 135M model (4K, ) at step 2000. Black line: .
Figure 10. per layer, pair and distance band, dense 135M model (4K, ) at step 4000. Black line: .
Figure 11. per layer, pair and distance band, dense 135M model (4K, ) at the end of training (step 5149). Black line: .
Figure 14. Relative influence (Equation (20)) per layer, pair and distance band, dense 135M model (4K, ) at step 2000; each pair divided by its maximum over the distance bands. Black line: .
Figure 15. Relative influence (Equation (20)) per layer, pair and distance band, dense 135M model (4K, ) at step 4000; each pair divided by its maximum over the distance bands. Black line: .
Figure 16. Relative influence (Equation (20)) per layer, pair and distance band, dense 135M model (4K, ) at the end of training (step 5149); each pair divided by its maximum over the distance bands. Black line: .
References
- [1]Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskyi, Y., Lebrón, F., and Sanghai, S. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4895–4901, 2023.
- [2]Barbero, F., Vitvitskyi, A., Perivolaropoulos, C., Pascanu, R., and Velicković, P. Round and round we go! What makes rotary positional encodings useful? In International Conference on Learning Representations (ICLR), 2025. arXiv:2410.06205.arxiv.org/abs/2410.06205
- [3]Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.arxiv.org/abs/2004.05150
- [4]Ben Allal, L., Lozhkov, A., and Bakouch, E. SmolLM – blazingly fast and remarkably powerful. Hugging Face blog, https://huggingface.co/blog/smollm, 2024.
- [5]Chen, S., Wong, S., Chen, L., and Tian, Y. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023.arxiv.org/abs/2306.15595
- [6]Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.arxiv.org/abs/1904.10509
- [7]Dao, T. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024. arXiv:2307.08691.arxiv.org/abs/2307.08691
- [8]Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 16344–16359, 2022. arXiv:2205.14135.arxiv.org/abs/2205.14135
- [9]DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026.arxiv.org/abs/2606.19348
- [11]Gao, T., Wettig, A., Yen, H., and Chen, D. How to train long-context language models (effectively). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers), pp. 7376–7399, 2025.
- [12]Gemma Team. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026.
- [13]Gokaslan, A. and Cohen, V. OpenWebText corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019.
- [14]Grattafiori, A., Dubey, A., Jauhari, A., et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.arxiv.org/abs/2407.21783
- [15]Hayashi, S.-i., Mukunoki, D., Hoshino, T., and Katagiri, T. ATFlash: Per-RoPE-wavelength attention windows for compute/memory-efficient LLM inference. arXiv preprint arXiv:2608.02947, 2026.arxiv.org/abs/2608.02947
- [16]Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Le Scao, T., Lavril, T., Wang, T., Lacroix, T., and El Sayed, W. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.arxiv.org/abs/2310.06825
- [17]Khandelwal, U., He, H., Qi, P., and Jurafsky, D. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 1: Long Papers), pp. 284–294, 2018.arxiv.org/abs/1805.04623
- [18]Liu, X., Yan, H., Zhang, S., An, C., Qiu, X., and Lin, D. Scaling laws of RoPE-based extrapolation. In International Conference on Learning Representations (ICLR), 2024. arXiv:2310.05209.arxiv.org/abs/2310.05209
- [20]Men, X., Xu, M., Wang, B., Zhang, Q., Lin, H., Han, X., and Chen, W. Base of RoPE bounds context length. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 87386–87410, 2024. arXiv:2405.14591.
- [21]Peng, B., Quesnelle, J., Fan, H., and Shippole, E. YaRN: Efficient context window extension of large language models. In International Conference on Learning Representations (ICLR), 2024. arXiv:2309.00071.arxiv.org/abs/2309.00071
- [22]Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations (ICLR), 2020. arXiv:1911.05507.arxiv.org/abs/1911.05507
- [23]Ribar, L., Chelombiev, I., Hudlass-Galley, L., Blake, C., Luschi, C., and Orr, D. SparQ attention: Bandwidth-efficient LLM inference. In International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pp. 42558–42583, 2024. arXiv:2312.04985.arxiv.org/abs/2312.04985
- [24]Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 68658–68685, 2024. arXiv:2407.08608.arxiv.org/abs/2407.08608
- [25]Singhania, P., Singh, S., He, S., Feizi, S., and Bhatale, A. LoKi: Low-rank keys for efficient sparse attention. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 16692–16723, 2024. arXiv:2406.02542.arxiv.org/abs/2406.02542
- [27]Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasicki, B., and Han, S. Quest: Query-aware sparsity for efficient long-context LLM inference. In International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pp. 47901–47911, 2024. arXiv:2406.10774.arxiv.org/abs/2406.10774
- [28]Team Olmo. Olmo 3. arXiv preprint arXiv:2512.13961, 2025.
- [29]Tillet, P., Kung, H. T., and Cox, D. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10–19, 2019.DOI
- [30]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017. arXiv:1706.03762.arxiv.org/abs/1706.03762
- [31]Wang, S., Zheng, Y., Zhu, T., Chen, S., Liu, S., Zheng, S., and Li, J. AdaRoPE: Not all attention heads should rotate and scale equally. In International Conference on Machine Learning (ICML), volume 306 of Proceedings of Machine Learning Research, pp. 132815–132865, 2026a. arXiv:2607.19363.arxiv.org/abs/2607.19363
- [32]Wang, Y., Wang, Y., Yue, Z., Zeng, H., Wang, Y., Lourentzou, I., Tu, Z., Chu, X., and McAuley, J. FASA: Frequency-aware sparse attention. In International Conference on Learning Representations (ICLR), 2026b. arXiv:2602.03152.arxiv.org/abs/2602.03152
- [33]Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), 2024. arXiv:2309.17453.arxiv.org/abs/2309.17453
- [34]Xie, Y., Wen, T., Huang, T., Chen, B., You, C., Jegelka, S., and Wang, Y. Scaling attention via feature sparsity. In International Conference on Learning Representations (ICLR), 2026. arXiv:2603.22300.arxiv.org/abs/2603.22300
- [35]Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., Khabsa, M., Fang, H., Mehddad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H. Effective long-context scaling of foundation models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) (Volume 1: Long Papers), pp. 4643–4663, 2024.
- [37]Yang, A., Li, A., Yang, B., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
- [38]Yang, S., Sheng, Y., Gonzalez, J. E., Stoica, I., and Zheng, L. Post-training sparse attention with Double Sparsity. arXiv preprint arXiv:2408.07092, 2024.arxiv.org/abs/2408.07092
- [39]Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y., Wang, L., Xiao, Z., Wang, Y., Ruan, C., Zhang, M., Liang, W., and Zeng, W. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23078–23097, 2025.arxiv.org/abs/2502.11089
- [40]Zaheer, M., Guruganesha, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., and Ahmed, A. Big Bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 17283–17297, 2020. arXiv:2007.14062.arxiv.org/abs/2007.14062
- [41]Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H₂O: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 34661–34710, 2023. arXiv:2306.14048.arxiv.org/abs/2306.14048
















